It is difficult to say what could be causing your managers to die. I recommend shutting them all down, relink fnd executables via adadmin, patch in the cpadmin.sh utility and run that instead of cmclean.sql.
EBS Concurrent Processing (CP) Analyzer (Doc ID 1411723.1)
Concurrent Processing - Command-Line Utility OR Recovery Wizard (Doc ID 134007.1)
21085588:R12.FND.B BACKPORT OF CONCURRENT MANAGER UTILITY
I have also faced this issue.
While bouncing, shutdown all services but I have start all again without checking that any process running on run or patch file system or not. If any process is running on run file system (mostly java one) and I have run the start all script then I used to face the same issue.
Bounce all services as following steps -
#1: stop all
#2: check if any process is running on fs1 or fs2
ps -ef | grep fs1
ps -ef | grep fs2
If anything running related to java and CM then kill them all.
#3: start all
Try this one and let me know.
If answer is helpful or correct, please mark it as you find the answer. It will help our community to get solution better.
It can happen when End dated a user (or) a responsibility of a user, but there is a scheduled request. Please refer the note above to get more details and Query to find out.
We downloaded and apply patch to get cpadmin... then ran cpadmin... but we still have the same issues... we now see 0 running with 10 pending... its really strange... any other thoughts?
Look at incompatibilities with all requests running, not just in the Standard Manager. For example, Gather Schema Statistics will make Launch Advanced Supply Chain Planning go to pending until it completes.
Based on experience, logical reason for situation "Actual=40, running=5. Pending=29." can be:
A} Programs are incompatible with each other. Please see the program definition of each such program in pending
B) Programs are self-incompatible. Means many program of one kind are sent to run all together and they are getting executed in sequence. We have seen it many times with our client
C) Unlikely but may be. some Parent-Child relation between the programs is keeping some programs in pending until child finishes. Need to understand it by reading program definition again.
You also need to look at ICM and Standard Manager logs closely for any errors.
Please look for any expensive DML as per Note 1075684.1
Please review AWR report (for 60 min interval) during the problematic period. This helps to find out any expensive DML/Query in the system (or) Concurrent Manger sub-system.
Turned out that it was a timing issue. ntp on one of two database nodes was not running. It was off by 6 minutes from actual time ....So jobs ready to run would wait 6 minutes before running. That was the cause of the issue. Once NTP was running and time was correct... Issues went away.