6 Replies Latest reply on Sep 16, 2011 2:49 PM by pouler

    Concurrent manager using lots of cpu & memory

      In the last few days, I have noticed the load on the production server as follows:

      load averages: 4.66, 4.86, 4.92 14:34:16
      333 processes: 327 sleeping, 6 on cpu
      CPU states: % idle, % user, % kernel, % iowait, % swap
      Memory: 32G real, 7591M free, 23G swap in use, 14G swap free

      20045 appsdapo 1 0 0 4073M 4029M cpu/2 165.0H 12.49% f60webmx
      1854 appsdapo 1 10 0 4082M 4060M cpu/0 240.0H 12.48% FNDLIBR
      2504 appsdapo 1 10 0 1872M 1869M cpu/19 1980.2 12.46% FNDLIBR
      2496 appsdapo 1 0 0 1876M 1873M cpu/1 1957.5 11.68% FNDLIBR
      18641 appsdapo 1 10 0 1512M 1509M cpu/16 1080.3 8.64% FNDLIBR

      I have come across Metalink note ID 114380.1 which says identify the C.M but how do I relate the FNDLIBR processes above with the specific C.Managers.

        • 1. Re: Concurrent manager using lots of cpu & memory
          Hussein Sawwan-Oracle
          Please post the details of the application release, database version and OS.

          Any changes have been done recently (patches, upgrade, installation, ..etc)?

          Concurrent Processing - How to Find Database Session & Process Associated with a Concurrent Program Which is Currently Running. [ID 735119.1]
          How to Retrieve SID Information For a Running Request [ID 280391.1]
          Concurrent Processing (CP) / APPS Reporting Scripts [ID 213021.1]
          Concurrent Processing - CCM.sql Diagnostic Script to Diagnose Common Concurrent Manager Issues [ID 171855.1]
          Scheduled Requests Running for END_DATED Users 100% CPU and Giving ORA-01002 [ID 1080047.1]
          Concurrent Processing - Troubleshooting Concurrent Manager Issues (Unix specific) [ID 104452.1]

          Also, search the forum for "Concurrent Manager" and "Tuning".

          • 2. Re: Concurrent manager using lots of cpu & memory

            Solaris 10

            Nothing has changed that I know as the dba.

            I have looked through some of the notes and none is saying how I can relate the FNDLIBR process with a c.manager etc

            Thanks Hussein, as always
            • 3. Re: Concurrent manager using lots of cpu & memory
              Hussein Sawwan-Oracle
              From the OS, issue "ps -ef | grep FNDLIBR" -- This will return many processes, so pick the PID of the process with a high CPU utilization and run this query (let's say PID is "495732"):
              SQL> SELECT d.sid, d.serial# ,d.osuser,d.process
              FROM apps.fnd_concurrent_processes b,
              v$process c,
              v$session d
              WHERE c.pid = b.oracle_process_id
              AND b.session_id=d.audsid
              AND d.process = '434372';
              • 4. Re: Concurrent manager using lots of cpu & memory

                Is this happening in your test instance also?

                Did you find if there is any major change in your query execution plan of the concurrent request ?

                Sridhar M
                • 5. Re: Concurrent manager using lots of cpu & memory
                  If the users who scheduled the concurrent requests are end-dated, then the concurrent managers will consume high cpu and memory. You can see the error and the concurrent request id in Concurrent Manager logs
                  • 6. Re: Concurrent manager using lots of cpu & memory

                    Thanks for the Post. This post helped me to solve my production issue.
                    My production application started hanging suddenly and no clue why it is behaving like this.
                    The symptom was FNDLIBR processes were on top cpu consumer on application server.
                    The database server also experienced heavy system i/o and cpu usage.
                    The database base log switches every minutes (500 MB) and there are no concurrent requests running.
                    No concurrent request errors. No OS errors (Cleaned the GNOME and restarted). No database errors.

                    But in sql area i saw lot of fnd service requests keep coming.

                    At the end the application server hangs because of high cpu and it caused the production down.

                    The problem was one of Our System Administrator disabled one responsibility with scheduled request.

                    One interesting thing we noticed, even you put the request in question Hold, it will start running while you restart the concurrent request.

                    You have to cancel the concurrent request and restart the concurrent manager can solve the issue.