1 2 Previous Next 29 Replies Latest reply on Jul 18, 2019 5:49 AM by Maaz Khan

    EBS 12.1.3 Performance Issue

    Beauty_and_dBest

      EBS 12.1.3

      11g

      OL6

       

      Hi ALL,

       

      We are currently encountering slow performance issue in our EBS instance when they are receiving items in Oracle.  Based on our initial assessment, the slowdown is caused by multiple create accounting jobs that was triggered from the receiving transaction that blocks the other jobs on queue from running (Pending status).  As a workaround we, advised users to limit the receiving to 10 items at a time.

      As mentioned, we need to come up with immediate solution to address this because there is an ongoing project that will require us to receive large number of items to be processed.

       

      Please  help how to manage performance for  multiple create accounting jobs and be able to receive and process large number of transactions.

      Do we need to create more concurrent managers to accommodate more create accounting jobs?

       

      Kind regards,

      jc

        • 1. Re: EBS 12.1.3 Performance Issue
          John_K

          Is it that the concurrent manager doesn't have enough slots for the number of jobs submitted, or is there an incompatibility that is preventing from running?

          Also, are they running on the standard manager or a custom one?

          • 2. Re: EBS 12.1.3 Performance Issue
            Kanda-Oracle

            Hi JC,

             

            You may create a 'specialized manager'  to process the specific requests (ex. create accounting jobs ) that are in question. whenever you submit that program "create accounting jobs" it would be proceed by the new concurrent manager that just created.

             

            You can increase and decrease the process based on the server capacity (CPU/RAM) of both DB/Apps tier. This helps to resolve or manage the situation better.    

             

            Note : Please note that If you do not have sufficient free CPU cycle & Free RAM, then adding new managers won't help

             

            Hope this helps!

             

            Thanks

             

            N Kandasamy

            • 3. Re: EBS 12.1.3 Performance Issue
              Kanda-Oracle

              Please ensure to "include" the program to the newly created concurrent manager and "Exclude" it from Standard Managers.

               

              So that whenever its submitted it would go to newly created manager.

              • 4. Re: EBS 12.1.3 Performance Issue
                Beauty_and_dBest

                Hi Kanda, John,

                 

                How do I know that we are running the following?:

                -That the concurrent manager doesn't have enough slots for the number of jobs submitted, or is there an incompatibility that is preventing from running?

                -That they running on the standard manager or a custom one?

                 

                Regading below remarks

                You may create a 'specialized manager'  to process the specific requests (ex. create accounting jobs ) that are in question. whenever you submit that program "create accounting jobs" it would be proceed by the new concurrent manager that just created.

                 

                How do I know when to create a "specialized manager".

                How do I know if it has already been created one?

                 

                 

                Kind regards,

                • 5. Re: EBS 12.1.3 Performance Issue
                  Kanda-Oracle

                  Hi JC

                   

                  1. Please mention whether  requests stay in Pending/standby pending/normal (or) Running/normal for longer time ?

                   

                  2. If you examine the screen shot of the "Administer Screen". This is to know whether they running on the standard manager or a custom one

                   

                  Navigation :- Sysadmin > Concurrent > Manager > Administer Screen

                   

                  Thanks!

                  • 6. Re: EBS 12.1.3 Performance Issue
                    Kanda-Oracle

                    Hi

                     

                    When to create a "specialized manager" ?

                     

                    1. One reason, You wanted to organize your requests to be processed in such a way that  critical short running jobs do not wait in the Queue for long time (pending/Normal)
                    2. Assume, There are 10 Standard Manager process. That means, it can process 10 requests at a time. Imaging there is a user who submitted one long running report (less important) 10 times.

                        So all the 10 standard manager processes would process only those 10 long running less important requests.  During that time, you can run any critical request ( example invoice printing..short running request).......In such scenario also you can create specialized manager.

                      2.1 You can create a different specialized/dedicated manager for the critical requests. so that they won't wait or disturbed by other requests.

                     

                    Hope this is clear !

                    • 7. Re: EBS 12.1.3 Performance Issue
                      Kanda-Oracle

                      Hi JC

                       

                      In a simpler word, Instead of having one single common Queue (Standard manager Queue), you are creating multiple Queues (by creating new specialized / dedicated managers). You can regulate the traffic better.

                        Hence you are now well organized in processing your incoming requests.   That means, the concurrent requests wait time in the Queue is reduced/manged well.

                       

                      Thanks!

                      • 8. Re: EBS 12.1.3 Performance Issue
                        Beauty_and_dBest

                        Thanks Kanda,

                         

                        Please confirm if my understanding is correct.

                         

                        Supposing our Standard Manager is 30 slots, then

                        I will create a new Custom  Manager for  Create Accounting jobs say 10 slots.

                         

                        Is it possible to have two sets of Standard Managers? with slots 30 each?

                         

                         

                        Kind regards,

                        • 9. Re: EBS 12.1.3 Performance Issue
                          Maaz Khan

                          Hi Jc,

                           

                          There will always be one standard manager. You can create a custom concurrent manager and slots will depend on you cpu cores.

                           

                          For Create accounting programs, there are module wise programs that are executed periodically depending on your environment and they have child programs known has Accounting Program that are spawned and they can be defined according to volume of data.

                           

                          So when you are defining slots, you must make sure that child programs(Accounting Program) are accommodated and they run in parallel under custom manager.

                           

                          Suppose Create Accounting program is defined with n child programs (Accounting Program), then you must have at least n+2 slots defined under your custom manager for optimal performance.

                           

                          Please refer below carefully before creating custom program. You will have to review your environment carefully and draft set of programs that will run under custom manager.

                           

                          1. 1. How to Create a Custom Concurrent Manager (Doc ID 170524.1)
                          2. 2. How to Make User Run Any Requests Using Standard Manager Except Specific Program Run by Custom Manager? (Doc ID 1297419.1)
                          3. 3. R12 SLA/FAH: How to Improve Performance in Subledger Accounting & Financials Accounting Hub (Doc ID 791049.1)

                           

                          Few tips on my blog as well when dealing with similar environment -

                           

                           

                          http://maazdba.blogspot.com/2016/11/performance-improvements-for-month.html

                          • 10. Re: EBS 12.1.3 Performance Issue
                            Kanda-Oracle

                            Hi JC,

                             

                            You may want to have 3 different ("specialized concurrent manager") ; each 10 process. The total number of the manager processes are same.

                            But you can use them for different purposes to regulate the incoming traffic.

                             

                            Note

                             

                            • Please note that If you do not have sufficient free CPU cycle & Free RAM, then adding new managers won't help
                            • Please note you can not have excessive "specialized concurrent manager".  It can degrade the performance due polling on queue tables. You need to create specialized managers only if there is a real requirement.

                             

                            Refer

                            =====

                            Best Practices for Performance for Concurrent Managers in E-Business Suite (Doc ID 1057802.1)

                             

                            Thanks!   

                            • 11. Re: EBS 12.1.3 Performance Issue
                              Kanda-Oracle

                              Please ensure to "include" the program to the newly created concurrent manager and "Exclude" it from Standard Managers.

                               

                              So that whenever its submitted it would go to newly created manager.

                               

                              • 12. Re: EBS 12.1.3 Performance Issue
                                Beauty_and_dBest

                                Thanks Kanda,

                                 

                                Do you have any Idea how much Memory is allocated to a new set of "Specialized Concurrent Managers"  with 10 members each?

                                I understand it is the AppsTier Server that needs to add CPU/Memory  and not the DatabaseTier Server.

                                 

                                What command to run to list if there is enough Free CPU/Memory to add more Conc. Managers?

                                Is Linux "top" command and "free"  can show it?

                                 

                                 

                                Kind regards,

                                • 13. Re: EBS 12.1.3 Performance Issue
                                  Maaz Khan

                                  Hi Jc,

                                   

                                  When you create specialized/custom concurrent manager, it  will comprise of processes serving a separate queue of concurrent programs 'included' under it. There are no memory arguments to be provided when defining specialized/custom concurrent manager.

                                   

                                   

                                  However, you can assign java memory as follows for Create Accounting programs as follows,

                                  Please note below -

                                  1.

                                  Any program INCLUDED in the CUSTOM MANAGER should be EXCLUED from the STANDARD MANAGER. Any program that is not included in CUSTOM MANAGER will be run by STANDARD MANAGER by default

                                  2.   

                                  Please take a look at the Sleep cycle of the Custom manager if that is abnormal then it will cause some performance issues.

                                  3.

                                  If a program is a parent program spawning multiple child programs then we must explicitly exclude child programs as well from Standard manager and add them under Custom Manager.

                                   

                                   

                                  • 14. Re: EBS 12.1.3 Performance Issue
                                    Beauty_and_dBest

                                    Thanks Maaz, Kanda, and ALL,

                                     

                                    What linux command to run, to check if we are running out of CPU? For Memory it is "free -m",  for  Storage it is "df -h". How about CPU?

                                     

                                    I think I have to agree with Kanda that we need to have enough Memory/CPU to accommodate new Custom Managers.

                                    And that it does NOT help improve performance if you create new managers and you have limited resources.

                                     

                                    Can you show sample  screenshot  where to "EXCLUDE" parameter from standard manager the Create Accounting program plus its children?

                                     

                                    Can you also help which log files for Concurrent Managers/Programs to check, if there are performance issues encountered?

                                     

                                     

                                    Kind regards,

                                    1 2 Previous Next