1 2 Previous Next 22 Replies Latest reply on Aug 15, 2016 6:01 AM by ProDBA

    EBS R12.2 : OACORE Going down frequently

    3203003

      Dear Experts,

       

      I have noticed that in my production box. OACORE Services are going down automatically. I have noticed this below message in the

       

      following log locaiton. Please review and advise.

       

      DB : 11.2.0.4

      EBS : R12.2.3

       

      /Instance_u01/erpapp/fs1/FMW_Home/user_projects/domains/EBS_domain_Instancename/servers/oacore_server2/logs

      oacore_server2.log

       

      main "EBS_domain_Instancename" running in Production Mode>

      <May 13, 2016 1:54:40 PM EDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RUNNING>

      <May 13, 2016 1:54:40 PM EDT> <Notice> <WebLogicServer> <BEA-000360> <Server started in RUNNING mode>

      **************************************************************************

      *** WARNING: Oracle BC4J debug build executing - do not use for timing ***

      **************************************************************************

      alta.desktop.xml: Parsing error, line 2, column 207:

      org.xml.sax.SAXParseException; systemId: alta.desktop.xml; lineNumber: 2; columnNumber: 207; Unable to locate base look and feel: skyros.desktop

              at oracle.cabo.share.xml.BaseNodeParser.logError(Unknown Source)

              at oracle.cabo.ui.laf.xml.parse.LookAndFeelExtensionParser.startElement(Unknown Source)

              at oracle.cabo.share.xml.TreeBuilder$Handler.startElement(Unknown Source)

              at oracle.xml.parser.v2.NonValidatingParser.parseElement(NonValidatingParser.java:1558)

              at oracle.xml.parser.v2.NonValidatingParser.parseRootElement(NonValidatingParser.java:442)

              at oracle.xml.parser.v2.NonValidatingParser.parseDocument(NonValidatingParser.java:388)

              at oracle.xml.parser.v2.XMLParser.parse(XMLParser.java:232)

              at oracle.cabo.share.xml.TreeBuilder.parse(Unknown Source)

              at oracle.cabo.share.xml.TreeBuilder.parse(Unknown Source)

              at oracle.cabo.ui.laf.xml.LookAndFeelUtils.createLookAndFeelExtension(Unknown Source)

              at oracle.cabo.ui.laf.xml.LookAndFeelUtils.createLookAndFeelExtension(Unknown Source)

                                                                                                          39,1          19%

      at oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:119)

              at java.security.AccessController.doPrivileged(Native Method)

              at oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:324)

              at oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:460)

              at oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:103)

              at oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:171)

              at oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:71)

              at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)

              at oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:163)

              at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)

              at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3748)

              at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3714)

              at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)

              at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)

              at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2283)

              at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2182)

              at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1491)

              at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)

              at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

      org.xml.sax.SAXParseException; systemId: alta.desktop.xml; lineNumber: 2; columnNumber: 207; Unable to locate base look and feel: skyros.desktop

              at oracle.cabo.share.xml.BaseNodeParser.logError(Unknown Source)

              at oracle.cabo.ui.laf.xml.parse.LookAndFeelExtensionParser.startElement(Unknown Source)

              at oracle.cabo.share.xml.TreeBuilder$Handler.startElement(Unknown Source)

              at oracle.xml.parser.v2.NonValidatingParser.parseElement(NonValidatingParser.java:1558)

              at oracle.xml.parser.v2.NonValidatingParser.parseRootElement(NonValidatingParser.java:442)

              at oracle.xml.parser.v2.NonValidatingParser.parseDocument(NonValidatingParser.java:388)

              at oracle.xml.parser.v2.XMLParser.parse(XMLParser.java:232)

              at oracle.cabo.share.xml.TreeBuilder.parse(Unknown Source)

              at oracle.cabo.share.xml.TreeBuilder.parse(Unknown Source)

              at oracle.cabo.ui.laf.xml.LookAndFeelUtils.createLookAndFeelExtension(Unknown Source)

              at oracle.cabo.ui.laf.xml.LookAndFeelUtils.createLookAndFeelExtension(Unknown Source)

              at oracle.cabo.ui.laf.config.ConfigLookAndFeelProvider.registerLookAndFeel(Unknown Source)

              at oracle.apps.fnd.framework.webui.laf.LafUtils.registerLookAndFeelProvider(LafUtils.java:239)

              at oracle.apps.fnd.framework.webui.laf.LafUtils.createLookAndFeel(LafUtils.java:205)

              at oracle.apps.fnd.framework.webui.laf.LafUtils.registerCustomLookAndFeels(LafUtils.java:156)

              at oracle.apps.fnd.framework.webui.OAConfigurationFactory.getConfiguration(OAConfigurationFactory.java:218)

              at oracle.apps.fnd.framework.webui.OARenderingContext.initConfiguration(OARenderingContext.java:439)

              at oracle.apps.fnd.framework.webui.OARenderingContext.configure(OARenderingContext.java:169)

              at oracle.apps.fnd.framework.webui.OARenderingContext.<init>(OARenderingContext.java:147)

              at oracle.apps.fnd.framework.webui.OAPageContextImpl.<init>(OAPageContextImpl.java:668)

              at oracle.apps.fnd.framework.webui.OAPageBean.createPageContext(OAPageBean.java:5267)

              at oracle.apps.fnd.framework.webui.OAPageBean.preparePage(OAPageBean.java:1375)

                                                                                               

      Regards

      Mohammed Abdul Muqeet

        • 1. Re: EBS R12.2 : OACORE Going down frequently
          3203003

          Also noticed this below error in the

           

          vi  $FMW_HOME/user_projects/domains/EBS_domain_$TWO_TASK/servers/oacore_server1/logs/oacore_server1.out

           

          Error

           

          <May 13, 2016 2:23:16 PM EDT> <Error> <ServletContext-/OA_HTML> <BEA-000000> <chain failed

          java.net.SocketException: Connection reset

                  at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:118)

                  at java.net.SocketOutputStream.write(SocketOutputStream.java:159)

                  at weblogic.servlet.internal.ChunkOutput.writeChunkNoTransfer(ChunkOutput.java:591)

                  at weblogic.servlet.internal.ChunkOutput.writeChunks(ChunkOutput.java:540)

                  at weblogic.servlet.internal.ChunkOutput.flush(ChunkOutput.java:427)

                  Truncated. see log file for complete stacktrace

          >

          <May 13, 2016 2:30:14 PM EDT> <Error> <ServletContext-/OA_HTML> <BEA-000000> <chain failed

          java.lang.IllegalStateException: Cannot resize buffer - 479 bytes have already been written (Servlet 2.3, sec. 5.1)

                  at weblogic.servlet.internal.ServletResponseImpl.setBufferSize(ServletResponseImpl.java:357)

                  at javax.servlet.ServletResponseWrapper.setBufferSize(ServletResponseWrapper.java:159)

                  at weblogic.servlet.jsp.PageContextImpl.initialize(PageContextImpl.java:74)

                  at weblogic.servlet.jsp.PageContextImpl.<init>(PageContextImpl.java:110)

                  at weblogic.servlet.jsp.JspFactoryImpl.getPageContext(JspFactoryImpl.java:40)

                  Truncated. see log file for complete stacktrace

          >

           

           

          Regards

          Mohammed.

          • 2. Re: EBS R12.2 : OACORE Going down frequently
            Jason Brincat

            Hi Mohammed,

            the first error is not the cause of your issue.

            The second error is related, however I feel it is not showing the root cause of the problem.

            How often is oacore going down? How many oacore's do you have in total?

            make sure all of these patches are applied:

            OACORE Managed Server Failing In 12.2 With Error 'java.lang.IllegalStateException: Cannot resize buffer' and 'java.lang.IllegalStateException: Strict Servlet API' Preventing User Login Until Services Are Restarted (Doc ID 2087130.1)

            If they are then let me know and I will give you the next steps

            thanks

            jason

            • 3. Re: EBS R12.2 : OACORE Going down frequently
              3203003

              Hi,

               

              Thanks for the reply.

               

              We have 6 OACORE defined in our Enviroment.

               

              Regards

              Mohammed.

              • 4. Re: EBS R12.2 : OACORE Going down frequently
                Jason Brincat

                Ok please ensure you have those fixes in place Mohammed

                • 5. Re: EBS R12.2 : OACORE Going down frequently
                  3203003

                  only 2 patches are not applied remaining are avialble in the system

                   

                   

                   

                  '19494816', - Yes

                  '20763474', - yes

                  '20026276', - Yes

                  '21619282', - Yes

                  '20027711',- Yes

                  '20823641',- Yes

                  '22512856', - NA

                  '19245366'; -NA

                   

                  Regards

                  Mohammed.

                  • 6. Re: EBS R12.2 : OACORE Going down frequently
                    Jason Brincat

                    Do you use WebADI?

                    What is your latest Delta Patchset installed?

                    You can check this via.

                    SELECT codelevel FROM AD_TRACKABLE_ENTITIES WHERE abbreviation in ('txk','ad');

                     

                    is oacore going down even when NOT under load?

                    • 8. Re: EBS R12.2 : OACORE Going down frequently
                      3203003

                      Yes - It is going down very frequently.


                      As I have told you that we have together 6 OACORES.

                       

                      We notice that every hour we have 1 or more oacores going down.

                       

                      regards

                      Mohammed

                      • 9. Re: EBS R12.2 : OACORE Going down frequently
                        Jason Brincat

                        Hi Mohammed,

                        Do you use WebADI?

                        Does oacore go down when there is no load on the system or ONLY under load. Please provide details

                        • 10. Re: EBS R12.2 : OACORE Going down frequently
                          3203003

                          Not Sure whether we have web ADI ?

                           

                          During the peak business hours 24x7 we have notice that it goes down. when there is less load during the week ends will will go down as well.

                          regards

                          Mohammed.

                          • 11. Re: EBS R12.2 : OACORE Going down frequently
                            3203003

                            We have scheduled OEM alert for this issue. When ever we have oacore down we get email in our mail box.

                             

                            we immediately stop and start the oacore using weblogic console and  it gets fix.

                             

                            but am sure this is not the permanent solution.

                             

                            regards

                            Mohammed.

                            • 12. Re: EBS R12.2 : OACORE Going down frequently
                              Jason Brincat

                              Ok sounds like you may have a memory leak.

                              Please use this note I wrote to collect the log files I will need for analysis

                               

                              12.2 Ebusiness Suite - Collecting Fusion Middleware Log Files

                              Note 1362900.1


                              thanks

                              Jason

                              • 13. Re: EBS R12.2 : OACORE Going down frequently
                                3203003

                                Due to compliance issue, I can not upload logs.

                                 

                                But i have noticed the following .

                                 

                                fs1/FMW_Home/user_projects/domains/EBS_domain_instance/servers/AdminServer/logs

                                EBS_domain_Instance.log

                                 

                                <Error> <oracle.as.management.mbeans.opmn> <> <AdminServer> <DmsThread-4> <weblogic> <> <f8f9eebad4a349d1:65ab49cf:154477185e4:-8000-000000000000010d> <1463354250793> <BEA-000000> <OPMN.queryProcessStatus failed with the following exception

                                oracle.as.management.opmn.optic.OpticBadConnectException: Too few bytes (1) received from OPMN response

                                        at oracle.as.management.opmn.optic.OpmnPhone.receiveResponse(OpmnPhone.java:826)

                                        at oracle.as.management.opmn.optic.OpmnPhone.makePhoneCall(OpmnPhone.java:331)

                                        at oracle.as.management.opmn.optic.OpmnPhone.request(OpmnPhone.java:270)

                                        at oracle.as.management.opmn.optic.OpmnQuery.getBuf(OpmnQuery.java:482)

                                        at oracle.as.management.opmn.optic.OpmnQuery.getDom(OpmnQuery.java:602)

                                        at oracle.as.management.opmn.optic.OpmnQuery.getProc(OpmnQuery.java:750)

                                        at oracle.as.management.mbeans.opmn.Opmn.queryProcessStatus(Opmn.java:328)

                                        at oracle.as.management.mbeans.opmn.Opmn.getStatus(Opmn.java:85)

                                        at oracle.as.management.mbeans.opmn.Opmn.getStatus(Opmn.java:95)

                                        at oracle.as.management.mbeans.register.OpmnInfoImpl.getComponentNames(OpmnInfoImpl.java:34)

                                        at oracle.dms.opmn.app.OpmnDiscoverer.gatherAddresses(OpmnDiscoverer.java:158)

                                        at oracle.dms.address.AddressBook$DiscoverAddressTask.call(AddressBook.java:814)

                                        at oracle.dms.address.AddressBook$DiscoverAddressTask.call(AddressBook.java:786)

                                        at java.util.concurrent.FutureTask.run(FutureTask.java:262)

                                        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

                                        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

                                        at oracle.dms.aggregator.DmsThreadFactory$1$1.run(DmsThreadFactory.java:56)

                                        at java.security.AccessController.doPrivileged(Native Method)

                                        at oracle.dms.aggregator.DmsThreadFactory$1.run(DmsThreadFactory.java:51)

                                        at java.lang.Thread.run(Thread.java:745)

                                 

                                Regards

                                Mohammed

                                • 14. Re: EBS R12.2 : OACORE Going down frequently
                                  3203003

                                  Also I have noticed the following error in our file AdminServer-diagnostic.log

                                   

                                  [2016-05-15T19:21:56.569-04:00] [AdminServer] [ERROR] [] [oracle.as.management.mbeans.opmn] [tid: DmsThread-4] [userId: weblogic] [ecid: f8f9eebad4a349d1:65ab49cf:154477185e4:-8000-000000000000010d,0] [APP: DMS Application#11.1.1.1.0] OPMN.queryProcessStatus failed with the following exception[[

                                  oracle.as.management.opmn.optic.OpticBadConnectException: Too few bytes (1) received from OPMN response

                                          at oracle.as.management.opmn.optic.OpmnPhone.receiveResponse(OpmnPhone.java:826)

                                          at oracle.as.management.opmn.optic.OpmnPhone.makePhoneCall(OpmnPhone.java:331)

                                          at oracle.as.management.opmn.optic.OpmnPhone.request(OpmnPhone.java:270)

                                          at oracle.as.management.opmn.optic.OpmnQuery.getBuf(OpmnQuery.java:482)

                                          at oracle.as.management.opmn.optic.OpmnQuery.getDom(OpmnQuery.java:602)

                                          at oracle.as.management.opmn.optic.OpmnQuery.getProc(OpmnQuery.java:750)

                                          at oracle.as.management.mbeans.opmn.Opmn.queryProcessStatus(Opmn.java:328)

                                          at oracle.as.management.mbeans.opmn.Opmn.getStatus(Opmn.java:85)

                                          at oracle.as.management.mbeans.opmn.Opmn.getStatus(Opmn.java:95)

                                          at oracle.as.management.mbeans.register.OpmnInfoImpl.getComponentNames(OpmnInfoImpl.java:34)

                                          at oracle.dms.opmn.app.OpmnDiscoverer.gatherAddresses(OpmnDiscoverer.java:236)

                                          at oracle.dms.address.AddressBook$DiscoverAddressTask.call(AddressBook.java:814)

                                          at oracle.dms.address.AddressBook$DiscoverAddressTask.call(AddressBook.java:786)

                                          at java.util.concurrent.FutureTask.run(FutureTask.java:262)

                                          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

                                          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

                                          at oracle.dms.aggregator.DmsThreadFactory$1$1.run(DmsThreadFactory.java:56)

                                          at java.security.AccessController.doPrivileged(Native Method)

                                          at oracle.dms.aggregator.DmsThreadFactory$1.run(DmsThreadFactory.java:51)

                                          at java.lang.Thread.run(Thread.java:745)

                                   

                                  Regards
                                  Mohammed.

                                  1 2 Previous Next