12 Replies Latest reply on Mar 11, 2016 4:11 PM by user12209669

    NullPointerException - OACORE SERVER STUCK ISSUE

    ProDBA

      Hi folks,

       

      We are running E-Business Suite R12.2.4. We had configured 2 non-shared application nodes with 7 oacore servers each. Recently, we are facing Oacore Server Stuck Thread issue. The following lines has been identified from  oacore log file:

       

      <Mar 3, 2016 2:47:41 PM PKT> <Error> <ServletContext-/OA_HTML> <BEA-000000> <chain failed

      java.lang.NullPointerException

        at _pages._jsp._fnd.__fnderror._jspService(__fnderror.java:485)

        at weblogic.servlet.jsp.JspBase.service(JspBase.java:34)

        at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)

        at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)

        at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:301)

        Truncated. see log file for complete stacktrace

      >

      <Mar 3, 2016 2:47:41 PM PKT> <Error> <ServletContext-/OA_HTML> <BEA-000000> <chain failed

      java.lang.NullPointerException

        at _pages._jsp._fnd.__fnderror._jspService(__fnderror.java:485)

        at weblogic.servlet.jsp.JspBase.service(JspBase.java:34)

        at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)

        at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)

        at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:301)

        Truncated. see log file for complete stacktrace

       

      However, I'm attaching complete log file for your kind review. Please, advice us the appropriate solution and cause of the problem.

       

      Regards,

      Ali Raza Memon

        • 1. Re: NullPointerException - OACORE SERVER STUCK ISSUE
          Srini Chavali-Oracle

          Suggest you open an SR. MOS Doc 2090612.1 may be related.

           

          Was this working before ? If so, what changes have been made recently ?

          • 2. Re: NullPointerException - OACORE SERVER STUCK ISSUE
            Ariz - Apps DBA

            Please post complete details of your environment, DB version, OS version etc

            • 3. Re: NullPointerException - OACORE SERVER STUCK ISSUE
              VishnuVinnakota

              I fixed the issue for one of customers applying the patches in the Doc ID: 2087130.1

               

              How many number of cocnurrent users do you have?

               

              What is the memory and cpu of your Application Node?

              • 4. Re: NullPointerException - OACORE SERVER STUCK ISSUE
                ProDBA

                Actually we have upgraded this system around 2 months ago. Since than we are facing many performance issues. Hence, many issues were resolved and system is being mature. We have already raised SR, and many problems were resolved. I suppose VishnuVinnakota pointed to the correct MOS Document. We're currently working on this.

                • 5. Re: NullPointerException - OACORE SERVER STUCK ISSUE
                  ProDBA

                  Half Rack Exadata with 2 non-shared application nodes over Linux box.

                  • 6. Re: NullPointerException - OACORE SERVER STUCK ISSUE
                    ProDBA

                    I suppose your Document points to correct resolution. We'll apply the suggested patch shortly. We have around 2000 users that are accessing application and in future they will be multiple. 256 GB of RAM and 80 CPUs with 10/core. Database on Exadata with half rack.

                    • 7. Re: NullPointerException - OACORE SERVER STUCK ISSUE
                      VishnuVinnakota

                      Yes. The environment looks similar. The client I mentioned had Super Cluster (obviously Exadata Database).

                       

                      • Apply those patches
                      • 2000 users... 10 OACOREs Ideally are required for 2000 users as 200 users per OACORE makes sense. Add required OACORE processes
                      • Last but not the least: Modify the Startup Arguments of all OACORE Managed Servers from WebLogic Console with Memory Xmx: 2048 and Permsize: 1024

                       

                      Let me know how it goes..

                      • 8. Re: NullPointerException - OACORE SERVER STUCK ISSUE
                        user12209669

                        Hi Vishnu,

                         

                        We have already 14 OACOREs (2 appl nodes with 7 oacores on each node), and current argument values for each oacore managed servers is '-XX:Permsize=128m -XX:MaxPermsize=384 -Xms3072m -Xmx3072m'. Now what you suggest  at this stage to overcome our the issue as this is occurring frequently and application is almost at dead situation unless restart the services.


                        Regards,

                        • 9. Re: NullPointerException - OACORE SERVER STUCK ISSUE
                          user12209669

                          Hi experts,

                           

                          Please update ...

                          • 10. Re: NullPointerException - OACORE SERVER STUCK ISSUE
                            VishnuVinnakota

                            This is solvable..

                             

                            1. Whats the exact OACORE Managed Server error when your application is crashing. Check this from EM Console of WebLogic (Not WebLogic Console). You can also check from logs from File System
                            2. What is the connection pool maximum value in EBS Data Source? This could be the root cause as well.
                            • 11. Re: NullPointerException - OACORE SERVER STUCK ISSUE
                              ProDBA

                              1. The following sort of errors appear within oacore servers logs.

                               

                              <Feb 19, 2016 3:22:29 PM PKT> <Warning> <JDBC> <BEA-001153> <Forcibly releasing inactive/harvested connection "weblogic.jdbc.wrapper.PoolConnection_oracle_jdbc_driver_T4CConnection@81c5f" back into the data source connection pool "EBSDataSource", currently reserved by: java.lang.Exception

                                      at weblogic.jdbc.common.internal.ConnectionEnv.setup(ConnectionEnv.java:366)

                                      at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:373)

                                      at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:339)

                                      at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:469)

                                      at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:363)

                                      at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:125)

                                      at weblogic.jdbc.common.internal.RmiDataSource.getPoolConnection(RmiDataSource.java:469)

                                      at weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:553)

                                      at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:513)

                                      at sun.reflect.GeneratedMethodAccessor191.invoke(Unknown Source)

                                      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

                                      at java.lang.reflect.Method.invoke(Method.java:606)

                                      at oracle.apps.fnd.security.DBConnObjWLSDSPool.getConnection(DBConnObjWLSDSPool.java:373)

                                      at oracle.apps.fnd.security.AppsConnectionManagerWLSDS.dbConnect(AppsConnectionManagerWLSDS.java:223)

                                      at oracle.apps.fnd.security.AppsConnectionManagerWLSDS.localAppsConnect(AppsConnectionManagerWLSDS.java:193)

                                      at oracle.apps.fnd.security.AppsConnectionManagerWLSDS.makeGuestConnection(AppsConnectionManagerWLSDS.java:68)

                                      at oracle.apps.fnd.security.DBConnObj.getLabelledConnection(DBConnObj.java:314)

                                      at oracle.apps.fnd.security.DBConnObj.<init>(DBConnObj.java:233)

                                      at oracle.apps.fnd.security.DBConnObjDS.<init>(DBConnObjDS.java:98)

                                      at sun.reflect.GeneratedConstructorAccessor142.newInstance(Unknown Source)

                                      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

                                      at java.lang.reflect.Constructor.newInstance(Constructor.java:526)

                                      at oracle.apps.fnd.common.Pool.createObject(Pool.java:1276)

                                      at oracle.apps.fnd.security.DBConnObjWLSDSPool.borrowObjectX(DBConnObjWLSDSPool.java:173)

                              ================

                              java.lang.NullPointerException

                                      at _pages._jsp._fnd.__fnderror._jspService(__fnderror.java:485)

                                      at weblogic.servlet.jsp.JspBase.service(JspBase.java:34)

                                      at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)

                                      at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)

                                      at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:301)

                                      Truncated. see log file for complete stacktrace

                              ==================

                              java.lang.IllegalStateException: Cannot resize buffer - 144 bytes have already been written (Servlet 2.3, sec. 5.1)

                                      at weblogic.servlet.internal.ServletResponseImpl.setBufferSize(ServletResponseImpl.java:357)

                                      at javax.servlet.ServletResponseWrapper.setBufferSize(ServletResponseWrapper.java:159)

                                      at weblogic.servlet.jsp.PageContextImpl.initialize(PageContextImpl.java:74)

                                      at weblogic.servlet.jsp.PageContextImpl.<init>(PageContextImpl.java:110)

                                      at weblogic.servlet.jsp.JspFactoryImpl.getPageContext(JspFactoryImpl.java:40)

                                      Truncated. see log file for complete stacktrace

                              >

                              48412

                              <Feb 19, 2016 1:19:57 PM PKT> <Error> <ServletContext-/OA_HTML> <BEA-000000> <chain failed

                              java.net.SocketException: Connection reset

                                      at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)

                                      at java.net.SocketOutputStream.write(SocketOutputStream.java:153)

                                      at weblogic.servlet.internal.ChunkOutput.writeChunkNoTransfer(ChunkOutput.java:591)

                                      at weblogic.servlet.internal.ChunkOutput.writeChunks(ChunkOutput.java:540)

                                      at weblogic.servlet.internal.ChunkOutput.flush(ChunkOutput.java:427)

                                      Truncated. see log file for complete stacktrace

                               

                              2. Please, review the attached screenshots.

                               

                              1.jpg2.jpg