9 Replies Latest reply on May 22, 2018 1:00 PM by Beauty_and_dBest

    Starting EBS 12.2.7 apps tier

    Beauty_and_dBest

      EBS 12.2.7

      12c

      OL6.8

       

       

      Our new cloned EBS 12.2.7 apps tier has error on oacore_server1 when started.

       

       

       

       

      [Service Control Execution Report]

      The report format is:

        <Service Group>            <Service>                                 <Script>                   <Status>

       

        Root Service                                                                                    Enabled

        Root Service               NodeManager                               adnodemgrctl.sh            Started

        Web Administration                                                                              Enabled

        Web Administration         AdminServer                               adadminsrvctl.sh           Started

        Web Administration         OracleTNSListenerAPPS_CRP_lehebsr12       adalnctl.sh                Started

        Web Entry Point Services                                                                        Enabled

        Web Entry Point Services   Oracle HTTP Server CRP_lehebsr12          adapcctl.sh                Started

        Web Entry Point Services   Oracle Process Manager for CRP_lehebsr12  adopmnctl.sh               Started

        Web Application Services                                                                        Enabled

        Web Application Services   oacore_server1                            admanagedsrvctl.sh         Failed

        Web Application Services   forms_server1                             admanagedsrvctl.sh         Started

        Web Application Services   oafm_server1                              admanagedsrvctl.sh         Started

        Web Application Services   forms-c4ws_server1                        admanagedsrvctl.sh         Disabled

        Web Application Services   oaea_server1                              admanagedsrvctl.sh         Disabled

        Batch Processing Services                                                                       Enabled

        Batch Processing Services  OracleConcMgrCRP_lehebsr12                adcmctl.sh                 Started

        Batch Processing Services  Oracle ICSM CRP_lehebsr12                 ieoicsm.sh                 Disabled

        Batch Processing Services  Oracle Fulfillment Server CRP_lehebsr12   jtffmctl.sh                Started

        Other Services                                                                                  Disabled

        Other Services             OracleFormsServer-Forms CRP_lehebsr12     adformsrvctl.sh            Disabled

        Other Services             Oracle MWA Service CRP_lehebsr12          mwactlwrpr.sh              Disabled

       

      Total time taken for execution is 2 mins 48 seconds.

       

       

      ServiceControl is exiting with status 1

      ===========

      ****************************************************

       

       

       

       

       

      Executing service control script:

      /u01/appcrp/CRP/fs2/inst/apps/CRP_lehebsr12/admin/scripts/admanagedsrvctl.sh start forms_server1 -nopromptmsg

      Timeout specified in context file: -1 second(s)

       

      script returned:

      ****************************************************

       

      You are running admanagedsrvctl.sh version 120.14.12020000.11

       

      Starting forms_server1...

       

      Server specific logs are located at /u01/appcrp/CRP/fs2/FMW_Home/user_projects/domains/EBS_domain_CRP/servers/forms_server1/logs

       

      admanagedsrvctl.sh: exiting with status 1

       

      admanagedsrvctl.sh: check the logfile /u01/appcrp/CRP/fs2/inst/apps/CRP_lehebsr12/logs/appl/admin/log/adformsctl.txt for more information ...

       

       

      .end std out.

       

      .end err out.

       

      ****************************************************

       

       

       

      Executing service control script:

      /u01/appcrp/CRP/fs2/inst/apps/CRP_lehebsr12/admin/scripts/admanagedsrvctl.sh start oacore_server1 -nopromptmsg

      Timeout specified in context file: -1 second(s)

       

      script returned:

      ****************************************************

       

      You are running admanagedsrvctl.sh version 120.14.12020000.11

       

      Starting oacore_server1...

       

      Server specific logs are located at /u01/appcrp/CRP/fs2/FMW_Home/user_projects/domains/EBS_domain_CRP/servers/oacore_server1/logs

       

      admanagedsrvctl.sh: exiting with status 1

       

      admanagedsrvctl.sh: check the logfile /u01/appcrp/CRP/fs2/inst/apps/CRP_lehebsr12/logs/appl/admin/log/adoacorectl.txt for more information ...

       

       

      .end std out.

       

      .end err out.

       

      ****************************************************

       

       

       

      Executing service control script:

      /u01/appcrp/CRP/fs2/inst/apps/CRP_lehebsr12/admin/scripts/admanagedsrvctl.sh start oafm_server1 -nopromptmsg

      Timeout specified in context file: -1 second(s)

       

       

       

      Please help

       

      Kind regards,

      jc

        • 1. Re: Starting EBS 12.2.7 apps tier
          Beauty_and_dBest

          Error log

           

          spolicy to get more information. Error message: unable to create new native thread

          weblogic.security.SecurityInitializationException: The loading of OPSS java security policy provider failed due to exception, see the exception stack trace or the server log file for root cause. If still see no obvious cause, enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: unable to create new native thread

                  at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.loadOPSSPolicy(CommonSecurityServiceManagerDelegateImpl.java:1402)

                  at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1022)

                  at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:888)

                  at weblogic.security.SecurityService.start(SecurityService.java:141)

                  at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)

                  Truncated. see log file for complete stacktrace

          Caused By: java.lang.OutOfMemoryError: unable to create new native thread

                  at java.lang.Thread.start0(Native Method)

                  at java.lang.Thread.start(Thread.java:714)

                  at oracle.security.jps.az.internal.runtime.notifier.RuntimeEventNotifierImpl.start(RuntimeEventNotifierImpl.java:198)

                  at oracle.security.jps.az.internal.runtime.notifier.RuntimeEventNotifierImpl.<init>(RuntimeEventNotifierImpl.java:90)

                  at oracle.security.jps.az.internal.runtime.service.PDPServiceImpl.initializeUncontrolledMode(PDPServiceImpl.java:785)

                  Truncated. see log file for complete stacktrace

          >

          <May 22, 2018 10:30:42 AM PHT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>

          <May 22, 2018 10:30:42 AM PHT> <Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down>

           

           

          INFO: Pushed ldap name and types info to libOvd. Ldaps : DefaultAuthenticator:idstore.ldap.provideridstore.ldap.

          <May 22, 2018 10:30:41 AM PHT> <Notice> <Security> <BEA-090082> <Security initializing using security realm myrealm.>

          <May 22, 2018 10:30:42 AM PHT> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: java.lang.OutOfMemoryError: unable to create new native thread

          java.lang.OutOfMemoryError: unable to create new native thread

                  at java.lang.Thread.start0(Native Method)

                  at java.lang.Thread.start(Thread.java:714)

                  at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:474)

                  at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:423)

                  at weblogic.store.admin.AdminHandler.activate(AdminHandler.java:126)

                  Truncated. see log file for complete stacktrace

          >

          <May 22, 2018 10:30:42 AM PHT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>

          • 2. Re: Starting EBS 12.2.7 apps tier
            VishnuVinnakota

            Hi,

             

              It is the issue with java memory allocation. Once you login to WebLogic console, for each managed server failed to start check the startup parameters reduce the java memory allocated and try to restart.

             

            Thanks,

            Vishnu

            • 3. Re: Starting EBS 12.2.7 apps tier
              Beauty_and_dBest

              Hi ALL,

               

              Is this cause by small ulimit? How do I increase my applmgr ulimit? and what parameter do I need to increase?

               

              [applmgr@ebsr12 ~]$ ulimit -a

              core file size          (blocks, -c) 0

              data seg size           (kbytes, -d) unlimited

              scheduling priority             (-e) 0

              file size               (blocks, -f) unlimited

              pending signals                 (-i) 515710

              max locked memory       (kbytes, -l) 64

              max memory size         (kbytes, -m) unlimited

              open files                      (-n) 4096

              pipe size            (512 bytes, -p) 8

              POSIX message queues     (bytes, -q) 819200

              real-time priority              (-r) 0

              stack size              (kbytes, -s) 10240

              cpu time               (seconds, -t) unlimited

              max user processes              (-u) 1024

              virtual memory          (kbytes, -v) unlimited

              file locks                      (-x) unlimited

               

              Kind regards,

              • 4. Re: Starting EBS 12.2.7 apps tier
                Beauty_and_dBest

                Hi Vishnu,

                 

                Re:

                It is the issue with java memory allocation. Once you login to WebLogic console, for each managed server failed to start check the startup parameters reduce the java memory allocated and try to restart.

                 

                How do I reduce it? Can I not increase it instead? I have been cloning this image many times and its the first time I got this error

                 

                Please help....

                • 5. Re: Starting EBS 12.2.7 apps tier
                  Beauty_and_dBest

                  Hi ALL,

                   

                  What command do I run to obtain this report list?

                   

                  [Service Control Execution Report]

                  The report format is:

                    <Service Group>            <Service>                                 <Script>                   <Status>

                   

                    Root Service                                                                                    Enabled

                    Root Service               NodeManager                               adnodemgrctl.sh            Started

                    Web Administration                                                                              Enabled

                    Web Administration         AdminServer                               adadminsrvctl.sh           Started

                    Web Administration         OracleTNSListenerAPPS_CRP_lehebsr12       adalnctl.sh                Started

                    Web Entry Point Services                                                                        Enabled

                    Web Entry Point Services   Oracle HTTP Server CRP_lehebsr12          adapcctl.sh                Started

                    Web Entry Point Services   Oracle Process Manager for CRP_lehebsr12  adopmnctl.sh               Started

                    Web Application Services                                                                        Enabled

                    Web Application Services   oacore_server1                            admanagedsrvctl.sh         Failed

                    Web Application Services   forms_server1                             admanagedsrvctl.sh         Started

                    Web Application Services   oafm_server1                              admanagedsrvctl.sh         Started

                    Web Application Services   forms-c4ws_server1                        admanagedsrvctl.sh         Disabled

                    Web Application Services   oaea_server1                              admanagedsrvctl.sh         Disabled

                    Batch Processing Services                                                                       Enabled

                    Batch Processing Services  OracleConcMgrCRP_lehebsr12                adcmctl.sh                 Started

                    Batch Processing Services  Oracle ICSM CRP_lehebsr12                 ieoicsm.sh                 Disabled

                    Batch Processing Services  Oracle Fulfillment Server CRP_lehebsr12   jtffmctl.sh                Started

                    Other Services                                                                                  Disabled

                    Other Services             OracleFormsServer-Forms CRP_lehebsr12     adformsrvctl.sh            Disabled

                    Other Services             Oracle MWA Service CRP_lehebsr12          mwactlwrpr.sh              Disabled

                   

                  Total time taken for execution is 2 mins 48 seconds.

                   

                   

                  Thanks a lot.

                  • 6. Re: Starting EBS 12.2.7 apps tier
                    VishnuVinnakota

                    Hi,

                     

                        Can you give me values for OACORE_SERVER1 and also FORMS_SERVER1 from WebLogic Console as below?

                     

                         oacore_mem.png

                     

                        Next time when you start, kill all processes of App user using kill -u applmgr or simply search with ps -ef|grep fs1 and ps -ef|grep fs2 and kill all prcoesses.

                     

                       Regarding ulimit values, values look fine. Should not be an issue.

                     

                    Thanks,

                    Vishnu

                    • 7. Re: Starting EBS 12.2.7 apps tier
                      Beauty_and_dBest

                      Thanks Vishnu and ALL,

                       

                      I forgot how to login to weblogic console, I have not used it for a long time.

                       

                      I tried this steps:

                       

                      1.)

                      $ grep listen-port $EBS_DOMAIN_HOME/config/config.xml

                          <listen-port>7022</listen-port>

                       

                      2.)

                      $ grep "s_wls_adminport" $CONTEXT_FILE

                               <wls_adminport oa_var="s_wls_adminport" oa_type="PORT" base="7001" step="1" range="-1" label="WLS Admin Server Port">7022</wls_adminport>

                       

                      Then http://server.domain:7022/console but I got error

                       

                      Please help....

                       

                      By the way, for the failed  process > Web Application Services   oacore_server1   admanagedsrvctl.sh   Failed

                       

                      If I restart it alone, it will now succeed, and I can login to the apps webpage.

                       

                      But why is that if I start it using adstrtal  it will fail with jave out of memory error?

                       

                       

                       

                      Kind regards,

                      • 8. Re: Starting EBS 12.2.7 apps tier
                        VishnuVinnakota

                        Hi,

                         

                          Do you remember password?

                         

                          Firstly, adadminsrvctl.sh status and then start.

                         

                         

                        Method 1

                         

                        Open the Application tier context file

                        vi $CONTEXTFILE

                         

                        Check the value of 'WLS Admin Server Port' from "s_wls_adminport" parameter

                         

                         

                        Method 2

                         

                        Open the EBS domain config file

                        vi $EBS_DOMAIN_HOME/config/config.xml

                         

                        Check the 'listen-port' value of the 'AdminServer'

                         

                         

                        Weblogic console URL

                         

                        http://<server name>. <domain name> : <weblogic Admin Port>/console

                         

                        Ex: http://oracle.test.com:7002/console

                         

                        Thanks,

                        Vishnu

                        • 9. Re: Starting EBS 12.2.7 apps tier
                          Beauty_and_dBest

                          Hi Vishnu and ALL,

                           

                           

                          Yes I tried that but it won't open on port 7022

                           

                          The password is just default welcome1, and the apps start successfully.

                           

                          How do I troubleshoot it why can not login?

                           

                           

                          Thanks.