5 Replies Latest reply on Jul 23, 2013 10:01 PM by user555668

    OC4J turns down automactically

    883788
      Hi have installed oracle 10.1.2.0.2

      i have installed both infra and middle tier ( BI ), on my system.

      oc4j security of infra-tier turns down automactically, which in turn shuts down oc4j_home, oc4j_portal, oc4j_BI forms components.

      $ ./opmnctl status

      Processes in Instance: BIinstance.cscwmeald199v.uk.emea.csc.com
      ------------------------------------------------+---------
      ias-component | process-type | pid | status
      ------------------------------------------------+---------
      LogLoader | logloaderd | N/A | Down
      dcm-daemon | dcm-daemon | 15397 | Alive
      DSA | DSA | N/A | Down
      OC4J | home | N/A | Down
      OC4J | OC4J_Portal | N/A | Down
      OC4J | OC4J_BI_Forms | N/A | Down
      HTTP_Server | HTTP_Server | 25037 | Alive
      WebCache | WebCache | 25039 | Alive
      WebCache | WebCacheAdmin | 25038 | Alive
      Discoverer | ServicesStatus | 25040 | Alive
      Discoverer | PreferenceServer | 25041 | Alive
      wireless | performance_server | 22115 | Alive
      wireless | messaging_server | N/A | Down
      wireless | OC4J_Wireless | N/A | Down

      $
      $ ./opmnctl status

      Processes in Instance: INFRA.cscwmeald199v.uk.emea.csc.com
      ------------------------------------------------+---------
      ias-component | process-type | pid | status
      ------------------------------------------------+---------
      LogLoader | logloaderd | N/A | Down
      dcm-daemon | dcm-daemon | 12813 | Alive
      DSA | DSA | N/A | Down
      OC4J | OC4J_SECURITY | N/A | Down
      HTTP_Server | HTTP_Server | 24331 | Alive
      OID | OID | 24338 | Alive

      $
      $
      $

      *$tail -30f OC4J~OC4J_SECURITY~default_island~1*$--------

      11/10/05 17:28:56 Start process
      --------
      11/10/05 17:29:36 Warning: Error reading transaction-log file (/opt/oracle/OraHome_1/j2ee/OC4J_SECURITY/persistence/OC4J_SECURITY_default_island_1/transaction.state) for recovery: premature end of file
      11/10/05 17:29:36 Forced or abrupt (crash etc) server shutdown detected, starting recovery process...
      11/10/05 17:29:36 Recovery completed, 0 connections committed and 0 rolled back...
      11/10/05 17:29:57 DAS servlet init enter
      11/10/05 17:29:57 oiddas: Release 10.1.2.1.0 Production Started
      11/10/05 17:30:41 DAS servlet init exit
      11/10/05 17:30:47 SSOLoginServlet.init: Starting up SSO server ...
      11/10/05 17:30:47 Initilize NLS utility...
      11/10/05 17:30:47 Reading SSO server policy....
      11/10/05 17:30:47 SSO: Entered FilePolicyManager constructor ...
      11/10/05 17:30:47 SSO: FilePolicyManager: done loading the /opt/oracle/OraHome_1/sso/conf/policy.properties file
      11/10/05 17:30:47 SSO: Leaving FilePolicyManager constructor ...
      11/10/05 17:30:47 SSODebug: Done loading the debug file, /opt/oracle/OraHome_1/sso/log/ssoServer.log
      11/10/05 17:31:08 SSOLoginServlet.init: SSO server started
      11/10/05 17:31:12 Oracle Application Server Containers for J2EE 10g (10.1.2.0.2) initialized

      --------
      11/10/05 22:14:41 Stop process--------

      --------
      11/10/05 22:57:10 Start process
      --------

      $
      $
      $
      $tail -50f ipm.log
      11/10/05 17:28:29 [4] Request 8152 Started. Command: start
      11/10/05 17:28:55 [4] Starting Process: OC4J~OC4J_SECURITY~default_island~1 (20185137:0)
      11/10/05 17:31:12 [4] Process Alive: OC4J~OC4J_SECURITY~default_island~1 (20185137:28692)
      11/10/05 17:31:12 [4] Request 8152 Completed. Command: start
      *11/10/05 17:53:24 [3] [libopmnoc4j] OC4J ping attempt timed out*
      *11/10/05 17:53:27 [4] [libopmnoc4j] Process Ping Failed: OC4J~OC4J_SECURITY~default_island~1 (20185137:28692)*
      *11/10/05 22:09:34 [3] [libopmnoc4j] OC4J ping attempt timed out*
      *11/10/05 22:09:49 [4] [libopmnoc4j] Process Ping Failed: OC4J~OC4J_SECURITY~default_island~1 (20185137:28692)*
      *11/10/05 22:12:09 [3] [libopmnoc4j] OC4J ping attempt timed out*
      *11/10/05 22:12:10 [4] [libopmnoc4j] Process Ping Failed: OC4J~OC4J_SECURITY~default_island~1 (20185137:28692)*
      *11/10/05 22:12:55 [3] [libopmnoc4j] OC4J ping attempt timed out*
      *11/10/05 22:13:11 [4] [libopmnoc4j] Process Ping Failed: OC4J~OC4J_SECURITY~default_island~1 (20185137:28692)*
      *11/10/05 22:14:19 [3] [libopmnoc4j] OC4J ping attempt timed out*
      *11/10/05 22:14:19 [4] [libopmnoc4j] Process Ping Failed: OC4J~OC4J_SECURITY~default_island~1 (20185137:28692)*
      *11/10/05 22:14:19 [4] [libopmnoc4j] Process Unreachable: OC4J~OC4J_SECURITY~default_island~1 (20185137:28692)*
      *11/10/05 22:14:19 [4] Restarting Process: OC4J~OC4J_SECURITY~default_island~1 (20185137:28692)*
      *11/10/05 22:16:19 [4] Stopping Process: OC4J~OC4J_SECURITY~default_island~1 (20185137:28692)*
      *11/10/05 22:16:19 [4] [libopmnoc4j] Forcefully Terminating Process: OC4J~OC4J_SECURITY~default_island~1 (20185137:28692)* 11/10/05 22:36:55 [4] [libopmnohs] Process Ping Failed: HTTP_Server~HTTP_Server~HTTP_Server~1 (20185130:24331) [The connection SSL receive timed out]
      11/10/05 22:36:59 [4] Job 0 123356 result:
      INFRA.cscwmeald199v.uk.emea.csc.com~OC4J~OC4J_SECURITY~default_island~20185137:28692
      Status: Stopped
      Operation: internal (oid dependency failed)
      ErrFile: /opt/oracle/OraHome_1/opmn/logs/OC4J~OC4J_SECURITY~default_island~1
      String: OID

      11/10/05 22:57:09 [4] Starting Process: OC4J~OC4J_SECURITY~default_island~1 (20185138:0)
      11/10/05 23:12:11 [4] Stopping Process: OC4J~OC4J_SECURITY~default_island~1 (20185138:11781)
      11/10/05 23:12:11 [4] [libopmnoc4j] Process Stop Error: OC4J~OC4J_SECURITY~default_island~1 (20185138:11781)
      11/10/05 23:12:11 [4] [libopmnoc4j] Forcefully Terminating Process: OC4J~OC4J_SECURITY~default_island~1 (20185138:11781)
      11/10/05 23:12:13 [4] Process Stopped: OC4J~OC4J_SECURITY~default_island~1 (20185138:11781)



      $
      $

      is it because of lack of memory, can anyone provide a solution for this ?

      i have just now started learning oas 10g
        • 1. Re: OC4J turns down automactically
          Raj Mareddi
          Can you also paste contents of /opt/oracle/OraHome_1/sso/log/ssoServer.log and application.log ?
          • 2. Re: OC4J turns down automactically
            883788
            Very Happy to see your response, kindly guide me to trouble shoot this error

            $ ./opmnctl status

            Processes in Instance: BIinstance.cscwmeald199v.uk.emea.csc.com
            ------------------------------------------------+---------
            ias-component | process-type | pid | status
            ------------------------------------------------+---------
            LogLoader | logloaderd | N/A | Down
            dcm-daemon | dcm-daemon | 15397 | Alive
            DSA | DSA | N/A | Down
            OC4J               | home               |     N/A | Down
            OC4J               | OC4J_Portal        |     N/A | Down
            OC4J               | OC4J_BI_Forms      |     N/A | Down
            HTTP_Server | HTTP_Server | 25037 | Alive
            WebCache | WebCache | 25039 | Alive
            WebCache | WebCacheAdmin | 25038 | Alive
            Discoverer | ServicesStatus | 25040 | Alive
            Discoverer | PreferenceServer | 25041 | Alive
            wireless | performance_server | 22115 | Alive
            wireless | messaging_server | N/A | Down
            wireless | OC4J_Wireless | N/A | Down

            $

            *$Infra tier*

            am unable to find any thing on ssoServer.log -> its empty -> not updated

            $ ls -ltr
            total 18
            -rw-rw-r-- 1 oracle oinstall 6653 Sep 22 13:17 ssoca.log
            -rw-rw-r-- 1 oracle oinstall 0 Sep 22 13:21 ssoreg.err
            -rw-rw-r-- 1 oracle oinstall 1359 Sep 22 13:24 ssoreg.log
            -rw-rw---- 1 oracle oinstall 0 Sep 22 13:31 ssoServer.log
            $ pwd
            /opt/oracle/OraHome_1/sso/log


            Application.log

            $ pwd
            /opt/oracle/OraHome_1/j2ee/OC4J_SECURITY/application-deployments/sso/OC4J_SECURITY_default_island_1
            $ tail -75f application.log

            11/10/04 12:13:25 web: SSOStatusServlet: init
            11/10/04 15:48:50 Started
            11/10/04 15:49:23 web: jsp: init
            11/10/04 15:49:24 web: auth: init
            11/10/04 15:49:39 web: Started
            11/10/04 15:49:40 ssohelp: jsp: init
            11/10/04 15:49:40 ssohelp: ohw: init
            11/10/04 15:49:42 ssohelp: Started
            11/10/04 15:49:47 web: SSOStatusServlet: init
            11/10/04 16:22:51 ssohelp: ohw: destroy
            11/10/04 16:22:51 ssohelp: Stopped
            11/10/04 16:22:51 web: Stopped
            11/10/04 16:22:51 Stopped (Shutdown executed by jazn.com/admin from 127.0.0.1 (localhost))
            11/10/04 16:23:31 Started
            11/10/04 16:24:05 web: jsp: init
            11/10/04 16:24:05 web: auth: init
            11/10/04 16:24:10 web: Started
            11/10/04 16:24:11 ssohelp: jsp: init
            11/10/04 16:24:11 ssohelp: ohw: init
            11/10/04 16:24:12 ssohelp: Started
            11/10/04 16:27:11 web: SSOStatusServlet: init
            11/10/05 09:40:22 Started
            11/10/05 09:41:05 web: jsp: init
            11/10/05 09:41:05 web: auth: init
            11/10/05 09:41:58 web: Started
            11/10/05 09:41:58 ssohelp: jsp: init
            11/10/05 09:41:58 ssohelp: ohw: init
            11/10/05 09:42:03 ssohelp: Started
            11/10/05 09:42:11 web: SSOStatusServlet: init
            11/10/05 10:36:51 Started
            11/10/05 10:39:31 web: jsp: init
            11/10/05 10:39:31 web: auth: init
            11/10/05 10:40:33 web: Started
            11/10/05 10:40:35 ssohelp: jsp: init
            11/10/05 10:40:35 ssohelp: ohw: init
            11/10/05 10:40:41 ssohelp: Started
            11/10/05 10:42:06 Started
            11/10/05 10:43:00 web: jsp: init
            11/10/05 10:43:01 web: auth: init
            11/10/05 10:43:11 web: Started
            11/10/05 10:43:11 ssohelp: jsp: init
            11/10/05 10:43:11 ssohelp: ohw: init
            11/10/05 10:43:12 ssohelp: Started
            11/10/05 10:43:13 web: SSOStatusServlet: init
            11/10/05 15:36:52 Started
            11/10/05 15:38:27 web: jsp: init
            11/10/05 15:38:28 web: auth: init
            11/10/05 15:39:03 web: Started
            11/10/05 15:39:03 ssohelp: jsp: init
            11/10/05 15:39:03 ssohelp: ohw: init
            11/10/05 15:39:13 ssohelp: Started
            11/10/05 15:42:12 web: SSOStatusServlet: init
            11/10/05 17:29:35 Started
            11/10/05 17:30:47 web: jsp: init
            11/10/05 17:30:47 web: auth: init
            11/10/05 17:31:08 web: Started
            11/10/05 17:31:08 ssohelp: jsp: init
            11/10/05 17:31:08 ssohelp: ohw: init
            11/10/05 17:31:12 ssohelp: Started
            11/10/05 17:32:11 web: SSOStatusServlet: init
            ^C$


            Middle tier

            $ pwd
            /export/home/BIinstance/j2ee/*OC4J_Portal*/application-deployments/portal/OC4J_Portal_default_island_1
            $ tail -75f application.log
            Started
            11/10/03 03:57:37 Started
            11/10/03 20:26:15 Started
            11/10/03 23:51:14 Started
            11/10/04 17:04:53 Started
            11/10/04 18:39:20 Started
            11/10/05 09:18:12 Started
            11/10/05 15:55:07 Started
            ^C$


            $ pwd
            /export/home/BIinstance/j2ee/*OC4J_BI_Forms*/application-deployments/*formsapp*/OC4J_BI_Forms_default_island_1
            $ tail -50f application.log
            11/10/05 11:47:00 formsweb: lservlet: destroy
            11/10/05 11:47:00 formsweb: Stopped
            11/10/05 11:47:01 Stopped (Shutdown executed by jazn.com/admin from 127.0.0.1 (localhost))
            11/10/05 11:55:29 Started
            11/10/05 16:03:43 Started
            11/10/05 17:35:26 Started
            11/10/05 17:38:09 formsweb: jsp: init
            11/10/05 17:38:09 formsweb: frmservlet: init
            11/10/05 17:38:09 formsweb: FormsServlet init():
            configFileName: /export/home/BIinstance/forms/server/formsweb.cfg
            testMode: false
            11/10/05 17:38:09 formsweb: Started
            11/10/05 17:38:51 formsweb: lservlet: init
            11/10/05 17:38:51 formsweb: ListenerServlet init()
            11/10/05 17:42:29 formsweb: lservlet: destroy
            11/10/05 17:42:29 formsweb: Stopped
            11/10/05 17:42:30 Stopped (Shutdown executed by jazn.com/admin from 127.0.0.1 (localhost))
            11/10/05 17:44:31 Started
            11/10/05 17:45:24 formsweb: jsp: init
            11/10/05 17:45:25 formsweb: frmservlet: init
            11/10/05 17:45:25 formsweb: FormsServlet init():
            configFileName: /export/home/BIinstance/forms/server/formsweb.cfg
            testMode: false
            11/10/05 17:45:25 formsweb: Started
            11/10/05 17:45:26 formsweb: lservlet: init
            11/10/05 17:45:26 formsweb: ListenerServlet init()
            11/10/05 17:48:17 formsweb: lservlet: destroy
            11/10/05 17:48:17 formsweb: Stopped
            11/10/05 17:48:18 Stopped (Shutdown executed by jazn.com/admin from 127.0.0.1 (localhost))
            11/10/05 17:49:40 Started
            11/10/05 17:50:18 formsweb: jsp: init
            11/10/05 17:50:18 formsweb: frmservlet: init
            11/10/05 17:50:18 formsweb: FormsServlet init():
            configFileName: /export/home/BIinstance/forms/server/formsweb.cfg
            testMode: false
            11/10/05 17:50:18 formsweb: Started
            11/10/05 17:50:19 formsweb: lservlet: init
            11/10/05 17:50:19 formsweb: ListenerServlet init()
            11/10/05 17:54:30 formsweb: lservlet: destroy
            11/10/05 17:54:30 formsweb: Stopped
            11/10/05 17:54:31 Stopped (Shutdown executed by jazn.com/admin from 127.0.0.1 (localhost))
            11/10/05 17:56:07 Started
            11/10/05 17:57:01 formsweb: jsp: init
            11/10/05 17:57:01 formsweb: frmservlet: init
            11/10/05 17:57:01 formsweb: FormsServlet init():
            configFileName: /export/home/BIinstance/forms/server/formsweb.cfg
            testMode: false
            11/10/05 17:57:01 formsweb: Started
            11/10/05 17:57:04 formsweb: lservlet: init
            11/10/05 17:57:04 formsweb: ListenerServlet init()
            ^C$

            ^C$ pwd
            /export/home/BIinstance/j2ee/*OC4J_BI_Forms*/application-deployments/*reports*/OC4J_BI_Forms_default_island_1

            $ tail -50f application.log
            11/10/05 16:08:24 web: Reports servlet - /reports/rwservlet: init
            11/10/05 16:10:48 web: Started
            11/10/05 16:11:20 reports_demo: jsp: init
            11/10/05 16:11:23 reports_demo: Started
            11/10/05 17:34:40 Started
            11/10/05 17:35:32 web: jsp: init
            11/10/05 17:35:34 web: Reports servlet - /reports/rwservlet: init
            11/10/05 17:35:37 web: Started
            11/10/05 17:35:38 reports_demo: jsp: init
            11/10/05 17:35:38 reports_demo: Started
            11/10/05 17:41:48 web: Broken pipe
            11/10/05 17:41:56 web: Broken pipe
            11/10/05 17:42:01 web: Broken pipe
            11/10/05 17:42:29 web: Stopped
            11/10/05 17:42:29 reports_demo: Stopped
            11/10/05 17:42:30 Stopped (Shutdown executed by jazn.com/admin from 127.0.0.1 (localhost))
            11/10/05 17:44:09 Started
            11/10/05 17:44:39 web: jsp: init
            11/10/05 17:44:40 web: Reports servlet - /reports/rwservlet: init
            11/10/05 17:44:49 web: Started
            11/10/05 17:44:50 reports_demo: jsp: init
            11/10/05 17:44:50 reports_demo: Started
            11/10/05 17:47:46 web: Broken pipe
            11/10/05 17:48:11 web: Broken pipe
            11/10/05 17:48:17 web: Stopped
            11/10/05 17:48:17 reports_demo: Stopped
            11/10/05 17:48:18 Stopped (Shutdown executed by jazn.com/admin from 127.0.0.1 (localhost))
            11/10/05 17:49:37 Started
            11/10/05 17:49:50 web: jsp: init
            11/10/05 17:49:50 web: Reports servlet - /reports/rwservlet: init
            11/10/05 17:49:56 web: Started
            11/10/05 17:49:56 reports_demo: jsp: init
            11/10/05 17:49:56 reports_demo: Started
            11/10/05 17:53:59 web: Broken pipe
            11/10/05 17:54:08 web: Broken pipe
            11/10/05 17:54:17 web: Broken pipe
            11/10/05 17:54:30 web: Stopped
            11/10/05 17:54:30 reports_demo: Stopped
            11/10/05 17:54:31 Stopped (Shutdown executed by jazn.com/admin from 127.0.0.1 (localhost))
            11/10/05 17:55:50 Started
            11/10/05 17:56:12 web: jsp: init
            11/10/05 17:56:12 web: Reports servlet - /reports/rwservlet: init
            11/10/05 17:56:14 web: Started
            11/10/05 17:56:14 reports_demo: jsp: init
            11/10/05 17:56:14 reports_demo: Started
            11/10/05 17:59:25 web: Broken pipe
            11/10/05 17:59:25 web: Broken pipe
            11/10/05 22:10:11 web: Broken pipe
            11/10/05 22:14:02 web: Broken pipe
            11/10/05 22:15:47 web: Broken pipe
            ^C$

            my observation from the logs is oc4j_security which is in infra would goes down automatically, which in turns brings oc4j_home, oc4j_portal and oc4j_BI_Forms down.

            i would like to know reason behind this and looking for a solution for this
            • 3. Re: OC4J turns down automactically
              Fabian
              Hi,

              Indeed you had good observation,
              "*my observation from the logs is oc4j_security which is in infra would goes down automatically, which in turns brings oc4j_home, oc4j_portal and oc4j_BI_Forms down*"

              Its true,Its because of oc4j_security,your entire oc4j_home, oc4j_portal and oc4j_BI_Forms goes down

              Try the below steps,
              And then bounce the entire Oracle Infra and BI home.

              1st step(Take a back up of opmn.xml file and do the below changes)
              Remove the line or comment the line from opmn.xml
              location: Infra Home/opmn/conf

              - <dependencies>
              <OID infrastructure="true" />
              </dependencies>

              2nd step

              validate your OMPN.xml using :-
              1)opmnctl validate

              update and resync console with new configuration:

              1)dcmctl updateconfig -ct opmn -v -d
              2)dcmctl resyncinstance -v -d

              Hope this helps ya.

              Regards
              Fabian
              • 4. Re: OC4J turns down automactically
                883788
                Thanks My issue is resolved :-)
                • 5. Re: OC4J turns down automactically
                  user555668

                  I am facing similar problem.... applied above steps...but no use...still problem remains same...

                  Any idea ???

                  here is my post...

                   

                  oc4j_security goes down automatically

                   

                  Regards

                  Masood