3 Replies Latest reply on Jan 26, 2010 10:00 AM by user121044

    Notification Mailer not starting.

    user121044
      Hi Guru's,

      Our production server has suddenly started misbehaving.
      We are running on two nodes (1 DB and the other is application) running on HP-UX Itanium release 12.0.4.
      Starting yesterday, the notification mailer in OAM has hung on status "starting" and when i try and stop the containers,
      they hang on status deactivating.
      The log files shows the following:-

      LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsc.Logger.Logger(String, int) : Logging to System.out until necessary parameters are retrieved for Logger to be properly started.
      LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsm.GSMSvcComponentContainer.initializeStateMachine() : BEGIN
      LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsc.SvcComponentContainer.getNewWorkflowContext() : BEGIN
      LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsc.SvcComponentContainer.getNewWorkflowContext() : BEGIN
      LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsc.SvcComponentContainer.loadGlobalParameters() : BEGIN
      LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsm.GSMSvcComponentContainer.loadContainerParameters() : BEGIN
      LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsc.SvcComponentContainer.start() : Successfully retrieved global and container parameters
      LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsc.SvcComponentContainer.validateParameterValues(Properties) : BEGIN
      LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsc.SvcComponentContainer.validateParameterValues(Properties) : ({SVC_COMP_MAX_ERROR_COUNT=10, SVC_COMP_MONITOR_LOOP_SLEEP=60, SVC_CONTAINER_LOOP_SLEEP=10, SVC_CONTAINER_READ_TIMEOUT=10, SVC_COMP_MONITOR_ONDEMAND_FREQ=300, SVC_CONTAINER_LOG_LEVEL=4, SVC_WRITE_DIAG_TO_GSM_LOG=Y})
      LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsc.SvcComponentContainer.start() : Successfully validated container parameters
      LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsm.GSMSvcComponentContainer.loadDetails(Connection) : BEGIN (oracle.jdbc.driver.T4CConnection@530cf2)
      LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsc.SvcComponentContainer.start() : Successfully retrieved container details
      LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsm.GSMSvcComponentContainer.startLogger() : BEGIN
      LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsc.SvcComponentContainer.startLogger() : BEGIN [default implementation]
      SVC-GSM-WFMLRSVC-120037-10006 : oracle.apps.fnd.cp.gsc.Logger.Logger(String, int) : Logging to System.out until necessary parameters are retrieved for Logger to be properly started.
      oracle.jms.AQjmsException: JMS-120: Dequeue failed
           at oracle.jms.AQjmsError.throwEx(AQjmsError.java:308)
           at oracle.jms.AQjmsConsumer.dequeue(AQjmsConsumer.java:2094)
           at oracle.jms.AQjmsConsumer.receiveFromAQ(AQjmsConsumer.java:922)
           at oracle.jms.AQjmsConsumer.receiveFromAQ(AQjmsConsumer.java:835)
           at oracle.jms.AQjmsConsumer.receive(AQjmsConsumer.java:776)
           at oracle.apps.fnd.wf.bes.ControlListener.run(ControlListener.java:74)

      [Linked-exception]
      java.sql.SQLException: ORA-00604: error occurred at recursive SQL level 1
      ORA-04031: unable to allocate 4080 bytes of shared memory ("shared pool","unknown object","sga heap(1,0)","library cache")

           at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
           at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:331)
           at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:288)
           at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:745)
           at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:215)
           at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:965)
           at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1170)
           at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3339)
           at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3384)
           at oracle.jms.AQjmsConsumer.dequeue(AQjmsConsumer.java:1601)
           at oracle.jms.AQjmsConsumer.receiveFromAQ(AQjmsConsumer.java:922)
           at oracle.jms.AQjmsConsumer.receiveFromAQ(AQjmsConsumer.java:835)
           at oracle.jms.AQjmsConsumer.receive(AQjmsConsumer.java:776)
           at oracle.apps.fnd.wf.bes.ControlListener.run(ControlListener.java:74)

      oracle.jms.AQjmsException: ORA-04031: unable to allocate 4080 bytes of shared memory ("shared pool","unknown object","sga heap(1,0)","library cache")

           at oracle.jms.AQjmsSession.removeSubscriber(AQjmsSession.java:4223)
           at oracle.jms.AQjmsSession.unsubscribe(AQjmsSession.java:4083)
           at oracle.apps.fnd.wf.bes.ControlConnection.unSubscribe(ControlConnection.java:209)
           at oracle.apps.fnd.wf.bes.ControlListener.run(ControlListener.java:118)




      i have tried pinning the objects into the shared pool but to no avail.
      I also tried bouncing the concurrent manager but it says :-



      Starting PROD_0105@PROD Internal Concurrent Manager -- shell process ID 3821

      logfile=/oracle/inst/apps/PROD_erpapps/logs/appl/conc/log/PROD_0105.mgr
      PRINTER=noprint
      mailto=applmgr
      restart=N
      diag=N
      sleep=30
      pmon=4
      quesiz=1
      Reviver is ENABLED



      Found dead process: spid=(29986), cpid=(119043), ORA pid=(50), manager=(0/4)


      Found dead process: spid=(29987), cpid=(119044), ORA pid=(52), manager=(0/5)


      Found dead process: spid=(29988), cpid=(119045), ORA pid=(48), manager=(275/40)




      Kindly assist me.