13 Replies Latest reply: Apr 30, 2013 12:58 AM by praveenps RSS

    The JMS and SUB adapters shutdown (RIB-SIM)

    Orlando De Abreu
      Products:
      Retail Merchandising System (RMS)
      Retail Integration Bus (RIB) 13.1.1
      Store Inventory Management (SIM)

      Platform: HP-UX Itanium
      Platform Version: 11.31

      The JMS and SUB Hospital Retry adapters of RIB-SIM are fallen with the following errors:

      Caused by: javax.ejb.EJBException: Unable to flush hospital.; Nested exception is: com.retek.platform.persistence.PersistenceException:
      Exception Description: Error while Obtaining information about the database. Please look at the nested exception for more details.; Nested exception is: Exception [TopLink-4019] (Oracle TopLink Essentials - 2.0.1 (Build b09d-fcs (12/06/2007))): oracle.toplink.essentials. exceptions.DatabaseException
      Exception Description: Error while Obtaining information about the database. Please look at the nested exception for more details.

      Caused by: java.sql.SQLException: javax.resource.ResourceException: RollbackException: Resource use was in. During abnormal shutdown of server processing and During subsequent recovery was unobtainable or failed Either To Have recovery permissions. If This issue logs Consult Continues to Occur after the next interval completes recovery processing
      at oracle.oc4j.sql.spi.ManagedConnectionImpl.setupTransaction (ManagedConnectionImpl.java: 841)
      at oracle.oc4j.sql.spi.ConnectionHandle.oc4j_intercept (ConnectionHandle.java: 305)
      at oracle_jdbc_driver_LogicalConnection_Proxy.getMetaData ()
      at


      The following note (basically restarting instances) was attempted but did not resolve:
      "Why Are tafr JMS Hospital Hospital Sub Retry Retry tafr And Shutting Down? [ID 1339893.1]"

      Customer attempted the following with same resutls:

      It carried out the following actions:
      1 - was restarted only container RIB-SIM. The error persists.
      2 - I restarted all of apps RIB (opmnctl). The error persists.
      3 - I restarted all RIB containers and shutdown the RIB database, start the RIB database and containers and the error persists.

      Despite this we get the same results.
        • 1. Re: The JMS and SUB adapters shutdown (RIB-SIM)
          praveenps
          Hi Orlando,

          Please provide more info

          1)Full error for one retry.
          2)How many records are pending in rib hospital.
          2)Check for xml files in rib log folder inside Oracle_home.

          Regards,
          PPS
          • 2. Re: The JMS and SUB adapters shutdown (RIB-SIM)
            Orlando De Abreu
            Hi praveenps,

            Regarding your questions, I got the following:

            1) For the JMS adapter:

            2013-04-16 13:55:46,850 [Timer-21] DEBUG com.retek.rib.j2ee.ErrorHospitalRetryTimerTask - >>>>>>>>>>>>>>>Time's up!<<2013-04-16 13:55:46,858 [Timer-21] DEBUG com.retek.rib.j2ee.ErrorHospitalRetryTimerTask - Calling Hospital Retry EJB begin.
            2013-04-16 13:55:46,858 [Timer-21] DEBUG com.retek.rib.j2ee.ErrorHospitalRetryTimerTask - Getting from JNDI Hospital Retry EJB Local Home.
            2013-04-16 13:55:46,858 [Timer-21] DEBUG com.retek.rib.j2ee.ErrorHospitalRetryTimerTask - Got Hospital Retry EJB Local Home from JNDI.
            2013-04-16 13:55:46,859 [Timer-21] DEBUG com.retek.rib.j2ee.ErrorHospitalRetryTimerTask - Created Remote Hospital Retry EJB instance.
            2013-04-16 13:55:46,859 [Timer-21] DEBUG com.retek.rib.j2ee.ErrorHospitalRetryEjb - $Header: /cvs/Integration/RIB/Rib_Support/src/com/retek/rib/j2ee/ErrorHospitalRetryEjb.java,v 1.38 2008/10/31 08:26:45 iyengam Exp $
            2013-04-16 13:55:46,860 [Timer-21] DEBUG com.retek.rib.j2ee.ErrorHospitalRetryEjb - ErrorHospitalRetryEjb.setSessionContext()
            2013-04-16 13:55:46,860 [Timer-21] DEBUG com.retek.rib.j2ee.ErrorHospitalRetryEjb - Initialization done.
            2013-04-16 13:55:46,861 [Timer-21] DEBUG com.retek.rib.j2ee.AdaptorInstance - $Header: /cvs/Integration/RIB/Rib_Support/src/com/retek/rib/j2ee/AdaptorInstance.java,v 1.8 2008/08/05 20:02:44 polavap Exp $
            2013-04-16 13:55:46,862 [Timer-21] DEBUG com.retek.rib.j2ee.AdaptorInstance - Found the text "jms_hosp_" starting at index 0 and ending at index 9.
            2013-04-16 13:55:46,863 [Timer-21] DEBUG com.retek.rib.j2ee.AdaptorInstance - adapterClassDef=jms_hosp
            2013-04-16 13:55:46,864 [Timer-21] DEBUG com.retek.rib.j2ee.AdaptorClassDef - $Header: /cvs/Integration/RIB/Rib_Support/src/com/retek/rib/j2ee/AdaptorClassDef.java,v 1.6 2008/07/25 14:45:01 polavap Exp $
            2013-04-16 13:55:46,865 [Timer-21] DEBUG com.retek.rib.j2ee.AdaptorClassDef - I found the text "jms_" starting at index 0 and ending at index 4.
            2013-04-16 13:55:46,865 [Timer-21] DEBUG com.retek.rib.j2ee.AdaptorClassDef - AdaptorClassShortName=jms
            2013-04-16 13:55:46,865 [Timer-21] DEBUG com.retek.rib.j2ee.AdaptorInstance - instanceNumber=0
            2013-04-16 13:55:46,866 [Timer-21] INFO com.retek.rib.j2ee.ManagedAdaptersFactory - JMS Server ID(jms1) found for adapter (jms_hosp_0).
            2013-04-16 13:55:46,867 [Timer-21] DEBUG com.retek.rib.j2ee.AdaptorInstance - $Header: /cvs/Integration/RIB/Rib_Support/src/com/retek/rib/j2ee/AdaptorInstance.java,v 1.8 2008/08/05 20:02:44 polavap Exp $
            2013-04-16 13:55:46,869 [Timer-21] DEBUG com.retek.rib.j2ee.AdaptorInstance - Found the text "jms_hosp_" starting at index 0 and ending at index 9.
            2013-04-16 13:55:46,870 [Timer-21] DEBUG com.retek.rib.j2ee.AdaptorInstance - adapterClassDef=jms_hosp
            2013-04-16 13:55:46,870 [Timer-21] DEBUG com.retek.rib.j2ee.AdaptorClassDef - $Header: /cvs/Integration/RIB/Rib_Support/src/com/retek/rib/j2ee/AdaptorClassDef.java,v 1.6 2008/07/25 14:45:01 polavap Exp $
            2013-04-16 13:55:46,873 [Timer-21] DEBUG com.retek.rib.j2ee.AdaptorClassDef - I found the text "jms_" starting at index 0 and ending at index 4.
            2013-04-16 13:55:46,874 [Timer-21] DEBUG com.retek.rib.j2ee.AdaptorClassDef - AdaptorClassShortName=jms
            2013-04-16 13:55:46,892 [Timer-21] DEBUG com.retek.rib.j2ee.AdaptorInstance - instanceNumber=0
            2013-04-16 13:55:46,893 [Timer-21] INFO com.retek.rib.j2ee.ManagedAdaptersFactory - JMS Server ID(jms1) found for adapter (jms_hosp_0).
            2013-04-16 13:55:46,894 [Timer-21] DEBUG com.retek.rib.j2ee.EjbUtil - Looking up TopicConnectionFactory (jms/Generic/Topic/XAConnectionFactoryjms1) from jndi.
            2013-04-16 13:55:46,894 [Timer-21] DEBUG com.retek.rib.j2ee.EjbUtil - Got TopicConnectionFactory from jndi oracle.j2ee.ra.jms.generic.XATopicConnectionFactoryWrapper
            2013-04-16 13:55:46,895 [Timer-21] DEBUG com.retek.rib.j2ee.ErrorHospitalRetryEjb - Topic Connection factory class (class oracle.j2ee.ra.jms.generic.XATopicConnectionFactoryWrapper).
            2013-04-16 13:55:46,896 [Timer-21] DEBUG com.retek.rib.collab.RibCollabController - $Header: /cvs/Integration/RIB/Rib_Support/src/com/retek/rib/collab/RibCollabController.java,v 1.9 2007/12/04 17:56:27 borapra Exp $
            2013-04-16 13:55:46,896 [Timer-21] DEBUG com.retek.rib.j2ee.JmsLocation - $Header: /cvs/Integration/RIB/Rib_Support/src/com/retek/rib/j2ee/JmsLocation.java,v 1.4 2007/06/12 18:03:32 borapra Exp $
            2013-04-16 13:55:46,897 [Timer-21] DEBUG com.retek.rib.j2ee.JmsLocation - I found the text "rib-sim_" starting at index 0 and ending at index 8.
            2013-04-16 13:55:46,898 [Timer-21] DEBUG com.retek.rib.j2ee.JmsLocation - appName=rib-sim
            2013-04-16 13:55:46,898 [Timer-21] DEBUG com.retek.rib.j2ee.AdaptorInstance - $Header: /cvs/Integration/RIB/Rib_Support/src/com/retek/rib/j2ee/AdaptorInstance.java,v 1.8 2008/08/05 20:02:44 polavap Exp $
            2013-04-16 13:55:46,899 [Timer-21] DEBUG com.retek.rib.j2ee.AdaptorInstance - Found the text "jms_hosp_" starting at index 0 and ending at index 9.
            2013-04-16 13:55:46,899 [Timer-21] DEBUG com.retek.rib.j2ee.AdaptorInstance - adapterClassDef=jms_hosp
            2013-04-16 13:55:46,900 [Timer-21] DEBUG com.retek.rib.j2ee.AdaptorClassDef - $Header: /cvs/Integration/RIB/Rib_Support/src/com/retek/rib/j2ee/AdaptorClassDef.java,v 1.6 2008/07/25 14:45:01 polavap Exp $
            2013-04-16 13:55:46,901 [Timer-21] DEBUG com.retek.rib.j2ee.AdaptorClassDef - I found the text "jms_" starting at index 0 and ending at index 4.
            2013-04-16 13:55:46,902 [Timer-21] DEBUG com.retek.rib.j2ee.AdaptorClassDef - AdaptorClassShortName=jms
            2013-04-16 13:55:46,902 [Timer-21] DEBUG com.retek.rib.j2ee.AdaptorInstance - instanceNumber=0
            2013-04-16 13:55:46,903 [Timer-21] DEBUG com.retek.rib.j2ee.J2eeCollabController - $Header: /cvs/Integration/RIB/Rib_Support/src/com/retek/rib/j2ee/J2eeCollabController.java,v 1.19 2008/07/25 14:45:02 polavap Exp $
            2013-04-16 13:55:46,903 [Timer-21] DEBUG com.retek.rib.j2ee.ErrorHospitalRetryEjb - ******************** retry() with reason code JMS called for thread Timer-21 ********************
            2013-04-16 13:55:46,904 [Timer-21] DEBUG com.retek.platform.persistence.DaoFactory - Looking up DAO impl for interface: com.retek.rib.domain.hospital.dao.HospitalDao
            2013-04-16 13:55:46,905 [Timer-21] DEBUG com.retek.rib.domain.hospital.dao.jpa.HospitalDaoImpl - $Header: /cvs/Integration/RIB/Rib_Support/src/com/retek/rib/domain/hospital/dao/jpa/HospitalDaoImpl.java,v 1.14 2009/01/22 10:26:35 nekarsh Exp $
            2013-04-16 13:55:46,906 [Timer-21] DEBUG com.retek.rib.domain.hospital.dao.jpa.HospitalDaoImpl - com.retek.rib.domain.hospital.dao.jpa.HospitalDaoImpl created.
            2013-04-16 13:55:46,906 [Timer-21] DEBUG com.retek.rib.collab.HospitalRetryController - new HospitaRetryController...moduleName: jms_hosp_0, jmsQueueID: jms1
            2013-04-16 13:55:46,910 [Timer-21] DEBUG com.retek.rib.util.RibProperties - Property key(numOfRecordsToRetry) value(20).
            2013-04-16 13:55:46,912 [Timer-21] DEBUG com.retek.rib.j2ee.ErrorHospitalRetryEjb - rib-system.properties loaded from file:/u01/app/oracle/product/10.1.3.1/OracleAS_1/j2ee/rib-sim/rib-sim/rib-system.properties
            2013-04-16 13:55:46,914 [Timer-21] DEBUG com.retek.rib.collab.HospitalRetryController - Doing garbage collection.
            2013-04-16 13:55:46,914 [Timer-21] DEBUG com.retek.rib.domain.hospital.dao.jpa.PersistenceManager - Looking up EntityManager as we did not find it in this thread.
            2013-04-16 13:55:46,915 [Timer-21] DEBUG com.retek.rib.domain.hospital.dao.jpa.PersistenceManager$ContainerManagedEntityManager - Looking up EntityManager from JNDI with name(java:comp/env/ribPersistenceUnit/EntityManager).
            2013-04-16 13:55:46,916 [Timer-21] DEBUG com.retek.rib.domain.hospital.dao.jpa.PersistenceManager$ContainerManagedEntityManager - Found EntityManager(com.evermind.server.ejb.persistence.EntityManagerProxy@15c4ee6) from JNDI with name(java:comp/env/ribPersistenceUnit/EntityManager).
            2013-04-16 13:55:46,922 [Timer-21] DEBUG com.retek.rib.domain.hospital.dao.jpa.PersistenceManager - Current EntityManager impl instance(com.evermind.server.ejb.persistence.EntityManagerProxy@15c4ee6).
            2013-04-16 13:55:46,925 [Timer-21] ERROR com.retek.rib.j2ee.ErrorHospitalRetryEjb - Exception thrown during retry.
            com.retek.platform.persistence.PersistenceException: Unable to execute delete query(select he from HospitalEntryImpl he where he.jmsQueueId = ?1 and he.deletePendingWorkAround = ?2 and he.inQueueWorkAround = ?3)
            exception id: 1366136746925
            at com.retek.rib.domain.hospital.dao.jpa.HospitalDaoImpl.removeMessagesReadyForDeletetion(HospitalDaoImpl.java:1164)
            at com.retek.rib.collab.HospitalRetryController.doGarbageCollection(HospitalRetryController.java:469)
            at com.retek.rib.j2ee.ErrorHospitalRetryEjb.garbageCollection(ErrorHospitalRetryEjb.java:213)
            at com.retek.rib.j2ee.ErrorHospitalRetryEjb.doRetry(ErrorHospitalRetryEjb.java:195)
            at com.retek.rib.j2ee.ErrorHospitalRetryEjb.retry(ErrorHospitalRetryEjb.java:155)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:585)
            at com.evermind.server.ejb.interceptor.joinpoint.EJBJoinPointImpl.invoke(EJBJoinPointImpl.java:35)
            at com.evermind.server.ejb.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:119)
            at com.evermind.server.ejb.interceptor.system.SetContextActionInterceptor.invoke(SetContextActionInterceptor.java:44)
            at com.evermind.server.ejb.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:119)
            at com.evermind.server.ejb.interceptor.system.DMSInterceptor.invoke(DMSInterceptor.java:52)
            at com.evermind.server.ejb.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:119)
            at com.evermind.server.ejb.interceptor.system.TxRequiredInterceptor.invoke(TxRequiredInterceptor.java:50)
            at com.evermind.server.ejb.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:119)
            at com.evermind.server.ejb.interceptor.system.DMSInterceptor.invoke(DMSInterceptor.java:52)
            at com.evermind.server.ejb.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:119)
            at com.evermind.server.ejb.InvocationContextPool.invoke(InvocationContextPool.java:55)
            at com.evermind.server.ejb.StatelessSessionEJBObject.OC4J_invokeMethod(StatelessSessionEJBObject.java:87)
            at ErrorHospitalRetry_LocalProxy_6787m9e.retry(Unknown Source)
            at com.retek.rib.j2ee.ErrorHospitalRetryTimerTask.callRemoteHospitalRetryEjb(ErrorHospitalRetryTimerTask.java:83)
            at com.retek.rib.j2ee.ErrorHospitalRetryTimerTask.run(ErrorHospitalRetryTimerTask.java:40)
            at java.util.TimerThread.mainLoop(Timer.java:512)
            at java.util.TimerThread.run(Timer.java:462)
            Caused by: javax.ejb.EJBException:
            Exception Description: Error while obtaining information about the database. Please look at the nested exception for more details.; nested exception is: Exception [TOPLINK-4019] (Oracle TopLink Essentials - 2.0.1 (Build b09d-fcs (12/06/2007))): oracle.toplink.essentials.exceptions.DatabaseException
            Exception Description: Error while obtaining information about the database. Please look at the nested exception for more details.
            at com.evermind.server.ejb.EJBUtils.createEJBException(EJBUtils.java:365)
            at com.evermind.server.ejb.EJBUtils.createEJBException(EJBUtils.java:356)
            at com.evermind.server.ejb.persistence.EntityManagerProxy.allocateEntityManager(EntityManagerProxy.java:92)
            at com.evermind.server.ejb.persistence.AbstractEntityManagerProxy.createQuery(AbstractEntityManagerProxy.java:101)
            at com.evermind.server.ejb.persistence.EntityManagerProxy.createQuery(EntityManagerProxy.java:69)
            at com.retek.rib.domain.hospital.dao.jpa.HospitalDaoImpl.removeMessagesReadyForDeletetion(HospitalDaoImpl.java:1142)
            ... 25 more
            Caused by: Exception [TOPLINK-4019] (Oracle TopLink Essentials - 2.0.1 (Build b09d-fcs (12/06/2007))): oracle.toplink.essentials.exceptions.DatabaseException
            Exception Description: Error while obtaining information about the database. Please look at the nested exception for more details.
            at oracle.toplink.essentials.exceptions.DatabaseException.errorRetrieveDbMetadataThroughJDBCConnection(DatabaseException.java:346)
            at oracle.toplink.essentials.internal.sessions.DatabaseSessionImpl.loginAndDetectDatasource(DatabaseSessionImpl.java:585)
            at oracle.toplink.essentials.ejb.cmp3.EntityManagerFactoryProvider.login(EntityManagerFactoryProvider.java:280)
            at oracle.toplink.essentials.internal.ejb.cmp3.EntityManagerSetupImpl.deploy(EntityManagerSetupImpl.java:229)
            at oracle.toplink.essentials.internal.ejb.cmp3.base.EntityManagerFactoryImpl.getServerSession(EntityManagerFactoryImpl.java:93)
            at oracle.toplink.essentials.internal.ejb.cmp3.base.EntityManagerFactoryImpl.createEntityManagerImpl(EntityManagerFactoryImpl.java:126)
            at oracle.toplink.essentials.internal.ejb.cmp3.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:99)
            at com.evermind.server.ejb.persistence.PersistenceUnitImpl.createTempPersistenceContext(PersistenceUnitImpl.java:84)
            at com.evermind.server.ejb.persistence.EntityManagerProxy.createEntityManager(EntityManagerProxy.java:99)
            at com.evermind.server.ejb.persistence.EntityManagerProxy.allocateEntityManager(EntityManagerProxy.java:84)
            ... 28 more
            Caused by: java.sql.SQLException: javax.resource.ResourceException: RollbackException: Resource was in use during abnormal shutdown of server and during subsequent recovery processing was either unobtainable or failed to have recovery permissions. Consult logs if this issue continues to occur after the next recovery processing interval completes
            at oracle.oc4j.sql.spi.ManagedConnectionImpl.setupTransaction(ManagedConnectionImpl.java:841)
            at oracle.oc4j.sql.spi.ConnectionHandle.oc4j_intercept(ConnectionHandle.java:305)
            at oracle_jdbc_driver_LogicalConnection_Proxy.getMetaData()
            at oracle.toplink.essentials.internal.sessions.DatabaseSessionImpl.loginAndDetectDatasource(DatabaseSessionImpl.java:583)
            ... 36 more
            Caused by: javax.resource.ResourceException: RollbackException: Resource was in use during abnormal shutdown of server and during subsequent recovery processing was either unobtainable or failed to have recovery permissions. Consult logs if this issue continues to occur after the next recovery processing interval completes
            at com.evermind.server.connector.ConnectionContext.setupForJTATransaction(ConnectionContext.java:374)
            at com.evermind.server.connector.ConnectionContext.setupForTransaction(ConnectionContext.java:301)
            at com.evermind.server.connector.ConnectionContext.setupForTransaction(ConnectionContext.java:286)
            at com.evermind.server.connector.ApplicationConnectionManager.lazyEnlist(ApplicationConnectionManager.java:2004)
            at oracle.j2ee.connector.OracleConnectionManager.lazyEnlist(OracleConnectionManager.java:285)
            at oracle.oc4j.sql.spi.ManagedConnectionFactoryImpl.enlist(ManagedConnectionFactoryImpl.java:532)
            at oracle.oc4j.sql.spi.ManagedConnectionImpl.setupTransaction(ManagedConnectionImpl.java:839)
            ...
            2013-04-16 13:55:46,939 [Timer-21] DEBUG com.retek.rib.domain.hospital.dao.jpa.PersistenceManager - Trying to close EntityManager.
            2013-04-16 13:55:46,944 [Timer-21] DEBUG com.retek.rib.domain.hospital.dao.jpa.PersistenceManager$ContainerManagedEntityManager - Close EntityManager(com.evermind.server.ejb.persistence.EntityManagerProxy@15c4ee6) called.
            2013-04-16 13:55:46,944 [Timer-21] DEBUG com.retek.rib.domain.hospital.dao.jpa.PersistenceManager - EntityManager(com.evermind.server.ejb.persistence.EntityManagerProxy@15c4ee6) closed.
            2013-04-16 13:55:46,945 [Timer-21] ERROR com.retek.rib.j2ee.ErrorHospitalRetryEjb - Unable to flush hospital.

            2) There are currently only 1195 records. All with just one retry (ATTEMPT_COUNT = 1). In RDMT this list is the number of messages retained in RIBAQ:

            JMS Topic Scan - Topics with Messages...

            Scanning for Topics with Subscribers...
            Found Topics with Subscribers: 85

            Scanning for Topics with message counts...
            -- Checking Threshold = 1000

            MSGCNT TOPIC SUBSCRIBER
            =============================================================================

            1066 ETASNINFROMRIBTOISO RIB_SIM_ASNIN_SUB_1 THRESHOLD WARNING
            1 ETDIFFSFROMRMS RIB_SIM_DIFFS_SUB_1
            227 ETEMRGNCYPRC SUBSCRIPTORCOLA
            40515 ETITEMLOCISO RIB_SIM_ITEMLOC_SUB_1 THRESHOLD WARNING
            40994 ETITEMLOCISO RIB_SIM_ITEMLOC_SUB_2 THRESHOLD WARNING
            40250 ETITEMLOCISO RIB_SIM_ITEMLOC_SUB_3 THRESHOLD WARNING
            40807 ETITEMLOCISO RIB_SIM_ITEMLOC_SUB_4 THRESHOLD WARNING
            41182 ETITEMLOCISO RIB_SIM_ITEMLOC_SUB_5 THRESHOLD WARNING
            35217 ETITEMSFROMRMS RIB_TAFR_ITEMSTOITMTL_TAFR_1 THRESHOLD WARNING
            8388 ETITEMSISO RIB_SIM_ITEMS_SUB_1 THRESHOLD WARNING
            120068 ETORDERFROMRMS RIB_TAFR_ORDERTOODRWH_TAFR_1 THRESHOLD WARNING
            16743 ETORDERISO RIB_SIM_ORDER_SUB_1 THRESHOLD WARNING
            25109 ETOTB RIB_RMS_OTB_SUB_1 THRESHOLD WARNING
            6236 ETPRMPRCCHGFROMRPM RIB_SIM_PRMPRCCHG_SUB_1 THRESHOLD WARNING
            4936 ETREGPRCCHGFROMRPM RIB_SIM_REGPRCCHG_SUB_1 THRESHOLD WARNING
            10 ETRTVREQISO RIB_SIM_RTVREQ_SUB_1
            447 ETSTOCKORDERSISO RIB_SIM_STOCKORDER_SUB_1
            84 ETVENDORFROMRMS RIB_SIM_VENDOR_SUB_1

            3) These are the files that I found, they mean?

            oracle@cmn1ap06:/u01/app/oracle/product/10.1.3.1/OracleAS_1/j2ee/rib-sim/log/rib-sim $ ll *xml
            -rw-r--r-- 1 oracle oinstall 15089 Apr 2 16:58 ASNIn_sub_120130402165807317.xml
            -rw-r--r-- 1 oracle oinstall 15170 Apr 2 17:49 ASNIn_sub_120130402174932935.xml
            -rw-r--r-- 1 oracle oinstall 29373 Apr 2 16:59 ItemLoc_sub_120130402165921980.xml
            -rw-r--r-- 1 oracle oinstall 29373 Apr 2 17:50 ItemLoc_sub_120130402175028052.xml
            -rw-r--r-- 1 oracle oinstall 29293 Apr 2 17:51 ItemLoc_sub_120130402175125457.xml
            -rw-r--r-- 1 oracle oinstall 31382 Apr 2 16:59 ItemLoc_sub_220130402165925463.xml
            -rw-r--r-- 1 oracle oinstall 31302 Apr 2 17:50 ItemLoc_sub_220130402175033674.xml
            -rw-r--r-- 1 oracle oinstall 29466 Apr 2 16:59 ItemLoc_sub_320130402165928956.xml
            -rw-r--r-- 1 oracle oinstall 29386 Apr 2 17:50 ItemLoc_sub_320130402175037180.xml
            -rw-r--r-- 1 oracle oinstall 29184 Apr 2 16:59 ItemLoc_sub_420130402165915033.xml
            -rw-r--r-- 1 oracle oinstall 29104 Apr 2 17:50 ItemLoc_sub_420130402175040624.xml
            -rw-r--r-- 1 oracle oinstall 27364 Apr 2 16:58 ItemLoc_sub_520130402165825624.xml
            -rw-r--r-- 1 oracle oinstall 27365 Apr 2 17:50 ItemLoc_sub_520130402175044050.xml
            -rw-r--r-- 1 oracle oinstall 11994 Apr 2 16:58 Items_sub_120130402165814827.xml
            -rw-r--r-- 1 oracle oinstall 9054 Apr 4 10:25 Items_sub_120130404102542730.xml
            -rw-r--r-- 1 oracle oinstall 9054 Apr 4 10:25 Items_sub_120130404102543083.xml
            -rw-r--r-- 1 oracle oinstall 9680 Mar 19 15:51 Order_sub_120130319155146466.xml
            -rw-r--r-- 1 oracle oinstall 9680 Mar 19 15:51 Order_sub_120130319155147063.xml
            -rw-r--r-- 1 oracle oinstall 18012 Apr 2 16:59 Order_sub_120130402165908053.xml
            -rw-r--r-- 1 oracle oinstall 13067 Apr 2 16:58 RTVReq_sub_120130402165822054.xml
            -rw-r--r-- 1 oracle oinstall 13068 Apr 2 17:50 RTVReq_sub_120130402175047487.xml
            -rw-r--r-- 1 oracle oinstall 12373 Apr 2 16:59 RegPrcChg_sub_120130402165935826.xml
            -rw-r--r-- 1 oracle oinstall 15202 Apr 2 16:59 StockOrder_sub_120130402165911538.xml
            -rw-r--r-- 1 oracle oinstall 13831 Apr 2 16:58 Vendor_sub_120130402165818459.xml
            -rw-r--r-- 1 oracle oinstall 13832 Apr 2 17:50 Vendor_sub_120130402175050964.xml

            Thank you for your support and fast response.
            • 3. Re: The JMS and SUB adapters shutdown (RIB-SIM)
              praveenps
              Hi Orlando,

              Try below steps.

              1)opmn stopall.... after this make sure opmn is not running.
              2)If production system back up all xml you found in log folder.
              3)Delete xml files from rib log folder.
              4)opmn startall.

              Let me know if error happens again.

              Regards,
              PPS
              • 4. Re: The JMS and SUB adapters shutdown (RIB-SIM)
                Orlando De Abreu
                Hi,

                I carried out your instructions but the problem persists, it will not be the problem. The connection to the RIB database through Oracle Application Server completes successfully, the instructions for Oracle Support is to make a complete stop/start of OR but do not think this solves anything.
                • 5. Re: The JMS and SUB adapters shutdown (RIB-SIM)
                  praveenps
                  Hi Orlando,

                  Please check the bugs documented as Bug:13072680/Bug 9173130.

                  MOS:ID 1087537.1

                  Regards,
                  PPS

                  Edited by: praveenps on Apr 17, 2013 11:23 PM
                  • 6. Re: The JMS and SUB adapters shutdown (RIB-SIM)
                    Orlando De Abreu
                    Hi,

                    Regarding your suggestion to Article 1087537.1, I see that is for version 13.2.4, and ours is 13.1.1. Although the hospital is empty (RIB_MESSAGE and RIB_MESSAGE_FAILURE) these adapters fall. The strange thing about this is that last Wednesday was solved this problem doing a node switchover RIB application (our application is under mode Service Guard). In yesterday got one of the nodes for maintenance and fell again.
                    • 7. Re: The JMS and SUB adapters shutdown (RIB-SIM)
                      praveenps
                      Hi Orlando,

                      I can see this issue has been reported first for 13.0.2 and later in 13.2.3.1 & 13.2.4. But this issue will not occur if hospital is empty. Can you please provide the error which you are getting when hospital is empty.

                      Regards,
                      PPS
                      • 8. Re: The JMS and SUB adapters shutdown (RIB-SIM)
                        Orlando De Abreu
                        Sorry, if there are errors in the hospital, but his ATTEMPT_COUNT = 1, ie, they have not processed nor even because the adapter is down. Now I am surprised that I have RIB_MESSAGE_FAILURE errors of this type:

                        "Caused by: java.sql.SQLException: Attempt to use an invalid handle 'oracle_jdbc_driver_LogicalConnection_Proxy @ bb5e2d'."

                        The JMS log tells me the following:

                        Caused by: javax.resource.ResourceException: RollbackException: Resource use was in. During abnormal shutdown of server processing and During subsequent recovery was unobtainable or failed Either To Have recovery permissions. If This issue logs Consult Continues to Occur after the next interval completes recovery processing
                        at com.evermind.server.connector.ConnectionContext.setupForJTATransaction (ConnectionContext.java: 374)
                        at com.evermind.server.connector.ConnectionContext.setupForTransaction (ConnectionContext.java: 301)
                        at com.evermind.server.connector.ConnectionContext.setupForTransaction (ConnectionContext.java: 286)
                        at
                        at oracle.j2ee.connector.OracleConnectionManager.lazyEnlist (OracleConnectionManager.java: 285)
                        at oracle.oc4j.sql.spi.ManagedConnectionFactoryImpl.enlist (ManagedConnectionFactoryImpl.java: 532)
                        at oracle.oc4j.sql.spi.ManagedConnectionImpl.setupTransaction (ManagedConnectionImpl.java: 839)

                        It seems that at some point it lost the connection to the database and this was in error.
                        • 9. Re: The JMS and SUB adapters shutdown (RIB-SIM)
                          praveenps
                          Hi Orlando,

                          It is tough to understand the issue without having access to servers :) Even then I will try.

                          "During abnormal shutdown of server processing and During subsequent recovery was unobtainable or failed Either To Have recovery permissions"

                          Can you check with DBA whether XA configuration steps are executed during intallation of RIB especially in rms-131-ig.pdf section "Verify that Database XA Resources are Configured for RIB" these steps give XA recovery permission for Servers.

                          Regards,
                          PPS
                          • 10. Re: The JMS and SUB adapters shutdown (RIB-SIM)
                            Orlando De Abreu
                            Hi,

                            I checked section you recommended and RIBAQ not had some permissions:

                            GRANT EXECUTE ON TO RIBAQ SYS.DBMS_AQ;
                            GRANT EXECUTE ON TO RIBAQ SYS.DBMS_AQADM;
                            GRANT EXECUTE ON TO RIBAQ SYS.DBMS_AQIN;
                            GRANT EXECUTE ON TO RIBAQ SYS.DBMS_AQJMS;

                            However, rebooted and the problem persists. You will know some recommendations RIB Guard Service mode?

                            Best Regards.
                            • 11. Re: The JMS and SUB adapters shutdown (RIB-SIM)
                              praveenps
                              Hi Orlando,

                              Please post the errors...

                              Regards,
                              PPS
                              • 12. Re: The JMS and SUB adapters shutdown (RIB-SIM)
                                Orlando De Abreu
                                Hi,

                                Last night we take certain actions and we hope permanent. We cleaned the following points and we go up the adapters:

                                1. Pending transactions were cleared of the database (dba_2pc_pending) involved in RIB and SIM.
                                2. I deleted the files *.lock and *.resources of the folder "$ORACLE_HOME/j2ee/rib-sim/xa-log" including those corresponding to node RIB homolog app.
                                3. Were cleaned files with *.xml folder "$ORACLE_HOME/j2ee/rib-sim/xa-log/OPMN_cmn1ap06.cm.com.ve_rib-rms.default_group.1." And "$ORACLE_HOME/j2ee/rib-sim/xa-log/OPMN_cmn2ap06.cm.com.ve_rib-rms.default_group.1. "
                                4. I deleted the files *.log from the folder "$ORACLE_HOME/j2ee/rib-sim/log/rib-sim_default_group_1"
                                5. Cleaned up old files from the directory "$ORACLE_HOME/opmn/logs"
                                5. He wiped his persistence folder "$ORACLE_HOME/j2ee/rib-sim/persistence"

                                Apparently there was some "garbage" in Oracle Application Server was trying to make a recovery of certain transactions that remained in permanent status "Recovering Committing". Hopefully these actions are durable and which are the true cause of the problem.

                                Thanks for your help! :)
                                • 13. Re: The JMS and SUB adapters shutdown (RIB-SIM)
                                  praveenps
                                  Hi Orlando,

                                  Good to know.

                                  Can you please check whether recovery is working fine in your environment. These steps cannot be executed in a PROD env without possible information loss.

                                  Regards,

                                  PPS