2 Replies Latest reply: Oct 17, 2013 12:03 AM by AjithPathiyil RSS

    Datasource pool issues in weblogic

    972017

      Frequently,I am coming accross this error in my weblogic server log:Any pointers on how to resolve this? any help on this

      Adminlog:

      <Oct 15, 2013 12:11:35 PM GMT> <Info> <JDBC> <EPGCRMITG1> <AdminServer> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <6a64a370db9cb4a0:-1d42ac6e:1419df56d93:-8000-0000000000003906> <1381839095305> <BEA-001128> <Connection for pool "LifeCycleDataSource-rac0" has been closed.>

      ####<Oct 15, 2013 12:11:35 PM GMT> <Info> <JDBC> <EPGCRMITG1> <AdminServer> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <6a64a370db9cb4a0:-1d42ac6e:1419df56d93:-8000-0000000000003906> <1381839095354> <BEA-001128> <Connection for pool "LifeCycleDataSource-rac0" has been closed.>

      ####<Oct 15, 2013 12:11:35 PM GMT> <Info> <JDBC> <EPGCRMITG1> <AdminServer> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <6a64a370db9cb4a0:-1d42ac6e:1419df56d93:-8000-0000000000003906> <1381839095356> <BEA-001128> <Connection for pool "LifeCycleDataSource-rac0" has been closed.>

      ####<Oct 15, 2013 12:12:36 PM GMT> <Warning> <JDBC> <EPGCRMITG1> <AdminServer> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <6a64a370db9cb4a0:-1d42ac6e:1419df56d93:-8000-0000000000003906> <1381839156379> <BEA-001129> <Received exception while creating connection for pool "LifeCycleDataSource-rac0": IO Error: The Network Adapter could not establish the connection.>

      ####<Oct 15, 2013 12:12:36 PM GMT> <Info> <JDBC> <EPGCRMITG1> <AdminServer> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <6a64a370db9cb4a0:-1d42ac6e:1419df56d93:-8000-0000000000003906> <1381839156381> <BEA-001156> <Stack trace associated with message 001129 follows:

       

      java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection

          at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:482)

          at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:678)

          at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:238)

          at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:34)

          at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:567)

          at weblogic.jdbc.common.internal.ConnectionEnvFactory.makeConnection(ConnectionEnvFactory.java:357)

          at weblogic.jdbc.common.internal.ConnectionEnvFactory.refreshResource(ConnectionEnvFactory.java:453)

          at weblogic.common.resourcepool.ResourcePoolImpl.refreshResource(ResourcePoolImpl.java:1841)

          at weblogic.common.resourcepool.ResourcePoolImpl.checkResource(ResourcePoolImpl.java:1741)

          at weblogic.common.resourcepool.ResourcePoolImpl.checkAndReturnResource(ResourcePoolImpl.java:1647)

          at weblogic.common.resourcepool.ResourcePoolImpl.checkAndReturnResource(ResourcePoolImpl.java:1636)

          at weblogic.common.resourcepool.ResourcePoolImpl.testUnusedResources(ResourcePoolImpl.java:2088)

          at weblogic.common.resourcepool.ResourcePoolImpl.access$1600(ResourcePoolImpl.java:41)

          at weblogic.common.resourcepool.ResourcePoolImpl$ResourcePoolMaintanenceTask.timerExpired(ResourcePoolImpl.java:2769)

          at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:273)

          at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:545)

          at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)

          at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

      Caused By: oracle.net.ns.NetException: The Network Adapter could not establish the connection

          at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:398)

          at oracle.net.resolver.AddrResolution.resolveAndExecute(AddrResolution.java:438)

          at oracle.net.ns.NSProtocol.establishConnection(NSProtocol.java:711)

          at oracle.net.ns.NSProtocol.connect(NSProtocol.java:257)

          at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1222)

          at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:330)

          at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:678)

          at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:238)

          at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:34)

          at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:567)

          at weblogic.jdbc.common.internal.ConnectionEnvFactory.makeConnection(ConnectionEnvFactory.java:357)

          at weblogic.jdbc.common.internal.ConnectionEnvFactory.refreshResource(ConnectionEnvFactory.java:453)

          at weblogic.common.resourcepool.ResourcePoolImpl.refreshResource(ResourcePoolImpl.java:1841)

          at weblogic.common.resourcepool.ResourcePoolImpl.checkResource(ResourcePoolImpl.java:1741)

          at weblogic.common.resourcepool.ResourcePoolImpl.checkAndReturnResource(ResourcePoolImpl.java:1647)

          at weblogic.common.resourcepool.ResourcePoolImpl.checkAndReturnResource(ResourcePoolImpl.java:1636)

          at weblogic.common.resourcepool.ResourcePoolImpl.testUnusedResources(ResourcePoolImpl.java:2088)

          at weblogic.common.resourcepool.ResourcePoolImpl.access$1600(ResourcePoolImpl.java:41)

          at weblogic.common.resourcepool.ResourcePoolImpl$ResourcePoolMaintanenceTask.timerExpired(ResourcePoolImpl.java:2769)

          at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:273)

          at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:545)

          at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)

          at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

      Caused By: java.net.SocketTimeoutException: connect timed out

          at java.net.PlainSocketImpl.socketConnect(Native Method)

          at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)

          at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)

          at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)

          at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)

          at java.net.Socket.connect(Socket.java:579)

          at oracle.net.nt.MetricsEnabledSocket.connect(TcpNTAdapter.java:631)

          at oracle.net.nt.TcpNTAdapter.connect(TcpNTAdapter.java:164)

       

      SOA Log:

      <BEA-001112> <Test "SELECT 1 FROM DUAL" set up for pool "ecoxproduct" failed with exception: "oracle.jdbc.xa.OracleXAException".>

      ####<Oct 16, 2013 3:53:48 PM GMT> <Info> <JDBC> <EPGCRMITG1> <soa_server1> <[ACTIVE] ExecuteThread: '62' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <6a64a370db9cb4a0:-3ffef6c2:141a29a60f1:-8000-00000000000ff280> <1381938828168> <BEA-001128> <Connection for pool "ecoxproduct" has been closed.>

      ####<Oct 16, 2013 3:53:48 PM GMT> <Warning> <JTA> <EPGCRMITG1> <soa_server1> <[ACTIVE] ExecuteThread: '62' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <6a64a370db9cb4a0:-3ffef6c2:141a29a60f1:-8000-00000000000ff280> <1381938828181> <BEA-110484> <The JTA health state has changed from HEALTH_OK to HEALTH_WARN with reason codes: Resource ecoxproduct_base_domain declared unhealthy.>

      ####<Oct 16, 2013 3:53:48 PM GMT> <Info> <JDBC> <EPGCRMITG1> <soa_server1> <[ACTIVE] ExecuteThread: '15' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <6a64a370db9cb4a0:-3ffef6c2:141a29a60f1:-8000-00000000000ff282> <1381938828653> <BEA-001128> <Connection for pool "AIAJMSDS-rac0" has been closed.>

      ####<Oct 16, 2013 3:53:54 PM GMT> <Error> <JDBC> <EPGCRMITG1> <soa_server1> <FabricScheduler_QuartzSchedulerThread> <<anonymous>> <> <0000K6YagLhE^M4pzSXBiW1ILeuk000002> <1381938834492> <BEA-001112> <Test "SELECT 1 FROM DUAL" set up for pool "SOALocalTxDataSource" failed with exception: "java.sql.SQLRecoverableException: IO Error: Connection reset".>

      ####<Oct 16, 2013 3:53:54 PM GMT> <Info> <JDBC> <EPGCRMITG1> <soa_server1> <FabricScheduler_QuartzSchedulerThread> <<anonymous>> <> <0000K6YagLhE^M4pzSXBiW1ILeuk000002> <1381938834493> <BEA-001128> <Connection for pool "SOALocalTxDataSource" has been closed.>

      ####<Oct 16, 2013 3:53:58 PM GMT> <Error> <JDBC> <EPGCRMITG1> <soa_server1> <MDSPollingThread-[soa-infra, jdbc/mds/MDS_LocalTxDataSource]> <<anonymous>> <> <0000K6YagLhE^M4pzSXBiW1ILeuk000002> <1381938838392> <BEA-001112> <Test "SELECT 1 FROM DUAL" set up for pool "mds-soa" failed with exception: "java.sql.SQLRecoverableException: IO Error: Connection reset".>

      ####<Oct 16, 2013 3:53:58 PM GMT> <Info> <JDBC> <EPGCRMITG1> <soa_server1> <MDSPollingThread-[soa-infra, jdbc/mds/MDS_LocalTxDataSource]> <<anonymous>> <> <0000K6YagLhE^M4pzSXBiW1ILeuk000002> <1381938838393> <BEA-001128> <Connection for pool "mds-soa" has been closed.>

      ####<Oct 16, 2013 3:53:58 PM GMT> <Error> <JDBC> <EPGCRMITG1> <soa_server1> <MDSPollingThread-[owsm, jdbc/mds/owsm]> <<anonymous>> <> <0000K6YagLhE^M4pzSXBiW1ILeuk000002> <1381938838393> <BEA-001112> <Test "SELECT 1 FROM DUAL" set up for pool "mds-owsm" failed with exception: "java.sql.SQLRecoverableException: IO Error: Connection reset".>

      ####<Oct 16, 2013 3:53:58 PM GMT> <Info> <JDBC> <EPGCRMITG1> <soa_server1> <MDSPollingThread-[owsm, jdbc/mds/owsm]> <<anonymous>> <> <0000K6YagLhE^M4pzSXBiW1ILeuk000002> <1381938838394> <BEA-001128> <Connection for pool "mds-owsm" has been closed.>

      ####<Oct 16, 2013 3:53:58 PM GMT> <Info> <JDBC> <EPGCRMITG1> <soa_server1> <[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <6a64a370db9cb4a0:-3ffef6c2:141a29a60f1:-8000-00000000000ff284> <1381938838414> <BEA-001128> <Connection for pool "AIAJMSDS-rac0" has been closed.>

      ####<Oct 16, 2013 3:54:04 PM GMT> <Warning> <JDBC> <EPGCRMITG1> <soa_server1> <FabricScheduler_QuartzSchedulerThread> <<anonymous>> <> <0000K6YagLhE^M4pzSXBiW1ILeuk000002> <1381938844505> <BEA-001129> <Received exception while creating connection for pool "SOALocalTxDataSource": IO Error: The Network Adapter could not establish the connection.>

      ####<Oct 16, 2013 3:54:04 PM GMT> <Info> <JDBC> <EPGCRMITG1> <soa_server1> <FabricScheduler_QuartzSchedulerThread> <<anonymous>> <> <0000K6YagLhE^M4pzSXBiW1ILeuk000002> <1381938844506> <BEA-001156> <Stack trace associated with message 001129 follows:

      This my connection pool parameters for  "ecoxproduct":

      Initial Capacity:1

      Maximum Capacity:5

      Minimun Capacity:1

      Statement Cache Type:LRU

      Statement Cache Size: 10

      Test Reserved Connections: uncheked

      Test Frequnct:120

      Test Table Name:SQL SELECT 1 FROM DUAL

      Seconds to trust ldle Poolconnection :10

      Shrink Frequency:900 SEC

      Connection cretion retry Frequncy   0

      Inactiv Connection time out:0

      Mixmunwaiting connetion:2147483647

      Connection Reserve Timeout: 10 SEC

      Thanku  you

      ram

        • 1. Re: Datasource pool issues in weblogic
          Joe Weinstein-Oracle

          The symptoms indicate a shutdown/failure of the DBMS. There is nothing you can do at the WLS end to help this,

          except I would recommend setting test reserved connections, as a general safer setting.

          • 2. Re: Datasource pool issues in weblogic
            AjithPathiyil

            Hi,

             

            The database going down is most unlikely to happen,

             

            Things to check here by DBA:-

             

            1) If the SERVICE_NAME used in the JDBC string - This service_name might be intermittently unregistering itself from database listener

            2) If the SID is used in the JDBC string - Check if you have used proper SID name in case of RAC

            3) if SCAN_NAME is used - The scan_name also intermittently breaks the client connection.