This discussion is archived
5 Replies Latest reply: Oct 8, 2013 9:10 AM by PHCharbonneau RSS

Disable and freeze users called on connection pool

jollysl - Oracle Newbie
Currently Being Moderated

Hi All,

 

Intermittently our WebLogic datasource (multi-datasource) could not create establish the connection to database due to Connection timed out (show in wl server log) like below stack trace.

 

 

####<Oct 3, 2013 10:14:56 AM WIT> <Warning> <JDBC> <cbtcmpap01.tc.xl.co.id> <CM15Server> <[ACTIVE] ExecuteThread: '38' for queue: 'weblogic.kernel.Default (self-tuning)

'> <EXT<Tksmau5EpICi6x6dS{Npa5dVBJG7XmOY43MhE1o0;appId=CM;>> <> <> <1380770096285> <BEA-001129> <Received exception while creating connection for pool "ABPRegularDataSo

urce1": IO Error: The Network Adapter could not establish the connection>

####<Oct 3, 2013 10:14:56 AM WIT> <Info> <JDBC> <cbtcmpap01.tc.xl.co.id> <CM15Server> <[ACTIVE] ExecuteThread: '38' for queue: 'weblogic.kernel.Default (self-tuning)'>

<EXT<Tksmau5EpICi6x6dS{Npa5dVBJG7XmOY43MhE1o0;appId=CM;>> <> <> <1380770096286> <BEA-001156> <Stack trace associated with message 001129 follows:

 

 

java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection

        at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:458)

        at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:546)

        at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:236)

        at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)

        at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:521)

        at weblogic.jdbc.common.internal.ConnectionEnvFactory.makeConnection(ConnectionEnvFactory.java:350)

        at weblogic.jdbc.common.internal.ConnectionEnvFactory.createResource(ConnectionEnvFactory.java:236)

        at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1249)

        at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1166)

        at weblogic.common.resourcepool.ResourcePoolImpl.reserveResourceInternal(ResourcePoolImpl.java:450)

        at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:342)

        at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:329)

        at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:417)

        at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:324)

        at weblogic.jdbc.common.internal.MultiPool.searchHighAvail(MultiPool.java:408)

        at weblogic.jdbc.common.internal.MultiPool.findPool(MultiPool.java:178)

        at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:89)

        at weblogic.jdbc.common.internal.RmiDataSource.getPoolConnection(RmiDataSource.java:350)

        at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:369)

.

.

.

        at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:667)

        at weblogic.rmi.cluster.ClusterableServerRef.invoke(ClusterableServerRef.java:230)

        at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:522)

        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)

        at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)

        at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:518)

        at weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:118)

        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)

        at weblogic.work.ExecuteThread.run(ExecuteThread.java:178)

Caused By: oracle.net.ns.NetException: The Network Adapter could not establish the connection

        at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:392)

        at oracle.net.resolver.AddrResolution.resolveAndExecute(AddrResolution.java:434)

        at oracle.net.ns.NSProtocol.establishConnection(NSProtocol.java:687)

        at oracle.net.ns.NSProtocol.connect(NSProtocol.java:247)

        at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1102)

        at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:320)

        at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:546)

        at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:236)

        at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)

        at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:521)

        at weblogic.jdbc.common.internal.ConnectionEnvFactory.makeConnection(ConnectionEnvFactory.java:350)

        at weblogic.jdbc.common.internal.ConnectionEnvFactory.createResource(ConnectionEnvFactory.java:236)

        at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1249)

        at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1166)

        at weblogic.common.resourcepool.ResourcePoolImpl.reserveResourceInternal(ResourcePoolImpl.java:450)

        at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:342)

        at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:329)

        at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:417)

        at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:324)

        at weblogic.jdbc.common.internal.MultiPool.searchHighAvail(MultiPool.java:408)

        at weblogic.jdbc.common.internal.MultiPool.findPool(MultiPool.java:178)

        at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:89)

        at weblogic.jdbc.common.internal.RmiDataSource.getPoolConnection(RmiDataSource.java:350)

        at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:369)

.

.

        at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:667)

        at weblogic.rmi.cluster.ClusterableServerRef.invoke(ClusterableServerRef.java:230)

        at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:522)

        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)

        at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)

        at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:518)

        at weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:118)

        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)

        at weblogic.work.ExecuteThread.run(ExecuteThread.java:178)

Caused By: java.net.ConnectException: Connection timed out

        at java.net.PlainSocketImpl.socketConnect(Native Method)

        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)

        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)

        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)

        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)

        at java.net.Socket.connect(Socket.java:529)

        at oracle.net.nt.TcpNTAdapter.connect(TcpNTAdapter.java:150)

        at oracle.net.nt.ConnOption.connect(ConnOption.java:133)

        at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:370)

        at oracle.net.resolver.AddrResolution.resolveAndExecute(AddrResolution.java:434)

        at oracle.net.ns.NSProtocol.establishConnection(NSProtocol.java:687)

        at oracle.net.ns.NSProtocol.connect(NSProtocol.java:247)

        at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1102)

        at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:320)

        at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:546)

        at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:236)

        at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)

        at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:521)

        at weblogic.jdbc.common.internal.ConnectionEnvFactory.makeConnection(ConnectionEnvFactory.java:350)

        at weblogic.jdbc.common.internal.ConnectionEnvFactory.createResource(ConnectionEnvFactory.java:236)

        at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1249)

        at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1166)

        at weblogic.common.resourcepool.ResourcePoolImpl.reserveResourceInternal(ResourcePoolImpl.java:450)

        at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:342)

        at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:329)

        at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:417)

        at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:324)

        at weblogic.jdbc.common.internal.MultiPool.searchHighAvail(MultiPool.java:408)

        at weblogic.jdbc.common.internal.MultiPool.findPool(MultiPool.java:178)

        at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:89)

        at weblogic.jdbc.common.internal.RmiDataSource.getPoolConnection(RmiDataSource.java:350)

        at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:369)

.

.

.

        at weblogic.rmi.cluster.ClusterableServerRef.invoke(ClusterableServerRef.java:230)

        at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:522)

        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)

        at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)

        at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:518)

        at weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:118)

        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)

        at weblogic.work.ExecuteThread.run(ExecuteThread.java:178)

>

 

 

 

Above error message then followed by below message:

 

####<Oct 3, 2013 10:15:00 AM WIT> <Info> <JDBC> <cbtcmpap01.tc.xl.co.id> <CM15Server> <[ACTIVE] ExecuteThread: '31' for queue: 'weblogic.kernel.Default (self-tuning)'>

<<WLS Kernel>> <> <> <1380770100870> <BEA-001116> <Disable and freeze users called on connection pool "ABPRegularDataSource1" by "<WLS Kernel>".>

####<Oct 3, 2013 10:15:00 AM WIT> <Info> <JDBC> <cbtcmpap01.tc.xl.co.id> <CM15Server> <[ACTIVE] ExecuteThread: '29' for queue: 'weblogic.kernel.Default (self-tuning)'>

<<WLS Kernel>> <> <> <1380770100870> <BEA-001116> <Disable and freeze users called on connection pool "ABPRegularDataSource1" by "<WLS Kernel>".>

####<Oct 3, 2013 10:15:00 AM WIT> <Info> <JDBC> <cbtcmpap01.tc.xl.co.id> <CM15Server> <[ACTIVE] ExecuteThread: '29' for queue: 'weblogic.kernel.Default (self-tuning)'>

<<WLS Kernel>> <> <> <1380770100871> <BEA-001116> <Disable and freeze users called on connection pool "ABPRegularDataSource2" by "<WLS Kernel>".>

 

 

I see that "disabled and freeze" message is an info, so i wondered whether this message is a default message when datasource cannot make connection to database.

 

Or does "disabled and freeze" message is telling something else.

 

Please if there is any insight about the case?

 

 

Thanks in advance.

 

Jolly

  • 1. Re: Disable and freeze users called on connection pool
    jollysl - Oracle Newbie
    Currently Being Moderated

    Additional information,

     

    This is multi-datasource, which is consist of datasource1 and datasource2. But for another issue we are facing, database instance where datasource2 connect to has been shutdown.

     

     

    Thanks,

    Jolly

  • 2. Re: Disable and freeze users called on connection pool
    Joe Weinstein Expert
    Currently Being Moderated

    Show me your config files for you datasources. I bet you're telling the driver to time out connection requests after 10000 milliseconds

    or something like that. Don't do that. SOmetimes the DBMS may be fine, working its current connections, but busy and slow to

    make new ones, and if the driver has been told not to wait, it'll throw an exception, and WLS takes a dim view of DBMSes that

    it cannot connect to.

  • 3. Re: Disable and freeze users called on connection pool
    jollysl - Oracle Newbie
    Currently Being Moderated

    Hi Joe, thanks for your reply.

     

     

    Below is the configuration:

     

    <?xml version='1.0' encoding='UTF-8'?>

    <jdbc-data-source xmlns="http://xmlns.oracle.com/weblogic/jdbc-data-source" xmlns:sec="http://xmlns.oracle.com/weblogic/security" xmlns:wls="http://xmlns.oracle.com/web

    logic/security/wls" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://xmlns.oracle.com/weblogic/jdbc-data-source http://xmlns.oracle.com/

    weblogic/jdbc-data-source/1.0/jdbc-data-source.xsd">

      <name>ABPRegularDataSource1</name>

      <jdbc-driver-params>

        <url>jdbc:oracle:thin:@cmpdb01v:1522:XLCM1A</url>

        <driver-name>oracle.jdbc.OracleDriver</driver-name>

        <properties>

          <property>

            <name>user</name>

            <value>XLAPPC</value>

          </property>

        </properties>

        <password-encrypted>{AES}vRuFVmb4u+B2saJOfqxHjZSkBbnPutNB1C3OCqzhTQk=</password-encrypted>

      </jdbc-driver-params>

      <jdbc-connection-pool-params>

        <initial-capacity>20</initial-capacity>

        <max-capacity>75</max-capacity>

        <capacity-increment>5</capacity-increment>

        <shrink-frequency-seconds>900</shrink-frequency-seconds>

        <highest-num-waiters>25</highest-num-waiters>

        <connection-creation-retry-frequency-seconds>60</connection-creation-retry-frequency-seconds>

        <connection-reserve-timeout-seconds>10</connection-reserve-timeout-seconds>

        <test-frequency-seconds>120</test-frequency-seconds>

        <test-connections-on-reserve>true</test-connections-on-reserve>

        <inactive-connection-timeout-seconds>180</inactive-connection-timeout-seconds>

        <test-table-name>SQL SELECT 1 FROM DUAL</test-table-name>

        <statement-cache-size>10</statement-cache-size>

        <statement-timeout>180</statement-timeout>

      </jdbc-connection-pool-params>

      <jdbc-data-source-params>

        <jndi-name>ABPRegularDataSource1</jndi-name>

        <global-transactions-protocol>None</global-transactions-protocol>

      </jdbc-data-source-params>

    </jdbc-data-source>

  • 4. Re: Disable and freeze users called on connection pool
    Joe Weinstein Expert
    Currently Being Moderated

    Hmmmm, odd. I've never seen a socket read time out for a connection request when the

    driver wasn't told to put a timeout on the socket read... Open an official support case.

  • 5. Re: Disable and freeze users called on connection pool
    PHCharbonneau Explorer
    Currently Being Moderated

    Hi,

     

    I would also recommend that you assess the health and availability of the remote Oracle listener, including from a network perspective. This is where a simple telnet or TNSPING can help. Reviewing the Oracle listener logs could also give you more details on the root cause.

     

     

    If we look more closely at the Exception stack trace, we can clearly see that the timeout was triggered during a Socket.connect() operation (connection creation time), not a query execution (Socket.read() timeout in this case). Tuning WLS JDBC DS may help, but from a root cause perspective, I recommend to focus on possible root causes of such intermittent Socket.connect() failure from WLS to Oracle listener (intermittent & increased network latency, Oracle listener problem/overload, DB server problem/overload etc.).

     

     

    Also please review the performance of the remote Oracle DB server itself, I have seen from past clients the exact same problem which was due to HW overload (CPU etc.) causing a delay to Socket.connect() operations with the remote Oracle listener and socket.connect() timeouts from the JDBC driver as a primary symptom. Other common scenarios I have seen, other than network, was related to an excessive # of connections created concurrently to a single Oracle listener, causing delays and ultimately socket.connect() timeouts from the JDBC driver.

     

     

    Caused By: java.net.ConnectException: Connection timed out

            at java.net.PlainSocketImpl.socketConnect(Native Method)

            at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)

            at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)

            at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)

            at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)

            at java.net.Socket.connect(Socket.java:529)

            at oracle.net.nt.TcpNTAdapter.connect(TcpNTAdapter.java:150)

            at oracle.net.nt.ConnOption.connect(ConnOption.java:133)

            at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:370)

            at oracle.net.resolver.AddrResolution.resolveAndExecute(AddrResolution.java:434)

            at oracle.net.ns.NSProtocol.establishConnection(NSProtocol.java:687)

            at oracle.net.ns.NSProtocol.connect(NSProtocol.java:247)

            at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1102)

            at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:320)

            at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:546)

            at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:236)

            at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)

            at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:521)

            at weblogic.jdbc.common.internal.ConnectionEnvFactory.makeConnection(ConnectionEnvFactory.java:350)

     

     

    Regards,

    P-H

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points