Forum Stats

  • 3,741,232 Users
  • 2,248,397 Discussions
  • 7,861,693 Comments

Discussions

Getting 'java.net.SocketException: Connection timed out' after long running process

User_CIP5O
User_CIP5O Member Posts: 8 Red Ribbon

Hi team,

We're getting 'java.net.SocketException: Connection timed out' in our application.

Below is the application stacktrace:

...Caused by: java.sql.SQLRecoverableException: Closed Connection        at oracle.jdbc.driver.PhysicalConnection.getAutoCommit(PhysicalConnection.java:2254) ~[ojdbc7.jar:12.1.0.2.0]        at oracle.jdbc.driver.PhysicalConnection.rollback(PhysicalConnection.java:2424) ~[ojdbc7.jar:12.1.0.2.0]        at weblogic.jdbc.wrapper.Connection.rollback(Connection.java:1785) ~[com.bea.core.datasource6.jar:12.2.1.2]        at org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.rollback(AbstractLogicalConnectionImplementor.java:115) ~[hibernate-core-5.1.1.Final.jar:5.1.1.Final]        at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.rollback(JdbcResourceLocalTransactionCoordinatorImpl.java:250) ~[hibernate-core-5.1.1.Final.jar:5.1.1.Final]        at org.hibernate.engine.transaction.internal.TransactionImpl.rollback(TransactionImpl.java:89) ~[hibernate-core-5.1.1.Final.jar:5.1.1.Final]        at org.hibernate.jpa.internal.TransactionImpl.rollback(TransactionImpl.java:101) ~[hibernate-entitymanager-5.1.1.Final.jar:5.1.1.Final]        at org.springframework.orm.jpa.JpaTransactionManager.doRollback(JpaTransactionManager.java:563) ~[spring-orm-5.0.6.RELEASE.jar:5.0.6.RELEASE]        ... 18 more

Further digging we found the following stacktrace related to the SocketException:

Caused by: java.sql.SQLRecoverableException: IO Error: Connection timed outat oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:790)at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:925)at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1111)at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:4798)at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:4845)at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1501)at weblogic.jdbc.wrapper.PreparedStatement.executeQuery(PreparedStatement.java:144)at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:70)... 47 moreCaused by: java.net.SocketException: Connection timed outat java.net.SocketInputStream.socketRead0(Native Method)at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)at java.net.SocketInputStream.read(SocketInputStream.java:170)at java.net.SocketInputStream.read(SocketInputStream.java:141)at oracle.net.ns.Packet.receive(Packet.java:311)at oracle.net.ns.DataPacket.receive(DataPacket.java:105)at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:305)at oracle.net.ns.NetInputStream.read(NetInputStream.java:249)at oracle.net.ns.NetInputStream.read(NetInputStream.java:171)at oracle.net.ns.NetInputStream.read(NetInputStream.java:89)at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:123)at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:79)at oracle.jdbc.driver.T4CMAREngineStream.unmarshalUB1(T4CMAREngineStream.java:429)at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:397)at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:257)at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:587)at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:225)at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:53)at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:774)

Below is the fact about the application:

1. It's deployed in Weblogic 12.2.1.2.0 running on Linux 64bit.

2. The error occurs only after long running process (>20mins).

3. The WL & DB server are in different subnet.

4. No idle timeout is specified in the firewall between WL & DB server.

5. Tcp dump shows that client sends FIN at the time the error occurs.

6. We've specified 'Test connection on reserved' with test freq. 120secs in WL.

The error happens when app tries to query using connection after leaving it idle for some time.

Any help would be greatly appreciated.

Regards,

James

Best Answer

  • Zlatko Sirotic
    Zlatko Sirotic Member Posts: 2,131 Gold Trophy
    edited Jan 27, 2019 9:56AM Accepted Answer

    It would be good to check the settings on the firewall once again.

    It has happened to me several times that network engineers have said that the firewall does not interrupt the connection.

    After an unsuccessful search for another culprit (database, connection pool, JDBC driver, application server ...), they checked in more detail and saw that they had made a mistake.

    Regards,

    Zlatko

Answers

  • Zlatko Sirotic
    Zlatko Sirotic Member Posts: 2,131 Gold Trophy
    edited Jan 25, 2019 7:37AM

    Because you said this:

    > 3. The WL & DB server are in different subnet.

    I would first suspect the firewall settings, but you also said this:

    > 4. No idle timeout is specified in the firewall between WL & DB server.

    so the firewall should not be a problem.

    I assume you are using a connection pool, but I do not know which (perhaps Oracle UCP).

    Check the setting that defines the maximum time after which the connection is returned to the connection pool, if the user is inactive.

    Regards,

    Zlatko

  • Zlatko Sirotic
    Zlatko Sirotic Member Posts: 2,131 Gold Trophy
    edited Jan 25, 2019 9:29AM

    Check:

    Inactive Connection Timeout

    The number of seconds of inactivity after which reserved connections will forcibly be released back into the pool. When set to 0 (the default), this feature is disabled.

    Regards,

    Zlatko

  • User_CIP5O
    User_CIP5O Member Posts: 8 Red Ribbon
    edited Jan 25, 2019 10:49AM

    Hi Zlatko,

    We've checked the setting you mentioned and it's the default (0).

    And I forgot to mention that not all long running process ends up with this error.

    Regards,

    James

  • Zlatko Sirotic
    Zlatko Sirotic Member Posts: 2,131 Gold Trophy
    edited Jan 25, 2019 11:09AM

    > And I forgot to mention that not all long running process ends up with this error.

    Do you use Connection Harvesting?

    https://docs.oracle.com/middleware/1212/wls/JDBCA/ds_harvesting.htm#JDBCA658

    You can specify a number of reserved connections to be released when a data source reaches a specified number of available connections.

    Harvesting helps to ensure that a specified number of connections are always available in the pool and improves performance by minimizing connection initialization ...

    Regards,

    Zlatko

  • User_CIP5O
    User_CIP5O Member Posts: 8 Red Ribbon
    edited Jan 25, 2019 11:16AM

    Hi Zlato,

    If using it means we have to change the setting other than default then the answer is no.

    Regards,

    James

  • Zlatko Sirotic
    Zlatko Sirotic Member Posts: 2,131 Gold Trophy
    edited Jan 25, 2019 12:13PM

    It's possible that the problem is related to Hibernate settings!?

    Unfortunately, I do not know almost anything about Hibernate

    (except that it, as JDBC in general, has AutoCommit = true as default, which in most cases is bad for data integrity).

    Regards,

    Zlatko

  • User_CIP5O
    User_CIP5O Member Posts: 8 Red Ribbon
    edited Jan 26, 2019 9:34AM

    Hi Zlatko,

    I doubt that Hibernate has something to do with it since it sits far above.

    We'll write standalone straight JDBC snippet and see if we can reproduce the errors.

    Regards,

    James

  • Zlatko Sirotic
    Zlatko Sirotic Member Posts: 2,131 Gold Trophy
    edited Jan 26, 2019 2:31PM

    > I doubt that Hibernate has something to do with it since it sits far above.

    I agree but, for example, Hibernate supports the javax.persistence.query.timeout query hint to call the setQueryTimeout() method on the JDBC Statement.

    I do not want to say that setQueryTimeout() is a "culprit" for closing a connection - this method does not close the connection (unless there is another code that explicitly does this, perhaps in exception handler).

    Regards,

    Zlatko

  • User_CIP5O
    User_CIP5O Member Posts: 8 Red Ribbon
    edited Jan 27, 2019 1:46AM

    Hi Zlatko,

    Thank you for your insight.

    Anyway, I'm still open for possibilities including Hibernate, I will look into it.

    Regards,

    James

  • User_CIP5O
    User_CIP5O Member Posts: 8 Red Ribbon
    edited Jan 27, 2019 9:07AM

    Hi Zlatko,

    I have written & run small snippets just to connect to DB and run query 'select 1 from DUAL'. The snippet will run the query then sleep for 1 hour after which it will run the same query again using the same connection.

    Below is the result:

    1. On my laptop to local XE DB -> SUCCESS.

    2. On our test environment where WL & DB server are in the same subnet -> SUCCESS.

    3. On production servers, where WL & DB are in different subnet -> FAILED.

    #3 failed with the following stacktrace:

    java.sql.SQLRecoverableException: IO Error: Connection timed out        at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:779) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]        at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:921) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]        at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1099) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]        at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3640) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]        at oracle.jdbc.driver.T4CPreparedStatement.executeInternal(T4CPreparedStatement.java:1384) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]        at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3687) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]        at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1165) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]...Caused by: java.io.IOException: Connection timed out        at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:1.8.0_71]        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:1.8.0_71]        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[?:1.8.0_71]        at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[?:1.8.0_71]        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[?:1.8.0_71]        at oracle.net.nt.TimeoutSocketChannel.read(TimeoutSocketChannel.java:178) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]        at oracle.net.ns.NSProtocolNIO.doSocketRead(NSProtocolNIO.java:544) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]        at oracle.net.ns.NIOPacket.readHeader(NIOPacket.java:234) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]        at oracle.net.ns.NIOPacket.readPacketFromSocketChannel(NIOPacket.java:174) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]        at oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:122) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]        at oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:100) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]        at oracle.net.ns.NIONSDataChannel.readDataFromSocketChannel(NIONSDataChannel.java:86) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]        at oracle.jdbc.driver.T4CMAREngineNIO.prepareForUnmarshall(T4CMAREngineNIO.java:762) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]        at oracle.jdbc.driver.T4CMAREngineNIO.unmarshalUB1(T4CMAREngineNIO.java:427) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]        at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:394) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]        at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:255) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]        at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:610) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]        at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:253) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]        at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:86) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]        at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:765) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]        ... 8 more

    This is either network related issue or DB setting.

    Regards,

    James

  • Zlatko Sirotic
    Zlatko Sirotic Member Posts: 2,131 Gold Trophy
    edited Jan 27, 2019 9:56AM Accepted Answer

    It would be good to check the settings on the firewall once again.

    It has happened to me several times that network engineers have said that the firewall does not interrupt the connection.

    After an unsuccessful search for another culprit (database, connection pool, JDBC driver, application server ...), they checked in more detail and saw that they had made a mistake.

    Regards,

    Zlatko

  • User_CIP5O
    User_CIP5O Member Posts: 8 Red Ribbon
    edited Jan 28, 2019 6:52AM

    Hi Zlato,

    I've made the network admin check the firewall, this time he added additional parameter that finally nailed the problem.

    Regards,

    James

  • User_7ZWP1
    User_7ZWP1 Member Posts: 1 Green Ribbon

    Hi James,

    I'm facing similar issue while connecting to DB server from WebLogic server via data source connection intermittently, both servers are in different subnet. Could you please help me with the network settings done on firewall to resolve.

    Regards,

    Ricky

Sign In or Register to comment.