Hi team,
We're getting 'java.net.SocketException: Connection timed out' in our application.
Below is the application stacktrace:
...
Caused by: java.sql.SQLRecoverableException: Closed Connection
at oracle.jdbc.driver.PhysicalConnection.getAutoCommit(PhysicalConnection.java:2254) ~[ojdbc7.jar:12.1.0.2.0]
at oracle.jdbc.driver.PhysicalConnection.rollback(PhysicalConnection.java:2424) ~[ojdbc7.jar:12.1.0.2.0]
at weblogic.jdbc.wrapper.Connection.rollback(Connection.java:1785) ~[com.bea.core.datasource6.jar:12.2.1.2]
at org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.rollback(AbstractLogicalConnectionImplementor.java:115) ~[hibernate-core-5.1.1.Final.jar:5.1.1.Final]
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.rollback(JdbcResourceLocalTransactionCoordina
torImpl.java:250) ~[hibernate-core-5.1.1.Final.jar:5.1.1.Final]
at org.hibernate.engine.transaction.internal.TransactionImpl.rollback(TransactionImpl.java:89) ~[hibernate-core-5.1.1.Final.jar:5.1.1.Final]
at org.hibernate.jpa.internal.TransactionImpl.rollback(TransactionImpl.java:101) ~[hibernate-entitymanager-5.1.1.Final.jar:5.1.1.Final]
at org.springframework.orm.jpa.JpaTransactionManager.doRollback(JpaTransactionManager.java:563) ~[spring-orm-5.0.6.RELEASE.jar:5.0.6.RELEASE]
... 18 more
Further digging we found the following stacktrace related to the SocketException:
Caused by: java.sql.SQLRecoverableException: IO Error: Connection timed out
at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:790)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:925)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1111)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:4798)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:4845)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1501)
at weblogic.jdbc.wrapper.PreparedStatement.executeQuery(PreparedStatement.java:144)
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:70)
... 47 more
Caused by: java.net.SocketException: Connection timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at oracle.net.ns.Packet.receive(Packet.java:311)
at oracle.net.ns.DataPacket.receive(DataPacket.java:105)
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:305)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:249)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:171)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:89)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:123)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:79)
at oracle.jdbc.driver.T4CMAREngineStream.unmarshalUB1(T4CMAREngineStream.java:429)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:397)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:257)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:587)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:225)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:53)
at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:774)
Below is the fact about the application:
1. It's deployed in Weblogic 12.2.1.2.0 running on Linux 64bit.
2. The error occurs only after long running process (>20mins).
3. The WL & DB server are in different subnet.
4. No idle timeout is specified in the firewall between WL & DB server.
5. Tcp dump shows that client sends FIN at the time the error occurs.
6. We've specified 'Test connection on reserved' with test freq. 120secs in WL.
The error happens when app tries to query using connection after leaving it idle for some time.
Any help would be greatly appreciated.
Regards,
James