This discussion is archived
5 Replies Latest reply: Oct 1, 2012 11:27 AM by 964907 RSS

Memory and Thread leak from getConnection when password is expired

964907 Newbie
Currently Being Moderated
I realize this isn't a support forum but thought I'd post here first on the chance there were any words of wisdom from somebody who's seen this issue.

We had an OutOfMemoryError thrown in our Java web app (JBoss, Oracle, Hibernate) which I have tracked down to an apparent memory leak inside the Oracle Java code that runs somewhere inside the OracleDataSource getConnection() method. The scenario that lead to the error was the fact that our database's user password had expired. Our app has two parts of it that each do a database access once every 30 seconds. Each one of these was incurring an "ORA-28001: the password has expired" exception. After a day or so OutOfMemoryError exceptions started getting thrown.

I isolated it into a stripped down test program (no JBoss, no Hibernate) and can reproduce the issue easily on my development machine which is runing Windows XP with Oracle Database 10g Express Edition Release 10.2.0.1.0. The test program keeps trying to get a connection to the database repeatedly, sleeping in between each try. The presumption is that the user's password has been artificially expired prior to running the program so as to make it incur an SQLException on the getConnection() call. The user and password I used were correct, except of course that the password had expired. It prints results every 500 tries just to show it's in porgress. The code takes two parameters. The first tells it whether or not to use connection caching - default "true". The second varies the sleep time between connection attempts - default 30 milliseconds. The output of a run with the default values follows:

Starting at: Thu Sep 27 13:35:33 EDT 2012
exception 0: java.sql.SQLException: ORA-28001: the password has expired
exception 500: java.sql.SQLException: ORA-28001: the password has expired
exception 1000: java.sql.SQLException: ORA-28001: the password has expired
exception 1500: java.sql.SQLException: ORA-28001: the password has expired
exception 2000: java.sql.SQLException: ORA-28001: the password has expired
exception 2500: java.sql.SQLException: ORA-28001: the password has expired
exception 3000: java.sql.SQLException: ORA-28001: the password has expired
exception 3500: java.sql.SQLException: ORA-28001: the password has expired
exception 4000: java.sql.SQLException: ORA-28001: the password has expired
exception 4500: java.sql.SQLException: ORA-28001: the password has expired
exception 4883: java.lang.OutOfMemoryError: unable to create new native thread
Done at: Thu Sep 27 13:38:42 EDT 2012

As you can see it gets the OutOfMemoryError after ~ 30 seconds.

In running the program in debug I noticed orphaned threads being created. Turns out that each call to getConnection() is leaving a thread that never runs down.

In JVisualVM I did a thread dump and here's an example of one of orphaned threads:

-----
"Thread-703" daemon prio=6 tid=0x09c5b400 nid=0x102e0 waiting on condition [0x11c0f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
     at java.lang.Thread.sleep(Native Method)
     at oracle.jdbc.pool.OracleImplicitConnectionCacheThread.run(OracleImplicitConnectionCacheThread.java:91)

Locked ownable synchronizers:
     - None
-----

Has anybody seen this? If so, is there a work around?

Here is the code:

<code>
import java.sql.Connection;
import java.sql.SQLException;
import java.util.Date;
import java.util.Properties;

import javax.sql.DataSource;

import oracle.jdbc.pool.OracleDataSource;

public class TestLeak {
private static final boolean DEFCONCACHE = true;
private static final int DEFINTERVAL = 30;

private static DataSource getDataSource(boolean connCache) throws SQLException {
OracleDataSource ds = new OracleDataSource();

ds.setURL("jdbc:oracle:thin:@localhost:1521:xe");
ds.setDriverType("oracle.jdbc.driver.OracleDriver");
ds.setUser("myuser");
ds.setPassword("mypassword");

if (connCache){
ds.setConnectionCachingEnabled(true);
ds.setConnectionCacheName("MyCache");
Properties cacheProps = new Properties();
cacheProps.setProperty("MinLimit", Integer.toString(5));
cacheProps.setProperty("MaxLimit", Integer.toString(20));
cacheProps.setProperty("InitialLimit", Integer.toString(5));
cacheProps.setProperty("InactivityTimeout", Integer.toString(900));
cacheProps.setProperty("ConnectionWaitTimeout", Integer.toString(10));
cacheProps.setProperty("PropertyCheckInterval", Integer.toString(60));
cacheProps.setProperty("ValidateConnection", "true");
ds.setConnectionCacheProperties(cacheProps);
}

return ds;
}

private static String time() {

return (new Date().toString());
}
public static void main(String[] args) {
boolean connCache = DEFCONCACHE;
Integer interval = new Integer(DEFINTERVAL);
DataSource ds;
int count = 0;
Connection conn1 = null;
boolean go = true;
System.out.println("Starting at: " + time());

if (args.length == 2) {
if (args[0].equalsIgnoreCase("true"))
connCache = true;
else if (args[0].equalsIgnoreCase("false"))
connCache = false;
else{
System.out
.println("Didn't understand P1=\""
+ args[0]
+ "\" Value must be true/fase for whether to use cxn pooling. Using value of "
+ connCache);
}

try {
interval = Integer.valueOf(args[1]);
}
catch (NumberFormatException nfe) {
System.out
.println("Didn't understand P2=\""
+ args[1]
+ "\" Value must be milliseconds to wait between connect tries. Using value of "
+ interval);
}
}

while (go) {
try {
ds = getDataSource(connCache);
/*Connection*/ conn1 = ds.getConnection();
if (conn1 != null) {
System.out.println("exception conn1.hashCode()=" + conn1.hashCode());
}
}
catch (OutOfMemoryError oom) {
System.out.println("exception " + count + ": " + oom.toString().trim());
go = false;
}
catch (SQLException e) {
if (count % 500 == 0)
System.out.println("exception " + count + ": " + e.toString().trim());
}

try {
Thread.sleep(interval);
}
catch (InterruptedException e) {
System.out.println("Sleep threw InterruptedException");
}

count++;
}
System.out.println("Done at: " + time());
}
}

// end file TestLeak.java
</code>
  • 1. Re: Memory and Thread leak from getConnection when password is expired
    Joe Weinstein Expert
    Currently Being Moderated
    Seems like a clear driver bug. Open a support case if this reproduces using the latest driver version
    you can download from Oracle.
  • 2. Re: Memory and Thread leak from getConnection when password is expired
    964907 Newbie
    Currently Being Moderated
    Oh duh. I forgot to check for driver updates. Just did though and we have the current driver for 10.2.0.1. Just for yuks I uploaded the most current driver for the last listed version of 10g R2 which is 10.2.0.5 and that behaved the same way. We'll make a support case.

    Thanks.
  • 3. Re: Memory and Thread leak from getConnection when password is expired
    dsurber Explorer
    Currently Being Moderated
    You should always use the latest version of the drivers, which is 11.2.0.3.0. The latest drivers always support all versions of the database that have not been desupported. In your case 10.2 database is still supported so the latest driver supports it.

    If your database has been desupported then you should use the latest version of the drivers that was released while your database was supported.

    Also you are using the driver's Implicit Connection Cache. That is deprecated in 11 in favor of the Universal Connection Pool. You can continue to use ICC with 11 but we are not actively supporting it. I'm not sure what support will do. Since 10 is still supported I guess (strictly a guess) they'll try to fix the problem in 10. It should be an easy fix. I don't see a bug that obviously corresponds to this problem, though there may be one.

    I suggest switching to the 11.2.0.3.0 driver and if feasible UCP.
  • 4. Re: Memory and Thread leak from getConnection when password is expired
    964907 Newbie
    Currently Being Moderated
    Thanks,

    I downloaded ojdbc6.jar from the 11.2.0.3 download page. After a re-test of the code unmodified it still behaves the same - leaks threads & gets OOME.

    Even if we could get the 11.2.0.3 driver to work though it may turn out to be really hard politically to switch to a driver that divergent from what we and our partners have tested on. Mod's to an existing older driver would probably be more palatible. (I could be wrong though. I will pursue this internally.)

    I will go off and look at UCP. I assume this is the place I need to be reading up:

    [http://docs.oracle.com/cd/E11882_01/java.112/e12265/toc.htm#BEGIN]

    Thanks for the help.
  • 5. Re: Memory and Thread leak from getConnection when password is expired
    964907 Newbie
    Currently Being Moderated
    I changed the test program to do things ala UCP. Changed code snippet to get the data source follows:
    PoolDataSource ds = PoolDataSourceFactory.getPoolDataSource();

    ds.setConnectionFactoryClassName("oracle.jdbc.pool.OracleDataSource");
    ds.setURL(URL);
    ds.setUser(USER);
    ds.setPassword(PASS);
    return ds;

    The change worked. Ran 10+ hours with no leak. Also, I was able to do this running our existing (albeit old) ojdbc14.jar file from 10.2.0.4 and only adding the ucp.jar file.

    I then integrated the new UCP approach into our application source. Unfortunately it exposed a known bug in Hibernate 3.3.1 (the version we’re using). The Hibernate code had a hardwired reference to the “oracle.jdbc.driver.OracleTypes” class which had been moved to the “oracle.jdbc” package in Oracle 11:

    Re: Problems of hibernate calling oracle stored procedure

    This bug is fixed in Hibernate 3.3.2. Unfortunately we cannot change our environment around at this point so we’re stuck with 3.3.1 so we need old-style OracleDataSource “IIC” to work. That’s not an issue for here though. Will make a support case.

    Thanks everyone for the help.

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points