5 Replies Latest reply: Dec 1, 2011 2:25 AM by 897266 RSS

    Memory leak in OCCI/libclnts on Linux RHEL 5.5 ? Please advise...

    897266
      Hi all,

      Starting with some background:
      I have an application that has been running on 32-bit Linux server for years without problem, a few weeks ago it was ported to 64-bit RHEL 5.5 (compiled with -m32). Now with the application in the new environment some processes are growing in memorysize.
      I have tried to pinpoint the memory leak with Valgrind and have numerous leaks reported in "libclntsh.so", "libocci.so" and "libnnz11.so"

      Examples:

      ==30114== Use of uninitialised value of size 4
      ==30114== at 0xA61BAD7: ztcedecb (in /soe3/opt/oracle-1/product/32bit-client-11.2.0.2/lib/libclntsh.so.11.1)
      ==30114== by 0xA61B1E2: ztcedencbk (in /soe3/opt/oracle-1/product/32bit-client-11.2.0.2/lib/libclntsh.so.11.1)
      ==30114== by 0xA61AA0A: ztcebn (in /soe3/opt/oracle-1/product/32bit-client-11.2.0.2/lib/libclntsh.so.11.1)
      ==30114== by 0xA61A561: ztcen (in /soe3/opt/oracle-1/product/32bit-client-11.2.0.2/lib/libclntsh.so.11.1)
      ==30114== by 0x8DA9725: ztceenc (in /soe3/opt/oracle-1/product/32bit-client-11.2.0.2/lib/libclntsh.so.11.1)
      ==30114== by 0x8E68920: ztcrbm (in /soe3/opt/oracle-1/product/32bit-client-11.2.0.2/lib/libclntsh.so.11.1)
      ==30114== by 0x8E684D5: ztcrbh (in /soe3/opt/oracle-1/product/32bit-client-11.2.0.2/lib/libclntsh.so.11.1)
      ==30114== by 0x8E68393: ztcrbp (in /soe3/opt/oracle-1/product/32bit-client-11.2.0.2/lib/libclntsh.so.11.1)
      ==30114== by 0x8E682C6: ztcr2seed (in /soe3/opt/oracle-1/product/32bit-client-11.2.0.2/lib/libclntsh.so.11.1)
      ==30114== by 0x8E6827F: ztcrseed3 (in /soe3/opt/oracle-1/product/32bit-client-11.2.0.2/lib/libclntsh.so.11.1)
      ==30114== by 0x8DAA24A: ztcsh (in /soe3/opt/oracle-1/product/32bit-client-11.2.0.2/lib/libclntsh.so.11.1)
      ==30114== by 0x99727D7: kpucpcreate (in /soe3/opt/oracle-1/product/32bit-client-11.2.0.2/lib/libclntsh.so.11.1)

      And:

      ==30114== Conditional jump or move depends on uninitialised value(s)
      ==30114== at 0x8E69BBC: ztvo5ke (in /soe3/opt/oracle-1/product/32bit-client-11.2.0.2/lib/libclntsh.so.11.1)
      ==30114== by 0x8D55E68: kpu8lgn (in /soe3/opt/oracle-1/product/32bit-client-11.2.0.2/lib/libclntsh.so.11.1)
      ==30114== by 0x8D53E56: kpuauthxa (in /soe3/opt/oracle-1/product/32bit-client-11.2.0.2/lib/libclntsh.so.11.1)
      ==30114== by 0x8D536DB: kpuauth (in /soe3/opt/oracle-1/product/32bit-client-11.2.0.2/lib/libclntsh.so.11.1)
      ==30114== by 0x996EB99: kpucpinithstnode (in /soe3/opt/oracle-1/product/32bit-client-11.2.0.2/lib/libclntsh.so.11.1)
      ==30114== by 0x996FFB1: kpucpcrecons (in /soe3/opt/oracle-1/product/32bit-client-11.2.0.2/lib/libclntsh.so.11.1)
      ==30114== by 0x9972871: kpucpcreate (in /soe3/opt/oracle-1/product/32bit-client-11.2.0.2/lib/libclntsh.so.11.1)
      ==30114== by 0x9821048: OCIConnectionPoolCreate (in /soe3/opt/oracle-1/product/32bit-client-11.2.0.2/lib/libclntsh.so.11.1)
      ==30114== by 0x7BFB896: oracle::occi::ConnectionPoolImpl::initialise(oracle::occi::EnvironmentImpl*, void*, unsigned int, void*, unsigned int, void*, unsigned int, unsigned int, unsigned int, unsigned int) (in /soe3/opt/oracle-1/product/32bit-client-11.2.0.2/lib/libocci.so.11.1)
      ==30114== by 0x7BFBB79: ZN6oracle4occi18ConnectionPoolImplC9EPNS015EnvironmentImplERKSsS5_S5_jjj (in /soe3/opt/oracle-1/product/32bit-client-11.2.0.2/lib/libocci.so.11.1)
      ==30114== by 0x7BFBAC7: oracle::occi::ConnectionPoolImpl::ConnectionPoolImpl(oracle::occi::EnvironmentImpl*, std::string const&, std::string const&, std::string const&, unsigned int, unsigned int, unsigned int) (in /soe3/opt/oracle-1/product/32bit-client-11.2.0.2/lib/libocci.so.11.1)
      ==30114== by 0x7BF2C13: oracle::occi::EnvironmentImpl::createConnectionPool(std::string const&, std::string const&, std::string const&, unsigned int, unsigned int, unsigned int) (in /soe3/opt/oracle-1/product/32bit-client-11.2.0.2/lib/libocci.so.11.1)

      And:

      ==30114== Conditional jump or move depends on uninitialised value(s)
      ==30114== at 0xAC3B2A5: CMP_RecomputeLength (in /soe3/opt/oracle-1/product/32bit-client-11.2.0.2/lib/libnnz11.so)
      ==30114== by 0xAC3B9EC: CMP_Divide (in /soe3/opt/oracle-1/product/32bit-client-11.2.0.2/lib/libnnz11.so)
      ==30114== by 0xAC3B5BA: CMP_ModularReduce (in /soe3/opt/oracle-1/product/32bit-client-11.2.0.2/lib/libnnz11.so)
      ==30114== by 0xAC3A2E6: Alg_ComputeModQ_GHash (in /soe3/opt/oracle-1/product/32bit-client-11.2.0.2/lib/libnnz11.so)
      ==30114== by 0xAC3AA30: A_X931RandomGenerateBytes (in /soe3/opt/oracle-1/product/32bit-client-11.2.0.2/lib/libnnz11.so)
      ==30114== by 0x8E681F0: ztcr2rnd (in /soe3/opt/oracle-1/product/32bit-client-11.2.0.2/lib/libclntsh.so.11.1)
      ==30114== by 0x8DAA260: ztcsh (in /soe3/opt/oracle-1/product/32bit-client-11.2.0.2/lib/libclntsh.so.11.1)
      ==30114== by 0x8CD0125: kpusattr (in /soe3/opt/oracle-1/product/32bit-client-11.2.0.2/lib/libclntsh.so.11.1)
      ==30114== by 0x8CE96CD: OCIAttrSet (in /soe3/opt/oracle-1/product/32bit-client-11.2.0.2/lib/libclntsh.so.11.1)
      ==30114== by 0x7BF6184: oracle::occi::ConnectionImpl::openConnection(OCIEnv*, OCIError*, void*, unsigned int, void*, unsigned int, void*, unsigned int, void*, unsigned int, unsigned int, void*, unsigned int, oracle::occi::Connection::Purity, oracle::occi::StatelessConnectionPool::PoolType) (in /soe3/opt/oracle-1/product/32bit-client-11.2.0.2/lib/libocci.so.11.1)
      ==30114== by 0x7BF9C29: ZN6oracle4occi14ConnectionImplC9EPNS018ConnectionPoolImplERKSsS5_ (in /soe3/opt/oracle-1/product/32bit-client-11.2.0.2/lib/libocci.so.11.1)
      ==30114== by 0x7BF9AD1: oracle::occi::ConnectionImpl::ConnectionImpl(oracle::occi::ConnectionPoolImpl*, std::string const&, std::string const&) (in /soe3/opt/oracle-1/product/32bit-client-11.2.0.2/lib/libocci.so.11.1)


      There has been quite a few changes due to the new server. New compiler (gcc) from version 2.96 to 4.1.2, new version of oracle client 10g to 11g (11.2.0.2), new version of OCCI (also 11.2.0.2). The applications sourcecode is pretty much unchanged though, and that is why I turn to all of you for help.

      Is there anyone who knows anything about this problem and how to even start to fix this leak (if it even is in OCCI)?

      I can supply sourcecode and/or logs if needed. It is quite a lot of complex code so it might take me a while to sort out the relevant code though...
        • 1. Re: Memory leak in OCCI/libclnts on Linux RHEL 5.5 ? Please advise...
          894085
          There are changes in the stateless connection pool in 11.2 which are clearly referenced there, do any of the 'pretty much unchanged' changes relate to your connection pool?

          I realise this thread is a little old now, but if you post up anything that's relevant to your pool management I'm happy to take a look?
          • 2. Re: Memory leak in OCCI/libclnts on Linux RHEL 5.5 ? Please advise...
            897266
            Thanks for your input! Tough this post is from some time ago this is still an issue for me I'm afraid, so I appreciate all the help I can get.
            I have stripped down the code to just the basic functions needed to replicate the problem. The pool creation and connections are the same as in our application (which is unchanged since before the porting). Though the following code makes no indications of a leak when run through Valgrind, it still grows when checked in "top". It could be that I stripped the code too much, but this is pretty much what our application does. It creates a database lock (not in this code), creates connection, (read table), terminates connection and release lock every 6 seconds to check if there is any new assignments added...


            #include <iostream>
            #include <string>
            #include <occi.h>

            using namespace oracle::occi;
            using namespace std;

            Environment *env;
            ConnectionPool *connPool;
            Connection *con;
            int x, loopnr;

            string usr = "test_user";
            string pwd = "test_pw";
            string url = "dbtest";

            void run(int x) { // Run loops
            for(int i=0; i<x; i++){ // Loop connect and disconnect x number of times
            try {
            fflush(stdout); // Update console row for count value
            con = connPool->createConnection (usr, pwd); // Connect
            usleep(100000);
            connPool->terminateConnection (con); // Disconnect
            cout << "\r"; // Update console row for count value
            cout << loopnr << endl; // Print number of loops done
            usleep(100000);
            loopnr++; // Count number of loops done
            }
            catch(SQLException ex) { // Exception handling
            cout<<"SQLException"<<endl;
            cout<<"Error number: "<< ex.getErrorCode() << endl;
            cout<<ex.getMessage() << endl;
            }
            }
            }

            int main(int argc,char* argv[]) {
            cout << "IN --- main()\n";
            if(argc == 2) { // Read number of loops as argument...
            x = atoi(argv[1]);
            }
            else { // ...or run loop 200 times if no args
            x = 200;
            }
            try{
            env = Environment::createEnvironment (Environment::DEFAULT); // Create environment
            loopnr = 1;
            connPool = env->createConnectionPool(usr, pwd, url, 1, 2, 1); // Create pool
            run(x); // Run loops
            }
            catch(SQLException ex) { // Exception handling
            cout<<"SQLException"<<endl;
            cout<<"Error number: "<< ex.getErrorCode() << endl;
            cout<<ex.getMessage() << endl;
            }
            env->terminateConnectionPool (connPool); // Cleanup Pool when done
            Environment::terminateEnvironment (env); // Cleanup Environment when done
            cout << "\nUT --- main()\n";
            return 0;
            }

            -----
            Above code is built with the following makefile:

            ORACLE_HOME=/opt/oracle/product/32bit-client-11.2.0.2

            CXXFLAGS = -Wall -m32 -g -fPIC \
            -I$(ORACLE_HOME)/rdbms/public

            LDLIBS = -L$(ORACLE_HOME)/lib/ -locci \
            -L$(ORACLE_HOME)/rdbms/lib/ -lclntsh
            LDFLAGS = -m32

            default: all

            Sample: Sample.cc

            all: Sample

            clean:
            rm -f Sample *.o

            Hopefully someone can make sense out of this....

            Edited by: StLa on 2011-nov-23 04:57
            • 3. Re: Memory leak in OCCI/libclnts on Linux RHEL 5.5 ? Please advise...
              894085
              Thanks,

              I'm having a look. I'll let you know what it looks like from this angle.
              • 4. Re: Memory leak in OCCI/libclnts on Linux RHEL 5.5 ? Please advise...
                897266
                Thanks, I keep working in my end too... I've called in a few other persons in my company, to see if they have any input in this matter.
                • 5. Re: Memory leak in OCCI/libclnts on Linux RHEL 5.5 ? Please advise...
                  897266
                  I've tested the above code with Helgrind and DRD and got reports about race conditions related to libclntsh.so.
                  Right now I'm not sure what to think about this, but I'll keep digging.
                  Any input would be greatly appreciated.