4 Replies Latest reply: Apr 29, 2014 11:28 AM by PatrickF RSS

    Potential Memory Leak and Other Errors in EDQ

    PatrickF

      We have recently migrated from a development environment with EDQ that was using PostgreSQL to a testing environment using Oracle DB. It is worth nothing that in the development environment, unit testing and some integration testing with Siebel occurred without error, and the traffic to EDQ was lower. It was only after the database and environment transition that we have started to encounter issues. Our error logs have been getting filled up with information similar to the following: (I can post the entire log if requested)

       

      Apr 21, 2014 2:08:54 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks

      SEVERE: The web application [/edq] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@4e78c870]) and a value of type [com.datanomic.userauth.server.Session] (value [com.datanomic.userauth.server.Session@4463db73]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.

      Apr 21, 2014 2:08:54 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks

      SEVERE: The web application [/edq] created a ThreadLocal with key of type [oracle.jdbc.driver.AutoKeyInfo$1] (value [oracle.jdbc.driver.AutoKeyInfo$1@77dc7e18]) and a value of type [oracle.jdbc.driver.OracleSql] (value [INSERT INTO dn_eventlog2 (event_id, event_type, event_sub_type, event_time, duration, severity, mission_execution_id, mission_name, task_execution_id, task_type, task_name, user_name, status, project_name, project_id, runlabel, run_profile, message_string, message) VALUES (seq_dn_eventlog2.nextval, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.

      Apr 21, 2014 2:08:54 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks

      SEVERE: The web application [/edq] created a ThreadLocal with key of type [oracle.jdbc.driver.AutoKeyInfo$1] (value [oracle.jdbc.driver.AutoKeyInfo$1@77dc7e18]) and a value of type [oracle.jdbc.driver.OracleSql] (value [INSERT INTO dn_eventlog2 (event_id, event_type, event_sub_type, event_time, duration, severity, mission_execution_id, mission_name, task_execution_id, task_type, task_name, user_name, status, project_name, project_id, runlabel, run_profile, message_string, message) VALUES (seq_dn_eventlog2.nextval, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.

      Apr 21, 2014 2:08:54 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks

      SEVERE: The web application [/edq] created a ThreadLocal with key of type [oracle.jdbc.driver.AutoKeyInfo$1] (value [oracle.jdbc.driver.AutoKeyInfo$1@77dc7e18]) and a value of type [oracle.jdbc.driver.OracleSql] (value [INSERT INTO dn_eventlog2 (event_id, event_type, event_sub_type, event_time, duration, severity, mission_execution_id, mission_name, task_execution_id, task_type, task_name, user_name, status, project_name, project_id, runlabel, run_profile, message_string, message) VALUES (seq_dn_eventlog2.nextval, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.

       

      We are also encountering the following error being displayed on one of our jobs:

       

      Process failed: A database error has occurred : IO Error: Socket read timed out. (Code: 200,302) (Code: 200,302)

       

      I have a feeling that these two issues may be related. I am waiting to implement a potential solution that I found which was to add an extra Java property to tomcat:

       

      -Doracle.mds.cache=simple


      Lastly, we have also started to encounter the following error when calling EDQ from Siebel:


      Call to PropertSet.GetChild() failed. The property set does not have any children. (SBL_EXL_00144)


      A temporary workaround for this issue is to restart both Siebel and EDQ.



      I am not sure if any of these issues are related, but they all started to occur at the same time. I will post any updates after the Java property is implemented as well, but in the mean time, does there appear to be anything obviously wrong?

        • 1. Re: Potential Memory Leak and Other Errors in EDQ
          Shilpi-Oracle

          HI

           

          Regarding error:

          Process failed: A database error has occurred : IO Error: Socket read timed out. (Code: 200,302) (Code: 200,302)

           

          This error indicate the application is not able to communicate to the database.

          This might be due to different reasons: firewall/proxy settings, network problem , some database problem especially TNS configuration problems.

          Reference- IO Error: Socket Read Timed Out. (Code: 200,302)" In Oracle Enterprise Data Quality (EDQ) (Doc ID 1488199.1).

           

          For the other reported error:

          Call to PropertSet.GetChild() failed. The property set does not have any children. (SBL_EXL_00144)

           

          Please check the input and output arguments for affected BS.

          If Business Service invoke a Workflow, then try to check the contents of the response property set ?

           

          Thanks,

          Shilpi

          • 2. Re: Potential Memory Leak and Other Errors in EDQ
            Mike-Matthews-Oracle

            Hi,

             

            It is not possible to tell for sure without the full logs but it sounds like the server encountered an error, probably in communicating with the database, which then led to it terminating in a non-clean manner. In normal operation, the application is very well tested for leaks of all kinds (connections/memory etc.)

             

            Make sure you have all the JVM parameters correctly set per the installation documentation here:

             

            Configuring EDQ Post-Installation

             

            as well as the required parameters for Tomcat as documented in Section 5.3 of the same document.

             

            If you get all these right, it is unlikely that you will encounter a memory leak especially if you are on one of the latest versions. Which version of EDQ do you have?

             

            Regards,

             

            Mike

            • 3. Re: Potential Memory Leak and Other Errors in EDQ
              Rde1-Oracle

              The warnings about memory leaks can be ignored.  They come from a feature in tomcat which detects the use of certain Java features which can cause memory problems if an application is shutdown but tomcat is not shutdown.  This is not supprted with EDQ - if you need to update the application you must stop and restart tomcat.

               

              Richard

              • 4. Re: Potential Memory Leak and Other Errors in EDQ
                PatrickF

                Hi Mike and Richard,

                 

                The full logs can be found here.

                 

                We are currently running EDQ 11.1.1.7.3 on a 64 bit architecture.

                 

                I will work with my team using that document to see if there are any set-up parameters that we have missed.

                 

                After some testing yesterday, we have determined that there may be a complex matching process that is causing some issues. It is approximately 120 processors and involves a fair amount of logic and branching. We have noticed that as we send more records to it (incrementally moving from 100 to ~600), the performance grinds to a complete halt. However, usually after waiting a significant amount of time, it appears that the process will start accepting new records once more. When the process is hanging like this, it will not cancel the real time job within EDQ either.