This discussion is archived
9 Replies Latest reply: Apr 29, 2010 6:54 AM by 843798 RSS

Memory leak under GNU/Linux when using exec()

843798 Newbie
Currently Being Moderated
Hi,

We detected that our application was taking all the free memory of the computer when we were using intensively and periodically the method exec() to execute some commands of the OS. The OS of the computer is a GNU/Linux based OS.

So, in order to do some monitoring we decided to wrote a simple program that called exec() infinite number of times, and using the profiler tool of Netbeans we saw a memory leak in the program because the number of surviving generations increased during all the execution time. The classes that have more surviving generations are java.lang.ref.Finalizer, java.io.FileDescriptor and byte[].

We also decided to test this simple program using Windows, and in that OS we saw that the memory leak disappeared: the number of surviving generations was almost stable.

I attach you the code of the program.
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;

public class testExec
{

    public static void main(String args[]) throws IOException, InterruptedException
    {

        Runtime runtime = Runtime.getRuntime();

        while (true)
        {
            Process process = null;
            InputStream is = null;
            InputStreamReader isr = null;
            BufferedReader br = null;

            try
            {
                process = runtime.exec("ls");
                //process = runtime.exec("cmd /c dir");
                is = process.getInputStream();
                isr = new InputStreamReader(is);
                br = new BufferedReader(isr);
                String line;

                while ((line = br.readLine()) != null)
                {
                    System.out.println(line);
                }
            }
            finally
            {
                process.waitFor();

                if (is != null)
                {
                    is.close();
                }
                if (isr != null)
                {
                    isr.close();
                }
                if (br != null)
                {
                    br.close();
                }
                if (process != null)
                {
                    process.destroy();
                }
            }
        }
    }
}
¿Is anything wrong with the test program we wrote? (we know that is not usual to call infinite times the command ls/dir, but it's just a test)
¿Why do we have a memory leak in Linux but not in Windows?

I will appreciate any help or ideas. Thanks in advance.
  • 1. Re: Memory leak under GNU/Linux when using exec()
    843798 Newbie
    Currently Being Moderated
    Hi Juan,

    I too have a very similar problem. I exec a Java program within another running java program, and the exec'd java program does IO operations as you mentioned with stdin and stdout. And more interestingly in tight similarity, this too runs on LINUX and takes up a huge memory of around 1.2GB ram, over a period of time.

    Did u get any hints on how this happens. ?

    Thanks & Regards
    Joby
  • 2. Re: Memory leak under GNU/Linux when using exec()
    843798 Newbie
    Currently Being Moderated
    Hi Joby,

    From our last profiling results, we haven't found yet a proper solution. We think that probably the problem is caused by the byte[]'s/FileInputStreams created by the class UNIXProcess that manage the stdin, stdout and stderr streams. It seems that these byte arrays cannot be removed correctly by the garbage collector and they become bigger and bigger, so at the end they took all the memory of the system.

    We downloaded the last version of OpenJDK 6 (build b19) and modified UNIXProcess.java.linux so when we call its method destroy(), we assign to null those streams. We did that because we wanted to indicate to the garbage collector that these objects could be removed, as we saw that the close() methods doesn't do anything on their implementation.
     
    public void destroy() {
         // There is a risk that pid will be recycled, causing us to
         // kill the wrong process!  So we only terminate processes
         // that appear to still be running.  Even with this check,
         // there is an unavoidable race condition here, but the window
         // is very small, and OSes try hard to not recycle pids too
         // soon, so this is quite safe.
         synchronized (this) {
             if (!hasExited)
              destroyProcess(pid);
         }
            try {
                stdin_stream.close();
                stdout_stream.close();
                stderr_stream.close();
    
                // LINES WE ADDED
                stdin_stream = null;
                stdout_stream = null;
                stderr_stream = null;
    
            } catch (IOException e) {
                // ignore
                e.printStackTrace();
            }
        }
    But this didn't work at all. We saw that we were able to execute for a long time our application and that the free memory of the system wasn't decreasing as before, but we did some profiling with this custom JVM and the test application and we still see more or less the same behaviour: lots of surviving generations, at some point increase of the used heap to the maximum allowed, and finally the crash of the test app.

    So sadly, we still don't have a solution for that problem. You could try to compile OpenJDK 6, modify it, and try it with your program to see if the last version works for you. Compiling OpenJDK 6 in Linux is quite easy: you just have to download the source and the binaries from here and configure your environment with something like this:
    export ANT_HOME=/opt/apache-ant-1.7.1/
    export ALT_BOOTDIR=/usr/lib/jvm/java-6-sun
    export ALT_OUTPUTDIR=/tmp/openjdk
    export ALT_BINARY_PLUGS_PATH=/opt/openjdk-binary-plugs/
    export ALT_JDK_IMPORT_PATH=/usr/lib/jvm/java-6-sun
    export LD_LIBRARY_PATH=
    export CLASSPATH=
    export JAVA_HOME=
    export LANG=C
    export CC=/usr/bin/gcc-4.3
    export CXX=/usr/bin/g++-4.3
    Hope it helps Joby :)

    Cheers.
  • 3. Re: Memory leak under GNU/Linux when using exec()
    843798 Newbie
    Currently Being Moderated
    Out of curiosity I took your program and tried on a CentOS 64 box running virtually nothing besides this test program.
    [ivan@khb5412 ~]$ uname -a
    Linux khb5412.khb.hu 2.6.18-164.15.1.el5 #1 SMP Wed Mar 17 11:30:06 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux
    
    
    [ivan@khb5412 ~]$ java -version
    java version "1.6.0"
    OpenJDK  Runtime Environment (build 1.6.0-b09)
    OpenJDK 64-Bit Server VM (build 1.6.0-b09, mixed mode)
    
      PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
      404 ivan      17   0 1284m  40m 7984 S 40.2  1.0   2:23.38 java
    ... after some time
      404 ivan      17   0 1285m  43m 7996 S 38.8  1.1   7:12.42 java
    ... after some time
      404 ivan      17   0 1285m  44m 7996 S 39.5  1.1   8:29.01 java
    ... after some time
      404 ivan      17   0 1285m  45m 7996 S 39.2  1.2   9:27.42 java
  • 4. Re: Memory leak under GNU/Linux when using exec()
    843798 Newbie
    Currently Being Moderated
                    is = process.getInputStream();
                    isr = new InputStreamReader(is);
                    br = new BufferedReader(isr);
    //
                    if (is != null)
                    {
                        is.close();
                    }
                    if (isr != null)
                    {
                        isr.close();
                    }
                    if (br != null)
                    {
                        br.close();
                    }
    When layering InputStreams one above the other, you should close (first and only) the outermost one, shouldn't you?

    Unfortunately these elementary tricks do not appear to help though.
  • 5. Re: Memory leak under GNU/Linux when using exec()
    843798 Newbie
    Currently Being Moderated
    After 300,000 iterations:
    22439 ivan      17   0 1285m  60m 8008 S 39.9  1.6  35:21.39 java
  • 6. Re: Memory leak under GNU/Linux when using exec()
    843798 Newbie
    Currently Being Moderated
    Hi,

    Thanks all for your replies.

    I would like to clarify that we saw the decreasing free memory of the system when using intensively the exec() in our "big" application and an old JVM (maybe from Java 6 update 13 to below). But the same code, and with a newer JVM, didn't have the same behaviour as we saw previously (the free memory of the system was almost stable to a value after we ran the application a whole weekend).

    We have made a more simple test program but this time I have used the Netbeans Profiler and a terminal. In the terminal the program has been able to run a whole night, but when running it in the Netbeans Profiler it crashed again in less than 40 minutes.

    So now I think that there was something that caused a memory leaked in old versions of the JVM when using exec(), but now maybe the problem is that the Netbeans profiler is interfering in the program in a way it shouldn't. I will report this to the Netbeans project, maybe they could help.
    import java.io.IOException;
    
    public class testFastExec
    {
        public static void main(String args[]) throws IOException, InterruptedException
        {
    
            Runtime runtime = Runtime.getRuntime();
    
            System.out.println("testFastExec");
    
            while (true)
            {
                Process process = null;
    
                try
                {
                    process = runtime.exec("ls /");
                    
                }
                finally
                {
                    process.waitFor();
                    
                    if (process != null)
                    {
                        process.destroy();
                    }
    
                    process = null;
                    
                    runtime.gc();
                }
            }
        }
    }
    Cheers and thanks for your attention and hints! :)

    Edited by: Juan_Chak on Apr 30, 2010 2:12 AM
  • 7. Re: Memory leak under GNU/Linux when using exec()
    843798 Newbie
    Currently Being Moderated
    Did you observe any change of the behaviour due to this?
     runtime.gc();
    Newer JVMs are told to take this only as a mild hint. Put it in a raw manner: it is not supposed to count any longer.
  • 8. Re: Memory leak under GNU/Linux when using exec()
    843798 Newbie
    Currently Being Moderated
    Hi Ivan,

    Thanks again for your reply :)

    I have profiled the program without the runtime.gc() and the result is the same (we get an OutOfMemoryError), but the behaviour is different. When using gc() the surviving generations and the used heap have more or less a lineal increment. But when not using the gc(), it seems that the number of surviving generations are stable, but there is a moment where the SG get higher and higher. The used heap also has a different behaviour: there are peaks of used heap from the beginning, but at the end the increment is "quite" lineal.


    Thanks and regards.
  • 9. Re: Memory leak under GNU/Linux when using exec()
    843798 Newbie
    Currently Being Moderated
    Hi,

    Have some news. It was a bug that has been resolved for the next release of Netbeans (6.9):

    http://netbeans.org/bugzilla/show_bug.cgi?id=185350


    Cheers.