8 Replies Latest reply: Jul 20, 2012 2:37 PM by 810618 RSS

    JDK 1.6.025 file descriptor leak

    919462
      Hi,

      We are using java.net.Socket to open a socket (snmp based communication) and we close the socket properly. But we are seeing "Too many open files" issue and this is happening in 1.6.0.25.

      We dont see any problem (with the same code base) in JDK1.6.0_U3. The application is running without any issues..Even with the latest Java (1.7U4) we are seeing this problem.

      I am seeing lot of file descriptor files opened under the /proc/<pid>/fd directory. But i couldnt read the file and all of them are UDP sockets.

      And also our CPU usage goes upto 100% and we are not sure how to debug this problem. Clearly this pbm is not there in 1.6.0.U3, only with the later versions.. Can anyone facing this issue or help me to debug this issue? for me it looks like a java pbm...

      Sam.
        • 1. Re: JDK 1.6.025 file descriptor leak
          919462
          As a additional info, the code works properly in JDK1.6U3 and the problem starts from JDK1.6U4..
          • 2. Re: JDK 1.6.025 file descriptor leak
            gimbal2
            You say "We are properly closing the socket" - I would make triple sure that you're doing that everywhere. I would also make triple sure that the exception is actually coming from socket handles and not something else like a file handle.

            If you are sure then the only way forward is to create a small example program and see if you can reproduce it. If you can then you have the base for a bug report you should be making. If you can you should also try running on a different environment (for example if you're running on Windows, try running on a Linux JDK) to see if that makes a difference.

            Just to be sure: this is not Oracle tech support. You're telling other Java programmers just like yourself.
            • 3. Re: JDK 1.6.025 file descriptor leak
              BIJ001
              "Too many open files"
              What OS is it?

              Anyway, your OS might help checking the resources held by the Java process.

              On Unix sockets, pipes, open files all use (and therefore compete for) file descriptors.

              oracle@izsak:/tmp> ls -l /proc/24544/fd
              total 48
              lrwx------ 1 oracle oinstall 64 Jul 17 15:49 0 -> /dev/pts/2
              l-wx------ 1 oracle oinstall 64 Jul 17 15:49 1 -> /tmp/ivan_R3.log
              l-wx------ 1 oracle oinstall 64 Jul 17 15:49 10 -> /home/oracle/oc4j_R3/oc4j.ivan/j2ee/home/log/oc4j/log.xml
              l-wx------ 1 oracle oinstall 64 Jul 17 15:49 11 -> /home/oracle/oc4j_R3/oc4j.ivan/j2ee/home/log/wsmgmt/auditing/log.xml
              l-wx------ 1 oracle oinstall 64 Jul 17 15:49 12 -> /home/oracle/oc4j_R3/oc4j.ivan/j2ee/home/log/wsmgmt/logging/log.xml
              l-wx------ 1 oracle oinstall 64 Jul 17 15:49 13 -> /home/oracle/oc4j_R3/oc4j.ivan/j2ee/home/log/system-application.log
              lrwx------ 1 oracle oinstall 64 Jul 17 15:49 14 -> socket:[22197049]
              l-wx------ 1 oracle oinstall 64 Jul 17 15:49 15 -> /home/oracle/oc4j_R3/oc4j.ivan/j2ee/home/log/global-application.log
              l-wx------ 1 oracle oinstall 64 Jul 17 15:49 16 -> /home/oracle/oc4j_R3/oc4j.ivan/j2ee/home/application-deployments/javasso/application.log
              l-wx------ 1 oracle oinstall 64 Jul 17 15:49 17 -> /home/oracle/oc4j_R3/oc4j.ivan/j2ee/home/log/ascontrol-application.log
              l-wx------ 1 oracle oinstall 64 Jul 17 15:49 18 -> /home/oracle/oc4j_R3/oc4j.ivan/j2ee/home/application-deployments/TokenStatic-dev/application.log
              l-wx------ 1 oracle oinstall 64 Jul 17 15:49 19 -> /home/oracle/oc4j_R3/oc4j.ivan/j2ee/home/application-deployments/client-token-dev-user/application.log
              l-wx------ 1 oracle oinstall 64 Jul 17 15:07 2 -> /tmp/ivan_R3.log
              l-wx------ 1 oracle oinstall 64 Jul 17 15:49 20 -> /home/oracle/oc4j_R3/oc4j.ivan/j2ee/home/application-deployments/client-token-dev-lra/application.log
              l-wx------ 1 oracle oinstall 64 Jul 17 15:49 21 -> /home/oracle/oc4j_R3/oc4j.ivan/j2ee/home/application-deployments/pgadmin-dev/application.log
              l-wx------ 1 oracle oinstall 64 Jul 17 15:49 22 -> /home/oracle/oc4j_R3/oc4j.ivan/j2ee/home/application-deployments/szep3-dev/application.log
              l-wx------ 1 oracle oinstall 64 Jul 17 15:49 23 -> /home/oracle/oc4j_R3/oc4j.ivan/j2ee/home/application-deployments/szep1-dev/application.log
              l-wx------ 1 oracle oinstall 64 Jul 17 15:49 24 -> /home/oracle/oc4j_R3/oc4j.ivan/j2ee/home/application-deployments/smea-dev/application.log
              l-wx------ 1 oracle oinstall 64 Jul 17 15:49 25 -> /home/oracle/oc4j_R3/oc4j.ivan/j2ee/home/application-deployments/psm-dev/application.log
              l-wx------ 1 oracle oinstall 64 Jul 17 15:49 26 -> /home/oracle/oc4j_R3/oc4j.ivan/j2ee/home/application-deployments/szep4-dev/application.log
              l-wx------ 1 oracle oinstall 64 Jul 17 15:49 27 -> /home/oracle/oc4j_R3/oc4j.ivan/j2ee/home/application-deployments/PaymentGateway-dev/application.log
              l-wx------ 1 oracle oinstall 64 Jul 17 15:49 28 -> /home/oracle/oc4j_R3/oc4j.ivan/j2ee/home/application-deployments/szep2-dev/application.log
              l-wx------ 1 oracle oinstall 64 Jul 17 15:49 29 -> /home/oracle/oc4j_R3/oc4j.ivan/j2ee/home/application-deployments/szep5-dev/application.log
              l-wx------ 1 oracle oinstall 64 Jul 17 15:49 3 -> /home/oracle/oc4j_R3/nohup.out
              lrwx------ 1 oracle oinstall 64 Jul 17 15:49 30 -> socket:[22210040]
              lr-x------ 1 oracle oinstall 64 Jul 17 15:49 31 -> /dev/random
              l-wx------ 1 oracle oinstall 64 Jul 17 15:49 32 -> /home/oracle/oc4j_R3/oc4j.ivan/j2ee/home/log/jms.log
              lrwx------ 1 oracle oinstall 64 Jul 17 15:49 33 -> /home/oracle/oc4j_R3/oc4j.ivan/j2ee/home/persistence/jms.state
              lrwx------ 1 oracle oinstall 64 Jul 17 15:49 34 -> socket:[22197058]
              lrwx------ 1 oracle oinstall 64 Jul 17 15:49 35 -> socket:[22197056]
              lrwx------ 1 oracle oinstall 64 Jul 17 15:49 36 -> /home/oracle/oc4j_R3/oc4j.ivan/j2ee/home/persistence/Oc4jJmsExceptionQueue
              l-wx------ 1 oracle oinstall 64 Jul 17 15:49 37 -> /home/oracle/oc4j_R3/oc4j.ivan/j2ee/home/log/rmi.log
              lrwx------ 1 oracle oinstall 64 Jul 17 15:49 38 -> socket:[22197107]
              lrwx------ 1 oracle oinstall 64 Jul 17 15:49 39 -> socket:[22197109]
              lr-x------ 1 oracle oinstall 64 Jul 17 15:49 4 -> pipe:[22197042]
              lr-x------ 1 oracle oinstall 64 Jul 17 15:49 40 -> pipe:[22197473]
              l-wx------ 1 oracle oinstall 64 Jul 17 15:49 41 -> pipe:[22197473]
              lrwx------ 1 oracle oinstall 64 Jul 17 15:49 42 -> socket:[22197474]
              l-wx------ 1 oracle oinstall 64 Jul 17 15:49 43 -> /home/oracle/oc4j_R3/oc4j.ivan/j2ee/home/log/ascontrol.log
              l-wx------ 1 oracle oinstall 64 Jul 17 15:49 44 -> /home/oracle/oc4j_R3/oc4j.ivan/j2ee/home/szep0.log
              l-wx------ 1 oracle oinstall 64 Jul 17 15:49 45 -> /home/oracle/oc4j_R3/oc4j.ivan/j2ee/home/szep2.log
              l-wx------ 1 oracle oinstall 64 Jul 17 15:49 49 -> /home/oracle/oc4j_R3/oc4j.ivan/j2ee/home/log/global-application.log
              l-wx------ 1 oracle oinstall 64 Jul 17 15:49 5 -> pipe:[22197042]
              l-wx------ 1 oracle oinstall 64 Jul 17 15:49 50 -> /home/oracle/oc4j_R3/oc4j.ivan/j2ee/home/application-deployments/ebank2-dev/application.log
              lr-x------ 1 oracle oinstall 64 Jul 17 15:49 6 -> /dev/random
              lr-x------ 1 oracle oinstall 64 Jul 17 15:49 7 -> /dev/urandom
              l-wx------ 1 oracle oinstall 64 Jul 17 15:49 8 -> /home/oracle/oc4j_R3/oc4j.ivan/j2ee/home/application-deployments/dev-sme/application.log
              l-wx------ 1 oracle oinstall 64 Jul 17 15:49 9 -> /home/oracle/oc4j_R3/oc4j.ivan/j2ee/home/log/server.log
              • 4. Re: JDK 1.6.025 file descriptor leak
                919462
                Yes, i did. Why i am saying this because the same code is working properly with JDK.16U3 and not with JDK1.6U4.
                • 5. Re: JDK 1.6.025 file descriptor leak
                  919462
                  Its Redhat. And as you said its purely UDP sockets opened by our application and though we are calling socket.close() , the reference is not completely cleaned up. And this pbm occurs only in JDK1.6U4 and not in JDK1.6U3.

                  I couldnt find anything in the JDK1.6U4 changelist, but i suspect some enhancements related to the GC ..
                  • 6. Re: JDK 1.6.025 file descriptor leak
                    810618
                    Are you absolutely sure you are executing the close() statement? Is your close() statement in a finally block?
                    • 7. Re: JDK 1.6.025 file descriptor leak
                      919462
                      Hi Kevin,

                      I am very sure. As i mentioned already, i have two different java installation ..When i run my product with JDK1.6U4 i am getting this problem , and not with JDK1.6U3. No code change, no environment change (apart from JAVA)

                      For me, when i switch to JDK1.6U4, getting this "Too many open files" , CPU goes to 98 %..
                      • 8. Re: JDK 1.6.025 file descriptor leak
                        810618
                        I guess I should've explained the thinking behind my question. If, for whatever reason, an unchecked Exception or Throwable is being thrown in the later JVMs that wasn't in the previous ones, and if you don't catch it, your Thread will just 'go away' possibly before the close() is executed. This happened to me on a server that was supposed to be monitoring a smoke alarm system. It was discovered after months of running that it stopped monitoring. I tracked it down to an uncaught NullPointerException causing the Thread to just go away. So now, after my known 'catches', I add a catch Throwable.

                        I know this is a long shot but if you've eliminated everything else, you might want to check to make sure there is no way this could be happening to you. It just seems to me that if this were a bug in the JVM, it would've been noticed long ago by others.

                        Kev