1 2 Previous Next 15 Replies Latest reply: May 3, 2012 6:41 AM by 800381 RSS

    Maximum threads managed by java

    561636
      Dear All,

      I am facing one problem in java threads. I have a program where there is necessity of creating threads. What i understand if we use 64 bit o.s and java, there are no limits with respect to memory (apart from what is available in the system). I have 128 GB Ram and so i can say that i have large amount of ram. I have been trying to create threads with stack size 256k. The moment the number of threads hitting around 45,000, the program is getting crashed with "out of memory error, unable to create native thread". I even tried to run a simple java program (which does create a thread and do suspend) to check how many number of maximum threads can be created. Again the moment the thread count is touching around 45,000, the program is exiting with the same error.

      I even changed the limits of linux (i am using Scientific Linux release 6.2 with kernel version 2.6.32-220 64 bit) and the following ulimits, i am using
      core file size (blocks, -c) unlimited
      data seg size (kbytes, -d) unlimited
      scheduling priority (-e) 0
      file size (blocks, -f) unlimited
      pending signals (-i) 1032024
      max locked memory (kbytes, -l) 256
      max memory size (kbytes, -m) unlimited
      open files (-n) 102400
      pipe size (512 bytes, -p) 8
      POSIX message queues (bytes, -q) 819200
      real-time priority (-r) 0
      stack size (kbytes, -s) 10240
      cpu time (seconds, -t) unlimited
      max user processes (-u) 65535
      virtual memory (kbytes, -v) unlimited
      file locks (-x) unlimited

      Even the values in "/proc/sys/kernel/threads-max" is 2064049.

      Can somebody tell me why i could create only threads around 45,000. I have so much ram available like 128GB. Am i missing some O.S settings?

      Thank you all,
      Vinod Kumar
        • 1. Re: Maximum threads managed by java
          561636
          sorry, i even raised the maximum user processes limit to 100000. still the same problem happening
          • 2. Re: Maximum threads managed by java
            EJP
            45000*256KB = 10.98GB. You're doing well to get even that far; you must have a huge disk.

            A thread is not a user process, and you are hitting a memory limit, not a user process limit, so raising it will naturally have no effect.
            • 3. Re: Maximum threads managed by java
              561636
              One thing, i had observed is the pid limit. I had raised the pid limit (and even map count -> for stack guard pages) and till now the memory error is not coming.
              Ofcourse, i need to test more to see that the error has been solved.
              • 4. Re: Maximum threads managed by java
                EJP
                The stack guard pages thing would help. Raising the pid limit would not.
                • 5. Re: Maximum threads managed by java
                  561636
                  I am not very much sure...but raising the pid limit helped...
                  i tested on a small machine (with around 3GB ram)...with raising pid limit ...the number of threads increased till (compared to default pid limit) system ran out of memory (with swap = 0)...
                  That's why i am testing in my server and till now, the peak threads reached around 58,000 and my program is running fine with out memory errors.
                  • 6. Re: Maximum threads managed by java
                    800381
                    EJP wrote:
                    The stack guard pages thing would help. Raising the pid limit would not.
                    Are you sure?

                    Doesn't Linux's NPTL implement threads at the process level using clone()?
                    • 7. Re: Maximum threads managed by java
                      EJP
                      The error is 'out of memory', not 'out of pids'.
                      • 8. Re: Maximum threads managed by java
                        561636
                        Hi,

                        I had set the /proc/sys/vm/max_map_count to 131072 and yesterday night, i had got the "stack guard pages deallocation failed" error came. After that the outofmemory error has come.
                        Can you please tell me why it is not possible to create large number of threads as i have a memory of 128GB (i found that theoretically, it is possible to create N number of threads where N is defined by the amount of memory available). If this is case, then i have 128GB of memory and so atleast i should be able to create lakhs of threads (as i can see, that memory is not fully used...only around 25-30GB is being used).

                        Is there any limit from the linux O.S? Or java is not capable of handling more than N number of threads? I could not understand what is restricting me in creating atleast 1 lakh of threads when i have the memory available for it (and ofcourse i have 48 cores machine).
                        I will again try with little higher value of "max_map_count".

                        Thanks for all the help.

                        Cheers,
                        Vinod Kumar Boppanna
                        • 9. Re: Maximum threads managed by java
                          gimbal2
                          EJP wrote:
                          The error is 'out of memory', not 'out of pids'.
                          Hm, that reminds me of this blog:

                          http://devgrok.blogspot.com/2012/03/resolving-outofmemoryerror-unable-to.html

                          Apparently the JVM can falsely report out of memory.
                          • 10. Re: Maximum threads managed by java
                            561636
                            Yes, java might be reporting wrong errors of memory over flow..that's why i am trying with every o.s parameter..already i had set the user process, open files to a large numbers....around 1 lakh..
                            and now trying with increasing pid and mmap limits..

                            Because i could not find exact O.S parameter due to which the java is facing problem in extending the number of threads.
                            Thanks
                            • 11. Re: Maximum threads managed by java
                              800381
                              You can run the JVM under strace and see what's failing at the system call level. Or, run pmap against the JVM and see how memory is laid out.

                              I'm guessing you're seeing a failed mmap() call for a thread stack, and it's failing because of the way the process address space is laid out - the JVM wants to put the memory in a specific location, but there's already something else there. Thus the mmap() call fails, and the error reported is "out of memory".

                              Once you find out what exactly is failing you can come up with a fix or workaround.
                              • 12. Re: Maximum threads managed by java
                                561636
                                I have tested my program by increasing the pid limit and mmap count into lakhs. My program reached a peak threads of around 73,000 and there are no memory errors (continuously run for the last two days) .
                                I even run a sample program (which just creates threads and keep the threads in suspend mode) and created threads into lakhs with having any problem of memory errors.

                                I am not sure why raising a pid limit is required for threads. I could see that all threads use the same PID and same PPID, but ofcourse have different Light weight thread ID. may be this id is also counted under pid limit.

                                Also, just for understanding, is there any limit on how much is allowed for mmap maximum count? or it can be any value?

                                Thanks to all for the help.

                                Vinod Kumar Boppanna
                                • 13. Re: Maximum threads managed by java
                                  EJP
                                  All these questions relate to the Linux operating system. They have nothing to do with Java.
                                  • 14. Re: Maximum threads managed by java
                                    sabre150
                                    I do wonder what algorithm requires 73,000+ threads! Even with Linux I would expect the context switching overhead to be enormous!
                                    1 2 Previous Next