I am facing one problem in java threads. I have a program where there is necessity of creating threads. What i understand if we use 64 bit o.s and java, there are no limits with respect to memory (apart from what is available in the system). I have 128 GB Ram and so i can say that i have large amount of ram. I have been trying to create threads with stack size 256k. The moment the number of threads hitting around 45,000, the program is getting crashed with "out of memory error, unable to create native thread". I even tried to run a simple java program (which does create a thread and do suspend) to check how many number of maximum threads can be created. Again the moment the thread count is touching around 45,000, the program is exiting with the same error.
I even changed the limits of linux (i am using Scientific Linux release 6.2 with kernel version 2.6.32-220 64 bit) and the following ulimits, i am using
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 1032024
max locked memory (kbytes, -l) 256
max memory size (kbytes, -m) unlimited
open files (-n) 102400
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 65535
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Even the values in "/proc/sys/kernel/threads-max" is 2064049.
Can somebody tell me why i could create only threads around 45,000. I have so much ram available like 128GB. Am i missing some O.S settings?
Thank you all,
45000*256KB = 10.98GB. You're doing well to get even that far; you must have a huge disk.
A thread is not a user process, and you are hitting a memory limit, not a user process limit, so raising it will naturally have no effect.
One thing, i had observed is the pid limit. I had raised the pid limit (and even map count -> for stack guard pages) and till now the memory error is not coming.
Ofcourse, i need to test more to see that the error has been solved.
I am not very much sure...but raising the pid limit helped...
i tested on a small machine (with around 3GB ram)...with raising pid limit ...the number of threads increased till (compared to default pid limit) system ran out of memory (with swap = 0)...
That's why i am testing in my server and till now, the peak threads reached around 58,000 and my program is running fine with out memory errors.
I had set the /proc/sys/vm/max_map_count to 131072 and yesterday night, i had got the "stack guard pages deallocation failed" error came. After that the outofmemory error has come.
Can you please tell me why it is not possible to create large number of threads as i have a memory of 128GB (i found that theoretically, it is possible to create N number of threads where N is defined by the amount of memory available). If this is case, then i have 128GB of memory and so atleast i should be able to create lakhs of threads (as i can see, that memory is not fully used...only around 25-30GB is being used).
Is there any limit from the linux O.S? Or java is not capable of handling more than N number of threads? I could not understand what is restricting me in creating atleast 1 lakh of threads when i have the memory available for it (and ofcourse i have 48 cores machine).
I will again try with little higher value of "max_map_count".
Thanks for all the help.
Vinod Kumar Boppanna
Yes, java might be reporting wrong errors of memory over flow..that's why i am trying with every o.s parameter..already i had set the user process, open files to a large numbers....around 1 lakh..
and now trying with increasing pid and mmap limits..
Because i could not find exact O.S parameter due to which the java is facing problem in extending the number of threads.
You can run the JVM under strace and see what's failing at the system call level. Or, run pmap against the JVM and see how memory is laid out.
I'm guessing you're seeing a failed mmap() call for a thread stack, and it's failing because of the way the process address space is laid out - the JVM wants to put the memory in a specific location, but there's already something else there. Thus the mmap() call fails, and the error reported is "out of memory".
Once you find out what exactly is failing you can come up with a fix or workaround.
I have tested my program by increasing the pid limit and mmap count into lakhs. My program reached a peak threads of around 73,000 and there are no memory errors (continuously run for the last two days) .
I even run a sample program (which just creates threads and keep the threads in suspend mode) and created threads into lakhs with having any problem of memory errors.
I am not sure why raising a pid limit is required for threads. I could see that all threads use the same PID and same PPID, but ofcourse have different Light weight thread ID. may be this id is also counted under pid limit.
Also, just for understanding, is there any limit on how much is allowed for mmap maximum count? or it can be any value?
Thanks to all for the help.
Vinod Kumar Boppanna