Environment – Java 1.5.0_11-b03 (32 bit vm), Solaris 10, M5000 server (64 cpu/128GB)
Application – custom webserver (more or less), uses java threads, uses shared memory through JNI interface.
Memory sizing: –Xms512M –Xmx1024M, usual running heap max at ~300M , ~164 threads
After running for a few days we are encountering OutOfMemoryErrors when attempting to allocate new threads:
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Unknown Source)
It is my understanding that this is a stack space issue as there is plenty of available heap.
We are allocating shared memory (fixed size (16k) sliding windows over ~400 files)
After the thread start failures we then start seeing shared memory allocation issues
OS_ERR = (12): Not enough space
(there is plenty of space on the file system, and I expect within the ~4GB process space)
We can make the issue occur quicker by allocating more shared memory. (still well under 4GB)
The current environment should be < 1GB shared memory.
The overall starting memory consumption is < 2GB
We are planning on reducing the default stack size but have yet to run a test with that since these errors are currently only occurring in production environments.
Could this be a fragmentation issue?
Is there any documentation that would help me understand where shared mem gets allocated and where native stacks live? (since they seem to be competing for space)
Any other hints or tools we could use to help diagnose this problem?