I have a very simple c++ program that dynamically loads the jvm and in a loop tries to allocate memory in chunks of 512K. The malloc fails after about 3.8 GB is consumed. I don't use the jvm in any way other than just loading it. If I specify a java max heap size of 4096 MB I can allocate up 29 GB and then the malloc fails. If I subsequently increase the max heap size to 32GB then I can allocate more than 100GB.
What is interesting is that this happens only on SunOS 11. Also it happens with java 1.6.0_26 and java 1.7 but not with java 1.6.0_22. If I don't load the jvm everything runs fine. This simple program runs without issues on SunOS 10, linux and windows with java 1.7.
I have used truss with this program and I get the following output