I have an issue where the CMS garbage collector is not executing, and I wonder if someone can clue me in on what could possibly cause such behavior. I'm running my program (using java 1.6.0_29) with the following JVM parameters:
In my gc.log file, I see the following CMS Entries:
2011-11-14T09:47:24.660-0500: 1104.878: [GC [1 CMS-initial-mark: 2758057K(3670016K)] 2796390K(4141888K), 0.0550771 secs] [Times: user=0.02 sys=0.03, real=0.06 secs]
Total time for which application threads were stopped: 0.0711610 seconds
2011-11-14T09:47:24.718-0500: 1104.936: [CMS-concurrent-mark-start]
So it looks like the CMS initiated. However, that's the last I ever heard from it. Memory kept expanding for the next 15 minutes or so until the heap was completely full, at which time the app hung on the following statement:
for many minutes until I finally just killed the JVM. So that last statement seems to indicate that the CMS wanted to do something...but at this point it was kinda too late.
If I change -XX:CMSInitiatingOccupancyFraction=75 to -XX:CMSInitiatingOccupancyFraction=30, this seems to work. However, I don't understand why.
Note that my program is running a Berkeley DB which I am giving 60% of the JVM memory to for cache. Thus, after a warmup period, 60% of the total heap will always be considered "live" for the cache alone.
It's probably due to fragmentation of the tenured space meaning that it has no more contiguous areas of memory to put objects it wants to store. However, I'm surprised it doesn't try to do a full GC to compact at this point. That's what we're experiencing at least and then the JVM crashes.
One setting we're trying is -XX:+CMSClassUnloadingEnabled and so far it seems to be helping. Next we plan to use this as well: -XX:CMSFullGCsBeforeCompaction=1.