We are running a service which used to frequently crash with JDK1.5.11. Once we upgraded to 1.5.17 the JDK crash stopped.
But we are facing a different issue.
The server runs normally for few hours(may be 10,12,15 hrs) and after that it starts to slow down. After struggling for half an hour, it becomes normal again. On analyzing we found that there are very frequent CMS collection when the server was non responsive. The frequency is such that there are no intermittent Young Generation collection.
The issue persisted even with jdk1.5.11. But most of the time the server crashed.
During normal behaviour the server CMS collection takes place approximately every 20-30 minutes collecting large chunk of memory. Our -Xmx is 4GB and -XMaxNewSize is 512MB and the CSMInitiatingOccupancy is 65%. So it used to reach 2.9GB of memory and then drop to 2.4GB or 2GB or even 1.8GB during CMS collection.
But daily for atleast half an hour(May be related to our traffic) the CMS runs very frequently (every 1 minute) without regaining much of memory. Say the memory drops from 2.9 to 2.85 or 2.8MB. The CMS tries to regain memory by running frequently(CMS runs as low as once per minute) without much success. Then after approximately half an hour it drops to 1.6-1.8GB. After this big drop the server begins to operate normally.
On analyzing the application we found that we are using a Cache which is a HashMap of SoftReferences. We believe the CMS struggles to clear these SoftReferences. The server is mostly stateless. In the sense we do not maintain anything specific for a user except for the SoftReference Cache. The other parameter we had configured is -XX:SoftRefLRUPolicyMSPerMB=60.
Yesterday when the CMS was struggling to regain memory we cleared the SoftReferences after approximately 15 minutes. Once we did that the CMS collected a huge chunk of memory and the used memory came to 1.5GB from 2.9GB.
We suspect its an issue with SoftReference. In this regard could you please shed some light on the behaviour of SoftReference. We have a Single ConcurrentHashMap in which we maintain this SoftReferences. Any pointers will be very helpful in proceeding further.