According to CMSInitiatingOccupancyFraction=60 , GC should work when reaching 60% of the memory. But that's not happening instead memory consumption goes above 4GB then GC works and memory comes down to 500MB ,during this our GUI screen hangs for few seconds.
I'm not as familiar with tuning the hotspot VM as i would like to be so the following may well be things you have already tried. If i had an extract from the GC logs that would probably help me visualize what is happening under the covers.
With CMSInitiatingOccupancyFraction=60, a Major GC will kick in when more than 60% of the tenured generation size is in use. Based on your description that appears to be the case here, if you set this to the default (92% iirc) does this help any with the pause or is it basically the same?
You may want to take a look at http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html (the "Starting a Concurrent Collection Cycle" section).
This could be one of those cases where the heap is just too big (probably not if you have 8 cores working through it) or it could be that you have in-flight threads that need to be brought to a safe point prior to processing and your CPU is pinned (what was CPU looking like before/during/after the GC cycle?) see:- http://mail.openjdk.java.net/pipermail/hotspot-gc-use/2011-August/000915.html
It might be worth investigating the use of UseCMSInitiatingOccupancyOnly (consider not using it see if that helps) http://docs.oracle.com/cd/E19543-01/820-2962/Performance_Tuning.html#wp1265817
Out of pure interest (for background) what is it you are doing in your GUI that is so memory intensive?