We have 62 applications (EARs) deployed. 21 of these application contains a single stateless session bean and somewhere between 3 and 40 entity beans each. In total we currently have 466 entity beans. The rest of the applications contain only a single stateful session bean each.
When all these applications are enabled we can run for a few minutes before getting an OutOfMemoryError. When we disable 20 of the latter kind, the ones with stateful session beans, the server seems to work OK.
We have not yet, for obvious reasons, rolled the system out to the users so only a few developers are using it. The clients are fat Swing clients that talk IIOP with the server.
In one of the beans we have a method that does three consecutive System.gc() and then measures the heap usage. With all applications enabled only around 30% of the maximum heap space is used, even right before the server crashes. The heap max is set to 512MB and since the usage rate is so low it doesn't seem meaningful to increase it. Also, if there is a leak then just increasing the heap may give us a longer uptime without really solving the problem. We suspect that there is some other limit somewhere, but since the error message in the log doesn't have a stack trace it is very difficult to narrow it down.
Has anyone had a similar problem and maybe found a solution?
what is the version of the application server and have u tested the apps individually by some profiling tools which shall tell u if at all the memory is being leaked by some applications. Jprobe or introscope could be helpful
We are using app server 7 2004Q2. As I wrote, we do monitor memory usage. Around 30% of the maximum heap space is used, right before the server crashes. That is with a heap max of 512MB. When I increased the maximum size to 1GB the behavior was the same, except of course that the reported heap usage was a much smaller fraction of the maximum.
Total heap size is not the only thing to look after. You can get an OutOfMemoryError when other memory space run out.
Try this: http://java.sun.com/docs/hotspot/PerformanceFAQ.html#171
This may also be a platform issue (things like heap per threads).
See the "Other Considerations" paragraph on permanent size also : http://java.sun.com/docs/hotspot/gc1.4.2/
Thanks for your tips. I read both links but couldn't find anything that we haven't already tried. Out heap is set to 1GB both initially and maximum so I don't think there is a heap resize problem. The machine has 3GB of physical memory and a 6GB swap file.
The following is a cutout from the log:
[04/Jun/2004:10:05:50] INFO ( 792): CORE3282: stdout: Memory Monitor: total=1012 MB, free=832 MB, used=180 MB.
[04/Jun/2004:10:06:54] INFO ( 792): CORE3282: stdout: Memory Monitor: total=1012 MB, free=832 MB, used=180 MB.
[04/Jun/2004:10:07:38] WARNING ( 792): CORE3283: stderr: java.lang.OutOfMemoryError
[04/Jun/2004:10:07:44] WARNING ( 792): CORE3283: stderr: java.lang.OutOfMemoryError
[04/Jun/2004:10:07:55] INFO ( 792): CORE3282: stdout: Memory Monitor: total=1012 MB, free=832 MB, used=180 MB.
[04/Jun/2004:10:08:57] INFO ( 792): CORE3282: stdout: Memory Monitor: total=1012 MB, free=832 MB, used=180 MB.
As you can see, there is no "requested <size> bytes" suffix on the OutOfMemoryError message, which would have indicated the heap resize bug. The memory monitor messages is from a thread that checks the heap values every second. Right after an OutOfMemoryError there is a message from this thread that says that there are over 800MB of free memory.
As for the other link, we have already tried disabling explicit GC with no success. We have also tried aggressive heap.
I suspect that the memory that the server is out of is not heap memory. I have tried increasing the stack size, lowering pool size and just about everything I can think of except memory enhancement drugs.
Yes, I understand, it is not a -Xmx setting issue.
What I suggested is that each thread has too much memory and that it was maybe a good idea to lower the stack size for each of them.
Other than that, look at the "Other considerations when scaling to a large number of threads" paragraph here: http://java.sun.com/docs/hotspot/threads/threads.html and give -XX:-UseTLAB a try.
You can also try alternate GC algorithms:
- �Throughput� (-XX:+UseParallelGC)
- �Concurrent� (-XX:+UseConcMarkGC)
Also, please provide the following:
- Hardware details
- OS version
- JVM version
- all JVM options used
I'm glad this helped.
Sun Support is obviously a very good place to ask questions as they have all the knowledge management tools to help customer.
You may want to try the jvmstat tools (http://developers.sun.com/dev/coolstuff/jvmstat/index.html) to monitor your running JVM and see the occupation of the various VM memory spaces.
Thanks for your tip. I have successfully run visualgc on AppServer 8 PE but I can't get it to connect to AppServer 7 Standard. I use the -XX:+UsePerfData option but jvmps doesn't find any JVM. When I start visualgc with the PID of appservd I get the following error:
Could not attach to 2260: Could not map vmid to user name
Running visualgc on itself (0) works fine. I have also tried starting the perfagent with the policy file based on the "All" template. The first three attempts failed: No listener on 1099 reported by netstat and after half a minute I got a "Fatal VM error" something popup. The forth time the listener started so I tried visualgc 2260@localhost and got this message:
Could not attach to 2260@localhost: Could not map vmid to user name
The OS is Win 2000. I downloaded jvmstat-2.0_b11.zip. When running visualgc on itself it says version 1.2 which is a bit strange.
Is it possible to use jvmstat with AppServer 7 SE. Is the fact that it is running as a service a problem?
Problem solved with the help of jvmstat-support! Since the Win2K machine has the FAT32 filesystem no memory mapped file was created. Tje JVM won't do that if it can't limit the access to it. The workaround is to use the option -XX:+PerfBypassFileSystemCheck on the app server. I also had to use srvany from Win2K toolikt to run the perfagent tool as a service. Now I can watch the innards of the appserver JVM both locally and remotely, and keep an eye on the critical Perm area!
I am using jvmstat to profile my application running in jboss server .This application is running as Windows Service in 2000 Server .Initially all services where running under SYSTEM account and i used to login to server with my account .So when i tried to invoke visualgc by providing the PID i got message "could not map vmid to username".So irequested to create a new account and i restarted all services with new name and i logged to server using same name .But still same problem .I am getting the same old messgae "could not map vmid to username"
I think it because of running java application as service .File system on windows machine is NTFS .Appreciate any help.
Edited by: Rajesh06 on May 14, 2008 11:36 AM