I've a problem debugging an application.
Background: Sometimes my application comes to a very unlikely state, which at the moment results in an error message. The stack trace alone has no great value, since this state is cause by the interaction of more than one thread. The state is resolved throwing an exception, the program continues normally.
Goal: If I reach this state, I want to suspend the application, dump the complete state of all java threads, objects, ... (complete java memory core dump) to analyse it later.
Question: Is there a possibility to generate such dumps?
One posibility would be, to send a signal from the vm to some external program (e.g. udp packet), this program attaches a standard OS-level debugger (e.g. gdb under linux), dumps the core, resumes app, detach. But I guess reconstruction of internal VM state from complete dump is rather hard, is it?
I've looked at jdb, to see if it has some java-core-dump functions, but it seems not to be so. Are there alternative implementations, or helper scripts that make jdb loop over all java object addresses and dump them?
Does someone known about the jdb remote debugging interface? Is the protocol public, can I implement such a feature on myself?
Thanks to Jim Holmlund from Sun I got it working the way I wanted to have it. It was all there in the linux jdk1.5 installation, he pointed out to me how to put the things together (sorry, works only for linux):
$ cat << END > gdb.commands
$ gdb --pid 26003 -x gdb.commands
$ jsadebugd ..../jdk1.5.0_06/bin/java tomcat.core DebugServer &
$ jdb -connect
You can inspect all the data (threads, stack frames, local variables,
objects) just as if you would have attached to a live VM, only
modifications do not work (step, continue, set...) because a VM core
dump is dead.
This is a really strange way to debug a java application, I never
thought that it could work that way.
I was looking for a way to dump the complete state of a JVM (all runing threads, objects, ...) and afterwards to analize the dumped data. I have found a thread(http://forum.java.sun.com/thread.jspa?threadID=735052) here in the forums with some explainations how this could be done using the Java 2 sdk 1.5.0 - troubleshooting tools (http://java.sun.com/j2se/1.5.0/docs/tooldocs/experimentaltools.html).
I've made a small application, which starts a thread and just prints out continously some messages to the console.
After that I attached with gdb to the running process as described, and generated a core file.
Unfortunatelly on calling jdsadebugd with the core dump it throws the following exception:
jsadebugd /..../jdk1.5.0_07/bin/java core.18294 DebServ
Attaching to core core.18294 from executable /.../jdk1.5.0_07/bin/java and starting RMI services, please wait...
Error attaching to core file or starting server: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the core file
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach0(Native Method)
Even if I try to print the shared objects mapping with jmap it fails with the following message:
jmap /.../jdk1.5.0_07/bin/java core.18294
Attaching to core core.18294 from executable /.../jdk1.5.0_07/bin/java, please wait...
Error attaching to core file: Can't attach to the core file
Does anyone have an idea what goes wrong here?
I am trying to do the same thing, only in Windows running Tomcat.
I am running 1.5_07, and most of the good tools, are not available for Windows until 1.6 (jmap, ect).
What can I use to get a full thread dump, memory dump, thread states?
May be a very dumb question -
Given a java1.5 core generated by gcore on solaris2.9, is it possible to convert it
to a hprof binary recognized by java6 jhat? Tried jmap -dump from java6, no luck. hprof agent in java1.5 too heavy for serious production heap dump - 5g heap in our case.
Any better way to handle this?
Thanks in advance.