This discussion is archived
5 Replies Latest reply: Mar 14, 2013 9:53 AM by gimbal2 RSS

thread dumps on fatal JVM error without intervention

954317 Newbie
Currently Being Moderated
Hi All,
I tried testing out the optional commands that are allowed on the JVM options to create thread dumps on fatal JVM error. Below are 2 cases to create a thread dump using kill -3 command 1) is when an OOM error occurs and 2) when any fatal error occurs and JVM goes down.
As you see from the output below, on OOM error the kill -3 command is working from “executing kill -3 9644’ where, %p is replaced by process id 9644. The same is not true for the case for OnErrror, my assumption was the OnError is triggered for all errors including OOM error. I am going to post the same to JVM news groups and check if, I can get help from there.
This is closest I have found to trigger a thread dump on fatal JVM error without intervention.

Thanks
1)

d:\Development\examples>java -D64 -XX:OnOutOfMemoryError="kill -3 %p " -Xmx10m -Xloggc:gc.log OOM
#
# java.lang.OutOfMemoryError: Java heap space
# -XX:OnOutOfMemoryError="kill -3 %p "
# Executing "kill -3 9644 "...
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at OOM.main(OOM.java:6)

2)
d:\Development\examples>java -D64 -XX:OnError="kill -3 %p " -Xmx10m -Xloggc:gc.log OOM
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at OOM.main(OOM.java:6)
  • 1. Re: thread dumps on fatal JVM error without intervention
    gimbal2 Guru
    Currently Being Moderated
    OOM is not necessarily fatal; it won't cause the JVM to terminate ungracefully. It will be thrown, the application may deal with it and as a result the application may stop. That means the application borked, but the JVM didn't.

    OOM may even be expected behavior. For example when you try to load a huge data set (for example a gigantic image), you may run out of heap space while doing so. An application can choose to react on that and continue as if it didn't happen.
  • 2. Re: thread dumps on fatal JVM error without intervention
    954317 Newbie
    Currently Being Moderated
    Thanks for quick reply. Here is what I am trying to accomplish. One of our JVM is shutting down unexpectedly without much logs. Below is all we see in the logs. We are trying to find out the root cause on this. Towards that we wanted to created thread dump when JVM goes down unexpectedly. On looking at the JVM options reference http://www.oracle.com/technetwork/java/javase/clopts-139448.html; thought -XX:OnError="kill -3 %p" will do the work for us and tested with OOM error; seems that is not the case (you explained that OOM is not necessarily fatal). Looking at the below logs using -XX:OnError="kill -3 %p" for unexpected JVM shutdown is the right direction?

    Thanks,



    14:23:17,860 INFO [STDOUT] Posting Shutdown Request to the server...
    14:23:17,862 INFO [org.jboss.bootstrap.impl.base.server.AbstractServer] Stopping: JBossAS [6.1.0.Final "Neo"]
  • 3. Re: thread dumps on fatal JVM error without intervention
    gimbal2 Guru
    Currently Being Moderated
    Yeah you see, that's both the application and the JVM shutting down gracefully - in this case JBoss itself is shutting down due to some event. To figure out WHY it is doing that you'll have to ask in the JBoss forums on the JBoss website, its not a global Java problem.
  • 4. Re: thread dumps on fatal JVM error without intervention
    jschellSomeoneStoleMyAlias Expert
    Currently Being Moderated
    gimbal2 wrote:
    OOM is not necessarily fatal; it won't cause the JVM to terminate ungracefully. It will be thrown, the application may deal with it and as a result the application may stop. That means the application borked, but the JVM didn't.

    OOM may even be expected behavior. For example when you try to load a huge data set (for example a gigantic image), you may run out of heap space while doing so. An application can choose to react on that and continue as if it didn't happen.
    Unfortunately that isn't true for many modern applications.

    There is a scenario where it works where one allocates a very large array of primitives. In such a case recovering from a OOM should be possible.

    But consider the case where one is attempting to build a very large tree (many small allocations) in Thread 1 and the applications is threaded. So Thread 1 allocates the one millionth object in the tree and then Thread 2 attempts an allocation of one small object (nothing to do with the tree) and it fails. (To recover one would need to retry every single allocation in every single thread and that is unlikely to be realistic.)

    And the second case is a more likely scenario than the first.
  • 5. Re: thread dumps on fatal JVM error without intervention
    gimbal2 Guru
    Currently Being Moderated
    jschell wrote:
    gimbal2 wrote:
    OOM is not necessarily fatal; it won't cause the JVM to terminate ungracefully. It will be thrown, the application may deal with it and as a result the application may stop. That means the application borked, but the JVM didn't.

    OOM may even be expected behavior. For example when you try to load a huge data set (for example a gigantic image), you may run out of heap space while doing so. An application can choose to react on that and continue as if it didn't happen.
    Unfortunately that isn't true for many modern applications.

    There is a scenario where it works where one allocates a very large array of primitives. In such a case recovering from a OOM should be possible.

    But consider the case where one is attempting to build a very large tree (many small allocations) in Thread 1 and the applications is threaded. So Thread 1 allocates the one millionth object in the tree and then Thread 2 attempts an allocation of one small object (nothing to do with the tree) and it fails. (To recover one would need to retry every single allocation in every single thread and that is unlikely to be realistic.)

    And the second case is a more likely scenario than the first.
    Right you are, in concurrent environments the results will be very erratic, depending on the timing of the threads that will do stuff at the same time. That may cause a server specific thread to go bang and henceforth the server may be designed to shutdown to prevent damage. But that is still server application logic doing its thing, not the JVM shutting down because of a fatal error.

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points