This discussion is archived
6 Replies Latest reply: Sep 14, 2011 3:40 AM by 802889 RSS

Weird 100% CPU usage behavior how can i monitor?

887650 Newbie
Currently Being Moderated
Hi, I have a server TCP/IP program written in JAVA and it was running for months on production with no issues, then it spiked up to 100% CPU usage even on low volume and it continued on 100% even after we stopped sending messages to it. We didn't turn it off we just stopped using it and diverted all network traffic to a back up server. It's as if we let go of the pedal but it got stuck at 100% throttle.

The application has been restarted and it seem to be working ok.

The question is now if it where to happen again, what could I do to check what caused the 100% CPU usage?
  • 1. Re: Weird 100% CPU usage behavior how can i monitor?
    jtahlborn Expert
    Currently Being Moderated
    the easiest thing to do first is get a stack dump from the process. (maybe a few over a short period of time). often the problem will jump out right there. next step is to hook jvisualvm to the process and look at the various monitors it provides (it has a "simple" sampler plugin). if still stuck, then you need to profile the application using a "full" profiler (i personally prefer yjp, but there are plenty of decent free options out there as well).
  • 2. Re: Weird 100% CPU usage behavior how can i monitor?
    gimbal2 Guru
    Currently Being Moderated
    884647 wrote:
    Hi, I have a server TCP/IP program written in JAVA and it was running for months on production with no issues, then it spiked up to 100% CPU usage even on low volume and it continued on 100% even after we stopped sending messages to it. We didn't turn it off we just stopped using it and diverted all network traffic to a back up server. It's as if we let go of the pedal but it got stuck at 100% throttle.
    What kind of server OS are we talking about here? A Windows variant by any chance?
  • 3. Re: Weird 100% CPU usage behavior how can i monitor?
    887650 Newbie
    Currently Being Moderated
    @jtahlborn

    1- How would I get the stack dump on a running application?
    2- And are you saying i can hook jvisualvm to an already running process without the need to have had started a profiling agent? i just wont get full class stats right?

    @gimbal2
    Yeah Windows 2003
    JDK 6 update 21 and yes it is running off of the java.exe in the JDK folder
    Also using Tanuki Wrapper to run as service.
  • 4. Re: Weird 100% CPU usage behavior how can i monitor?
    gimbal2 Guru
    Currently Being Moderated
    884647 wrote:
    @gimbal2
    Yeah Windows 2003
    JDK 6 update 21 and yes it is running off of the java.exe in the JDK folder
    Also using Tanuki Wrapper to run as service.
    Hm. I know that in Windows environments there is an issue with the way the current tickcount is kept, which after ~48 days of runtime will overflow and return to 0. At this point the 'next tick' is actually before the 'previous tick' in code that uses this, which can be things like thread sleep code and what not that will start to misbehave.

    I'm not saying this is the problem you are facing, it might be something completely different. Its just the first thing that popped into my head when I saw your problem description. My main point is that you may not be able to fix the issue at all because you can't even be sure that the problem is in either the JVM or your own application.

    But first things first, connect a profiler to the application when it starts to misbehave again because perhaps it is something that can be fixed; its always the silly things you don't think about when the problem seems 'weird'. But it may take weeks of runtime again!
  • 5. Re: Weird 100% CPU usage behavior how can i monitor?
    887650 Newbie
    Currently Being Moderated
    lol it happened after months. Also using netty for the server code.

    And here is another funny issue if running with the Tanuki Wrapper I cannot hook jvisualvm to the application... I think it had to do with something where the temp folder of the running application vs where jvisualvm looks.
  • 6. Re: Weird 100% CPU usage behavior how can i monitor?
    802889 Explorer
    Currently Being Moderated
    884647 wrote:
    @gimbal2
    Yeah Windows 2003
    JDK 6 update 21 and yes it is running off of the java.exe in the JDK folder
    Could you try the lastest Java 6 JVM? A few months I reproduced a bug reported for Jaybird, where terminating the Firebird server caused 100% CPU-usage on the client when the client was listening for (callback) events from the server. This was reproducable with Java 6 update 18, but recently I was unable to reproduce that same issue with Java 6 update 26 or 27.

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points