6 Replies Latest reply: May 20, 2010 5:01 PM by 807557 RSS

    Jitter

    807557
      I'm evaluating real time java for our application.
      Installed it accordingly to instructions, applied Quick Start Guide recommendations. Made working thread to be RealtimeThread
      Still measurements show significant jitter for our operations:
      latency (nanoseconds):
      max : 5374368.0
      mean : 384078.0
      median: 362537.0
      min : 221003.0

      The linux we use:
      CentOS release 5.4 (Final)
      2.6.24.7-149.ay #1 SMP PREEMPT RT Tue Mar 23 14:01:51 PDT 2010 x86_64 x86_64 x86_64 GNU/Linux

      Is OS ok for real time java? What else are we missing?
        • 1. Re: Jitter
          807557
          CentOS is not a supported Linux platform for Java RTS. We have no idea what real-time capabilities it has.

          What kind of operation are you measuring? What does the code do? Have you used ITC? Is GC an issue?

          David Holmes
          Oracle Corporation
          • 2. Re: Jitter
            807557
            The operation prepares a message (creates object, populates it with fields), validates it (just regular "if/else" checks). After that it writes message to db and sends it, but I commented this part to keep things simple. So no db, no remote connection.

            ITC - I used Step 3 from Quick Start and applied suggested params:
            -XX:+RTSJBuildCompilationList \
            -XX:+RTSJBuildClassInitializationList \
            -XX:CompilationList=nhrt.precompile \
            -XX:PreInitList=itc.preinit \
            Is it enough?

            GC - I'm not sure. We suspect GC since on java6 GC tuning helped with jitter a bit. Since I made thread RealtimeThread it should not be interrupted by GC as far as I understand, is it correct?
            • 3. Re: Jitter
              807557
              Kostya wrote:
              The operation prepares a message (creates object, populates it with fields), validates it (just regular "if/else" checks). After that it writes message to db and sends it, but I commented this part to keep things simple. So no db, no remote connection.
              Ok.
              ITC - I used Step 3 from Quick Start and applied suggested params:
              -XX:+RTSJBuildCompilationList \
              -XX:+RTSJBuildClassInitializationList \
              -XX:CompilationList=nhrt.precompile \
              -XX:PreInitList=itc.preinit \
              Is it enough?
              It's the most you can do. Note that for 2.2 you can simply specify -XX:+UseITC to replace all the separate flags.
              GC - I'm not sure. We suspect GC since on java6 GC tuning helped with jitter a bit. Since I made thread RealtimeThread it should not be interrupted by GC as far as I understand, is it correct?
              Not quite. You need to set the priority correctly. If you've used the default priority for your RTT then it will be higher priority than the GC in its normal (non-boosted) operating mode. However if memory gets low enough and the GC goes into boosted mode, then under default settings your RTT would now be preempted. If memory gets low enough then even a critical RTT (one with priority greater than the GC) can get preempted and/or blocked on an allocation.

              Can you add some tracing to the operation (eg start/end print statements) and turn on GC tracing -XX:+PrintGC to see if GC is occurring during the operation?

              How many processors do you have?

              David Holmes
              Oracle Corporation
              • 4. Re: Jitter
                807557
                Sorry for late answer.
                Applied -XX:+UseITC instead of those 4 params, added -XX:+PrintGC
                This is output:

                debug: [May 20 13:36:12.696 CDT] [RealtimeThread-2#58] Sender thread 1 : Starting sending
                debug: [May 20 13:36:16.387 CDT] [RealtimeThread-2#58] Sender thread 1 : End sending.

                Number of orders = 100000
                Time in nanoseconds = 3.677884179E9
                Throughput = Msg/sec = 27189.545709726386 Sending latency:
                max : 5.536827E7
                mean : 31437.0
                median: 29859.0
                min : 27179.0

                No GC activity shown. Also jitter is reproduced on small waves, when memory should not be an issue:

                Number of orders = 100
                Time in nanoseconds = 3859810.0
                Throughput = Msg/sec = 25908.011016086286 Sending latency:
                max : 281426.0
                mean : 33816.0
                median: 30426.5
                min : 29613.0


                We have 2 CPUs
                • 5. Re: Jitter
                  807557
                  GettingStarted.zip samples however show good results on our box.
                  Deterministic sample for example shows

                  Mean execution time: 804 microseconds
                  Best execution time: 789 microseconds
                  Worst execution time: 884 microseconds
                  • 6. Re: Jitter
                    807557
                    As an experiment you might try creating a cpuset for use by your Realtime-Threads. However with only 2 CPUs this may reduce jitter at the unacceptable cost of a loss in throughput.

                    You might try using -XX:+LogJitterProneEvents, in conjunction with the com.sun.rtsjx.SteadyMode class to mark in the log when you are in steady-mode. That might give further insight, if you can constrain the log to the time period when the jitter is occurring. The jitter log is less than ideal however and is quite verbose.

                    Beyond that, given the environment you are working in, there is nothing specific I can advise with regard to the jitter.

                    David Holmes