This discussion is archived
1 2 Previous Next 24 Replies Latest reply: Jul 15, 2010 10:41 PM by 807557 RSS

causes of jitter

807557 Newbie
Currently Being Moderated
Hi,
I am using -XX:+LogJitterProneEvents command-line option to find out whats the cause of a great jitter in my application(10 ms!)
After looking up the file I found that the most propable cause of the jitter is the:
"patching call, RealtimeThread-5, invokespecial at bci 43 in method virtual jobject sun.reflect.GeneratedMethodAccessor65.invoke(jobject, jobject) for method virtual void java.lang.Boolean.<init>(jboolean)"
Any idea why this is happening? What exactly is the patching call?What about the invokespecial?

Thanks.
  • 1. Re: causes of jitter
    807557 Newbie
    Currently Being Moderated
    gn_164 wrote:
    After looking up the file I found that the most propable cause of the jitter is the:
    "patching call, RealtimeThread-5, invokespecial at bci 43 in method virtual jobject sun.reflect.GeneratedMethodAccessor65.invoke(jobject, jobject) for method virtual void java.lang.Boolean.<init>(jboolean)"
    Any idea why this is happening? What exactly is the patching call?What about the invokespecial?
    I don't know the details of the actual patch but it is probably occurring because you are using reflection and the reflection inflation mechanism is kicking in. With reflection initially native code is used to perform method/constructor calls, however this is slower than executing the equivalent bytecode. So there is an inflation mechanism whereby after a certain threshold of calls to a given method/constructor is reached, the native accessor is replaced with a generated accessor - which is in effect a class written just to perform the requested invocation. That class is dynamically loaded and then subject to JIT compilation like a normal class. Consequently, this process can introduce significant jitter effects:
    - dynamic creation and loading of the class
    - subsequent JIT compilation of the class, including things like patching

    Two things to try here:

    1. Use the generated method accessor from the outset by specifying -Dsun.reflect.noInflation=true

    This might not remove the patching but should cause it to occur at the very first reflection usage, which might be in a non-critical path

    2. Try to avoid using the generated accessor at all and stick with the native accessor by setting -Dsun.reflect.inflationThreshold=N where N is a large enough number that you won't reach it for any of the reflection calls you are using. Of course if the use of reflection continues over time no count may be big enough to avoid the switch.

    Also you should be using JRTS 2.2 as 2.1 had some known issues with unexpected patching.

    HTH.

    David Holmes
  • 2. Re: causes of jitter
    807557 Newbie
    Currently Being Moderated
    Hi,

    Is there any way we can disable patching. We are testing our trading system on JRTS 2.2 and there is a constant jitter of 50-100ms when first order is executed. On enabling +XX:LogJitterProneEvents I can see huge number of lines printed in log looks like following. 
    +"patching call, IBA_GEN_1, invokeinterface at bci 4 in method virtual void <some method(jobject)> for method virtual void <some method>"+

    We are using -XX:+RTSJBuildCompilationList to compile all classes needed at start up.

    Regards,
    Prashant
  • 3. Re: causes of jitter
    807557 Newbie
    Currently Being Moderated
    Hi Prashant,

    As well as building a compilation list you also need a pre-initialization list - are you using that?

    David holmes
  • 4. Re: causes of jitter
    807557 Newbie
    Currently Being Moderated
    Yes we are doing a Class.forName() for all classes in classpath programmatically. I would like to know if I can turn off patching calls some how.

    Prashant
  • 5. Re: causes of jitter
    807557 Newbie
    Currently Being Moderated
    Hi all,

    I just discussed with one of our compiler expert who cannot unfortunately access this forum right now. Here is what I understood :-)

    First, you can reduce the number of patching by using -XX:-RTSJUseInlineCaches

    However, what may impact you is that, in order to generate patchless calls, the compiler needs to have all the information available at compile time. This means that compilation of a method depends on whether other classes have already been loaded.

    This is done automatically when you use JavaRTS preinit list mechanism. With a preinit list, JavaRTS loads the classes before compiling them, avoiding the need for future patching.

    With a class.forName approach, you are more likely to compile classes before all their dependencies have been loaded. The missing dependencies will have to be patched at a later time. If I understood correctly, -XX:-RTSJUseInlineCaches might help but will not completely eliminate patching in that case.

    Regards,

    Bertrand DELSART
    JavaRTS Technical Leader
  • 6. Re: causes of jitter
    807557 Newbie
    Currently Being Moderated
    While we very much recommend you try to use the JavaRTS pre-init mechanism, note that you can achieve a similar effect in your code by using two Class.forName loops - in the first loop specify not to initialize ie

    Class.forName(cls, false, classloader);

    then in the second loop trigger initialization by passing true as the second arg, or using the short-form Class.forName(cls)

    I wouldn't worry about messing with the inline cache setting until after you've used a proper pre-initialization process, and then only if you find a problem still exists.

    David Holmes
  • 7. Re: causes of jitter
    807557 Newbie
    Currently Being Moderated
    Thanks David/Bertrand

    Reason why I am reluctant to use JavaRTS preinit list mechanism is because i am getting ExceptionInInitialization error while using it and as number of classes is huge I don't think I can modify the list manually.

    I will try both suggestions and post results here.

    Prashant

    Edited by: Prashant-Goel on Mar 29, 2010 10:41 PM
  • 8. Re: causes of jitter
    807557 Newbie
    Currently Being Moderated
    Can you tell us which class causes the ExceptionInInitializer problem? If it's a JDK class we may be able to suggest how to modify the list (but yes you'd have to apply this modification when it was regenerated - unless there's a programmatic tweak that can be used to get it into the generated list in a different place.)

    That said the list is supposed to be created using the order in which classes were loaded by the application, so you shouldn't get this kind of error.

    Also note that rather than using Class.forName you could use the programmatic API for ITC to preinit all the classes (which preloads them first):

    http://java.sun.com/javase/technologies/realtime/reference/doc_2.2/release/ext-docs/com/sun/rtsjx/Compiler.html

    David Holmes
  • 9. Re: causes of jitter
    807557 Newbie
    Currently Being Moderated
    I guess it is following class java/nio/charset/CoderResult, Classes in my itc.preinit are in following order
    java/lang/String
    java/lang/String
    java/lang/Object
    java/lang/System
    java/lang/Math
    java/lang/Number
    java/util/AbstractCollection
    java/lang/Error
    java/lang/Double
    java/lang/Thread
    java/util/HashMap
    java/lang/ref/Reference
    java/io/InputStream
    sun/misc/Unsafe
    java/lang/Long
    java/lang/StringBuffer
    java/lang/AbstractStringBuilder
    java/lang/Integer
    java/lang/Character
    java/lang/Boolean
    java/lang/ThreadLocal
    java/lang/StringBuilder
    java/lang/Enum
    java/lang/Integer$IntegerCache
    java/util/HashMap$Entry
    sun/nio/cs/ISO_8859_1$Encoder
    java/nio/ByteBuffer
    java/nio/Buffer
    java/nio/CharBuffer
    java/nio/charset/CoderResult
    sun/nio/cs/Surrogate$Parser
    java/io/File
    java/lang/NullPointerException
    java/lang/IllegalMonitorStateException
    sun/net/www/ParseUtil
    java/util/BitSet
    java/util/Locale
    java/lang/CharacterDataLatin1
    java/util/ArrayList
    java/lang/ThreadLocal$ThreadLocalMap
    java/lang/ThreadLocal$ThreadLocalMap$Entry
    java/util/concurrent/locks/ReentrantLock
    java/util/concurrent/locks/ReentrantLock$NonfairSync
    java/util/concurrent/locks/ReentrantLock$Sync
    If I am understanding it correctly patching is happening because my class loading and initialization code kicks in only after RTJ is done with precompiling.

    Prashant
  • 10. Re: causes of jitter
    807557 Newbie
    Currently Being Moderated
    Prashant-Goel wrote:
    I guess it is following class java/nio/charset/CoderResult, Classes in my itc.preinit are in following order
    I don't see anything obvious that would cause an issue. Did the ExceptionInInitializerError report the underlying exception?
    If I am understanding it correctly patching is happening because my class loading and initialization code kicks in only after RTJ is done with precompiling.
    Patching occurs for a number of reasons. One reason is that when class A is compiled and makes a call to Class B but Class B has not been loaded, then code is generated that has to be patched later, when the code is used, when B has been loaded (or would be forced to load). So you need to load all your classes first and then commence initialization, which in turn performs initialization-time-compilation.

    David Holmes
  • 11. Re: causes of jitter
    807557 Newbie
    Currently Being Moderated
    davidholmes wrote:
    I don't see anything obvious that would cause an issue. Did the ExceptionInInitializerError report the underlying exception?
    No VM exits with following message
    Error occurred during initialization of VM
    java.lang.ExceptionInInitializerError

    Also I am not able to map the -XX:+PrintCompilation output to itc.preinit file. So I am not sure which class is causing this.

    Patching occurs for a number of reasons. One reason is that when class A is compiled and makes a call to Class B but Class B has not been loaded, then code is generated that has to be patched later, when the code is used, when B has been loaded (or would be forced to load). So you need to load all your classes first and then commence initialization, which in turn performs initialization-time-compilation.
    I am doing following steps

    Step 1 : Run my code with -XX:+RTSJBuildCompilationList
    Step 2 : Run my code again with -XX:+RTSJBuildCompilationList -Drtsj.precompile=nhrt.precompile

    Change code to add the nhrt.precompile file into the application jar, remove -XX:+RTSJBuildCompilationList -Drtsj.precompile=nhrt.precompile,
    Mark methods in nhrt.precompile using com.sun.rtsjx.Compiler class for compilation during initialization.
    Class.forName(classname, false, null) for all classes in classpath,
    Class.forName(classname, true, null)

    Step 3 : Run code again without any VM argument for class loading or initializing.


    I am still seeing jitter of around 40-50 ms. Am I doing anything in wrong order.

    Prashant

    Edited by: Prashant-Goel on Mar 30, 2010 7:08 AM

    Edited by: Prashant-Goel on Mar 30, 2010 7:11 AM
  • 12. Re: causes of jitter
    807557 Newbie
    Currently Being Moderated
    Prashant-Goel wrote:
    No VM exits with following message
    Error occurred during initialization of VM java.lang.ExceptionInInitializerErrorAh I see - that's not very informative, I'll file a RFE to see if we can fix that.
    The alternative is to use java_g and set -XX:+TraceExceptions. That will be a bit "noisy" but will show what exception leads to the ExceptionInInitializerError.
    I am doing following steps

    Step 1 : Run my code with -XX:+RTSJBuildCompilationList
    Step 2 : Run my code again with -XX:+RTSJBuildCompilationList -Drtsj.precompile=nhrt.precompile

    Change code to add the nhrt.precompile file into the application jar, remove -XX:+RTSJBuildCompilationList -Drtsj.precompile=nhrt.precompile,
    Mark methods in nhrt.precompile using com.sun.rtsjx.Compiler class for compilation during initialization.
    Class.forName(classname, false, null) for all classes in classpath,
    Class.forName(classname, true, null)

    Step 3 : Run code again without any VM argument for class loading or initializing.

    I am still seeing jitter of around 40-50 ms. Am I doing anything in wrong order.It's not clear to me that what you are doing creates a complete compilation list, that covers all the code accessed by RTTs. The usual process here is as follows:
    Step 1: Generate the initial compilation list
     java -XX:+RTSJBuildCompilationList <app> <args>
    Step 2: Expand compilation list using code-coverage
     java -XX:+RTSJBuildCompilationList -Drtsj.precompile=nhrt.precompile <app> <newargs>
    Step 3: Based on the compilation list generate the list of classes that need pre-loading & initializing
     java -XX:+RTSJBuildClassInitializationList  -Drtsj.precompile=nhrt.precompile <app> <args>
    Step 4: Use both sets of lists to run the application
    java -Drtsj.precompile=nhrt.precompile  -Drtsj.preinit=itc.preinit <app> <args>
    Also you should be able to take your preinit list that causes the ExceptionInInitializerError and simply delete CoderResult from it. That class will be initialized during VM initialization anyway.
  • 13. Re: causes of jitter
    807557 Newbie
    Currently Being Moderated
    Hi,

    I am able to use -XX:+RTSJBuildClassInitializationList and XX:+RTSJBuildCompilationList command line options to generate preinit and precompile files and using them while running my application. I tried using -XX:-RTSJUseInlineCaches as well but patching calls hence jitter is still happening.

    Prashant
  • 14. Re: causes of jitter
    807557 Newbie
    Currently Being Moderated
    When you say you have 50ms of jitter, what exactly are you measuring and how are you measuring it? Does the first run perform any "initialization" actions? Note that many libraries perform lazy initialization so pre-initialization of classes etc can't help with that.

    It is also possible that the events you see in the LogJitterProneEvents log are not the causes of your jitter. What system are you on? If Solaris then DTrace and our TSV tool would be useful in diagnosing things further.

    David Holmes
1 2 Previous Next