4 Replies Latest reply: May 16, 2009 2:30 AM by 700344 RSS

    Help with speeding up Hadoop application with Jrockit vm

    700344
      Hi all,

      This is Grace. I am using JRockit JVM to run some Hadoop tests. In the test, there are many short time(within 1 minute) processes running. Comparing with the Sun JVM (with -Xmx option only), the test is completed much more slowly by Jrockit JVM. I have already tried to adjust some of the Jrockit basic options, such as –Xmx, -XXaggressive and etc. But there is no big difference. Have anyone seen this problem before? And does anyone can offer some advices about optimization or tuning? Thanks a lot for your time and help.

      Here is some basic information of the testing environment:
      OS: Redhat 5.3 x86_64
      Sun vm version is 1.6.0_11 (64bits), Jrockit vm version is 3.1.0-1.6.0(64bits)

      Regards, Grace
        • 1. Re: Help with speeding up Hadoop application with Jrockit vm
          680605
          I have not tuned for the Hadoop application (maybe someone else out there have?). However, we have collected all our tuning knowledge in a Tuning and Diagnostics Guide I can recommend you: http://download.oracle.com/docs/cd/E13150_01/jrockit_jvm/jrockit/geninfo/diagnos/part_02.html

          Especially the session "Tuning for better applicatin throughput" could help you: http://download.oracle.com/docs/cd/E13150_01/jrockit_jvm/jrockit/geninfo/diagnos/tune_app_thruput.html

          Good luck!

          Cheers,
          Tuva - JRockit PM
          • 2. Re: Help with speeding up Hadoop application with Jrockit vm
            700344
            Thanks for your kind suggestions.

            I have tried some of the options according to the guides already. But there is no big difference. I have also run the Mission Control to find the possible cause. From the results, there are many short duration processes. And these processes allocated 90% large object and 10% small object. Most hot classes/methods are jrockit.jvm.ArrayCopy.memCopyBlock.

            And here is my option list:
            -Xms512m -Xmx1024m -Xns=250m -XgcPrio:throughput -XXaggressive -XXtlaSize:min=10k,preferred=200m -XXlazyUnlocking

            Is there any idea about such kind of use-case above? thanks a lot.

            Regards,
            Grace
            • 3. Re: Help with speeding up Hadoop application with Jrockit vm
              680605
              Most of your objects are large and goes to the old space. When you are running with a nursery all large objects are allocated in the old space. (An object larger than 2k is typically considered large). This will result in a lot of GC of the old space. I guess that this is causing a lof of memCopyBlock() which is the performance blocker.

              So, I have a few suggestions:
              * Always set initial heap size equal to maximum heap size. -Xms=Xmx (1024m in your case) Set the maximal heap size as large as possible (without resulting in page faults).

              * With such high amount of large objects you could profit a static GC, for example -Xgc:singlepar (or -Xgc:genpar with a very small nursery).

              * If the large objects in your application live shortly then you may still profit from using a nursery.
              ** Set GC strategy to -Xgc:genpar (or -Xgc:gencon) to get a nursery.
              ** You will also need to modify the threshold for large objects by setting: -XXlargeObjectLimit="a limit, a little larger than the size of your objects".
              ** Tune nursery size. You will probably have to test a few nursery sizes before you find the best suited nursery size
              ** I see that you tune TLA. TLA tuning is preferably combined with tuning the large object limit for example: -XXtlasize:min=4k, preferred=128k - XXlargeObjectLimit=4k

              I would not use -XXaggressive if it does not give you performance improvements. The same for LazyUnlcoking - it gives a performance boost for some applications, but not all.

              Edited by: TuvaPalm on May 12, 2009 6:08 AM
              • 4. Re: Help with speeding up Hadoop application with Jrockit vm
                700344
                I'd highly appreciate for your comments and suggestion. According to your suggestion, I have made some updates of parameter list and also tested its performance result. This tuning does improve the performance to some extent. On the other hand, because my use cases are mainly processes with short duration, it seems not quite suitable although part of it is very valuable.

                I'd like to try other methods which will jump out of Jrocket to see whether there is some expected improvement.
                Many thanks again for your time and help.