0 Replies Latest reply: Jan 17, 2012 4:12 AM by astonishingONE RSS

    help regarding weblogic 10.3 SUN JVM 1.6 tuning

    astonishingONE
      Hi,

      i am having issues with tuning garbage collect.

      Server is intel itaium quad core 64 bit

      Weblogic 10.3.0

      JVM is Sun hotspot 1.6 upadate 14 1.6.0_14 (6u14)


      32 GB ram

      My JVM memory options

      old using CMS -concurrent collector had issues

      /jdk1.6.0_14/bin/java -server -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -XX:+PrintTenuringDistribution -Xms7168m -Xmn3000m -Xmx7168m -Xss768k -XX:+AggressiveOpts -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=65 -XX:CMSFullGCsBeforeCompaction=1 -XX:+UseCMSInitiatingOccupancyOnly -XX:+CMSScavengeBeforeRemark -XX:NewRatio=6 -XX:TargetSurvivorRatio=90 -XX:+CMSParallelRemarkEnabled -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=2 -XX:+DisableExplicitGC -Xloggc:/var/log/gc.log -XX:PretenureSizeThreshold=1m -XX:PermSize=256m -XX:MaxPermSize=512m


      We have to restart after 12 hours as

      the young gc is ok and when 65% is completed the first Full GC occurs clear lot of memory , then send FC at 65 % full old only 10% then before after third time then starts lot of FC GC every 6 sec.

      The old generation graph keep going up .

      The overall heap graph start to allocate more and more memory and reach zero memory where lot of FC GC occurs when gradually increase in pause time and old genration does not clears .

      Even after the restart of weblogic server the memory remain same , goes away after reboot harware server.

      Object genrated they keep increasing

      num #instances #bytes class name

      ----------------------------------------------

      1: 2114960 101518080 edu.emory.mathcs.backport.java.util.concurrent.ConcurrentHashMap$Segment

      2: 663190 79657592 [Ljava.lang.Object;

      3: 2114960 67808400 [Ledu.emory.mathcs.backport.java.util.concurrent.ConcurrentHashMap$HashEntry;

      4: 2114960 67678720 edu.emory.mathcs.backport.java.util.concurrent.locks.ReentrantLock$NonfairSync

      5: 397258 61014680 [Ljava.util.HashMap$Entry;

      6: 234262 31675816 <constMethodKlass>

      7: 234262 28122688 <methodKlass>

      8: 384602 27691344 java.util.HashMap

      9: 195759 23090832 [C

      10: 18615 22282568 <constantPoolKlass>

      11: 132185 20092120 [Ledu.emory.mathcs.backport.java.util.concurrent.ConcurrentHashMap$Segment;

      12: 36616 17372320 [I

      13: 292169 16261512 <symbolKlass>

      14: 397258 15890320 java.util.HashMap$FrontCache

      15: 18615 15382280 <instanceKlassKlass>

      16: 13981 12352992 <constantPoolCacheKlass>

      17: 132185 10574800 edu.emory.mathcs.backport.java.util.concurrent.ConcurrentHashMap

      18: 203460 8138400 java.lang.String

      19: 114872 6432832 java.util.HashMap$Entry

      20: 40960 6385680 [B

      21: 40248 5795712 org.apache.axis2.description.AxisBindingMessage

      22: 32037 5638512 org.apache.axis2.description.AxisMessage

      23: 137209 5488360 java.util.ArrayList

      24: 120238 4809520 org.apache.axis2.description.PolicySubject

      25: 98902 4747296 edu.emory.mathcs.backport.java.util.concurrent.ConcurrentHashMap$HashEntry

      26: 117830 4713200 org.apache.axis2.engine.Phase

      27: 120504 3856128 java.util.Date

      28: 24249 3685848 java.lang.reflect.Method

      29: 20028 3685152 java.lang.Class

      30: 25254 3030480 org.apache.axis2.description.AxisBindingOperation

      31: 30859 2898576 [[I

      32: 120415 2889960 org.apache.axis2.description.ParameterIncludeImpl

      33: 117838 2828112 edu.emory.mathcs.backport.java.util.concurrent.CopyOnWriteArrayList

      34: 3725 2172104 <methodDataKlass>

      35: 28281 2102792 [S

      36: 24494 1763568 java.util.LinkedHashMap$Entry

      37: 32864 1577472 java.util.Hashtable$Entry


      Now using Followed sun java jvm recommendations (parallel collector )

      /jdk1.6.0_14/bin/java -server -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -XX:+PrintTenuringDistribution -Xms8192m -Xmn5000m -Xmx8192m -Xss768k -XX:+AggressiveOpts -XX:+AggressiveHeap -XX:MaxGCPauseMillis=3500 -XX:GCTimeRatio=19 -XX:InterpreterProfilePercentage=10 -XX:CompileThreshold=35000 -XX:OnStackReplacePercentage=80 -XX:ReservedCodeCacheSize=128m -XX:AllocatePrefetchStepSize=128 -XX:AllocatePrefetchStyle=2 -XX:AllocatePrefetchDistance=1024 -Xloggc:/var/log/gc.log -XX:MaxTenuringThreshold=25 -XX:+UseParallelOldGC -XX:PermSize=256m -XX:MaxPermSize=512m -XX:ParallelGCThreads=4 -XX:SurvivorRatio=3 -XX:+PrintTenuringDistribution


      In both above the real issuews starts when the old generation becomes full , its not cleared at all.

      my gc log for current
      49.272: [GC
      Desired survivor size 1073741824 bytes, new threshold 7 (max 25)
      [PSYoungGen: 3145726K->77824K(4194304K)] 3145726K->77824K(7602176K), 0.1589830 secs] [Times: user=0.33 sys=0.05, real=0.16 secs]
      53.401: [GC
      Desired survivor size 1073741824 bytes, new threshold 7 (max 25)
      [PSYoungGen: 3223539K->71680K(4194304K)] 3223539K->71680K(7602176K), 0.0837190 secs] [Times: user=0.21 sys=0.03, real=0.08 secs]
      72.459: [GC
      Desired survivor size 1073741824 bytes, new threshold 7 (max 25)
      [PSYoungGen: 3217393K->120832K(4194304K)] 3217393K->120832K(7602176K), 0.2130480 secs] [Times: user=0.66 sys=0.01, real=0.21 secs]
      82.518: [GC
      Desired survivor size 1073741824 bytes, new threshold 7 (max 25)
      [PSYoungGen: 3266548K->126976K(4194304K)] 3266548K->126976K(7602176K), 0.1503580 secs] [Times: user=0.46 sys=0.02, real=0.15 secs]
      100.193: [GC
      Desired survivor size 168689664 bytes, new threshold 6 (max 25)
      [PSYoungGen: 3272704K->141312K(4194304K)] 3272704K->141312K(7602176K), 0.1695430 secs] [Times: user=0.51 sys=0.02, real=0.17 secs]
      596.617: [GC
      Desired survivor size 324796416 bytes, new threshold 5 (max 25)
      [PSYoungGen: 3287040K->163840K(4773248K)] 3287040K->394568K(8181120K), 0.8661540 secs] [Times: user=1.83 sys=0.18, real=0.87 secs]
      1436.642: [GC
      Desired survivor size 491716608 bytes, new threshold 4 (max 25)
      [PSYoungGen: 4772352K->315392K(4787008K)] 5003080K->755155K(8194880K), 1.0644550 secs] [Times: user=2.64 sys=0.22, real=1.07 secs]
      2555.211: [GC
      Desired survivor size 651952128 bytes, new threshold 3 (max 25)
      [PSYoungGen: 4787008K->317440K(4288512K)] 5226771K->1078915K(7696384K), 1.0683070 secs] [Times: user=3.46 sys=0.22, real=1.07 secs]
      3597.979: [GC
      Desired survivor size 720371712 bytes, new threshold 2 (max 25)
      [PSYoungGen: 4286976K->526336K(4385152K)] 5048451K->1352756K(7793024K), 0.8806620 secs] [Times: user=3.11 sys=0.03, real=0.88 secs]
      4732.952: [GC
      Desired survivor size 755957760 bytes, new threshold 1 (max 25)
      [PSYoungGen: 4385152K->493568K(4469888K)] 5211572K->1570806K(7877760K), 1.0933950 secs] [Times: user=3.88 sys=0.10, real=1.09 secs]
      5779.276: [GC
      Desired survivor size 717094912 bytes, new threshold 1 (max 25)
      [PSYoungGen: 4259956K->321536K(4504640K)] 5337194K->1614102K(7912512K), 0.8668480 secs] [Times: user=2.94 sys=0.11, real=0.87 secs]
      6708.765: [GC
      Desired survivor size 679149568 bytes, new threshold 1 (max 25)
      [PSYoungGen: 4087936K->296960K(4579648K)] 5380502K->1602262K(7987520K), 0.4922540 secs] [Times: user=1.55 sys=0.01, real=0.49 secs]
      7678.425: [GC
      Desired survivor size 649330688 bytes, new threshold 1 (max 25)
      [PSYoungGen: 4176319K->270336K(4542592K)] 5481621K->1585265K(7950464K), 0.4383880 secs] [Times: user=1.36 sys=0.00, real=0.44 secs]
      8686.739: [GC
      Desired survivor size 622919680 bytes, new threshold 1 (max 25)
      [PSYoungGen: 4149696K->323584K(4634560K)] 5464625K->1646577K(8042432K), 0.5048630 secs] [Times: user=1.60 sys=0.01, real=0.50 secs]
      9536.132: [GC
      Desired survivor size 604962816 bytes, new threshold 1 (max 25)
      [PSYoungGen: 4324032K->231424K(4608768K)] 5647025K->1562417K(8016640K), 0.3650570 secs] [Times: user=1.14 sys=0.00, real=0.36 secs]
      10291.285: [GC
      Desired survivor size 586022912 bytes, new threshold 1 (max 25)
      [PSYoungGen: 4231864K->235520K(4670592K)] 5562857K->1729073K(8078464K), 0.6045870 secs] [Times: user=2.02 sys=0.07, real=0.60 secs]
      11415.725: [GC
      Desired survivor size 559742976 bytes, new threshold 1 (max 25)
      [PSYoungGen: 4315328K->278528K(4652096K)] 5808881K->1776414K(8059968K), 0.4509800 secs] [Times: user=1.40 sys=0.01, real=0.45 secs]
      12592.377: [GC
      Desired survivor size 542834688 bytes, new threshold 1 (max 25)
      [PSYoungGen: 4358336K->315392K(4712768K)] 5856222K->2020766K(8120640K), 0.7958210 secs] [Times: user=2.68 sys=0.09, real=0.80 secs]
      14298.975: [GC
      Desired survivor size 564199424 bytes, new threshold 1 (max 25)
      [PSYoungGen: 4481536K->413696K(4579840K)] 6186910K->2354270K(7987712K), 1.0072620 secs] [Times: user=3.32 sys=0.11, real=1.01 secs]
      15965.584: [GC
      Desired survivor size 554500096 bytes, new threshold 1 (max 25)
      [PSYoungGen: 4579808K->346112K(4701376K)] 6520383K->2299550K(8109248K), 0.5760710 secs] [Times: user=1.78 sys=0.01, real=0.58 secs]
      17859.134: [GC
      Desired survivor size 558563328 bytes, new threshold 1 (max 25)
      [PSYoungGen: 4496512K->382976K(4691904K)] 6449950K->2346654K(8099776K), 0.6244400 secs] [Times: user=1.93 sys=0.01, real=0.62 secs]
      19772.901: [GC
      Desired survivor size 549322752 bytes, new threshold 1 (max 25)
      [PSYoungGen: 4533376K->354304K(4706432K)] 6497054K->2605278K(8114304K), 0.9921450 secs] [Times: user=3.32 sys=0.14, real=0.99 secs]
      22155.372: [GC
      Desired survivor size 529924096 bytes, new threshold 1 (max 25)
      [PSYoungGen: 4515264K->305152K(4697408K)] 6766238K->2820894K(8105280K), 0.8972460 secs] [Times: user=3.01 sys=0.12, real=0.90 secs]
      25154.898: [GC
      Desired survivor size 557907968 bytes, new threshold 1 (max 25)
      [PSYoungGen: 4466112K->446464K(4670720K)] 6981854K->2971934K(8078592K), 0.7218790 secs] [Times: user=2.25 sys=0.00, real=0.72 secs]
      27501.304: [GC
      Desired survivor size 567148544 bytes, new threshold 1 (max 25)
      [PSYoungGen: 4599680K->413696K(4566912K)] 7125150K->3280990K(7974784K), 1.1696970 secs] [Times: user=3.94 sys=0.16, real=1.17 secs]
      30416.690: [GC
      Desired survivor size 560726016 bytes, new threshold 1 (max 25)
      [PSYoungGen: 4566912K->380928K(4695296K)] 7434206K->3557004K(8103168K), 1.0760010 secs] [Times: user=3.57 sys=0.15, real=1.08 secs]
      30417.766: [Full GC [PSYoungGen: 380928K->0K(4695296K)] [ParOldGen: 3176076K->2779206K(3407872K)] 3557004K->2779206K(8103168K) [PSPermGen: 137410K->136090K(524288K)], 21.0202100 secs] [Times: user=46.75 sys=0.07, real=21.02 secs]
      33352.524: [GC
      Desired survivor size 553844736 bytes, new threshold 1 (max 25)
      [PSYoungGen: 4141440K->380928K(4689024K)] 6920646K->3160134K(8096896K), 0.6223370 secs] [Times: user=1.88 sys=0.00, real=0.62 secs]
      36004.625: [GC
      Desired survivor size 549781504 bytes, new threshold 1 (max 25)
      [PSYoungGen: 4522368K->389120K(4705984K)] 7301574K->3181254K(8113856K), 0.6389750 secs] [Times: user=1.96 sys=0.00, real=0.64 secs]
      39204.905: [GC
      Desired survivor size 550371328 bytes, new threshold 2 (max 25)
      [PSYoungGen: 4554240K->403456K(4702016K)] 7346374K->3209094K(8109888K), 0.6556140 secs] [Times: user=2.03 sys=0.00, real=0.66 secs]
      42363.832: [GC
      Desired survivor size 610271232 bytes, new threshold 1 (max 25)
      [PSYoungGen: 4568576K->536576K(4588416K)] 7374214K->3369488K(7996288K), 0.8709560 secs] [Times: user=2.73 sys=0.00, real=0.87 secs]
      46949.686: [GC
      Desired survivor size 592707584 bytes, new threshold 2 (max 25)
      [PSYoungGen: 4587520K->350208K(4646912K)] 7420432K->3445502K(8054784K), 0.9147430 secs] [Times: user=3.10 sys=0.00, real=0.92 secs]
      46950.601: [Full GC [PSYoungGen: 350208K->0K(4646912K)] [ParOldGen: 3095294K->2808499K(3407872K)] 3445502K->2808499K(8054784K) [PSPermGen: 136551K->136506K(524288K)], 15.1227450 secs] [Times: user=26.32 sys=0.07, real=15.12 secs]
      55410.580: [GC
      Desired survivor size 572522496 bytes, new threshold 3 (max 25)
      [PSYoungGen: 4050944K->376832K(4683776K)] 6859443K->3185331K(8091648K), 0.6224830 secs] [Times: user=1.87 sys=0.00, real=0.62 secs]
      68327.028: [GC
      Desired survivor size 568393728 bytes, new threshold 4 (max 25)
      [PSYoungGen: 4481792K->290816K(4664064K)] 7290291K->3099315K(8071936K), 0.4951140 secs] [Times: user=1.49 sys=0.00, real=0.50 secs]
      73297.028: [GC
      Desired survivor size 583204864 bytes, new threshold 5 (max 25)
      [PSYoungGen: 4395776K->456704K(4658880K)] 7204275K->3265203K(8066752K), 0.7185530 secs] [Times: user=2.24 sys=0.00, real=0.72 secs]
      75459.941: [GC
      Desired survivor size 598671360 bytes, new threshold 6 (max 25)
      [PSYoungGen: 4560512K->468992K(4572800K)] 7369011K->3277491K(7980672K), 0.7232830 secs] [Times: user=2.30 sys=0.00, real=0.72 secs]
      76889.410: [GC
      Desired survivor size 633339904 bytes, new threshold 7 (max 25)
      [PSYoungGen: 4572799K->530432K(4590528K)] 7381299K->3338931K(7998400K), 0.8254890 secs] [Times: user=2.63 sys=0.00, real=0.83 secs]
      78409.695: [GC
      Desired survivor size 636420096 bytes, new threshold 8 (max 25)
      [PSYoungGen: 4536319K->466944K(4472832K)] 7344819K->3275443K(7880704K), 0.7111590 secs] [Times: user=2.32 sys=0.00, real=0.71 secs]
      79956.284: [GC
      Desired survivor size 641269760 bytes, new threshold 9 (max 25)
      [PSYoungGen: 4472832K->477184K(4611904K)] 7281331K->3285683K(8019776K), 0.7282880 secs] [Times: user=2.37 sys=0.00, real=0.73 secs]
      81227.200: [GC
      Desired survivor size 642842624 bytes, new threshold 9 (max 25)
      [PSYoungGen: 4467584K->475136K(4465536K)] 7276083K->3283635K(7873408K), 0.7101730 secs] [Times: user=2.34 sys=0.00, real=0.71 secs]
      82594.075: [GC
      Desired survivor size 630915072 bytes, new threshold 9 (max 25)
      [PSYoungGen: 4465536K->444416K(4626752K)] 7274035K->3252915K(8034624K), 0.6827300 secs] [Times: user=2.24 sys=0.00, real=0.68 secs]
      83901.039: [GC
      Desired survivor size 628883456 bytes, new threshold 9 (max 25)
      [PSYoungGen: 4443392K->468992K(4615104K)] 7251891K->3296758K(8022976K), 0.7387020 secs] [Times: user=2.43 sys=0.01, real=0.74 secs]
      85270.381: [GC
      Desired survivor size 614006784 bytes, new threshold 9 (max 25)
      [PSYoungGen: 4467968K->438272K(4643264K)] 7295734K->3276214K(8051136K), 0.6958560 secs] [Times: user=2.26 sys=0.00, real=0.70 secs]
      86584.768: [GC
      Desired survivor size 668860416 bytes, new threshold 8 (max 25)
      [PSYoungGen: 4467392K->598016K(4627136K)] 7305334K->3456887K(8035008K), 0.9053490 secs] [Times: user=3.17 sys=0.00, real=0.91 secs]
      87875.175: [GC
      Desired survivor size 804454400 bytes, new threshold 7 (max 25)
      [PSYoungGen: 4627136K->614400K(4287424K)] 7486007K->3719504K(7695296K), 1.2332520 secs] [Times: user=4.46 sys=0.00, real=1.23 secs]
      87876.409: [Full GC [PSYoungGen: 614400K->0K(4287424K)] [ParOldGen: 3105104K->3308576K(3407872K)] 3719504K->3308576K(7695296K) [PSPermGen: 137144K->137084K(524288K)], 18.4937680 secs] [Times: user=37.63 sys=3.48, real=18.49 secs]
      89034.197: [Full GC [PSYoungGen: 3671680K->84388K(4287424K)] [ParOldGen: 3308576K->3407861K(3407872K)] 6980256K->3492250K(7695296K) [PSPermGen: 137107K->137088K(524288K)], 29.4977940 secs] [Times: user=48.68 sys=8.78, real=29.50 secs]
      90124.178: [Full GC [PSYoungGen: 3671680K->49214K(4287424K)] [ParOldGen: 3407861K->3407871K(3407872K)] 7079541K->3457086K(7695296K) [PSPermGen: 137179K->137151K(524288K)], 23.7582660 secs] [Times: user=45.72 sys=2.53, real=23.76 secs]
      91290.429: [Full GC [PSYoungGen: 3671680K->120269K(4287424K)] [ParOldGen: 3407871K->3407869K(3407872K)] 7079551K->3528139K(7695296K) [PSPermGen: 137170K->137151K(262144K)], 30.4431980 secs] [Times: user=48.70 sys=1.09, real=30.44 secs]
      92410.006: [Full GC [PSYoungGen: 3671680K->90688K(4287424K)] [ParOldGen: 3407869K->3407868K(3407872K)] 7079549K->3498557K(7695296K) [PSPermGen: 137176K->137158K(262144K)], 30.6326750 secs] [Times: user=46.20 sys=18.09, real=30.63 secs]
      93488.443: [Full GC [PSYoungGen: 3671680K->96348K(4287424K)] [ParOldGen: 3407868K->3407871K(3407872K)] 7079548K->3504220K(7695296K) [PSPermGen: 137178K->137161K(262144K)], 28.1295570 secs]

      Please advice .