3 Replies Latest reply: Aug 24, 2012 4:23 AM by Sylvain Duloutre-Oracle RSS

    ODSEE 11g - DPS Directory proxy server suddenly increase load average

    877148
      Hi all

      Recently upgraded from directory server 5.2 to ODSEE 11g, one directory proxy configure to one master directory server and one consumer directory server.
      all the three instances are in the same sparc t3 machine.

      Directory proxy server alerts server load average on the machine is above 6.00 normally it is 0.66. I'm not sure what is causing the sudden burst in the load ? the traffic is normal there is no abnormal requests coming to the server. proxy performance degrades over the span of 24 hours ....and Once i restart the proxy services (dpsadm restart) all load averages comes to normal and directory proxy runs normal for the next two - three weeks. The same cycle continues ...I'm not sure what was causing the sudden load increase.

      I increased the JVM heap size from 1GB to 2 GB still continue to have the problem ..did anyone else experience similar problem. How did you fix it....

      Any input or advise in the right direction is much appreciated.

      Thank you.
        • 1. Re: ODSEE 11g - DPS Directory proxy server suddenly increase load average
          Sylvain Duloutre-Oracle
          Hi,

          Not sure I understand all the details:

          Could you please clarify the statement "Directory proxy server alerts server load average on the machine is above 6.00 normally it is 0.66" ?
          What do you mean by "alerts" ? Is it related to the DPS administrative alerts as described in http://docs.oracle.com/cd/E35622_01/html/821-1222/gbece.html#scrolltoc? Is the "server load average" a metric computed by DPS (and exposed in the dps monitoring info) or is it a metric you compute by other means ?

          You also mention that the proxy performance degrades over the span of 24 hours. What are the symptoms ? increase of etimes as perceived by the client applications? increase in CPU usage ? anything else ?

          Thanks

          -Sylvain
          • 2. Re: ODSEE 11g - DPS Directory proxy server suddenly increase load average
            877148
            server load I'm referring to "prstat command" - server load average suddenly shoot up from 0.66 to 6.00 ie) the CPU usage. Alert is from our server monitoring tool not related to directory proxy.
            Clients report connections time out (etime goes from etime=0 ..2..4.....) over 24 hours i can see the etime increases and eventually the proxy server get hung and non responsive. Once I restart all the performance back to normal at-least for another two weeks.

            I suspect there might be a memory leak or JVM Garbage collection issue -- any expert input how to figure this out will help.

            Here is the JVM args in the proxy server "Xms2g -Xmx2g -Xmn1g -XX:SurvivorRatio=4 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC"


            Here is a jstat during the problem

            ./jstat -gcutil -t 25365 2s 30
            Timestamp S0 S1 E O P YGC YGCT FGC FGCT GCT
            982106.4 0.00 26.17 4.26 92.25 59.52 523 60.979 689 1002.587 1063.566
            982108.4 0.00 26.17 4.40 92.25 59.52 523 60.979 689 1002.587 1063.566
            982110.4 0.00 26.17 4.80 92.25 59.52 523 60.979 689 1002.587 1063.566
            982112.4 0.00 26.17 5.10 92.25 59.52 523 60.979 690 1002.719 1063.698
            982114.4 0.00 26.17 5.15 92.25 59.52 523 60.979 690 1002.719 1063.698
            982116.4 0.00 26.17 5.32 92.25 59.52 523 60.979 691 1003.009 1063.988
            982118.4 0.00 26.17 5.72 92.25 59.52 523 60.979 691 1003.009 1063.988
            982120.4 0.00 26.17 5.80 92.25 59.52 523 60.979 691 1003.009 1063.988
            982122.4 0.00 26.17 5.93 92.25 59.52 523 60.979 692 1003.168 1064.146
            982124.4 0.00 26.17 6.03 92.25 59.52 523 60.979 692 1003.168 1064.146
            982126.4 0.00 26.17 6.15 92.25 59.52 523 60.979 693 1003.481 1064.460
            982128.5 0.00 26.17 6.18 92.25 59.52 523 60.979 693 1003.481 1064.460
            982130.5 0.00 26.17 6.25 92.25 59.52 523 60.979 693 1003.481 1064.460
            982132.5 0.00 26.17 6.29 92.25 59.52 523 60.979 694 1003.656 1064.635
            982134.5 0.00 26.17 6.31 92.25 59.52 523 60.979 694 1003.656 1064.635
            982136.5 0.00 26.17 6.36 92.25 59.52 523 60.979 695 1003.988 1064.967
            982138.5 0.00 26.17 6.89 92.25 59.52 523 60.979 695 1003.988 1064.967
            982140.5 0.00 26.17 6.99 92.25 59.52 523 60.979 695 1003.988 1064.967
            982142.5 0.00 26.17 7.08 92.25 59.52 523 60.979 696 1004.187 1065.165
            982144.5 0.00 26.17 7.31 92.25 59.52 523 60.979 696 1004.187 1065.165
            982146.5 0.00 26.17 7.82 92.25 59.52 523 60.979 697 1004.553 1065.531
            982148.5 0.00 26.17 7.92 92.25 59.52 523 60.979 697 1004.553 1065.531
            982150.5 0.00 26.17 8.01 92.25 59.52 523 60.979 697 1004.553 1065.531
            982152.5 0.00 26.17 8.17 92.25 59.52 523 60.979 698 1004.786 1065.764
            982154.5 0.00 26.17 8.26 92.25 59.52 523 60.979 698 1004.786 1065.764
            982156.5 0.00 26.17 8.38 92.25 59.52 523 60.979 699 1005.174 1066.153
            982158.5 0.00 26.17 8.74 92.25 59.52 523 60.979 699 1005.174 1066.153
            982160.5 0.00 26.17 8.88 92.25 59.52 523 60.979 699 1005.174 1066.153
            982162.5 0.00 26.17 8.96 92.25 59.52 523 60.979 700 1005.433 1066.412
            982164.5 0.00 26.17 9.09 92.25 59.52 523 60.979 700 1005.433 1066.412

            jstat after the restart

            ./jstat -gcutil -t 10084 2s 30
            Timestamp S0 S1 E O P YGC YGCT FGC FGCT GCT
            40312.6 0.00 25.13 88.49 1.98 63.68 21 2.366 0 0.000 2.366
            40314.6 0.00 25.13 88.58 1.98 63.68 21 2.366 0 0.000 2.366
            40316.6 0.00 25.13 88.71 1.98 63.68 21 2.366 0 0.000 2.366
            40318.6 0.00 25.13 88.99 1.98 63.68 21 2.366 0 0.000 2.366
            40320.6 0.00 25.13 89.31 1.98 63.68 21 2.366 0 0.000 2.366
            40322.6 0.00 25.13 89.36 1.98 63.68 21 2.366 0 0.000 2.366
            40324.6 0.00 25.13 89.42 1.98 63.68 21 2.366 0 0.000 2.366
            40326.6 0.00 25.13 89.53 1.98 63.68 21 2.366 0 0.000 2.366
            40328.6 0.00 25.13 89.60 1.98 63.68 21 2.366 0 0.000 2.366
            40330.6 0.00 25.13 89.72 1.98 63.68 21 2.366 0 0.000 2.366
            40332.6 0.00 25.13 90.11 1.98 63.68 21 2.366 0 0.000 2.366
            40334.6 0.00 25.13 90.56 1.98 63.68 21 2.366 0 0.000 2.366
            40336.6 0.00 25.13 90.67 1.98 63.68 21 2.366 0 0.000 2.366
            40338.6 0.00 25.13 90.75 1.98 63.68 21 2.366 0 0.000 2.366
            40340.6 0.00 25.13 91.09 1.98 63.68 21 2.366 0 0.000 2.366
            40342.6 0.00 25.13 91.36 1.98 63.68 21 2.366 0 0.000 2.366
            40344.6 0.00 25.13 91.47 1.98 63.68 21 2.366 0 0.000 2.366
            40346.6 0.00 25.13 91.53 1.98 63.68 21 2.366 0 0.000 2.366
            40348.7 0.00 25.13 91.64 1.98 63.68 21 2.366 0 0.000 2.366
            40350.7 0.00 25.13 91.77 1.98 63.68 21 2.366 0 0.000 2.366
            40352.7 0.00 25.13 91.87 1.98 63.68 21 2.366 0 0.000 2.366
            40354.7 0.00 25.13 91.95 1.98 63.68 21 2.366 0 0.000 2.366
            40356.7 0.00 25.13 92.11 1.98 63.68 21 2.366 0 0.000 2.366
            40358.7 0.00 25.13 92.19 1.98 63.68 21 2.366 0 0.000 2.366
            40360.7 0.00 25.13 92.24 1.98 63.68 21 2.366 0 0.000 2.366
            40362.7 0.00 25.13 92.85 1.98 63.68 21 2.366 0 0.000 2.366
            40364.7 0.00 25.13 93.19 1.98 63.68 21 2.366 0 0.000 2.366
            40366.7 0.00 25.13 93.40 1.98 63.68 21 2.366 0 0.000 2.366
            40368.7 0.00 25.13 93.44 1.98 63.68 21 2.366 0 0.000 2.366
            40370.7 0.00 25.13 93.47 1.98 63.68 21 2.366 0 0.000 2.366


            Any one else had similar behavior. Any input to the right direction is highly appreciated.

            Thanks.
            • 3. Re: ODSEE 11g - DPS Directory proxy server suddenly increase load average
              Sylvain Duloutre-Oracle
              Hi,

              What exact release are you using ? 11g may correspond to 11.1.1.3.0 11gR1 , 11.1.1.5.0 11gR1PS1 or the newly released 11.1.1.5.1 (bundle patch) ?
              I would encourage you to switch to the latest bundle patch if you are using an older release.

              The GC seeems to be the culprit, but it should not trigger so often, so I would recommand to contact the Oracle support to open a case.

              -Sylvain