4 Replies Latest reply: Apr 9, 2013 9:16 AM by 997468 RSS

    managed server is is causing HIGH CPU utilization

    user10594123
      Team,

      In One of the managed server, I found HIGH CPU utilization. I use “ TOP “ command and found, three process are culprit.

      PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
      9781 apirdrb0 15 0 4723m 3.0g 3748 S 208.8 9.5 31552:56 java
      4534 apirdrb0 17 0 4456m 2.8g 3724 S 6.6 8.7 746:49.23 java
      9641 apirdrb0 17 0 4096m 1.3g 3728 S 4.0 4.1 372:42.65 java

      Later I grep the process and found, some java process in soa_server1 is causing HIGH CPU utilization.


      Now, I want to know, what actually is running behind that java process. So, I can guide to client in right track.


      Kindly share your views




      Check the below log :-


      Ps –ef | grep 9781


      ps -ef | grep 9781
      59001 9781 8110 99 Mar29 ? 21-21:41:00 /u01/APPLTOP/fusionapps/jdk6/bin/java -jrockit -Xms512m -Xmx2048m -Xgc:genpar -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/u01/APPLTOP/instance/debug -XX:+ExitOnOutOfMemoryError -Xverbosedecorations=level,module,timestamp,millis,pid -Xverbose:gc -Dweblogic.Name=soa_server1 -Djava.security.policy=/u01/APPLTOP/fusionapps/wlserver_10.3/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Dweblogic.system.BootIdentityFile=/irdrb010/APPLTOP/instance/domains/primary.oracleoutsourcing.com/FinancialDomain/servers/soa_server1/data/nodemanager/boot.properties -Dweblogic.nodemanager.ServiceEnabled=true -Dweblogic.security.SSL.ignoreHostnameVerification=true -Dweblogic.ReverseDNSAllowed=false -DJDBCProgramName=DS/FinancialDomain/soa_server1 -Dserver.group=FIN_SOACluster -Dtangosol.coherence.localhost=primary.oracleoutsourcing.com -Dtangosol.coherence.localport=8089 -Dtangosol.coherence.wka1=primary.oracleoutsourcing.com -Dtangosol.coherence.wka1.port=8089 -Dtangosol.coherence.wka3=secondary-scale2.oracleoutsourcing.com -Dtangosol.coherence.wka3.port=8089 -Dtangosol.coherence.wka4=secondary-scale3.oracleoutsourcing.com -Dtangosol.coherence.wka4.port=8089 -Dtangosol.coherence.wka5=secondary-scale4.oracleoutsourcing.com -Dtangosol.coherence.wka5.port=8089 -Dtangosol.coherence.wka6=secondary-scale5.oracleoutsourcing.com -Dtangosol.coherence.wka6.port=8089 -Dtangosol.coherence.wka7=secondary-scale6.oracleoutsourcing.com -Dtangosol.coherence.wka7.port=8089 -Dtangosol.coherence.wka8=secondary-scale7.oracleoutsourcing.com -Dtangosol.coherence.wka8.port=8089 -Dtangosol.coherence.wka9=secondary-scale8.oracleoutsourcing.com -Dtangosol.coherence.wka9.port=8089 -Dtangosol.coherence.wka10=secondary-scale9.oracleoutsourcing.com -Dtangosol.coherence.wka10.port=8089 -da -Dplatform.home=/u01/APPLTOP/fusionapps/wlserver_10.3 -Dwls.home=/u01/APPLTOP/fusionapps/wlserver_10.3/server -Dweblogic.home=/u01/APPLTOP/fusionapps/wlserver_10.3/server -Dcommon.components.home=/u01/APPLTOP/fusionapps/oracle_common -Djrf.version=11.1.1 -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger -Ddomain.home=/u01/APPLTOP/instance/domains/primary.oracleoutsourcing.com/FinancialDomain -Djrockit.optfile=/u01/APPLTOP/fusionapps/oracle_common/modules/oracle.jrf_11.1.1/jrocket_optfile.txt -Doracle.server.config.dir=/u01/APPLTOP/instance/domains/primary.oracleoutsourcing.com/FinancialDomain/config/fmwconfig/servers/soa_server1 -Doracle.domain.config.dir=/u01/APPLTOP/instance/domains/primary.oracleoutsourcing.com/FinancialDomain/config/fmwconfig -Digf.arisidbeans.carmlloc=/u01/APPLTOP/instance/domains/primary.oracleoutsourcing.com/FinancialDomain/config/fmwconfig/carml -Digf.arisidstack.home=/u01/APPLTOP/instance/domains/primary.oracleoutsourcing.com/FinancialDomain/config/fmwconfig/arisidprovider -Doracle.security.jps.config=/u01/APPLTOP/instance/domains/primary.oracleoutsourcing.com/FinancialDomain/config/fmwconfig/jps-config.xml -Doracle.deployed.app.dir=/u01/APPLTOP/instance/domains/primary.oracleoutsourcing.com/FinancialDomain/servers/soa_server1/tmp/_WL_user -Doracle.deployed.app.ext=/- -Dweblogic.alternateTypesDirectory=/u01/APPLTOP/fusionapps/oracle_common/modules/oracle.ossoiap_11.1.1,/u01/APPLTOP/fusionapps/oracle_common/modules/oracle.oamprovider_11.1.1 -Djava.protocol.handler.pkgs=oracle.mds.net.protocol|oracle.fabric.common.classloaderurl.handler|oracle.fabric.common.uddiurl.handler|oracle.bpm.io.fs.protocol -Dweblogic.jdbc.remoteEnabled=false -Datgpf.oracle.home=/u01/APPLTOP/fusionapps/atgpf -Doracle.fusion.appsMode=true -Dbip.client.config.dir=/u01/APPLTOP/instance/domains/primary.oracleoutsourcing.com/FinancialDomain/config -Dweblogic.alternateTypesDirectory=/u01/APPLTOP/fusionapps/bi/bifoundation/javahost/lib/obisintegration/adf -DUSE_JAAS=false -Djps.policystore.hybrid.mode=false -Djps.combiner.optimize.lazyeval=true -Djps.combiner.optimize=true -Djps.auth=ACC -Doracle.core.ojdl.logging.usercontextprovider=oracle.core.ojdl.logging.impl.UserContextImpl -noverify -Doracle.webcenter.analytics.disable-native-partitioning=fal
      59001 14066 29425 0 06:47 pts/1 00:00:00 grep 9781
        • 1. Re: managed server is is causing HIGH CPU utilization
          997468
          Hi,

          The process id/details you have gathered will just tell you pid on which your JVM is running. For debugging high CPU caused by weblogic managed server, you should be checking on the applications deployed on the server.
          At time when you see high CPU utilization, take thread dump on the pid and then using top command get the culprit thread id and check what that thread is executing which caused high CPU.

          For taking thread dump refer to - http://middlewaresupport.wordpress.com/2013/01/23/thread-dump-what-when-and-how/
          For debugging high CPU refer to article - http://middlewaremagic.com/weblogic/?p=4884


          Thanks,
          Ranjan
          • 2. Re: managed server is is causing HIGH CPU utilization
            user10594123
            Dear Ranjan,

            That link is very helpfull.

            But, some how, I could not find jstack in /u01/APPLTOP/fusionapps/jdk6/bin.

            Can you please let me know, how i need to overcome it
            • 3. Re: managed server is is causing HIGH CPU utilization
              user10594123
              Dear Ranjan,


              As per the Documnets http://middlewaremagic.com/weblogic/?p=4884. Step8). convert those child PIDs into Hexadecimal values and search in thread dump log.

              I tried those method but somehow, I could not see any nid values in my thread dump log.

              so, my question shall we consider "Active thread " is the culprit for HIGH CPU utilization. If i m wrong guide me correct path.
              • 4. Re: managed server is is causing HIGH CPU utilization
                997468
                Hi,

                It could be that jstack is not present with the version you are using, but no worries you can select other options mentioned in blog [ http://middlewaresupport.wordpress.com/2013/01/23/thread-dump-what-when-and-how/ ] to gather thread dump.

                For the "nid" not matching your thread dump "nid", did you verified the pid on which you gathered thread dump and top command output. If you are not sure about the process id which is running weblogic server you can use $JAVA_HOME/bin/jps -v | grep -i "weblogic.Server" to get the id. If server has been started using nodemanager then go to servers/<server>/data/nodemanager and check for the pid file.

                Also remember to gather thread dump and top command output simultaneously.
                Steps in blog http://middlewaremagic.com/weblogic/?p=4884 are useful, I was able to match the nid for such scenario in past.

                ## rank it if found useful :) ##

                Thanks,
                Ranjan

                Edited by: Ranjan K on 9 Apr, 2013 7:15 AM