This discussion is archived
4 Replies Latest reply: Apr 9, 2013 7:16 AM by 997468 RSS

managed server is is causing HIGH CPU utilization

user10594123 Newbie
Currently Being Moderated
Team,

In One of the managed server, I found HIGH CPU utilization. I use “ TOP “ command and found, three process are culprit.

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9781 apirdrb0 15 0 4723m 3.0g 3748 S 208.8 9.5 31552:56 java
4534 apirdrb0 17 0 4456m 2.8g 3724 S 6.6 8.7 746:49.23 java
9641 apirdrb0 17 0 4096m 1.3g 3728 S 4.0 4.1 372:42.65 java

Later I grep the process and found, some java process in soa_server1 is causing HIGH CPU utilization.


Now, I want to know, what actually is running behind that java process. So, I can guide to client in right track.


Kindly share your views




Check the below log :-


Ps –ef | grep 9781


ps -ef | grep 9781
59001 9781 8110 99 Mar29 ? 21-21:41:00 /u01/APPLTOP/fusionapps/jdk6/bin/java -jrockit -Xms512m -Xmx2048m -Xgc:genpar -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/u01/APPLTOP/instance/debug -XX:+ExitOnOutOfMemoryError -Xverbosedecorations=level,module,timestamp,millis,pid -Xverbose:gc -Dweblogic.Name=soa_server1 -Djava.security.policy=/u01/APPLTOP/fusionapps/wlserver_10.3/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Dweblogic.system.BootIdentityFile=/irdrb010/APPLTOP/instance/domains/primary.oracleoutsourcing.com/FinancialDomain/servers/soa_server1/data/nodemanager/boot.properties -Dweblogic.nodemanager.ServiceEnabled=true -Dweblogic.security.SSL.ignoreHostnameVerification=true -Dweblogic.ReverseDNSAllowed=false -DJDBCProgramName=DS/FinancialDomain/soa_server1 -Dserver.group=FIN_SOACluster -Dtangosol.coherence.localhost=primary.oracleoutsourcing.com -Dtangosol.coherence.localport=8089 -Dtangosol.coherence.wka1=primary.oracleoutsourcing.com -Dtangosol.coherence.wka1.port=8089 -Dtangosol.coherence.wka3=secondary-scale2.oracleoutsourcing.com -Dtangosol.coherence.wka3.port=8089 -Dtangosol.coherence.wka4=secondary-scale3.oracleoutsourcing.com -Dtangosol.coherence.wka4.port=8089 -Dtangosol.coherence.wka5=secondary-scale4.oracleoutsourcing.com -Dtangosol.coherence.wka5.port=8089 -Dtangosol.coherence.wka6=secondary-scale5.oracleoutsourcing.com -Dtangosol.coherence.wka6.port=8089 -Dtangosol.coherence.wka7=secondary-scale6.oracleoutsourcing.com -Dtangosol.coherence.wka7.port=8089 -Dtangosol.coherence.wka8=secondary-scale7.oracleoutsourcing.com -Dtangosol.coherence.wka8.port=8089 -Dtangosol.coherence.wka9=secondary-scale8.oracleoutsourcing.com -Dtangosol.coherence.wka9.port=8089 -Dtangosol.coherence.wka10=secondary-scale9.oracleoutsourcing.com -Dtangosol.coherence.wka10.port=8089 -da -Dplatform.home=/u01/APPLTOP/fusionapps/wlserver_10.3 -Dwls.home=/u01/APPLTOP/fusionapps/wlserver_10.3/server -Dweblogic.home=/u01/APPLTOP/fusionapps/wlserver_10.3/server -Dcommon.components.home=/u01/APPLTOP/fusionapps/oracle_common -Djrf.version=11.1.1 -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger -Ddomain.home=/u01/APPLTOP/instance/domains/primary.oracleoutsourcing.com/FinancialDomain -Djrockit.optfile=/u01/APPLTOP/fusionapps/oracle_common/modules/oracle.jrf_11.1.1/jrocket_optfile.txt -Doracle.server.config.dir=/u01/APPLTOP/instance/domains/primary.oracleoutsourcing.com/FinancialDomain/config/fmwconfig/servers/soa_server1 -Doracle.domain.config.dir=/u01/APPLTOP/instance/domains/primary.oracleoutsourcing.com/FinancialDomain/config/fmwconfig -Digf.arisidbeans.carmlloc=/u01/APPLTOP/instance/domains/primary.oracleoutsourcing.com/FinancialDomain/config/fmwconfig/carml -Digf.arisidstack.home=/u01/APPLTOP/instance/domains/primary.oracleoutsourcing.com/FinancialDomain/config/fmwconfig/arisidprovider -Doracle.security.jps.config=/u01/APPLTOP/instance/domains/primary.oracleoutsourcing.com/FinancialDomain/config/fmwconfig/jps-config.xml -Doracle.deployed.app.dir=/u01/APPLTOP/instance/domains/primary.oracleoutsourcing.com/FinancialDomain/servers/soa_server1/tmp/_WL_user -Doracle.deployed.app.ext=/- -Dweblogic.alternateTypesDirectory=/u01/APPLTOP/fusionapps/oracle_common/modules/oracle.ossoiap_11.1.1,/u01/APPLTOP/fusionapps/oracle_common/modules/oracle.oamprovider_11.1.1 -Djava.protocol.handler.pkgs=oracle.mds.net.protocol|oracle.fabric.common.classloaderurl.handler|oracle.fabric.common.uddiurl.handler|oracle.bpm.io.fs.protocol -Dweblogic.jdbc.remoteEnabled=false -Datgpf.oracle.home=/u01/APPLTOP/fusionapps/atgpf -Doracle.fusion.appsMode=true -Dbip.client.config.dir=/u01/APPLTOP/instance/domains/primary.oracleoutsourcing.com/FinancialDomain/config -Dweblogic.alternateTypesDirectory=/u01/APPLTOP/fusionapps/bi/bifoundation/javahost/lib/obisintegration/adf -DUSE_JAAS=false -Djps.policystore.hybrid.mode=false -Djps.combiner.optimize.lazyeval=true -Djps.combiner.optimize=true -Djps.auth=ACC -Doracle.core.ojdl.logging.usercontextprovider=oracle.core.ojdl.logging.impl.UserContextImpl -noverify -Doracle.webcenter.analytics.disable-native-partitioning=fal
59001 14066 29425 0 06:47 pts/1 00:00:00 grep 9781
  • 1. Re: managed server is is causing HIGH CPU utilization
    997468 Explorer
    Currently Being Moderated
    Hi,

    The process id/details you have gathered will just tell you pid on which your JVM is running. For debugging high CPU caused by weblogic managed server, you should be checking on the applications deployed on the server.
    At time when you see high CPU utilization, take thread dump on the pid and then using top command get the culprit thread id and check what that thread is executing which caused high CPU.

    For taking thread dump refer to - http://middlewaresupport.wordpress.com/2013/01/23/thread-dump-what-when-and-how/
    For debugging high CPU refer to article - http://middlewaremagic.com/weblogic/?p=4884


    Thanks,
    Ranjan
  • 2. Re: managed server is is causing HIGH CPU utilization
    user10594123 Newbie
    Currently Being Moderated
    Dear Ranjan,

    That link is very helpfull.

    But, some how, I could not find jstack in /u01/APPLTOP/fusionapps/jdk6/bin.

    Can you please let me know, how i need to overcome it
  • 3. Re: managed server is is causing HIGH CPU utilization
    user10594123 Newbie
    Currently Being Moderated
    Dear Ranjan,


    As per the Documnets http://middlewaremagic.com/weblogic/?p=4884. Step8). convert those child PIDs into Hexadecimal values and search in thread dump log.

    I tried those method but somehow, I could not see any nid values in my thread dump log.

    so, my question shall we consider "Active thread " is the culprit for HIGH CPU utilization. If i m wrong guide me correct path.
  • 4. Re: managed server is is causing HIGH CPU utilization
    997468 Explorer
    Currently Being Moderated
    Hi,

    It could be that jstack is not present with the version you are using, but no worries you can select other options mentioned in blog [ http://middlewaresupport.wordpress.com/2013/01/23/thread-dump-what-when-and-how/ ] to gather thread dump.

    For the "nid" not matching your thread dump "nid", did you verified the pid on which you gathered thread dump and top command output. If you are not sure about the process id which is running weblogic server you can use $JAVA_HOME/bin/jps -v | grep -i "weblogic.Server" to get the id. If server has been started using nodemanager then go to servers/<server>/data/nodemanager and check for the pid file.

    Also remember to gather thread dump and top command output simultaneously.
    Steps in blog http://middlewaremagic.com/weblogic/?p=4884 are useful, I was able to match the nid for such scenario in past.

    ## rank it if found useful :) ##

    Thanks,
    Ranjan

    Edited by: Ranjan K on 9 Apr, 2013 7:15 AM

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points