Integration

how is swap? do you have more than 6gb in RAm?


Regards.

Hi,

We have 16GB RAM on the server. so this is not SWAP problem ?

Thank you,
xtanto

what is the limits of the user?
I remember there is a kernel limit for the user. I dont remember if it's ulimit -a. Try to check that is not in unlimited or a high value.


Regards

Edited by: Roberto Barrera on Jan 14, 2009 10:37 AM

Edited by: Roberto Barrera on Jan 14, 2009 10:39 AM

Hi,

I am not at the site now, I am sorry not getting it, but what is the relation between kernel limit for user and CPU utilized 99.9 % ?

Thank you,
xtanto

I been in something like this before but in a Solaris, in solaris you create your kernel projects, and in this you set the maximum amounts for the user in cpu, memory (RAM), file sizes and so on, I can't tell for linux is such as kernel projects, but you should be available to set this amounts, that may not be specificly for kernel. Sorry if I'm confussing you with kernel, but the thing is that you should see if the user is available to use a lot of cpu or memory.

Regards.

Maybe this could help.

Regars

From note 467624.1

Applies to:
Oracle Containers for J2EE - Version: 10.1.2.0.0 to 11.0.0.0
Linux x86
Purpose
To find out the java thread that is taking high CPU usage in OC4J

This article is specific to Linux due to its threading model and makes specific use of parameters for the ps command that only exist in Linux but not in other operating systems. The general idea from this note might, however, be used on other platforms also, but then requires deeper knowledge of that operating system.
Scope and Application

If you are using the JDK 1.4 then you need to set the LD_ASSUME_KERNEL value to 2.4.1 to get the unique nid value.

In JDK 1.5 setting the LD_ASSUME_KERNEL to 2.4.1 is not required.

Add this variable to ORACLE_HOME\opmn\conf\opmn.xml file as follows:
<ias-instance id="j2ee3.host.domain">
<environment>
<variable id="TMP" value="/tmp"/>
<variable id="LD_ASSUME_KERNEL" value="2.4.1"/>
</environment>




Find Out The Thread That Is Taking High CPU Usage In OC4J

Step 1: Find out the Process ID of the OC4J container

You can find out the pid from the output of the following command:

ORACLE_HOME\opmn\bin> opmnctl status

For example
O_H\opmn\bin> opmnctl status

Processes in Instance: j2ee3.host.domain
------------------------------------------------+---------
ias-component | process-type | pid | status
------------------------------------------------+---------
DSA | DSA | N/A | Down
LogLoader | logloaderd | N/A | Down
dcm-daemon | dcm-daemon | N/A | Down
OC4J | home | 28370 | Alive
WebCache | WebCache | 28380 | Alive
WebCache | WebCacheAdmin | 28371 | Alive
HTTP_Server | HTTP_Server | 28373 | Alive

In this note, we will try to trace the java thread that is using high CPU in OC4J 'home'. The pid of 'home' container is 28370 according to the example.

Step 2: Find the actual process ID

Use the following command to find out the actual process ID:
ps -ef | grep 28370
rac 28370 28354 0 11:23 ? 00:00:01 /home/rac/j2ee2/jdk/bin/java
rac 28377 28370 0 11:23 ? 00:00:00 /home/rac/j2ee2/jdk/bin/java

In this case the actual java process is 28377.

Step 3: Find out the thread which takes lot of CPU time

Run the command
watch "ps -eLo pid,ppid,tid,pcpu,comm | grep 28377"

The thread ID of the thread that consumes high CPU time will be shown in the first column of the output


Step 4: Take thread dump to trace the java thread with the thread ID.

Take the thread dump using the following command:
kill -3 28370 (process-id-from-opmn-status-output)

The thread dump will be written to the log file at ORACLE_HOME\opmn\logs\OC4J~<OC4J_Container_Name>~default_island~1.


Step 5: Trace the java thread.

Convert the Thread ID you got from the command,
ps -eLo pid,ppid,tid,pcpu,comm | grep 28377

to hexadecimal and then search for this value in the thread dump.

For example, if you get the thread id '5828' in the first column of this command, convert '5828' to hexadecimal:
5828 (dec) corresponds to 0x16c4 (hex)
Now search for nid=0x16c4 in the thread dump, and you may find something like:
"RMIServer [0.0.0.0:12401] count:1" prio=1 tid=0xa7c1ee80 nid=0x16c4 runnable [a777e000..a777e22c]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0xac0d5b50> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at com.evermind.server.rmi.RMIServer.run(RMIServer.java:464)
at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:186)
at java.lang.Thread.run(Thread.java:534)

Hi Roberto,
Thank you for your very helpful answer..

Another question is, can I do the same thing like your answer using AD4J ?

Thank you,
xtanto

Thats a good question, I'm not familiar with AD4J. But it shoud work with as it work with OC4J as well

Regards

Hi Mr Roberto,

After following the step, here is I get from the thread dump :

"DestroyJavaVM" prio=1 tid=0x0000002ba444a4e0 nid=0x7e58 waiting on condition [0x0000000000000000..0x0000007fbfffd2d0]

This is the process that took 51% CPU.

and still the total cpu taken by Java is 99.99%.

Is there any clue what does this mean ?

Thank you very much,
xtanto

Did you implement a garbage collector optiocn for java? This might be a reason, that proces wants to destroy the JVM but it can't or take to much time for it, did you try to make any king of tuning for OC4J container where you have this application?

Greetings.

Hi Mr Roberto,
No, I havent done any GC tuning, should I do incremental GC ?
YES, below is the setting from opmn.xml :
<category id="start-parameters">
<data id="java-options" value="-server -XX:CompileCommand=exclude,oracle/sql/NUMBER,toBytes -ms3072M
-mx3072M -XX:+AggressiveHeap -XX:MaxPermSize=1024M -XX:AppendRatio=3
-Djava.security.policy=$ORACLE_HOME/j2ee/ittas/config/java2.policy -Djava.awt.headless=true -Dhttp.webdir.enable=false"/>
</category>


Thank you,
xtanto