10 Replies Latest reply: Dec 17, 2012 6:15 AM by Ganesh.. RSS

    The WL server causes the use of CPU to reach to 100%

    Ferez
      Hi

      I am using WebLogic Server Version 10.3.4.0. I got 3 applications deployed in this server, two of which developed by ADF technology and the other one by GWT. I start the server with 10GB of RAM assigned to it and after a while (sometimes one day, some other times 5 hours) the weblogic server causes the use of the CPU to reach to 100% and as a result the server sort of doesn't response anymore. The number of users is not very large, maybe 10~20 users for each application. I have no idea what is the cause of this, configuration of wl server or one of the applications. How could I find out? Do you have any idea?

      Anyway, my system info is as follows:
      CPU: Intel(R) Xeon(R) CPU 5160 @ 3.00GHz with 4 cores
      RAM: 32GB
      OS: Red Hat Enterprise Linux Server release 5.7, x86_64, 2.6.18

      Thanks,
      Ferez
        • 1. Re: The WL server causes the use of CPU to reach to 100%
          Kalyan Pasupuleti-Oracle
          Hi,


          1. Get the top output and find the PID associated with your userID that started WLS that is using up the CPU.

          2. Take several thread dumps of the WebLogic Server via kill -3 <PID>

          3. Convert the PID number from the first step to a hex value.

          (The JVM for Linux implements Java threads as native threads, which results in each thread being a separate Linux process.)

          4. Search the thread dumps for the corresponding value after "nid=" that match the hex number from the previous step.

          This will give you the thread that is causing the High CPU issue.

          5. Determine what exactly happening that particular class.


          Regards,
          Kal
          • 2. Re: The WL server causes the use of CPU to reach to 100%
            Pierluigi Vernetto
            mmm you might find it simpler to use a profiling tool - even JConsole, better still yourkit - to identify hot methods
            • 3. Re: The WL server causes the use of CPU to reach to 100%
              147443
              Also to add to the list :

              - Your sever may have 10G RAM but you need to find out how much memory is dedicated to each managed server.
              ps -ef |grep -i weblogic

              - Please check managed server log and find out how often Java Garbage collection is run

              - What Java vendor you are using - JRockIt, Sun or ....

              ps -ef |grep -i java
              which java
              java -version
              • 4. Re: The WL server causes the use of CPU to reach to 100%
                Ganesh..
                Hi,
                You can follow this link
                http://gskblogs.blogspot.in/2012/09/thread-analysis-in-weblogic-on-linux.html
                It will give you the exact lines of code in which the cpu utilization is high.
                • 5. Re: The WL server causes the use of CPU to reach to 100%
                  Ferez
                  Thank you all for the replies, they were very helpful and I actually used every single one of them to find out the following:

                  Part of the "ps –ef | java" command’s output shows that I am using jrockit-jdk1.6.0_20-R28.1.0-4.0.1 and -Xms10240m, -Xmx10240m options are set.

                  Using "top –H –p 3889" command (3889 is the process id obtained from the previous "ps –ef | java" command) shows that 4 "GC Worker Thread" process is run for each CPU core and they take 100% of the CPU time.

                  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
                  3895 root 25 0 13.1g 11g 21m R 99.9 38.1 15:54.95 (GC Worker Thre
                  3896 root 25 0 13.1g 11g 21m R 99.9 38.1 15:50.55 (GC Worker Thre
                  3893 root 25 0 13.1g 11g 21m R 98.6 38.1 15:42.09 (GC Worker Thre
                  3894 root 19 0 13.1g 11g 21m R 98.6 38.1 15:43.10 (GC Worker Thre
                  3899 root 20 4 13.1g 11g 21m S 1.3 38.1 7:07.59 (VM Periodic Ta
                  3889 root 18 0 13.1g 11g 21m S 0.0 38.1 0:00.00 java
                  3890 root 16 0 13.1g 11g 21m S 0.0 38.1 0:04.96 java
                  3891 root 16 0 13.1g 11g 21m S 0.0 38.1 0:00.12 (Signal Handler
                  3892 root 15 0 13.1g 11g 21m S 0.0 38.1 0:17.36 (OC Main Thread


                  I used JRockit Mission Control to see what happens. It turns out that after a while the 10GB memory gets occupied, GC starts to work to make room for the app but it could free up memory just about 1GB and the memory gets occupied soon and GC comes into play again and this process gets repeated again and again and it keeps the CPU usage near 100%. What is wrong? Is it because of memory leaks? How could I find out which application (and which part in the application) causes this? I am stuck and I really appreciate your help.

                  The following is part of Thread Stack Dump of the server:

                  This page displays the current stacks for each thread.

                  ===== FULL THREAD DUMP ===============

                  Sun Nov 11 12:27:49 2012

                  Oracle JRockit(R) R28.1.0-123-138454-1.6.0_20-20101014-1350-linux-x86_64

                  "Main Thread" id=1 idx=0x4 tid=3890 prio=5 alive, waiting, native_blocked

                  -- Waiting for notification on: weblogic/t3/srvr/T3Srvr@0x22985f770[fat lock]

                  at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

                  at java/lang/Object.wait(J)V(Native Method)

                  at java/lang/Object.wait(Object.java:485)

                  at weblogic/t3/srvr/T3Srvr.waitForDeath(T3Srvr.java:979)

                  ^-- Lock released while waiting: weblogic/t3/srvr/T3Srvr@0x22985f770[fat lock]

                  at weblogic/t3/srvr/T3Srvr.run(T3Srvr.java:488)

                  at weblogic/Server.main(Server.java:71)

                  at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                  -- end of trace

                  "(Signal Handler)" id=2 idx=0x8 tid=3891 prio=5 alive, native_blocked, daemon

                  "(OC Main Thread)" id=3 idx=0xc tid=3892 prio=5 alive, native_blocked, daemon

                  "(GC Worker Thread 1)" id=? idx=0x10 tid=3893 prio=5 alive, daemon

                  "(GC Worker Thread 2)" id=? idx=0x14 tid=3894 prio=5 alive, daemon

                  "(GC Worker Thread 3)" id=? idx=0x18 tid=3895 prio=5 alive, daemon

                  "(GC Worker Thread 4)" id=? idx=0x1c tid=3896 prio=5 alive, daemon

                  "(Code Generation Thread 1)" id=4 idx=0x20 tid=3897 prio=5 alive, native_waiting, daemon

                  "(Code Optimization Thread 1)" id=5 idx=0x24 tid=3898 prio=5 alive, native_waiting, daemon

                  "(VM Periodic Task)" id=6 idx=0x28 tid=3899 prio=10 alive, in native, daemon

                  "Finalizer" id=7 idx=0x2c tid=3900 prio=8 alive, native_blocked, daemon

                  at jrockit/memory/Finalizer.waitForFinalizees(J[Ljava/lang/Object;)I(Native Method)
                           
                                  at jrockit/memory/Finalizer.access$700(Finalizer.java:12)
                           
                                  at jrockit/memory/Finalizer$4.run(Finalizer.java:189)
                           
                                  at java/lang/Thread.run(Thread.java:619)
                           
                                  at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
                           
                                  -- end of trace
                           
                              "Reference Handler" id=8 idx=0x30 tid=3901 prio=10 alive, native_blocked, daemon
                           
                                  at java/lang/ref/Reference.waitForActivatedQueue(J)Ljava/lang/ref/Reference;(Native Method)
                           
                                  at java/lang/ref/Reference.access$100(Reference.java:11)
                           
                                  at java/lang/ref/Reference$ReferenceHandler.run(Reference.java:82)
                           
                                  at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
                           
                                  -- end of trace
                           
                              "(Sensor Event Thread)" id=9 idx=0x34 tid=3902 prio=5 alive, native_blocked, daemon
                           
                              "VM JFR Buffer Thread" id=10 idx=0x38 tid=3903 prio=5 alive, in native, daemon
                           
                              "Timer-0" id=13 idx=0x3c tid=3906 prio=5 alive, waiting, native_blocked, daemon
                           
                                  -- Waiting for notification on: java/util/TaskQueue@0x221261da8[fat lock]

                  at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

                  at java/lang/Object.wait(J)V(Native Method)

                  at java/lang/Object.wait(Object.java:485)

                  at java/util/TimerThread.mainLoop(Timer.java:483)

                  ^-- Lock released while waiting: java/util/TaskQueue@0x221261da8[fat lock]

                  at java/util/TimerThread.run(Timer.java:462)

                  at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                  -- end of trace

                  "Timer-1" id=14 idx=0x40 tid=3907 prio=5 alive, waiting, native_blocked, daemon

                  -- Waiting for notification on: java/util/TaskQueue@0x221262160[fat lock]

                  at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

                  at java/lang/Object.wait(J)V(Native Method)

                  at java/util/TimerThread.mainLoop(Timer.java:509)

                  ^-- Lock released while waiting: java/util/TaskQueue@0x221262160[fat lock]

                  at java/util/TimerThread.run(Timer.java:462)

                  at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                  -- end of trace

                  "[STUCK] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" id=15 idx=0x44 tid=3908 prio=1 alive, native_blocked, daemon

                  at jrockit/net/SocketNativeIO.readBytesPinned(Ljava/io/FileDescriptor;[BIII)I(Native Method)
                           
                                  at jrockit/net/SocketNativeIO.socketRead(SocketNativeIO.java:32)[inlined]

                  at java/net/SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(SocketInputStream.java)[inlined]

                  at java/net/SocketInputStream.read(SocketInputStream.java:129)[optimized]

                  at oracle/net/nt/MetricsEnabledInputStream.read(TcpNTAdapter.java:718)[optimized]

                  at oracle/net/ns/Packet.receive(Packet.java:295)[optimized]

                  at oracle/net/ns/DataPacket.receive(DataPacket.java:106)

                  at oracle/net/ns/NetInputStream.getNextPacket(NetInputStream.java:317)[optimized]

                  at oracle/net/ns/NetInputStream.read(NetInputStream.java:262)[optimized]

                  at oracle/net/ns/NetInputStream.read(NetInputStream.java:187)[inlined]

                  at oracle/net/ns/NetInputStream.read(NetInputStream.java:104)[optimized]

                  at oracle/jdbc/driver/T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:126)[inlined]

                  at oracle/jdbc/driver/T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:82)[optimized]

                  at oracle/jdbc/driver/T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1177)[optimized]

                  at oracle/jdbc/driver/T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1153)[inlined]

                  at oracle/jdbc/driver/T4CTTIfun.receive(T4CTTIfun.java:312)[optimized]

                  at oracle/jdbc/driver/T4CTTIfun.doRPC(T4CTTIfun.java:204)[optimized]

                  at oracle/jdbc/driver/T4C8Oall.doOALL(T4C8Oall.java:540)[inlined]

                  at oracle/jdbc/driver/T4CPreparedStatement.doOall8(T4CPreparedStatement.java:217)[inlined]

                  at oracle/jdbc/driver/T4CPreparedStatement.fetch(T4CPreparedStatement.java:1168)[optimized]

                  at oracle/jdbc/driver/OracleResultSetImpl.close_or_fetch_from_next(OracleResultSetImpl.java:628)[inlined]

                  at oracle/jdbc/driver/OracleResultSetImpl.next(OracleResultSetImpl.java:490)[optimized]

                  ^-- Holding lock: oracle/jdbc/driver/T4CConnection@0x285c6cfe0[recursive]

                  ^-- Holding lock: oracle/jdbc/driver/T4CConnection@0x285c6cfe0[thin lock]

                  at weblogic/jdbc/wrapper/ResultSet_oracle_jdbc_driver_OracleResultSetImpl.next()Z(Unknown Source)[optimized]

                  at oracle/jbo/server/QueryCollection.hasNextInResultSet(QueryCollection.java:4611)[inlined]

                  at oracle/jbo/server/ViewObjectImpl.hasNextForCollection(ViewObjectImpl.java:6899)[optimized]

                  at oracle/jbo/server/QueryCollection.hasNext(QueryCollection.java:4579)[inlined]

                  at oracle/jbo/server/QueryCollection.populateRow(QueryCollection.java:3553)[optimized]

                  at oracle/jbo/server/QueryCollection.fetch(QueryCollection.java:3387)[optimized]

                  ^-- Holding lock: oracle/jbo/JboSyncLock@0x2ca5f7460[recursive]

                  at oracle/jbo/server/QueryCollection.get(QueryCollection.java:2188)[optimized]

                  at oracle/jbo/server/ViewRowSetImpl.getRow(ViewRowSetImpl.java:5016)[optimized]

                  at oracle/jbo/server/ViewRowSetImpl.getRow(ViewRowSetImpl.java:3242)

                  at oracle/jbo/server/ViewObjectImpl.activateCurrentRow(ViewObjectImpl.java:18786)

                  at oracle/jbo/server/ViewRowSetIteratorImpl.activateIteratorState(ViewRowSetIteratorImpl.java:3947)[optimized]

                  ^-- Holding lock: oracle/jbo/JboSyncLock@0x2ca5f7460[recursive]

                  at oracle/jbo/server/ViewRowSetImpl.activateIteratorState(ViewRowSetImpl.java:7235)

                  at oracle/jbo/server/ViewRowSetImpl.doCreateAndInitRow(ViewRowSetImpl.java:2465)

                  ^-- Holding lock: oracle/jbo/JboSyncLock@0x2ca5f7460[thin lock]

                  at oracle/jbo/server/ViewRowSetImpl.createRow(ViewRowSetImpl.java:2455)

                  at oracle/jbo/server/ViewObjectImpl.createRow(ViewObjectImpl.java:10631)

                  at org/nict/oms11g/view/backingbeans/exit/ReturnCartableBean.processRow(ReturnCartableBean.java:435)

                  at org/nict/oms11g/view/backingbeans/exit/ReturnCartableBean.notUsed(ReturnCartableBean.java:102)

                  at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                  at jrockit/vm/Reflect.invokeMethod(Ljava/lang/Object;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Native Method)
                           
                                  at sun/reflect/NativeMethodAccessorImpl.invoke0(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Native Method)
                           
                                  at sun/reflect/NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                           
                                  at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[optimized]

                  at java/lang/reflect/Method.invoke(Method.java:597)[optimized]

                  at com/sun/el/parser/AstValue.invoke(Lcom/sun/el/lang/EvaluationContext;[Ljava/lang/Class;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)
                           
                                  at com/sun/el/MethodExpressionImpl.invoke(Ljavax/el/ELContext;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)
                           
                                  at org/apache/myfaces/trinidadinternal/taglib/util/MethodExpressionMethodBinding.invoke(MethodExpressionMethodBinding.java:53)
                           
                                  at org/apache/myfaces/trinidad/component/UIXComponentBase.broadcastToMethodBinding(UIXComponentBase.java:1256)
                           
                                  at org/apache/myfaces/trinidad/component/UIXCommand.broadcast(UIXCommand.java:183)
                           
                                  at oracle/adf/view/rich/component/fragment/ContextSwitchingComponent$1.run(ContextSwitchingComponent.java:92)
                           
                                  at oracle/adf/view/rich/component/fragment/ContextSwitchingComponent._processPhase(ContextSwitchingComponent.java:361)
                           
                                  at oracle/adf/view/rich/component/fragment/ContextSwitchingComponent.broadcast(ContextSwitchingComponent.java:96)
                           
                                  at oracle/adf/view/rich/component/fragment/UIXInclude.broadcast(UIXInclude.java:102)
                           
                                  at oracle/adf/view/rich/component/fragment/ContextSwitchingComponent$1.run(ContextSwitchingComponent.java:92)
                           
                                  at oracle/adf/view/rich/component/fragment/ContextSwitchingComponent._processPhase(ContextSwitchingComponent.java:361)
                           
                                  at oracle/adf/view/rich/component/fragment/ContextSwitchingComponent.broadcast(ContextSwitchingComponent.java:96)
                           
                                  at oracle/adf/view/rich/component/fragment/UIXInclude.broadcast(UIXInclude.java:96)
                           
                                  at oracle/adf/view/rich/component/fragment/UIXRegion.broadcast(UIXRegion.java:148)
                           
                                  at oracle/adf/view/rich/component/fragment/ContextSwitchingComponent$1.run(ContextSwitchingComponent.java:92)
                           
                                  at oracle/adf/view/rich/component/fragment/ContextSwitchingComponent._processPhase(ContextSwitchingComponent.java:361)
                           
                                  at oracle/adf/view/rich/component/fragment/ContextSwitchingComponent.broadcast(ContextSwitchingComponent.java:96)
                           
                                  at oracle/adf/view/rich/component/fragment/UIXInclude.broadcast(UIXInclude.java:102)
                           
                                  at oracle/adf/view/rich/component/fragment/ContextSwitchingComponent$1.run(ContextSwitchingComponent.java:92)
                           
                                  at oracle/adf/view/rich/component/fragment/ContextSwitchingComponent._processPhase(ContextSwitchingComponent.java:361)
                           
                                  at oracle/adf/view/rich/component/fragment/ContextSwitchingComponent.broadcast(ContextSwitchingComponent.java:96)
                           
                                  at oracle/adf/view/rich/component/fragment/UIXInclude.broadcast(UIXInclude.java:96)
                           
                                  at oracle/adfinternal/view/faces/lifecycle/LifecycleImpl.broadcastEvents(LifecycleImpl.java:879)
                           
                                  at oracle/adfinternal/view/faces/lifecycle/LifecycleImpl._executePhase(LifecycleImpl.java:312)[optimized]

                  at oracle/adfinternal/view/faces/lifecycle/LifecycleImpl.execute(LifecycleImpl.java:185)

                  at javax/faces/webapp/FacesServlet.service(FacesServlet.java:265)

                  at weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)

                  at weblogic/servlet/internal/StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)[inlined]

                  at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:300)[optimized]

                  at weblogic/servlet/internal/TailFilter.doFilter(TailFilter.java:26)[optimized]

                  at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]

                  at oracle/adf/model/servlet/ADFBindingFilter.doFilter(ADFBindingFilter.java:205)[optimized]

                  at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]

                  at oracle/adfinternal/view/faces/webapp/rich/RegistrationFilter.doFilter(RegistrationFilter.java:106)

                  at org/apache/myfaces/trinidadinternal/webapp/TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:446)

                  at oracle/adfinternal/view/faces/activedata/AdsFilter.doFilter(AdsFilter.java:60)

                  at org/apache/myfaces/trinidadinternal/webapp/TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:446)

                  at org/apache/myfaces/trinidadinternal/webapp/TrinidadFilterImpl._doFilterImpl(TrinidadFilterImpl.java:271)

                  at org/apache/myfaces/trinidadinternal/webapp/TrinidadFilterImpl.doFilter(TrinidadFilterImpl.java:177)

                  at org/apache/myfaces/trinidad/webapp/TrinidadFilter.doFilter(TrinidadFilter.java:92)

                  at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]

                  at oracle/jheadstart/controller/jsf/AuthenticationFilter.doFilter(AuthenticationFilter.java:282)

                  at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]

                  at oracle/security/jps/ee/http/JpsAbsFilter$1.run(JpsAbsFilter.java:111)

                  at jrockit/vm/AccessController.doPrivileged(AccessController.java:254)[inlined]

                  at oracle/security/jps/util/JpsSubject.doAsPrivileged(JpsSubject.java:313)[inlined]

                  at oracle/security/jps/ee/util/JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:413)[inlined]

                  at oracle/security/jps/ee/http/JpsAbsFilter.runJaasMode(JpsAbsFilter.java:94)[inlined]

                  at oracle/security/jps/ee/http/JpsAbsFilter.doFilter(JpsAbsFilter.java:161)[optimized]

                  at oracle/security/jps/ee/http/JpsFilter.doFilter(JpsFilter.java:71)

                  at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]

                  at oracle/dms/servlet/DMSServletFilter.doFilter(DMSServletFilter.java:136)[optimized]

                  at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]

                  at weblogic/servlet/internal/RequestEventsFilter.doFilter(RequestEventsFilter.java:27)

                  at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[inlined]

                  at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3715)[optimized]

                  at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)

                  at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)[inlined]

                  at weblogic/security/service/SecurityManager.runAs(SecurityManager.java:120)[optimized]

                  at weblogic/servlet/internal/WebAppServletContext.securedExecute(WebAppServletContext.java:2277)

                  at weblogic/servlet/internal/WebAppServletContext.execute(WebAppServletContext.java:2183)

                  at weblogic/servlet/internal/ServletRequestImpl.run(ServletRequestImpl.java:1454)[optimized]

                  at weblogic/work/ExecuteThread.execute(ExecuteThread.java:207)[optimized]

                  at weblogic/work/ExecuteThread.run(ExecuteThread.java:176)

                  at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

                  -- end of trace

                  ...
                  • 6. Re: The WL server causes the use of CPU to reach to 100%
                    Ganesh..
                    Hi,
                    The stacktrace shows that the thread 3908 is stuck waiting for the reply from the database server.
                    A couple of things :
                    1. Check the database and network performance.
                    2. Which GC strategy are you using?
                    You can use Jrockit memory leak detector to check any memory leaks in your code.
                    • 7. Re: The WL server causes the use of CPU to reach to 100%
                      Ferez
                      Hi

                      1- It seems that the database and network performance is fine.

                      2- Here is my VM and GC related info as is shown in the WLS console

                      --->Java Virtual Machine Memory Utilization Statistics

                      Heap Size Current: 10737418240
                      Heap Free Current: 2967018664
                      Heap Free Percent: 27
                      Heap Size Max: 10737418240
                      Total Physical Memory: 33752182784

                      --->Garbage Collection Statistics

                      GC Algorithm: Dynamic GC, current strategy: Generational Parallel Mark & Sweep, generational=true, sweep=parallel, mark=parallel
                      Total GC Count: 6
                      Last GC Start: 12/11/12 10:22:54 AM IRST
                      Last GC End: 12/11/12 10:22:55 AM IRST
                      Total GC Time: 1349
                      Handles Compaction: true
                      Concurrent: false
                      Generational: true
                      Incremental: false
                      Parallel: true

                      --->Java Virtual Machine Processor Utilization Statistics

                      Total Number Of Threads: 66
                      Number Of Daemon Threads: 60
                      Number Of Processors: 4
                      All Processors Average Load: 0.25
                      JVM Processor Load: 0.25
                      --------------------------------------------------------

                      Going through thread 3908 stack trace shows that line 435 in ReturnCartableBean.java in my application is sort of responsible of this. Am I right? I guess it from the following section in the dump:

                      at org/nict/oms11g/view/backingbeans/exit/ReturnCartableBean.processRow(ReturnCartableBean.java:435)

                      This line in my application is:
                      oracle.jbo.Row newRow = myViewObject.createRow();

                      What is wrong with this statement? Is this problematic? It seems that it works fine at first but it causes problems after a while.

                      Cheers,
                      Ferez
                      • 8. Re: The WL server causes the use of CPU to reach to 100%
                        Ganesh..
                        Hi,

                        at oracle/jbo/server/ViewObjectImpl.createRow(ViewObjectImpl.java:10631)

                        at org/nict/oms11g/view/backingbeans/exit/ReturnCartableBean.processRow(ReturnCartableBean.java:435)

                        You are right buddy, this is the line where the thread is getting stuck. And there is nothing wrong with the statement you gave.
                        If you follow the stacktrace, it is waiting for T4CPreparedStatement.fetch over network.
                        In general case scenario, ViewObject.createRow should not cause stuck threads. I will recommend you to check for application performance tuning, mainly AppModule tuning. Check this link :
                        http://docs.oracle.com/cd/E15523_01/web.1111/b31974/bcampool.htm
                        • 9. Re: The WL server causes the use of CPU to reach to 100%
                          Ferez
                          Hi

                          Thank you very much Ganesh, I am going to check the AM tuning as you suggested but one more question. How could you get the fact that T4CPreparedStatement.fetch is the cause of problem? I mean when I follow the stacktrace to the top (starting from ViewObjectImpl.createRow) I could see lines such as:
                          ^-- Holding lock: oracle/jbo/JboSyncLock@0x2ca5f7460[thin lock]
                          or this one
                          ^-- Holding lock: oracle/jdbc/driver/T4CConnection@0x285c6cfe0[thin lock]

                          but for me there is nothing wrong with this line
                          at oracle/jdbc/driver/T4CPreparedStatement.fetch(T4CPreparedStatement.java:1168)[optimized]

                          How do you know what is the cause of problem?

                          Regards,
                          Ferez
                          • 10. Re: The WL server causes the use of CPU to reach to 100%
                            Ganesh..
                            Hi,

                            From the stacktrace it is visible that the thread is stuck in createRow() method. This method is trying to fetch some data from DB with T4CPreparedStatement.fetch() this call. Going deeper, you can say the thread is stuck while fetching some data which is not something very often. So this means, there is seriously something wrong with either the connection or the tuning of application or database.