This discussion is archived
10 Replies Latest reply: Dec 4, 2012 1:44 PM by 976109 RSS

Weblogic Health status goes to "Warning"

976109 Newbie
Currently Being Moderated
Hello All,

I am working in a company in which we receive thousands of requests per day, when there are peek times, some of our weblogic servers reports the health status as "warning", we are using clustering and load balancing, i have searched for possible solutions on the internet but they all indicates the tuning of weblogic e.g increasing the time interval, stuck thread count etc etc.....

is there any solid solution available to this problem, means which could really solve this, i tried many of these but unable to solve the problem, every day to solve the problem i have to bounce back "restart the weblogic instance", so please help me in this problem so that i do not have to restart my weblogic server instance again and again.

your help is highly appreciated.

P.s: we are using weblogic 10.3.5

Regards.

Mansoor.
  • 1. Re: Weblogic Health status goes to "Warning"
    Fabian Pro
    Currently Being Moderated
    Hi Mansoor,

    Can you share the traces of your logs when the managed server goes to warning state and before restarting the instances.
    So that we can analyze the reason why does the server goes to warning state.

    Regards
    Fabian
  • 2. Re: Weblogic Health status goes to "Warning"
    919205 Newbie
    Currently Being Moderated
    Warning state of server indicate the server's idle threads threshold level has reached(In another phrase more stuck threads in the pool), so fixing the issue you may have to look at the thread dumps and Tune from [Server-> Configuration -> Overload]

    -Aj
  • 3. Re: Weblogic Health status goes to "Warning"
    976109 Newbie
    Currently Being Moderated
    Hello,

    here is some memory dump log is given..

    please analyze..


    2012/11/28 04:17:48.668 INFO <(SOAPHandler.java:116)> [1112013899] -
    ******** soapMessage in fault ******** SOAPMessageImpl[SOAPPartImpl[SOAPEnvelopeImpl[nullSOAPBody[<env:Fault><faultcode>env:Server</faultcode><faultstring>Exception during processing: java.lang.OutOfMemoryError: allocLargeObjectOrArray - Object size: 4112, Num elements: 1024 (see Fault Detail for stacktrace)</faultstring><detail><bea_fault:stacktrace xmlns:bea_fault="http://www.bea.com/servers/wls70/webservice/fault/1.0.0">java.lang.OutOfMemoryError: allocLargeObjectOrArray - Object size: 4112, Num elements: 1024
         at weblogic.utils.collections.CircularQueue.expandQueue(CircularQueue.java:90)
         at weblogic.utils.collections.CircularQueue.add(CircularQueue.java:108)
         at weblogic.xml.babel.stream.XMLInputStreamBase.add(XMLInputStreamBase.java:151)
         at weblogic.xml.babel.adapters.DOMAdapter.adaptElement(DOMAdapter.java:160)
         at weblogic.xml.babel.adapters.DOMAdapter.adapt(DOMAdapter.java:71)
         at weblogic.xml.babel.adapters.DOMAdapter.recurseChildren(DOMAdapter.java:197)
         at weblogic.xml.babel.adapters.DOMAdapter.adaptElement(DOMAdapter.java:161)
         at weblogic.xml.babel.adapters.DOMAdapter.adapt(DOMAdapter.java:71)
         at weblogic.xml.babel.adapters.DOMAdapter.recurseChildren(DOMAdapter.java:197)
         at weblogic.xml.babel.adapters.DOMAdapter.adaptElement(DOMAdapter.java:161)
         at weblogic.xml.babel.adapters.DOMAdapter.adapt(DOMAdapter.java:71)
         at weblogic.xml.babel.adapters.DOMAdapter.recurseChildren(DOMAdapter.java:197)
         at weblogic.xml.babel.adapters.DOMAdapter.adaptElement(DOMAdapter.java:161)
         at weblogic.xml.babel.adapters.DOMAdapter.adapt(DOMAdapter.java:71)
         at weblogic.xml.babel.adapters.DOMAdapter.adaptDocument(DOMAdapter.java:96)
         at weblogic.xml.babel.adapters.DOMAdapter.adapt(DOMAdapter.java:64)
         at weblogic.xml.babel.stream.DOMInputStream.open(DOMInputStream.java:53)
         at weblogic.xml.babel.stream.XMLInputStreamFactoryImpl.newInputStream(XMLInputStreamFactoryImpl.java:95)
         at weblogic.xml.schema.binding.internal.builtin.DocumentCodec.serialize(DocumentCodec.java:53)
         at weblogic.xml.schema.binding.RuntimeUtils.invoke_serializer(RuntimeUtils.java:192)
         at weblogic.xml.schema.binding.RuntimeUtils.invoke_serializer(RuntimeUtils.java:178)
         at weblogic.webservice.core.DefaultPart.invokeSerializer(DefaultPart.java:335)
         at weblogic.webservice.core.DefaultPart.toXML(DefaultPart.java:304)
         at weblogic.webservice.core.DefaultMessage.toXML(DefaultMessage.java:651)
         at weblogic.webservice.core.handler.InvokeHandler.handleRequest(InvokeHandler.java:151)
         at weblogic.webservice.core.HandlerChainImpl.handleRequest(HandlerChainImpl.java:144)
         at weblogic.webservice.core.DefaultOperation.process(DefaultOperation.java:551)
         at weblogic.webservice.server.Dispatcher.process(Dispatcher.java:204)
         at weblogic.webservice.server.Dispatcher.doDispatch(Dispatcher.java:175)
         at weblogic.webservice.server.Dispatcher.dispatch(Dispatcher.java:97)
         at weblogic.webservice.server.WebServiceManager.dispatch(WebServiceManager.java:102)
         at weblogic.webservice.server.servlet.WebServiceServlet.serverSideInvoke(WebServiceServlet.java:321)
         at weblogic.webservice.server.servlet.ServletBase.doPost(ServletBase.java:454)
         at weblogic.webservice.server.servlet.WebServiceServlet.doPost(WebServiceServlet.java:293)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:821)
         at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
         at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:292)
         at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3594)
         at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
         at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2108)
         at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1432)
         at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
         at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
    </bea_fault:stacktrace></detail></env:Fault>]]]]

    2012/11/28 04:21:38.893 INFO <(SOAPHandler.java:116)> [2145221434] -
    ******** soapMessage in fault ******** SOAPMessageImpl[SOAPPartImpl[SOAPEnvelopeImpl[nullSOAPBody[<env:Fault><faultcode>env:Server</faultcode><faultstring>Exception during processing: java.lang.OutOfMemoryError: allocLargeObjectOrArray - Object size: 235368, Num elements: 58837 (see Fault Detail for stacktrace)</faultstring><detail><bea_fault:stacktrace xmlns:bea_fault="http://www.bea.com/servers/wls70/webservice/fault/1.0.0">java.lang.OutOfMemoryError: allocLargeObjectOrArray - Object size: 235368, Num elements: 58837
         at java.util.Arrays.copyOf(Arrays.java:2760)
         at java.util.Arrays.copyOf(Arrays.java:2734)
         at java.util.ArrayList.ensureCapacity(ArrayList.java:167)
         at java.util.ArrayList.add(ArrayList.java:351)
         at weblogic.xml.xmlnode.XMLNode.addChild(XMLNode.java:479)
         at weblogic.xml.xmlnode.XMLNode.addChild(XMLNode.java:435)
         at weblogic.xml.xmlnode.XMLNode.addChild(XMLNode.java:514)
         at weblogic.xml.xmlnode.XMLNodeWriter.write(XMLNodeWriter.java:55)
         at weblogic.xml.xmlnode.XMLNodeWriter.write(XMLNodeWriter.java:105)
         at weblogic.xml.babel.stream.XMLOutputStreamBase.writeXMLEvent(XMLOutputStreamBase.java:77)
         at weblogic.xml.babel.stream.XMLOutputStreamBase.write(XMLOutputStreamBase.java:99)
         at weblogic.xml.babel.stream.XMLOutputStreamBase.flush(XMLOutputStreamBase.java:208)
         at weblogic.xml.babel.stream.XMLOutputStreamBase.flush(XMLOutputStreamBase.java:215)
         at weblogic.xml.babel.stream.XMLOutputStreamBase.addXMLEvent(XMLOutputStreamBase.java:125)
         at weblogic.xml.babel.stream.XMLOutputStreamBase.add(XMLOutputStreamBase.java:173)
         at weblogic.xml.schema.binding.internal.builtin.DocumentCodec.serialize(DocumentCodec.java:71)
         at weblogic.xml.schema.binding.RuntimeUtils.invoke_serializer(RuntimeUtils.java:192)
         at weblogic.xml.schema.binding.RuntimeUtils.invoke_serializer(RuntimeUtils.java:178)
         at weblogic.webservice.core.DefaultPart.invokeSerializer(DefaultPart.java:335)
         at weblogic.webservice.core.DefaultPart.toXML(DefaultPart.java:304)
         at weblogic.webservice.core.DefaultMessage.toXML(DefaultMessage.java:651)
         at weblogic.webservice.core.handler.InvokeHandler.handleRequest(InvokeHandler.java:151)
         at weblogic.webservice.core.HandlerChainImpl.handleRequest(HandlerChainImpl.java:144)
         at weblogic.webservice.core.DefaultOperation.process(DefaultOperation.java:551)
         at weblogic.webservice.server.Dispatcher.process(Dispatcher.java:204)
         at weblogic.webservice.server.Dispatcher.doDispatch(Dispatcher.java:175)
         at weblogic.webservice.server.Dispatcher.dispatch(Dispatcher.java:97)
         at weblogic.webservice.server.WebServiceManager.dispatch(WebServiceManager.java:102)
         at weblogic.webservice.server.servlet.WebServiceServlet.serverSideInvoke(WebServiceServlet.java:321)
         at weblogic.webservice.server.servlet.ServletBase.doPost(ServletBase.java:454)
         at weblogic.webservice.server.servlet.WebServiceServlet.doPost(WebServiceServlet.java:293)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:821)
         at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
         at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:292)
         at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3594)
         at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
         at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2108)
         at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1432)
         at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
         at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
    </bea_fault:stacktrace></detail></env:Fault>]]]]

    Your help is highly appreciated.

    Regards.

    Mansoor
  • 4. Re: Weblogic Health status goes to "Warning"
    Fabian Pro
    Currently Being Moderated
    Hi ,

    There is Out of Memory happening on your each jvm
    java.lang.OutOfMemoryError: allocLargeObjectOrArray - Object size: 4112, Num elements: 1024
    Can you give us more details.
    1) You have installed weblogic on 32bit machine or 64 bit machine??
    2) what is the memory arguments for your managed server???

    Just for knowledge

    If you are using 64 bit,I would suggest to use 2GB Minimum or 8Gb Max so ideal one would be like this.
    If you are using 32 bit then we cannot increase more than 2 GB there.

    Regards
    Fabian
  • 5. Re: Weblogic Health status goes to "Warning"
    976109 Newbie
    Currently Being Moderated
    Here are some facts given.


    Java Version: 1.6.0_14


    OS Name:     Linux

    OS Version:2.6.18-194.11.4.el5

    JACC Enabled:     false


    Heap Size Current:1572864000

    Heap Free Current: 1152664240

    Heap Free Percent:73

    Heap Size Max:     1572864000

    Total Physical Memory: 8237264896

    please give suggestions.

    Regards.

    Mansoor
  • 6. Re: Weblogic Health status goes to "Warning"
    Fabian Pro
    Currently Being Moderated
    Hi,

    From the information Provided by u,I just got the below details,
    Ur using linux box and your max and min memory allocated to each manage server is 1.5Gb(We can increase this,but need more information)

    To determine whether your weblogic is 32 bit or 64 bit.
    execute the below command
    1)java -version
    2)java -d64 -version

    Let me know whether ur using jdk or jrockit??

    Also give will the last few lines from the server logs when your server state changes to warning.Need entire thread stack for OutOfMemoryError.

    Enable Garbage collection logging(This would be needing a restart of services)

    1.Go to DOMAIN_HOME/bin
    2.Edit setDomainEnv.sh
    3.Go to JAVA_OPTIONS parameter definition

    If your using Jrockit

    Make the changes as below
    JAVA_OPTIONS="${JAVA_OPTIONS} -verbose:gc -Xverboselog:/tmp/gc.log -Xverbose:memory"
    export JAVA_OPTIONS

    if your using Jdk

    Make the changes as below

    JAVA_OPTIONS="${JAVA_OPTIONS} -XX:+PrintGCTimeStamps -Xloggc:/tmp/gc.log -XX:+PrintGCDetails"
    export JAVA_OPTIONS

    4. Save the file
    5. Stop WebLogic managed server
    6. Start WebLogic server using the script.

    By doing the above changes, gc logs will be created in /tmp/gc.log which will help us to analyse whether GC is happening effectively.



    Regards,
    Fabian
  • 7. Re: Weblogic Health status goes to "Warning"
    976109 Newbie
    Currently Being Moderated
    Hi,

    Some of the information which i can access is given below (this task is assigned to me but i am not weblogic admin)

    - we are using jrocket.
    -weblogic is 32 bit.

    A snapshot of server log is given below

    [memory ][Mon Nov 26 14:46:12 2012][07144] Running with 32 bit heap and compressed references.
    [memory ][Mon Nov 26 14:46:12 2012][07144] GC mode: Static, strategy: Generational Concurrent Mark & Sweep
    [memory ][Mon Nov 26 14:46:12 2012][07144] heap size: 1536000K, maximal heap size: 1536000K, nursery size: 384000K
    [memory ][Mon Nov 26 14:46:12 2012][07144] <s>-<end>: GC <before>K-><after>K (<heap>K), <pause> ms
    [memory ][Mon Nov 26 14:46:12 2012][07144] <s/start> - start time of collection (seconds since jvm start)
    [memory ][Mon Nov 26 14:46:12 2012][07144] <end> - end time of collection (seconds since jvm start)
    [memory ][Mon Nov 26 14:46:12 2012][07144] <before> - memory used by objects before collection (KB)
    [memory ][Mon Nov 26 14:46:12 2012][07144] <after> - memory used by objects after collection (KB)
    [memory ][Mon Nov 26 14:46:12 2012][07144] <heap> - size of heap after collection (KB)
    [memory ][Mon Nov 26 14:46:12 2012][07144] <pause> - total sum of pauses during collection (milliseconds)
    [memory ][Mon Nov 26 14:46:12 2012][07144] run with -Xverbose:gcpause to see individual pauses
    [JRockit] Management server started on port 57032, ssl=false, authenticate=false.
    [memory ][Mon Nov 26 14:46:19 2012][07144] 6.891: parallel nursery GC 432860K->161756K (1536000K), 56.218 ms
    [memory ][Mon Nov 26 14:47:14 2012][07144] 62.086: parallel nursery GC 480873K->201412K (1536000K), 35.692 ms

    [memory ][Wed Nov 28 07:51:50 2012][07144] 147933.627-147938.089: GC 1281600K->1276830K (1536000K), sum of pauses 557.423 ms
    [memory ][Wed Nov 28 07:51:55 2012][07144] 147938.168-147942.452: GC 1280000K->1276826K (1536000K), sum of pauses 41.780 ms
    [memory ][Wed Nov 28 07:52:07 2012][07144] 147942.533-147954.871: GC 1286400K->1276836K (1536000K), sum of pauses 8197.144 ms
    [memory ][Wed Nov 28 07:52:07 2012][07144] Throwing OutOfMemory: allocLargeObjectOrArray - Object size: 16777232, Num elements: 4194304
    [memory ][Wed Nov 28 07:52:10 2012][07144] 147954.956-147957.579: GC 1283200K->757041K (1536000K), sum of pauses 340.881 ms
    [memory ][Wed Nov 28 07:52:12 2012][07144] Changing GC strategy to Generational Concurrent Mark & Sweep
    [memory ][Wed Nov 28 07:52:13 2012][07144] 147957.659-147960.479: GC 760241K->861430K (1536000K), sum of pauses 488.982 ms
    [memory ][Wed Nov 28 07:52:16 2012][07144] 147960.566-147963.376: GC 863268K->761200K (1536000K), sum of pauses 366.075 ms
    [memory ][Wed Nov 28 07:53:09 2012][07144] 148013.685-148016.781: GC 813168K->920501K (1536000K), sum of pauses 773.052 ms
    [memory ][Wed Nov 28 08:06:33 2012][07144] 148820.343: parallel nursery GC 1168533K->1018744K (1536000K), 893.296 ms
    [memory ][Wed Nov 28 08:06:34 2012][07144] 148821.616: parallel nursery GC 1377488K->1201126K (1536000K), 503.901 ms
    [memory ][Wed Nov 28 08:06:35 2012][07144] 148822.326: parallel nursery GC 1489509K->1536000K (1536000K), 408.912 ms
    [memory ][Wed Nov 28 08:06:38 2012][07144] Changing GC strategy to Generational Concurrent Mark, Parallel Sweep
    [memory ][Wed Nov 28 08:06:39 2012][07144] 148821.258-148826.868: GC 1052344K->1281600K (1536000K), sum of pauses 1236.701 ms
    [memory ][Wed Nov 28 08:06:44 2012][07144] 148826.964-148831.537: GC 1433600K->1393957K (1536000K), sum of pauses 1067.819 ms
    [memory ][Wed Nov 28 08:06:48 2012][07144] 148831.627-148835.880: GC 1536000K->1483200K (1536000K), sum of pauses 819.866 ms
    [memory ][Wed Nov 28 08:06:52 2012][07144] 148835.958-148840.043: GC 1502400K->1428227K (1536000K), sum of pauses 696.051 ms
    [memory ][Wed Nov 28 08:06:58 2012][07144] 148840.121-148845.840: GC 1428227K->1421591K (1536000K), sum of pauses 2038.479 ms
    [memory ][Wed Nov 28 08:07:12 2012][07144] 148845.919-148859.476: GC 1422400K->1297787K (1536000K), sum of pauses 9836.237 ms
    [memory ][Wed Nov 28 08:07:12 2012][07144] Throwing OutOfMemory: allocLargeObjectOrArray - Object size: 353040, Num elements: 88256
    [memory ][Wed Nov 28 08:07:13 2012][07144] 148859.555-148861.120: GC 1302400K->665578K (1536000K), sum of pauses 226.632 ms
    [memory ][Wed Nov 28 08:07:13 2012][07144] Changing GC strategy to Generational Concurrent Mark & Sweep
    [memory ][Wed Nov 28 08:07:14 2012][07144] 148861.199-148861.419: GC 680200K->157010K (1536000K), sum of pauses 137.983 ms
    [memory ][Wed Nov 28 12:19:20 2012][07144] 163987.323: parallel nursery GC 449982K->141655K (1536000K), 60.356 ms
    [memory ][Wed Nov 28 16:17:21 2012][07144] 178268.807: parallel nursery GC 450605K->163172K (1536000K), 67.881 ms
    [memory ][Wed Nov 28 19:57:51 2012][07144] 191498.731: parallel nursery GC 451275K->163737K (1536000K), 54.393 ms

    A snapshot of managed server thread dump is given below

    2012/11/28 05:08:49.762 INFO <(DBProcessor.java:158)> [1149165250] -
    Execution Time: 4.014

    2012/11/28 05:10:06.928 INFO <(ARInvoiceWebservice.java:81)> [1149165250] -
    Inquiry Webservice overall request execution time: 4821.064

    2012/11/28 05:10:30.652 INFO <(SOAPHandler.java:116)> [1149165250] -
    ******** soapMessage in fault ******** SOAPMessageImpl[SOAPPartImpl[SOAPEnvelopeImpl[nullSOAPBody[<env:Fault><faultcode>env:Server</faultcode><faultstring>Exception during processing: java.lang.OutOfMemoryError: allocLargeObjectOrArray - Object size: 65552, Num elements: 16384 (see Fault Detail for stacktrace)</faultstring><detail><bea_fault:stacktrace xmlns:bea_fault="http://www.bea.com/servers/wls70/webservice/fault/1.0.0">java.lang.OutOfMemoryError: allocLargeObjectOrArray - Object size: 65552, Num elements: 16384
         at weblogic.utils.collections.CircularQueue.expandQueue(CircularQueue.java:90)
         at weblogic.utils.collections.CircularQueue.add(CircularQueue.java:108)
         at weblogic.xml.babel.stream.XMLInputStreamBase.add(XMLInputStreamBase.java:151)
         at weblogic.xml.babel.adapters.DOMAdapter.adaptNamespace(DOMAdapter.java:107)
         at weblogic.xml.babel.adapters.DOMAdapter.adaptAttribute(DOMAdapter.java:127)
         at weblogic.xml.babel.adapters.DOMAdapter.adaptElement(DOMAdapter.java:158)
         at weblogic.xml.babel.adapters.DOMAdapter.adapt(DOMAdapter.java:71)
         at weblogic.xml.babel.adapters.DOMAdapter.recurseChildren(DOMAdapter.java:197)
         at weblogic.xml.babel.adapters.DOMAdapter.adaptElement(DOMAdapter.java:161)
         at weblogic.xml.babel.adapters.DOMAdapter.adapt(DOMAdapter.java:71)
         at weblogic.xml.babel.adapters.DOMAdapter.recurseChildren(DOMAdapter.java:197)
         at weblogic.xml.babel.adapters.DOMAdapter.adaptElement(DOMAdapter.java:161)
         at weblogic.xml.babel.adapters.DOMAdapter.adapt(DOMAdapter.java:71)
         at weblogic.xml.babel.adapters.DOMAdapter.recurseChildren(DOMAdapter.java:197)
         at weblogic.xml.babel.adapters.DOMAdapter.adaptElement(DOMAdapter.java:161)
         at weblogic.xml.babel.adapters.DOMAdapter.adapt(DOMAdapter.java:71)
         at weblogic.xml.babel.adapters.DOMAdapter.adaptDocument(DOMAdapter.java:96)
         at weblogic.xml.babel.adapters.DOMAdapter.adapt(DOMAdapter.java:64)
         at weblogic.xml.babel.stream.DOMInputStream.open(DOMInputStream.java:53)
         at weblogic.xml.babel.stream.XMLInputStreamFactoryImpl.newInputStream(XMLInputStreamFactoryImpl.java:95)
         at weblogic.xml.schema.binding.internal.builtin.DocumentCodec.serialize(DocumentCodec.java:53)
         at weblogic.xml.schema.binding.RuntimeUtils.invoke_serializer(RuntimeUtils.java:192)
         at weblogic.xml.schema.binding.RuntimeUtils.invoke_serializer(RuntimeUtils.java:178)
         at weblogic.webservice.core.DefaultPart.invokeSerializer(DefaultPart.java:335)
         at weblogic.webservice.core.DefaultPart.toXML(DefaultPart.java:304)
         at weblogic.webservice.core.DefaultMessage.toXML(DefaultMessage.java:651)
         at weblogic.webservice.core.handler.InvokeHandler.handleRequest(InvokeHandler.java:151)
         at weblogic.webservice.core.HandlerChainImpl.handleRequest(HandlerChainImpl.java:144)
         at weblogic.webservice.core.DefaultOperation.process(DefaultOperation.java:551)
         at weblogic.webservice.server.Dispatcher.process(Dispatcher.java:204)
         at weblogic.webservice.server.Dispatcher.doDispatch(Dispatcher.java:175)
         at weblogic.webservice.server.Dispatcher.dispatch(Dispatcher.java:97)
         at weblogic.webservice.server.WebServiceManager.dispatch(WebServiceManager.java:102)
         at weblogic.webservice.server.servlet.WebServiceServlet.serverSideInvoke(WebServiceServlet.java:321)
         at weblogic.webservice.server.servlet.ServletBase.doPost(ServletBase.java:454)
         at weblogic.webservice.server.servlet.WebServiceServlet.doPost(WebServiceServlet.java:293)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:821)
         at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
         at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:292)
         at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3594)
         at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
         at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2108)
         at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1432)
         at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
    </bea_fault:stacktrace></detail></env:Fault>]]]]

    2012/11/28 05:22:07.188 INFO <(SOAPHandler.java:116)> [2145221434] -
    ******** soapMessage in fault ******** SOAPMessageImpl[SOAPPartImpl[SOAPEnvelopeImpl[nullSOAPBody[<env:Fault><faultcode>env:Server</faultcode><faultstring>Exception during processing: java.lang.OutOfMemoryError: allocLargeObjectOrArray - Object size: 235368, Num elements: 58837 (see Fault Detail for stacktrace)</faultstring><detail><bea_fault:stacktrace xmlns:bea_fault="http://www.bea.com/servers/wls70/webservice/fault/1.0.0">java.lang.OutOfMemoryError: allocLargeObjectOrArray - Object size: 235368, Num elements: 58837
         at java.util.Arrays.copyOf(Arrays.java:2760)
         at java.util.Arrays.copyOf(Arrays.java:2734)
         at java.util.ArrayList.ensureCapacity(ArrayList.java:167)
         at java.util.ArrayList.add(ArrayList.java:351)
         at weblogic.xml.xmlnode.XMLNode.addChild(XMLNode.java:479)
         at weblogic.xml.xmlnode.XMLNode.addChild(XMLNode.java:435)
         at weblogic.xml.xmlnode.XMLNode.addChild(XMLNode.java:514)
         at weblogic.xml.xmlnode.XMLNodeWriter.write(XMLNodeWriter.java:55)
         at weblogic.xml.xmlnode.XMLNodeWriter.write(XMLNodeWriter.java:105)
         at weblogic.xml.babel.stream.XMLOutputStreamBase.writeXMLEvent(XMLOutputStreamBase.java:77)
         at weblogic.xml.babel.stream.XMLOutputStreamBase.write(XMLOutputStreamBase.java:99)
         at weblogic.xml.babel.stream.XMLOutputStreamBase.flush(XMLOutputStreamBase.java:208)
         at weblogic.xml.babel.stream.XMLOutputStreamBase.flush(XMLOutputStreamBase.java:215)
         at weblogic.xml.babel.stream.XMLOutputStreamBase.addXMLEvent(XMLOutputStreamBase.java:125)
         at weblogic.xml.babel.stream.XMLOutputStreamBase.add(XMLOutputStreamBase.java:173)
         at weblogic.xml.schema.binding.internal.builtin.DocumentCodec.serialize(DocumentCodec.java:71)
         at weblogic.xml.schema.binding.RuntimeUtils.invoke_serializer(RuntimeUtils.java:192)
         at weblogic.xml.schema.binding.RuntimeUtils.invoke_serializer(RuntimeUtils.java:178)
         at weblogic.webservice.core.DefaultPart.invokeSerializer(DefaultPart.java:335)
         at weblogic.webservice.core.DefaultPart.toXML(DefaultPart.java:304)
         at weblogic.webservice.core.DefaultMessage.toXML(DefaultMessage.java:651)
         at weblogic.webservice.core.handler.InvokeHandler.handleRequest(InvokeHandler.java:151)
         at weblogic.webservice.core.HandlerChainImpl.handleRequest(HandlerChainImpl.java:144)
         at weblogic.webservice.core.DefaultOperation.process(DefaultOperation.java:551)
         at weblogic.webservice.server.Dispatcher.process(Dispatcher.java:204)
         at weblogic.webservice.server.Dispatcher.doDispatch(Dispatcher.java:175)
         at weblogic.webservice.server.Dispatcher.dispatch(Dispatcher.java:97)
         at weblogic.webservice.server.WebServiceManager.dispatch(WebServiceManager.java:102)
         at weblogic.webservice.server.servlet.WebServiceServlet.serverSideInvoke(WebServiceServlet.java:321)
         at weblogic.webservice.server.servlet.ServletBase.doPost(ServletBase.java:454)
         at weblogic.webservice.server.servlet.WebServiceServlet.doPost(WebServiceServlet.java:293)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:821)
         at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
         at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:292)
         at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3594)
         at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
         at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2108)
         at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1432)
         at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
         at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
    </bea_fault:stacktrace></detail></env:Fault>]]]]

    Ps: " Do you consider that maximizing the heap space will solve the problem definetly??"

    Hope you will understand and will sugest a better solution.

    your help is highly appreciated.

    Regards.

    Mansoor.
  • 8. Re: Weblogic Health status goes to "Warning"
    Fabian Pro
    Currently Being Moderated
    Hi Mansoor,

    From your Logs,Seems like your heap size if fully getting used,No space for new objects
    heap size: 1536000K, maximal heap size: 1536000K, nursery size: 384000K
    Your Using Generational Concurrent Mark & Sweep
    In which means,GC allows most of the Java Threads to keep running during most of the garbage collection(GC) Process with few pause.
    And it seams like Even After Garbage collection the memory is not recovered and even the Garbage Collection is taking to long. pauses 557.423 ms
    147933.627-147938.089: GC 1281600K->1276830K (1536000K), sum of pauses 557.423 ms
    You can give a try by setting below parameter ,here were are declaring heap size to 2gb,and Setting Young space to 512mb(This is for new objects)
    And we are even changing the strategy of garbage collection,ie Parallel time Mode(Which performs Parallel Mark and Sweep) and we have set pause time for garbage collection should not exceed 200 ms

    -Xms2048m -Xmx2048m -Xns512m -XgcPrio:pausetime -XpauseTarget:200ms

    Details About garbage collection

    http://docs.oracle.com/cd/E13188_01/jrockit/docs50/userguide/memman.html

    Have you enabled Garbage collection logging which i had mentioned in my earlier post.
    JAVA_OPTIONS="${JAVA_OPTIONS} -verbose:gc -Xverboselog:/tmp/gc.log -Xverbose:memory"
    export JAVA_OPTIONS
    If yes,Then provide us the gc.log for further analysis

    Or you can analyze it by urself,Link for the same

    http://middlewaremagic.com/weblogic/?p=5131


    Regards
    Fabian
  • 9. Re: Weblogic Health status goes to "Warning"
    976109 Newbie
    Currently Being Moderated
    Hi,

    Below is some information regarding GC is given.

    [WARN ] Use of -Djrockit.optfile is deprecated and discouraged.
    [memory ][Tue Nov 27 17:48:51 2012][06986] Running with 32 bit heap and compressed references.
    [memory ][Tue Nov 27 17:48:51 2012][06986] GC mode: Garbage collection optimized for short pausetimes, strategy: Generational Concurrent Mark & Sweep.
    [memory ][Tue Nov 27 17:48:51 2012][06986] Heap size: 1536000KB, maximal heap size: 1536000KB, nursery size: 384000KB.
    [memory ][Tue Nov 27 17:48:51 2012][06986] <start>-<end>: <type> <before>KB-><after>KB (<heap>KB), <time> ms, sum of pauses <pause> ms.
    [memory ][Tue Nov 27 17:48:51 2012][06986] <start> - start time of collection (seconds since jvm start).
    [memory ][Tue Nov 27 17:48:51 2012][06986] <type> - OC (old collection) or YC (young collection).
    [memory ][Tue Nov 27 17:48:51 2012][06986] <end> - end time of collection (seconds since jvm start).
    [memory ][Tue Nov 27 17:48:51 2012][06986] <before> - memory used by objects before collection (KB).
    [memory ][Tue Nov 27 17:48:51 2012][06986] <after> - memory used by objects after collection (KB).
    [memory ][Tue Nov 27 17:48:51 2012][06986] <heap> - size of heap after collection (KB).
    [memory ][Tue Nov 27 17:48:51 2012][06986] <time> - total time of collection (milliseconds).
    [memory ][Tue Nov 27 17:48:51 2012][06986] <pause> - total sum of pauses during collection (milliseconds).
    [memory ][Tue Nov 27 17:48:51 2012][06986] Run with -Xverbose:gcpause to see individual phases.
    [memory ][Tue Nov 27 17:48:51 2012][06986] Running on hypervisor Unknown (but probably running on a hypervisor)
    [mgmnt  ][Tue Nov 27 17:48:52 2012][06986] Remote JMX connector started at address sje00128.ute.fedex.com:57032
    [mgmnt  ][Tue Nov 27 17:48:52 2012][06986] Local JMX connector started
    <Nov 27, 2012 11:48:53 AM CST> <Info> <Security> <BEA-090905> <Disabling CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true>
    <Nov 27, 2012 11:48:54 AM CST> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG to FIPS186PRNG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true>
    <Nov 27, 2012 11:48:55 AM CST> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with Oracle JRockit(R) Version R28.1.4-7-144370-1.6.0_26-20110617-2130-linux-x86_64 from Oracle Corporation>
    <Nov 27, 2012 11:48:57 AM CST> <Info> <Management> <BEA-141107> <Version: WebLogic Server 10.3.5.0 Fri Apr 1 20:20:06 PDT 2011 1398638 >
    <Nov 27, 2012 11:49:01 AM CST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>
    <Nov 27, 2012 11:49:01 AM CST> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool>
    [memory ][Tue Nov 27 17:49:02 2012][06986] [YC#1] 11.168-11.259
    : YC 414780KB->142943KB (1536000KB), 0.092 s, sum of pauses 91.529 ms, longest pause 91.529 ms.
    [memory ][Tue Nov 27 17:49:08 2012][06986] [YC#2] 17.288-17.406: YC 446196KB->169302KB (1536000KB), 0.119 s, sum of pauses 116.815 ms, longest pause 116.815 ms.
    [memory ][Tue Nov 27 17:49:14 2012][06986] [YC#3] 23.406-23.441: YC 473754KB->190716KB (1536000KB), 0.035 s, sum of pauses 34.627 ms, longest pause 34.627 ms.
    [memory ][Tue Nov 27 17:49:28 2012][06986] [YC#4] 37.678-37.756: YC 505886KB->225024KB (1536000KB), 0.079 s, sum of pauses 78.453 ms, longest pause 78.453 ms.
    [memory ][Tue Nov 27 17:49:36 2012][06986] [YC#5] 45.360-45.434: YC 523630KB->246993KB (1536000KB), 0.074 s, sum of pauses 74.243 ms, longest pause 74.243 ms.
    [memory ][Wed Nov 28 10:41:54 2012][06986] [YC#6] 60775.410-60783.147: YC 535807KB->254945KB (1536000KB), 7.737 s, sum of pauses 7735.633 ms, longest pause 7735.633 ms.
    [memory ][Wed Nov 28 20:40:52 2012][06986] [YC#7] 96721.146-96721.683: YC 544511KB->257889KB (1536000KB), 0.537 s, sum of pauses 536.991 ms, longest pause 536.991 ms.
    [memory ][Thu Nov 29 15:46:30 2012][06986] [YC#8] 165459.460-165459.507: YC 546436KB->259814KB (1536000KB), 0.047 s, sum of pauses 46.992 ms, longest pause 46.992 ms.

    Hope you understand the problem and will help me to solve.

    your help is highly appreciated.

    Regards.

    Mansoor
  • 10. Re: Weblogic Health status goes to "Warning"
    976109 Newbie
    Currently Being Moderated
    Hi Fabian,

    I am not looking any "Parallel mark and sweep option" in the command code that you provided, could you please look at your command code once again and verify that is this the same code that you want me to try!!

    -Xms2048m -Xmx2048m -Xns512m -XgcPrio:pausetime -XpauseTarget:200ms

    your help is highly appreciated.

    Regards.

    Mansoor.

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points