5 Replies Latest reply: Jul 21, 2012 1:40 PM by Kalyan Pasupuleti-Oracle RSS

    Managed Servers in Warning Statue due to Stuck Threads

    gibbsworadba
      My goal of this post is to understand why I am having the below issue. I am running Oracle WebLogic Server 11g v10.3.4.0 on Solaris. This WebLogic installation is running a single AdminServer and 2 managed servers.

      My AdminServer is fine, it is running and it's health is labeled as OK in the WebLogic Administration Console. Both of my managed servers are having issues right now. Managed server (MS1) and managed server (MS2)
      are in a warning state due to stuck threads. My .ear (deployed application) is still up and running, my users can log into the site and do the majority of the functionality in the site, but one page is an issue, a page that the users
      are saying they cannot access.

      So I log into the WebLogic Administration Console, click on the "Warning" hyperlink in the "System Status" panel and am directed to the "Monitoring\Health" tab page. I click on my MS1 managed server hyperlink, then click
      on the threadpool hyperlink to drill down into the problem. It then displays an html page with 24 records in it, with all of the thread information. I can see in the first column of this HTML table that the some of the names of
      the threads are begining with [STUCK], [ACTIVE], and [STANDBY]. I'm assuming that the most valuable information that is going to help me in diagnosing this issue are the rows with a thread name beginig with STUCK.
      So with that in mind, I look at the values in the Application column of those rows and do see that it has the name of the my .ear (my deployed application) containing the page that my users are complaining about.
      The "Current Request" column data in this HTML table does show me the page (the URL) in my application that is having the problem, this is the same URL that my users have been complaining about. I don't understand all of
      the data in this thread table in the administration console (I've included one record below, one labeled as stuck, it's in bold font).

      Below is a single record from this thread table, all of the records with a thread name begining with [STUCK] seem to reference the same HTML page. Any help is greatly appreciated. From here I am unsure what direction to
      take to fix the problem (or know exactly what Java code is causing the issue).

      Name: [STUCK] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'

      Total Requests: 84958

      Current Request:
      weblogic.servlet.internal.ServletRequestImpl@57d38af0[ GET /englink/teams/missions/team/viewTeam.seam?p_cid=6682 HTTP/1.1 Accept: application/x-ms-application, image/jpeg, application/xaml+xml, image/gif, image/pjpeg, application/x-ms-xbap, application/msword, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/vnd.xfdl; version=7.6.1.123, / Referer: https://..........mil/englink/teams/manageTeams.seam Accept-Language: en-US User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Win64; x64; Trident/5.0; .NET CLR 2.0.50727; SLCC2; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET4.0C; .NET4.0E; InfoPath.2; MS-RTC LM 8; Tablet PC 2.0) UA-CPU: AMD64 Accept-Encoding: gzip, deflate Cookie: __utma=201804309.579273823.1335813579.1340907577.1340970347.54; __utmz=201804309.1335813579.1.1.utmccn=(direct)|utmcsr=(direct)|utmcmd=(none); JSESSIONID=9J9fQGPJMn4CvGrZt5mwpsLmjKsZKjkW12Z8nnhGhPX8nyGGkpfF!-792419015 ECID-Context: 1.00018^XE50JFk34MRUADUP0002oH005e4j;kXjE1ZDLIPHPj5QRbSPQiRRBoLPOhJVBhPRSnPRRiJVBoLPOhVATdLRTKLPOhUBS_HPR Connection: Keep-Alive Proxy-Client-IP: 155.82.164.228 X-Forwarded-For: 155.82.164.228 X-WebLogic-KeepAliveSecs: 30 X-WebLogic-Request-ClusterInfo: true x-weblogic-cluster-hash: bMs97kIQ/2k25w5aHnbxY7KyGO0 ]

      Transaction: No value in this cell
      User: <anonymous>     
      Idle: false     
      Stuck: true     
      Hogger: true     
      Standby: false     
      Application: englink_v1-59_20120709v2     
      Module: englink     
      Workmanager: default

      Edited by: we**** on Jul 18, 2012 1:25 PM
        • 1. Re: Managed Servers in Warning Statue due to Stuck Threads
          Kalyan Pasupuleti-Oracle
          Hi,

          It looks like that you are try to download some files from following request.

          here are few step for drill down the issue.

          Enable tunneling on every managed server along with Admin server.

          you will find Tunneling under Server >> Protocol, there you will find check box say Enable Tunneling.


          Secondly try to increase chunk size.


          -Dweblogic.Chunksize=8192 -Dweblogic.utils.io.chunkpoolsize=8192


          To know more details about the chunk please see following link.

          http://docs.oracle.com/cd/E13222_01/wls/docs90/perform/topten.html#1132666


          for tuning check following link.

          http://docs.oracle.com/cd/E13222_01/wls/docs90/perform/WLSTuning.html#1152088


          set these two options in your environment and try to check again.

          Note: check with your memory arguments too that will also be one major issue.


          Regards,
          Kal
          • 2. Re: Managed Servers in Warning Statue due to Stuck Threads
            gibbsworadba
            Have a quick easy to understand explanation of what "tunneling enabling" does?

            I see from the URL you posted that weblogic.utils.io.chunkpoolsize defaults to 2048, but the weblogic.Chunksize doesn't mention a default value, is there a way
            in the WebLogic admin console, how can I tell what the current weblogic.Chunksizesize is?

            The URL you provided ("http://docs.oracle.com/cd/E13222_01/wls/docs90/perform/topten.html#1132666") says this:
            For applications that handle large amounts of data per request, increasing the value on both the client and server sides can boost performance.

            I understand to test this suggestion of yours that I would have to start WebLogic with those 2 chunksize attributes, but what does the above comment
            about changing this also on the client mean? In my architecture I have OHS which directs to the request to the WebLogic Application Server.... do I need to make
            this chunksize change on any other pieces of the architecture?

            Edited by: we**** on Jul 19, 2012 8:01 PM
            • 3. Re: Managed Servers in Warning Statue due to Stuck Threads
              Kalyan Pasupuleti-Oracle
              Hi,


              You need not change any thing from OHS side.

              simple set the chunk size and chunk pool size from weblogic side.


              Coming to default chunk size is 2k.

              Where as Tunneling is procedure to check your protocol channel for every frequent time and make refresh channels and make avoid peer to peer communication or protocol violation.

              Hope I address your query.

              Regards,
              Kal
              • 4. Re: Managed Servers in Warning Statue due to Stuck Threads
                gibbsworadba
                Thank you for your responses, they are clear.

                When you say:
                -Dweblogic.Chunksize=8192 -Dweblogic.utils.io.chunkpoolsize=8192

                was that just an example or are you suggesting that I use these 8192 values? After my initial read of http://docs.oracle.com/cd/E13222_01/wls/docs90/perform/WLSTuning.html#1152088
                I would'nt think to use the same value for both of these parameters.

                Also, do you agree that it only makes sense to start my 2 managed servers with these chunk parameters, not the admin server since my application .ear
                is not deployed to the admin server, it's only deployed to the two managed servers?

                Edited by: we**** on Jul 20, 2012 9:50 AM
                • 5. Re: Managed Servers in Warning Statue due to Stuck Threads
                  Kalyan Pasupuleti-Oracle
                  Hi,

                  Please set these parameter where your App is going to deployed.

                  If Admin server is not using this App then it is not required.

                  Because Every server instance will hold different port and different network channels so we need to set parameter accordingly.

                  Regards,
                  Kal