8 Replies Latest reply: Apr 12, 2013 10:17 AM by Luis RSS

    Issues with HttpClusterServlet: Caught exception:  status line is null

    839066
      I have an environment with two boxes running weblogic 10.3.2, configured in a cluster.
      I have another environment running weblogic 10.3.2, completely separate from the other one, on a separate box, where the HttpClusterServlet is deployed for load balancing.
      When I connect to the individual servers in the cluster I can access my application with no problems. However, when I connect thru the HttpClusterServlet quite often I get an exception in the HttpClusterServlet log: status line is null which causes it to switch to the other server.
      How can I fix this?

      Here is the stack trace from the log:
      <Sat Feb 12 12:31:06 EST 2011> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'>: Cookie: JSESSIONID=PGqHNWDhzD8TzjzfnPzGCrn8DTHc5bHCRp2jxh26WxDLBHZYRWNh!-162124595
      <Sat Feb 12 12:31:06 EST 2011> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'>: Successfully send request headers to server: -162124595!192.168.223.250!8011!-1
      <Sat Feb 12 12:31:06 EST 2011> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'>: Caught exception while reading response status : weblogic.servlet.proxy.HalfOpenSocketRetryException: status line is null
           at weblogic.servlet.proxy.GenericProxyServlet.readStatus(GenericProxyServlet.java:787)
           at weblogic.servlet.proxy.HttpClusterServlet.sendResponse(HttpClusterServlet.java:1450)
           at weblogic.servlet.proxy.HttpClusterServlet.service(HttpClusterServlet.java:300)
           at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
           at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
           at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
           at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:292)
           at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:175)
           at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3594)
           at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
           at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
           at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2202)
           at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2108)
           at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1432)
           at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
           at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)

      <Sat Feb 12 12:31:06 EST 2011> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'>: marked bad: -162124595!192.168.223.250!8011!-1
      <Sat Feb 12 12:31:06 EST 2011> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'>: Error reading response code, or got 503. Trying next server
      <Sat Feb 12 12:31:06 EST 2011> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'>: Sleeping for 2 secs .....
      <Sat Feb 12 12:31:08 EST 2011> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'>: attempt #1 out of a max of 5
      <Sat Feb 12 12:31:08 EST 2011> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'>: #### Trying to connect with server 871690966!192.168.223.244!8011!-1


      The web.xml for the HttpClusterServlet is as follows:
      <!DOCTYPE web-app PUBLIC
           '-//Sun Microsystems, Inc.//DTD Web Application 2.3//EN'
           'http://java.sun.com/dtd/web-app_2_3.dtd'>

      <web-app>
      <servlet>
      <servlet-name>HttpClusterServlet</servlet-name>
      <servlet-class>
      weblogic.servlet.proxy.HttpClusterServlet
      </servlet-class>
      <init-param>
      <param-name>WebLogicCluster</param-name>
      <param-value>192.168.223.244:8011|192.168.223.250:8011</param-value>
      </init-param>
      <init-param>
      <param-name>DebugConfigInfo</param-name>
      <param-value>ON</param-value>
      </init-param>
      <init-param>
      <param-name>Debug</param-name>
      <param-value>ON</param-value>
      </init-param>
      <init-param>
      <param-name>WLLogFile</param-name>
      <param-value>/data/weblogic/bea1032/user_projects/domains/TNP81/servers/HTTPClusterServlet/logs/wlproxy.log</param-value>
      </init-param>
      <init-param>
      <param-name>verbose</param-name>
      <param-value>true</param-value>
      </init-param>
      </servlet>
      <servlet-mapping>
      <servlet-name>HttpClusterServlet</servlet-name>
      <url-pattern>/</url-pattern>
      </servlet-mapping>
      <servlet-mapping>
      <servlet-name>HttpClusterServlet</servlet-name>
      <url-pattern>*.jsp</url-pattern>
      </servlet-mapping>
      <servlet-mapping>
      <servlet-name>HttpClusterServlet</servlet-name>
      <url-pattern>*.htm</url-pattern>
      </servlet-mapping>
      <servlet-mapping>
      <servlet-name>HttpClusterServlet</servlet-name>
      <url-pattern>*.html</url-pattern>
      </servlet-mapping>
      </web-app>

      I would appreciate any input.
      TIA,
      Ion Moraru
        • 1. Re: Issues with HttpClusterServlet: Caught exception:  status line is null
          René van Wijk
          Are you also able to connect to the individual server from where the HttpClusterServlet is running?

          Maybe there are some network problems between the server on which HttpClusterServlet is running and the
          servers n your clusters.
          • 2. Re: Issues with HttpClusterServlet: Caught exception:  status line is null
            839066
            Yes, I can connect to the servers. In fact only some of the requests return this exception. There is no consistency in which one returns exceptions. One time it failed on retrieving a *.jpeg file other time loading a *.js, etc.
            I don't know if the problem is on the clustered machines or in the HttpClusterServlet configuration.
            Ion Moraru
            • 3. Re: Issues with HttpClusterServlet: Caught exception:  status line is null
              René van Wijk
              If it is constantly the same jpeg and js you should review the paths in your application which is deployed on your weblogic servers.

              Or is this completely random?
              • 4. Re: Issues with HttpClusterServlet: Caught exception:  status line is null
                Karunanidhi-Oracle
                do you have oracle support account , log into support.oracle.com and check for the bug
                9711123
                • 5. Re: Issues with HttpClusterServlet: Caught exception:  status line is null
                  895964
                  Hi

                  Did you ever get a resolution to this, I have a very similar issue?
                  • 6. Re: Issues with HttpClusterServlet: Caught exception:  status line is null
                    tbeets
                    I ran into this issue. I'm using HttpClusterServlet (10.3.4). I was seeing weird behavior where if I was using Internet Explorer or Firefox performance of my clustered (session-replicated) webapp was great. Same app with Chrome was a dog. What was going on?

                    In the managed server logs I could see that requests in the same session (using Chrome only) were switching servers. The app was working in Chrome but bad performance.

                    Turned on HttpClusterServlet debug and found that Chrome has an issue trying to find /favicon.ico on every request.

                    [Google Chrome favicon issue|http://stackoverflow.com/questions/1003350/why-is-chrome-searching-for-my-favicon-ico-when-i-serve-up-a-file-from-asp-net-m]

                    I don't have that file on my servers. For some reason, this is tickling the weblogic.servlet.proxy.HalfOpenSocketRetryException and causing HttpClusterServlet to mark the primary server bad and fail the session over to the other...

                    Does not happen using IE and Firefox.
                    <Wed Dec 21 10:13:38 PST 2011> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'>: ===New Request===GET /favicon.ico HTTP/1.1
                    
                    <Wed Dec 21 10:13:38 PST 2011> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'>: Found session from url: null
                    <Wed Dec 21 10:13:38 PST 2011> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'>: attempt #0 out of a max of 5
                    <Wed Dec 21 10:13:38 PST 2011> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'>: #### Trying to connect with server -19992525!minty!11002!-1
                    <Wed Dec 21 10:13:38 PST 2011> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'>: Returning recycled connection: ProxyConnection(isSecureProxy=false):  minty:11002, keep-alive='20'secs
                    <Wed Dec 21 10:13:38 PST 2011> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'>: In-bound headers: 
                    <Wed Dec 21 10:13:38 PST 2011> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'>: Host: minty:11000
                    <Wed Dec 21 10:13:38 PST 2011> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'>: Accept: */*
                    <Wed Dec 21 10:13:38 PST 2011> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'>: User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.63 Safari/535.7
                    <Wed Dec 21 10:13:38 PST 2011> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'>: Accept-Encoding: gzip,deflate,sdch
                    <Wed Dec 21 10:13:38 PST 2011> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'>: Accept-Language: en-US,en;q=0.8
                    <Wed Dec 21 10:13:38 PST 2011> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'>: Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
                    <Wed Dec 21 10:13:38 PST 2011> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'>: Cookie: JSESSIONID=mgVkTySSh3RLdWdkpwhWvkTz3J6RshJQ5TQS9GrT1vjH1v2hpRpD!1918860886!-19992525
                    <Wed Dec 21 10:13:38 PST 2011> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'>: Successfully send request headers to server: -19992525!minty!11002!-1
                    <Wed Dec 21 10:13:38 PST 2011> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'>: Caught exception while reading response status : weblogic.servlet.proxy.HalfOpenSocketRetryException: status line is null
                         at weblogic.servlet.proxy.GenericProxyServlet.readStatus(GenericProxyServlet.java:852)
                         at weblogic.servlet.proxy.HttpClusterServlet.sendResponse(HttpClusterServlet.java:1621)
                         at weblogic.servlet.proxy.HttpClusterServlet.service(HttpClusterServlet.java:404)
                         at javax.servlet.http.HttpServlet.service(HttpServlet.java:821)
                         at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
                         at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
                         at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:300)
                         at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:184)
                         at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3717)
                         at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)
                         at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
                         at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
                         at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2277)
                         at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2183)
                         at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1454)
                         at weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)
                         at weblogic.work.ExecuteThread.run(ExecuteThread.java:176)
                    
                    <Wed Dec 21 10:13:38 PST 2011> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'>: marked bad: -19992525!minty!11002!-1
                    <Wed Dec 21 10:13:38 PST 2011> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'>: Error reading response code, or got 503. Trying next server
                    <Wed Dec 21 10:13:38 PST 2011> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'>: Sleeping for 2 secs .....
                    • 7. Re: Issues with HttpClusterServlet: Caught exception:  status line is null
                      Luis
                      Hello everybody,

                      We are suffering a very similar issue but with the weblogic.servlet.proxy.HttpProxyServlet and using Weblogic 12c

                      Our scenario is a little bit different, we are using this proxy as a bridge for our APEX applications. These ones require a cookie with the username value. So for fulfil this requirement we have set up a filter in front of the HttpProxyServlet. Basically this filter overrides the getHeader... methods in a class that extends HttpServletRequestWrapper.

                      The problem is that sometimes the cookie is not reaching the APEX application. In the proxy log (wlproxy.log) we can see this:
                      weblogic.servlet.proxy.HalfOpenSocketRetryException: status line is null
                              at weblogic.servlet.proxy.GenericProxyServlet.readStatus(GenericProxyServlet.java:856)
                              at weblogic.servlet.proxy.GenericProxyServlet.sendResponse(GenericProxyServlet.java:890)
                              at weblogic.servlet.proxy.HttpProxyServlet.sendResponse(HttpProxyServlet.java:24)
                              at weblogic.servlet.proxy.GenericProxyServlet.service(GenericProxyServlet.java:433)
                              at weblogic.servlet.proxy.HttpProxyServlet.service(HttpProxyServlet.java:24)
                              at javax.servlet.http.HttpServlet.service(HttpServlet.java:845)
                              at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:242)
                              at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:216)
                              at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:132)
                              at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:338)
                              at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:25)
                              at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:75)
                              at ch.cern.sso.weblogic.filters.SsoFilter.doFilter(SsoFilter.java:114)
                              at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:75)
                              at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3296)
                              at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3262)
                              at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:57)
                              at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2171)
                              at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2099)
                              at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2075)
                              at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1514)
                              at weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:255)
                              at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
                              at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
                      The behaviour seems to be the next:

                      * The filter code is executed, and our custom cookies are added: *weblogic.kernel.Default (self-tuning)'>: Cook
                      ie: ORA_WWV_APP_ JSESSIONID=........... ; AI_USERNAME=LURODRIG;*

                      * The exception is thrown: * weblogic.servlet.proxy.HalfOpenSocketRetryException: status line is null*

                      * The proxy retries to send the request: weblogic.kernel.Default (self-tuning)'>: Doing retry. but unfortunately without our custom cookies. It seems that in this retry the filter is not being executed.

                      Any ideas?

                      Thanks in advance,

                      Luis

                      ps: I have taken a look at the patch. It is for 10.3 versions, regarding that I am using 12c should I supposed that it is included for this release?

                      Edited by: Luis on Mar 7, 2013 6:12 PM
                      • 8. Re: Issues with HttpClusterServlet: Caught exception:  status line is null
                        Luis
                        Hello there,
                        <init-param>
                        <param-name>KeepAliveEnabled</param-name>
                        <param-value>false</param-value>
                        </init-param>
                        ... it should do the trick...

                        Hope it helps,

                        Luis