3 Replies Latest reply: Oct 14, 2009 9:57 AM by 711333 RSS

    NestedRuntimeException: Cannot parse POST parameters of request

    711333
      Get the following exception intermittently when I submit (POST method) a form. of course when this happens the request is not processed.

      ####<Oct 2, 2009 9:14:22 AM EDT> <Error> <KeyCommerce System> <SIMSAPP1> <AdminServer> <[ACTIVE] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <BEA1-176AA6AF1C35> <> <1254489262078> <000000> <WarehouseLoginTask
      weblogic.utils.NestedRuntimeException: Cannot parse POST parameters of request: *'/warehouse/workflow/orderupdate_import_process'*
      at weblogic.servlet.internal.ServletRequestImpl$RequestParameters.mergePostParams(ServletRequestImpl.java:1812)
      at weblogic.servlet.internal.ServletRequestImpl$RequestParameters.parseQueryParams(ServletRequestImpl.java:1699)
      at weblogic.servlet.internal.ServletRequestImpl$RequestParameters.getQueryParams(ServletRequestImpl.java:1652)
      at weblogic.servlet.internal.ServletRequestImpl.getParameter(ServletRequestImpl.java:752)
      at kc1.web.account.WarehouseLoginTask.execute(WarehouseLoginTask.java:36)
      at kc1.web.workflow.WorkflowServlet.executeWorkflow(WorkflowServlet.java:175)
      at kc1.web.workflow.WorkflowServlet.doGet(WorkflowServlet.java:98)
      at kc1.web.workflow.WorkflowServlet.doPost(WorkflowServlet.java:69)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:763)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
      at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
      at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
      at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:230)
      at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:175)
      at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3201)
      at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
      at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
      at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2002)
      at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:1861)
      at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1362)
      at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
      at weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
      java.net.SocketTimeoutException: Read timed out
      at jrockit.net.SocketNativeIO.socketRead(Ljava.io.FileDescriptor;[BIII)I(Unknown Source)
      at java.net.SocketInputStream.socketRead0(Ljava.io.FileDescriptor;[BIII)I(SocketInputStream.java:???)
      at java.net.SocketInputStream.read(SocketInputStream.java:113)
      at weblogic.servlet.internal.PostInputStream.read(PostInputStream.java:177)
      at weblogic.servlet.internal.ServletInputStreamImpl.read(ServletInputStreamImpl.java:211)
      at weblogic.servlet.internal.ServletRequestImpl$RequestParameters.mergePostParams(ServletRequestImpl.java:1787)
      at weblogic.servlet.internal.ServletRequestImpl$RequestParameters.parseQueryParams(ServletRequestImpl.java:1699)
      at weblogic.servlet.internal.ServletRequestImpl$RequestParameters.getQueryParams(ServletRequestImpl.java:1652)
      at weblogic.servlet.internal.ServletRequestImpl.getParameter(ServletRequestImpl.java:752)
      +at kc1.web.account.WarehouseLoginTask.execute(WarehouseLoginTask.java:36)+
      at kc1.web.workflow.WorkflowServlet.executeWorkflow(WorkflowServlet.java:175)
      at kc1.web.workflow.WorkflowServlet.doGet(WorkflowServlet.java:98)
      at kc1.web.workflow.WorkflowServlet.doPost(WorkflowServlet.java:69)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:763)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
      at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
      at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
      at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:230)
      at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:175)
      at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3201)
      at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
      at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
      at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2002)
      at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:1861)
      at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1362)
      at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
      at weblogic.work.ExecuteThread.run(ExecuteThread.java:181)

      The text in bold is the target action of the JSP which has the form with large number of fields. the text in italic is the place in code where i am doing (which generates the exception):
      String var_name = request.getParameter("param_name");

      On googling this it seems that the 'SocketTimeoutException' exception is the one to look for. Many sources suggest that it is basically WebLogic rejecting requests when they go over the limit. Currently the 'Post Timeout' (under HTTP protocol settings) value for the WebLogic server isntance is 30 (default). Some also say that it can be the 'TCP max connection' issue on the server (Windows 2003 Server ). But I didn't find any solution for this. There was also a suggestion about verifying whether the form is being submitted twice. I did check that but that was not the case. Moreover, if it was indeed true then shouldn't this exception occur all the time when the form is submitted and not sporadically?

      The application is hosted on WebLogic 9.2. JVM being used is the JRockIt which comes with the WebLogic server. OS is Windows 2003 Server 64bit. *+The most frustrating thing is that this issue is not consistent and happens every now and then so it is hard to track or debug.+*

      _Dual submission of forms:_
      -There is 2 kinds of strategies being used through JavaScript.
      -HTML button with type 'submit' and on 'onclick' event of this button a small JS method is called which validates stuff and returs true or false. 'form.submit()' is not being called within that JS method.
      -HTML button with type 'submit' and a small JS method is called on the 'onsubmit' event of the form which validates stuff and returns true or false. 'form.submit()' is not being called within that JS  method.
      -I read at one place that if we are submitting a form through JS 'submit' function then we should not provide the 'action' attribute in the form element. DO this instead in the JS method. As it might cause double submission. But again, I will re-ask the same question: if the form was being submitted twice, wouldn't this error be consistent?

      _Logging and debugging:_
      -I tried Charles and it was recording requests from the client (browser) as we know. But as told earlier the issue is not consistent at all, so during all of my tests all requests went through fine.

      -Then I tried Jmeter. Created a HTTP Request sampler. Simulated that request with appropriate parameters and made like 200-300 calls but again all went through ok. The thing is that I tried to simulate the sampler's parameters as production-like as possible but of course they were not 100% exact. I used graphic listener (which honestly I couldn't get) and Aggregate Report and Summary but all seem to report successes. I also tried the Access Log Sampler which reads from any log files (I supplied some production log files) but it can't simulate Post requests!

      -IIS logging: There is IIS at the front-end working as a Web Server (2 machines infact for load-balancing) for the application. WebLogic-IIS plugin is used to redirect request from IIS to WebLogic. I created a trace at the Web Server level. I used the 'logman' utility which generates .etl files and then 'logparser' to read them. Did this on both the Web Server machines. Logged all components ("IIS: WWW Server" 0xFFFFFFFE 5). But got nothing new or significant in those files. POST parameters are also not logged there.

      _Some misc. information:_
      -There is no clustering in the application and WebLogic deployment. The one AdminServer is also running the application. This mostly happens in the morning when there are many users connected.
      -The actual users (which are in hundreds) are all over the country (I am in US) and connect through firewalls and vpn's and some across the internet. Can this be network related thingy which I should investigate (involve network guys)? I tried 'Wireshark' but as I am  poor in networks I couldn't make it work.

      -I have conifgured 'extended log format' and added couple of ip/dns fields too but they are not helping. There is an entry of each Post request but that's it. The status code is 200 which is success.

      -I am catching an exception, printing it on the server console and then throwing it again. In addition to this I am also redirecting the user to the error page. So the user simply sees the error page and we get the exception details from the server log. The framework is customized (and very simple) based on FrontController pattern. Nothing fancy going on here. A central servlet which processes each URL pattern and invokes the corresponding action/task class.

      -Can this be a network issue? what should I take to the network guys which shows that this might be a network issue? What should they be looking for at the network side? I don't seem to have anything to pin-point in that direction.

      -This application was migrated from WebLogic 8.1 to 9.2 a year or 2 back (before I came). Can this have some affect if that was not performed properly?

      Thanks a lot.

      *Cross posted at: http://www.coderanch.com/t/464137/BEA-Weblogic/Limit-Data-Key-Value-Pairs*

      Edited by: haqyunus on Oct 12, 2009 12:12 PM

      Edited by: haqyunus on Oct 12, 2009 12:13 PM                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
        • 1. Re: NestedRuntimeException: Cannot parse POST parameters of request
          711333
          I've now placed the following snoop code in the catch block where this exception is handled (in WarehouseLoginTask.java above!!!):

          try
          {  
          .........
          String varname = request.getParameter("user_name");//---THE MAIN EXCEPTION ABOVE, GENERATED HERE
          .........
          }
          catch (Exception ex) {  

          ////SNOOP CODE START
          try {  
          Logger.info("**************************REQUEST ATTRIBUTES*********************************");
          java.util.Enumeration e = request.getAttributeNames();
          while (e.hasMoreElements()) {  
          String key = (String)e.nextElement();
          Object value = request.getAttribute(key);
          Logger.info(" " + key + " = " + value);
          }
          Logger.error("**************************REQUEST HEADERS**************************");
          Enumeration en = request.getHeaderNames();
          while (en.hasMoreElements()) {  
          String key = (String)en.nextElement();
          String value = request.getHeader(key);
          Logger.info(" <b>" + key + "</b>: " + value);
          }

          Logger.error("**************************MISC. REQUEST INFO.**************************");
          //Logger.info("Servlet Name: " + getServletContext().getsgetServletName());
          Logger.info("Protocol: " + request.getProtocol().trim());
          Logger.info("Scheme: " + request.getScheme());
          Logger.info("Server Name: " + request.getServerName());
          Logger.info("Server Port: " + request.getServerPort());
          Logger.info("Server Info: " + getServletContext().getServerInfo());
          Logger.info("Remote Addr: " + request.getRemoteAddr());
          Logger.info("Remote Host: " + request.getRemoteHost());
          Logger.info("Character Encoding: " + request.getCharacterEncoding());
          Logger.info("Content Length: " + request.getContentLength());
          Logger.info("Content Type: "+ request.getContentType());
          Logger.info("Locale: "+ request.getLocale());
          Logger.info("Default Response Buffer: "+ response.getBufferSize());
          Logger.info("Request Is Secure: " + request.isSecure());
          Logger.info("Auth Type: " + request.getAuthType());
          Logger.info("HTTP Method: " + request.getMethod());
          Logger.info("Remote User: " + request.getRemoteUser());
          Logger.info("Request URI: " + request.getRequestURI());
          Logger.info("Context Path: " + request.getContextPath());
          Logger.info("Servlet Path: " + request.getServletPath());
          Logger.info("Path Info: " + request.getPathInfo());
          Logger.info("Path Trans: " + request.getPathTranslated());
          Logger.info("Query String: " + request.getQueryString());

          Logger.info("**************************PARAMETER NAMES AND VALUES**************************");
          //THIS SHOULD AGAIN THROW THE ABOVE-MENTIONED EXCEPTION...!!???
          java.util.Enumeration en1 = request.getParameterNames();
          while (en1.hasMoreElements()) {  
          String key = (String)en1.nextElement();
          String[] values = request.getParameterValues(key);
          Logger.info(" " + key + " = ");
          for(int i1 = 0; i1 < values.length; i1++) {  
          Logger.info(values[i1] + " ");
          }

          }

          }catch(Exception ex1) {  
          Logger.error("!!!!!!!!!!!!!!!!!EXCEPTION IN SNOOP/DEBUG CODE!!!!!!!!!!!!!!!!!");
          Logger.error(ex1.getMessage());
          Logger.error("!!!!!!!!!!!!!!!!!EXITING THIS REQUEST'S SNOOP/DEBUG!!!!!!!!!!!!!!!!!");
          }//SNOOP CODE END

          request.setAttribute("task_result", "error");
          Logger.error("WarehouseLoginTask", ex);
          throw new WorkflowException(ex.getMessage());
          }



          Here is the output of one occurence of the exception from today morning from the snoop code:


          <**************************REQUEST ATTRIBUTES*********************************>
          < weblogic.servlet.network_channel.port = 7001>
          <**************************REQUEST HEADERS**************************>
          < <b>Cache-Control</b>: no-cache>
          < <b>Content-Type</b>: application/x-www-form-urlencoded>
          < <b>Accept</b>: image/gif, image/jpeg, image/pjpeg, image/pjpeg, application/x-shockwave-flash, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */*>
          < <b>Accept-Encoding</b>: gzip, deflate>
          < <b>Accept-Language</b>: en-us>
          < <b>Cookie</b>: JSESSIONID=ThvvKJDDCHHhyb0w3bc6gJKnL3PdQZBlWGJQQCYqtlF669KjtTcN!2081454430>
          < <b>Host</b>: +{server name which hosts the app.}>+
          < <b>Referer</b>: +{full https request url}>+
          < <b>User-Agent</b>: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; msn OptimizedIE8;ENUS)>
          < <b>Content-Length</b>: 7187>
          < <b>Connection</b>: Keep-Alive>
          < <b>Proxy-Client-IP</b>: +{client ip}>+
          < <b>X-Forwarded-For</b>: +{client ip}>+
          < <b>X-WebLogic-KeepAliveSecs</b>: 30>
          < <b>X-WebLogic-Force-JVMID</b>: 2081454430>
          <**************************MISC. REQUEST INFO.**************************>
          <Protocol: HTTP/1.1>
          <Scheme: https>
          <Server Name: +{server name which hosts the app.}>+
          <Server Port: 7002>
          <Server Info: WebLogic Server 9.2 MP2 Mon Jun 25 01:32:01 EDT 2007 952826 >
          <Remote Addr: +{client ip}>+
          <Remote Host: +{full client machine address}>+
          <Character Encoding: null>
          <Content Length: 7187>
          <Content Type: application/x-www-form-urlencoded>
          <Locale: en_US>
          <Default Response Buffer: 12216>
          <Request Is Secure: true>
          <Auth Type: null>
          <HTTP Method: POST>
          <Remote User: null>
          <Request URI: /warehouse/workflow/update_mfg_shipment>
          <Context Path: /warehouse>
          <Servlet Path: /workflow>
          <Path Info: /update_mfg_shipment>
          <Path Trans: null>
          <Query String: null>
          <**************************PARAMETER NAMES AND VALUES**************************>

          As we can see that nothing was printed when we tried to get 'request.parameterNames'. The values in italics are just informatory and have been masked for confidentiality.

          But interesting to notice is that the following didn't seem to get executed at all:

          catch(Exception ex1) {     
          Logger.error("!!!!!!!!!!!!!!!!!EXCEPTION IN SNOOP/DEBUG CODE!!!!!!!!!!!!!!!!!");
          Logger.error(ex1.getMessage());
          Logger.error("!!!!!!!!!!!!!!!!!EXITING THIS REQUEST'S SNOOP/DEBUG!!!!!!!!!!!!!!!!!");
          }//SNOOP CODE END

          If getParameterNames generated an exception (which it seems to have done) then shouldn't we something from this catch.....i am losing my mind gradually....

          Edited by: haqyunus on Oct 13, 2009 10:31 AM
          • 2. Re: NestedRuntimeException: Cannot parse POST parameters of request
            david.karr
            It might be useful to try constructing a piece of client code that sends an http request to this servlet which starts to write out data, then sleeps for 40 seconds or so, then finishes writing the output. See if you get the same error as you've been getting. This is just an experiment to see if this pattern matches your results.

            Concerning your additional debugging, you seem to have expected the last "getParameterNames()" to throw an exception, but it didn't appear to. Just for confirmation, you might want to put a log statement that prints the value of "en1.hasMoreElements()" right after you call that.
            • 3. Re: NestedRuntimeException: Cannot parse POST parameters of request
              711333
              Thanks David.

              I was using JMeter for simulating requests to the server. I think I can cause this 'delay-scenario' through that too. I will look into it. Also printing out the hasMoreElements method value is interesting idea as well. If it prints false, then this will mean that we are not getting any POST parameters-- a more mysterious mystery....:)