This discussion is archived
5 Replies Latest reply: Mar 29, 2010 9:00 PM by 807573 RSS

random 501,403 errors and HTTP2228 and HTTP3068 errors

807573 Oracle ACE
Currently Being Moderated
I have deployed a SJWP 4.0.13 with SJWS 7.0u5 origin server. While it is working fine for the most part, users have been occasionally getting 501 errors - here is a snippet from the access log
99.236.238.99 - "-" - [08/Mar/2010:17:13:48 -0500] "GET /licensees/licensee1/js/gam.html?1268086402696 HTTP/1.1" 501 148 "http://kidshealth.org/parent/pregnancy_calendar/pregnancy_calendar_intro.html" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30618)" "__gads=ID=272df0c712d874c7:T=1268086229:S=ALNI_MaA7MUxIUuhlCUmNBLCSJcillEuWw; textSize=1; KHCookie=KH_881268086204840" GET /licensees/licensee1/js/gam.html 1268086402696 "HTTP/1.1"
75.108.135.141 - "-" - [08/Mar/2010:17:13:48 -0500] "GET /parent/pregnancy_calendar/images_70716/1107202764157.viewAll.gif HTTP/1.1" 501 148 "http://kidshealth.org/parent/pregnancy_calendar/week14.html" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6" "__gads=ID=a61f5303cd4fcad6:T=1268082350:S=ALNI_MYij8da766gtAY_6R_nheKniiTokA; textSize=1; KHCookie=KH_221268082351821" GET /parent/pregnancy_calendar/images_70716/1107202764157.viewAll.gif - "HTTP/1.1"
167.128.220.96 - "-" - [08/Mar/2010:17:13:49 -0500] "GET /licensees/licensee1/js/gam.html?1268086428703 HTTP/1.1" 501 148 "http://kidshealth.org/parent/infections/skin/rocky.html" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; GTB0.0; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)" "__gads=ID=83aaaa6e1e32280b:T=1268085754:S=ALNI_MamLasKCOCPiWSZmecvvVqRiRjcVA" GET /licensees/licensee1/js/gam.html 1268086428703 "HTTP/1.1"
24.187.125.82 - "-" - [08/Mar/2010:17:13:50 -0500] "GET /kid/watch/er/headers_25350/sweat.gif HTTP/1.1" 502 292 "http://74.125.47.132/search?q=cache:eCkNVYKY9tkJ:endoflifecare.tripod.com/juvenilehuntingtonsdisease/id51.html+ways+prevent+dehydration&cd=7&hl=en&ct=clnk&gl=us" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; InfoPath.1; .NET CLR 1.1.4322)" "-" GET /kid/watch/er/headers_25350/sweat.gif - "HTTP/1.1"
206.170.178.32 - "-" - [08/Mar/2010:17:13:51 -0500] "GET /licensees/licensee1/js/gam.html?1268086595578 HTTP/1.1" 501 148 "http://kidshealth.org/teen/diseases_conditions/cancer/cancer_center.html" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; OfficeLiveConnector.1.3; OfficeLivePatch.0.0)" "KHCookie=KH_1001268085452718; __gads=ID=e68ffde799f78393:T=1268085328:S=ALNI_MYBvGiLh8FRIrpyFX0QiO2Q5QfHyQ; textSize=1" GET /licensees/licensee1/js/gam.html 1268086595578 "HTTP/1.1"
24.187.125.82 - "-" - [08/Mar/2010:17:13:50 -0500] "GET /parent/firstaid_safe/emergencies/headers_27373/firstaid_dehyd1.jpg HTTP/1.1" 502 292 "http://74.125.47.132/search?q=cache:eCkNVYKY9tkJ:endoflifecare.tripod.com/juvenilehuntingtonsdisease/id51.html+ways+prevent+dehydration&cd=7&hl=en&ct=clnk&gl=us" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; InfoPath.1; .NET CLR 1.1.4322)" "-" GET /parent/firstaid_safe/emergencies/headers_27373/firstaid_dehyd1.jpg - "HTTP/1.1"
66.249.71.37 - "-" - [08/Mar/2010:17:13:52 -0500] "GET /PageManager.jsp?dn=familydoctor&lic=44&cat_id=20253&article_set=32826&ps=104 HTTP/1.1" 501 148 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-" GET /PageManager.jsp dn=familydoctor&lic=44&cat_id=20253&article_set=32826&ps=104 "HTTP/1.1"
209.235.199.130 - "-" - [08/Mar/2010:17:13:51 -0500] "GET http://kidshealth.org/kid/htbw/_bfs_DSmoviesource.html HTTP/1.1" 403 142 "http://school.nettrekker.com" "Noelios-Restlet-Engine/1.1.6" "__utma=245211015.1365694104.1268084226.1268084226.1268084226.1;  __utmb=245211015.4.10.1268084226;  __utmc=245211015;  __utmz=245211015.1268084226.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none);  eduDymSessionId=682164954017634;  ttNode=41363;  cnt_ids=;  rightbar=visible;  rAloudPlayer=true" GET http://kidshealth.org/kid/htbw/_bfs_DSmoviesource.html - "HTTP/1.1"
208.87.137.231 - "-" - [08/Mar/2010:17:13:52 -0500] "GET /licensees/licensee1/js/gam.html?1268086575429 HTTP/1.1" 501 148 "http://kidshealth.org/parent/firstaid_safe/home/household_checklist_walls.html" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; GTB6.3)" "textSize=1; KHCookie=KH_691240337640513" GET /licensees/licensee1/js/gam.html 1268086575429 "HTTP/1.1"
216.54.146.95 - "-" - [08/Mar/2010:17:13:53 -0500] "GET /licensees/licensee1/images/IRParents/layout/kh_parents_enewsBanner.png HTTP/1.1" 501 148 "http://kidshealth.org/parent/infections/lung/whooping_cough.html" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; AskTB5.6)" "-" GET /licensees/licensee1/images/IRParents/layout/kh_parents_enewsBanner.png - "HTTP/1.1"
has anyone else run into anything like this?

Also, i see the following in the errors logs - not sure if they are related.
[08/Mar/2010:17:22:10] failure ( 3760): HTTP3068: Error receiving request from 216.249.145.230 (Connection refused)
[08/Mar/2010:17:22:10] failure ( 3760): HTTP3068: Error receiving request from 216.249.145.230 (Connection refused)
[08/Mar/2010:17:22:13] warning ( 3760): for host 66.244.4.46 trying to GET /updates/K_featured.gif, finish-response reports: HTTP2228: Response content length mismatch (0 bytes with a content length of 27760)
[08/Mar/2010:17:22:19] warning ( 3760): for host 66.244.4.46 trying to GET /updates/K_featured.gif, finish-response reports: HTTP2228: Response content length mismatch (0 bytes with a content length of 27760)
any ideas/thoughts/suggestions are welcome.

thanks
JB                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
  • 1. Re: random 501,403 errors and HTTP2228 and HTTP3068 errors
    807573 Oracle ACE
    Currently Being Moderated
    More information about the 501 errors - the reason for the 501 errors a malformed request on the origin-server - "ET" request instead of "GET" but i have no idea why it is happening. I have a open ticket with Sun support. I tried version 4.0.12 with a vanilla configuration (hust the reverse mapping ) but i have the same issue. here is a snippet of the access log from the origin -server


    zz.z.zz.zz - - [11/Mar/2010:10:30:25 -0500] "ET /licensees/licensee1/images/IRParents/banners/donateNew.gif HTTP/1.1" 501 148 
    zz.z.zz.zz - - [11/Mar/2010:10:30:27 -0500] "ET /licensees/licensee1/images/IRParents/banners/donateNew.gif HTTP/1.1" 501 148 zz.z.zz.zz - - [11/Mar/2010:10:30:30 -0500] "ET /licensees/licensee1/images/IRParents/banners/donateNew.gif HTTP/1.1" 501 148 zz.z.zz.zz - - [11/Mar/2010:10:30:33 -0500] "ET /category_headers/category27/growth_development.jpg HTTP/1.1" 501 148 
  • 2. Re: random 501,403 errors and HTTP2228 and HTTP3068 errors
    807573 Oracle ACE
    Currently Being Moderated
    Update on this issue.

    Turns out that there is bug in both the proxy server and the web server. Sun has filed a bug and provided me with a FVB for the proxy that seems to be working.
  • 3. Re: random 501,403 errors and HTTP2228 and HTTP3068 errors
    807573 Oracle ACE
    Currently Being Moderated
    Can you be more specific about which problem you described as well as what bug was identified by Sun?
  • 4. Re: random 501,403 errors and HTTP2228 and HTTP3068 errors
    807573 Oracle ACE
    Currently Being Moderated
    here is the note from the Sun engineer.


    Here's an update.  I've reviewed the customer's proxy snoop data, and it appears that when we get a request that turns out to be a "ET"request on the origin server, prior to that, there is a POST request.  The POST request gets to the proxy server from their client meeting all HTTP protocol requirements.  It has the Content-Length and the request entity body arrives as expected from the client.  However, it appears our proxy server also has a bug.  It forwards the POST header info without the POST entity body. NOTE that the POST entity body actually arrives in a seperate TCP packet,milliseconds later.  Since the request had been forwarded on to the origin server, the origin server processes the POST with a 200 success response without having processed the POST entity body.

    After this, the GET request arrives at the proxy and gets forwarded onto the origin server.  The origin server now malforms the request and makes it a "ET" request.

    So the way I see it we have 2 bugs:
    +1. proxy server is not forwarding the POST entity body to the origin server.+
    +2. origin server WS7.0u8, malforms the next request if the pior request is a POST request without POST entity body.+
  • 5. Re: random 501,403 errors and HTTP2228 and HTTP3068 errors
    807573 Oracle ACE
    Currently Being Moderated
    Thank you for the details.