Is there anyone else using a Google Search Appliance for there web site search? Are you seeing any issues in indexing pages hosted on NetSuite?
The reason I ask is we recently noticed that the Google Search Appliance we run for our site search was not returning the expected results for pages hosted on NetSuite. In looking at the Retrieval Errors: on the Google Search Appliance it shows "Error Excluded: Other 4xx HTTP response code." for a high percentage of the pages it tried to retrieve from NetSuites site. But if I go to the link it in a browser works fine. Our guess is sometime in the recent past (1-2 months) something changed at NetSuite to where our Google Appliance was no longer fully indexing the portion of our site that is hosted on NetSuite.
So after some testing and digging and getting it to where I can appear like a search appliance to NetSuite I find that if you do a request looking like the search appliance that NetSuite returns an error of 503 Service Unavailable a hight percentage of the time. But if you look like a normal browser, you can request all the pages you want and not get an error. These results lead me to test as though I were a web crawler (Yahoo,Google) with similar results. This leads me to believe that NetSuite is throttling results to web crawlers which is causing problem with the Google Search Appliance and makes me wonder if it is causing similar issues with web crawlers and search engines.
Test looking like search appliance - Error
curl www2.promax.com/EditShare-3-84TB-Storage -v -A 'gsa-crawler (Enterprise; MID-02096; admin@promax.com)' -0 * About to connect() to www2.promax.com port 80 * Trying 63.209.28.41... * connected * Connected to www2.promax.com (63.209.28.41) port 80 > GET /EditShare-3-84TB-Storage HTTP/1.0 User-Agent: gsa-crawler (Enterprise; MID-02096; admin-promax.com) Host: www2.promax.com Pragma: no-cache Accept: */* < HTTP/1.1 503 Service Unavailable < Date: Wed, 28 Nov 2007 01:13:34 GMT < Server: Oracle-Application-Server-10g < Set-Cookie: NS_VER=2007.1.0; domain=www2.promax.com; path=/ < P3P: CP="CAO PSAa OUR BUS PUR" < Vary: User-Agent < NS_RTIMER_COMPOSITE: 34913981:73686F702D6A6176613030382E7376616C652E6E65746C65646765722E636F6D:80 < Content-Location: /s.nl < Connection: close < Content-Type: text/html; charset=utf-8 * Closing connection #0 <HTML><HEAD><TITLE>503 Service Unavailable</TITLE></HEAD><BODY><H1>503 Service Unavailable</H1></BODY></HTML>
Test looking like a common browser - OKcurl www2.promax.com/EditShare-3-84TB-Storage -v -0 * About to connect() to www2.promax.com port 80 * Trying 63.209.28.41... * connected * Connected to www2.promax.com (63.209.28.41) port 80 > GET /EditShare-3-84TB-Storage HTTP/1.0 User-Agent: curl/7.13.1 (powerpc-apple-darwin8.0) libcurl/7.13.1 OpenSSL/0.9.7l zlib/1.2.3 Host: www2.promax.com Pragma: no-cache Accept: */* < HTTP/1.1 200 OK < Date: Wed, 28 Nov 2007 01:33:08 GMT < Server: Oracle-Application-Server-10g < Set-Cookie: NS_VER=2007.1.0; domain=www2.promax.com; path=/ < P3P: CP="CAO PSAa OUR BUS PUR" < Vary: User-Agent < Set-Cookie: JSESSIONID=0a0002491f437da88a6e2273435298b8c48e3b70ffd7.e3eTaxmKbNaNe34Pa38Ta38Qbxf0; path=/ < Set-Cookie: NLShopperId=GPgu0nHwAMPT4qWf; Domain=.promax.com; Expires=Wed, 05-Dec-2007 01:33:09 GMT; Path=/ < Set-Cookie: NLVisitorId=GPgu0nDwACkDxFKD; Domain=.promax.com; Expires=Tue, 18-Nov-2008 01:33:09 GMT; Path=/ < NS_RTIMER_COMPOSITE: 25329392:73686F702D6A6176613030342E7376616C652E6E65746C65646765722E636F6D:80 < Content-Location: /s.nl < Expires: 0 < Pragma: No-Cache < Cache-Control: No-Cache,no-store < Connection: close < Content-Type: text/html; charset=utf-8 <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> <title>ProMax | EditShare 3.84TB Storage</title> <meta http-equiv="keywords" content="EditShare, 3.84TB, Storage, (12, drives, x, 320GB, plus, one, 12-port, RAID, card),EditShare ,ESSTOR-38BX12 " /> <meta http-equiv="description" content="EditShare 3.84TB Storage (12 drives x 320GB plus one 12-port RAID card) EditShare ESSTOR-38BX12 214161 " /> <!-- Begin Meta Tag HTML --><!-- End Meta Tag HTML --> <script language='JavaScript'>window.status='Loading...';</script> <script type='text/javascript' src='/javascript/NLUtil.jsp__NS_VER=2007.1.0&locale=en_US.nlqs'></script> <script language='JavaScript'>window.status='';</script> <!-- v=2007.1.0.38 db=8 c=628153 n=1 rg=1 e=0 r=17 sc=2 cat= id=A:4427 thm=100 bll= cart=374390 isstore=T price=5 cur=1 lang=en_US partner=0 promo=0 https=F page(candbc=T fromdbc=T v=2007.1 vol=F) --> <script> PositionX = 10; PositionY = 10; defaultWidth = 600; defaultHeight = 400; . . . .
Test looking like Yahoo's Web Crawler's - Errorcurl www2.promax.com/EditShare-3-84TB-Storage -v -A 'Yahoo-MMCrawler/3.x (mms dash mmcrawler dash support at yahoo dash inc dot com)' -0 * About to connect() to www2.promax.com port 80 * Trying 63.209.28.41... * connected * Connected to www2.promax.com (63.209.28.41) port 80 > GET /EditShare-3-84TB-Storage HTTP/1.0 User-Agent: Yahoo-MMCrawler/3.x (mms dash mmcrawler dash support at yahoo dash inc dot com) Host: www2.promax.com Pragma: no-cache Accept: */* < HTTP/1.1 503 Service Unavailable < Date: Wed, 28 Nov 2007 01:47:52 GMT < Server: Oracle-Application-Server-10g < Set-Cookie: NS_VER=2007.1.0; domain=www2.promax.com; path=/ < P3P: CP="CAO PSAa OUR BUS PUR" < Vary: User-Agent < NS_RTIMER_COMPOSITE: 16538278:73686F702D6A6176613031312E7376616C652E6E65746C65646765722E636F6D:80 < Content-Location: /s.nl < Connection: close < Content-Type: text/html; charset=utf-8 * Closing connection #0 <HTML><HEAD><TITLE>503 Service Unavailable</TITLE></HEAD><BODY><H1>503 Service Unavailable</H1></BODY></HTML>