Below is my first try at a robots.txt starter kit for NetSuite webstores. Robots or bots for short are what Google, Yahoo, etc use to crawl the links on your site. Why care how the robots crawl your site?
1) SEO
2) Don't waste the bot's time on non-essential pages link contact us, about us, etc.
3) Don't waste your bandwidth
Note: I could not figure out how to handle the URL's below. Help would be great:
http://shopping.netsuite.com/c.999999
https://checkout.netsuite.com/c.999999
https://system.netsuite.com/c.999999
Where c.999999 is your NetSuite customer number. These are the links to checkout, customer service, etc.
I have read NetSuite will exclude these links for bots but have not been able to get a positive test.
Robots.txt starts on next line.
#Block Googlebot-Image
User-agent: Googlebot-Image
Disallow: /
#
User-agent: *
Disallow: /s.nl #NetSuite Cart and other stuff
Disallow: /core/media/media.nl #NetSuite image handler
#* is wildcard for Googlebots
Disallow: /*?sort1desc* #Sorts on item lists
Disallow: /*jsessionid=* #Netsuite should handle jsessionid
Disallow: /*?sc=* #For items in multiple categories.
#Below unique to our implementation
#Nothing in site or subfolder
Disallow: /site/
Below is a link to Google's information on robots.txt
http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40360