Discussions
Read on for the latest updates including:
• Agenda Builder launch
• SuiteWorld On Air registration
• New NetSuite Prompt Studio Contest
• And more!
Check out this thread to learn more!
robots.txt - Crawl-Delay: 20
Anybody notice a new addition to their robots.txt?
NetSuite is adding:
Crawl-Delay: 20
This translates into requesting that any spider / bot which comes through to wait 20 seconds between requests.
Its not really a big deal, I would not want a bot going any faster than this anyway.
That said, the rogue bots are why you would do such a command but those rogue bots will not respect it anyway.
From what I have read, Google does not recognize the command. Yahoo and MSN apparently do. Anybody know the answer for Ask.com?
Does anyone recall a announcement that robots.txt was being modified?