Discussions
Stay up-to-date with the latest news from NetSuite. You’ll be in the know about how to connect with peers and take your business to new heights at our virtual, in-person, on demand events, and much more.
New AI Community Guidelines. Please review and follow them to ensure AI use stays safe, accurate, and compliant.
Update: Narrative Insights has been restored and is now available.
Narrative Insights is Temporarily Unavailable due to an Infrastructure Issue. Learn how This Impacts Your Account and What to Expect While the Feature is Disabled.
Narrative Insights is Temporarily Unavailable due to an Infrastructure Issue. Learn how This Impacts Your Account and What to Expect While the Feature is Disabled.
robots.txt - Crawl-Delay: 20
Anybody notice a new addition to their robots.txt?
NetSuite is adding:
Crawl-Delay: 20
This translates into requesting that any spider / bot which comes through to wait 20 seconds between requests.
Its not really a big deal, I would not want a bot going any faster than this anyway.
That said, the rogue bots are why you would do such a command but those rogue bots will not respect it anyway.
From what I have read, Google does not recognize the command. Yahoo and MSN apparently do. Anybody know the answer for Ask.com?
Does anyone recall a announcement that robots.txt was being modified?
0