megaphone
Update your Profile with your Support type to get your Support Type badge.
Nominate Your Peers for NetSuite Support Community's Choice of the Quarter! Submit your nomination today.
Stay in the know of how NetSuite can help grow your business with our guides, webinars, and events. Subscribe Here
What Topic Should We Cover Next? Your idea could be our next feature—drop your suggestion now!
No Limits. Just possibilities. Join us for complimentary one-day events around the world and step into a future fueled by AI and limitless potential. Discover what's next at SuiteConnect Tour 2026.
Try Intelligent Payment Automation – Fee Free For Your First Month For more information, visit this thread.
New AI Community Guidelines. Please review and follow them to ensure AI use stays safe, accurate, and compliant.
Check out the Announcements page to stay updated with the latest news, updates, upcoming events, and important community information!
What if your ERP could help surface the next step before you go looking for it? Watch here to learn more!

Discussions
Stay up-to-date with the latest news from NetSuite. You’ll be in the know about how to connect with peers and take your business to new heights at our virtual, in-person, on demand events, and much more.
New AI Community Guidelines. Please review and follow them to ensure AI use stays safe, accurate, and compliant.
Keep an eye out for upcoming NetSuite events, including meetups, workshops, and webinars. These sessions are a great way to connect with peers, learn from experts, and stay current on the latest NetSuite updates and best practices. Registration links are provided in each event.

robots.txt - Crawl-Delay: 20

edited Nov 27, 2019 7:42PM in Web Site / E-Commerce 147 comments

Anybody notice a new addition to their robots.txt?

NetSuite is adding:
Crawl-Delay: 20

This translates into requesting that any spider / bot which comes through to wait 20 seconds between requests.

Its not really a big deal, I would not want a bot going any faster than this anyway.

That said, the rogue bots are why you would do such a command but those rogue bots will not respect it anyway.

From what I have read, Google does not recognize the command. Yahoo and MSN apparently do. Anybody know the answer for Ask.com?

Does anyone recall a announcement that robots.txt was being modified?

Howdy, Stranger!

Log In

To view full details, sign in.

Register

Don't have an account? Click here to get started!

Leaderboard

Community Whiz

Quarter 2 (Apr-Jun 2026)

This Week's Leaders

This Month's Leaders

All Time Leaders