megaphone
Update your Profile with your Support type to get your Support Type badge.
Nominate Your Peers for NetSuite Support Community's Choice of the Quarter! Submit your nomination today.
Stay in the know of how NetSuite can help grow your business with our guides, webinars, and events. Subscribe Here
What Topic Should We Cover Next? Your idea could be our next feature—drop your suggestion now!
No Limits. Just possibilities. Join us for complimentary one-day events around the world and step into a future fueled by AI and limitless potential. Discover what's next at SuiteConnect Tour 2026.
Try Intelligent Payment Automation – Fee Free For Your First Month For more information, visit this thread.
New AI Community Guidelines. Please review and follow them to ensure AI use stays safe, accurate, and compliant.
Expand your NetSuite knowledge by joining our Ask A Guru Live sessions. RSVP on this event now.

Discussions
Stay up-to-date with the latest news from NetSuite. You’ll be in the know about how to connect with peers and take your business to new heights at our virtual, in-person, on demand events, and much more.
New AI Community Guidelines. Please review and follow them to ensure AI use stays safe, accurate, and compliant.
Please note that on Saturday, April 11, 2026, at 8:00 PM Pacific time, our Case Management System will undergo a scheduled maintenance for approximately 30 minutes. During this time, case creation via SuiteAnswers will be unavailable and inbound calls will be routed to Customer Service.

robots.txt - Crawl-Delay: 20

edited Nov 27, 2019 7:42PM in Web Site / E-Commerce 147 comments

Anybody notice a new addition to their robots.txt?

NetSuite is adding:
Crawl-Delay: 20

This translates into requesting that any spider / bot which comes through to wait 20 seconds between requests.

Its not really a big deal, I would not want a bot going any faster than this anyway.

That said, the rogue bots are why you would do such a command but those rogue bots will not respect it anyway.

From what I have read, Google does not recognize the command. Yahoo and MSN apparently do. Anybody know the answer for Ask.com?

Does anyone recall a announcement that robots.txt was being modified?

Howdy, Stranger!

Log In

To view full details, sign in.

Register

Don't have an account? Click here to get started!

Leaderboard

Community Whiz

Quarter 2 (Apr-Jun 2026)

This Week's Leaders

This Month's Leaders

All Time Leaders