megaphone
Update your Profile with your Support type to get your Support Type badge.
Nominate Your Peers for NetSuite Support Community's Choice of the Quarter! Submit your nomination today.
Expand your NetSuite knowledge by joining our Ask A Guru Live sessions. RSVP on this event now.
Stay in the know of how NetSuite can help grow your business with our guides, webinars, and events. Subscribe Here
What Topic Should We Cover Next? Your idea could be our next feature—drop your suggestion now!
No Limits. Just possibilities. Join us for complimentary one-day events around the world and step into a future fueled by AI and limitless potential. Discover what's next at SuiteConnect Tour 2026.
Try Intelligent Payment Automation – Fee Free For Your First Month For more information, visit this thread.

Discussions
Stay up-to-date with the latest news from NetSuite. You’ll be in the know about how to connect with peers and take your business to new heights at our virtual, in-person, on demand events, and much more.
Now is the time to ask your NetSuite-savvy friends and colleagues to join the NetSuite Support Community! Refer now! Click here to watch and learn more!
Stay in the Know

Be sure you're subscribed to NetSuite communication to stay in the know about monthly happenings, updates and announcements. Subscribe

robots.txt - Crawl-Delay: 20

edited Nov 27, 2019 7:42PM in Web Site / E-Commerce 147 comments

Anybody notice a new addition to their robots.txt?

NetSuite is adding:
Crawl-Delay: 20

This translates into requesting that any spider / bot which comes through to wait 20 seconds between requests.

Its not really a big deal, I would not want a bot going any faster than this anyway.

That said, the rogue bots are why you would do such a command but those rogue bots will not respect it anyway.

From what I have read, Google does not recognize the command. Yahoo and MSN apparently do. Anybody know the answer for Ask.com?

Does anyone recall a announcement that robots.txt was being modified?

Howdy, Stranger!

Log In

To view full details, sign in.

Register

Don't have an account? Click here to get started!

Leaderboard

Community Whiz

Quarter 1 (Jan-Mar 2026)

This Week's Leaders

This Month's Leaders

All Time Leaders