Discussions
Stay up-to-date with the latest news from NetSuite. You’ll be in the know about how to connect with peers and take your business to new heights at our virtual, in-person, on demand events, and much more.
Now is the time to ask your NetSuite-savvy friends and colleagues to join the NetSuite Support Community! Refer now! Click here to watch and learn more!
Update your Profile with your Support type to get your Support Type badge.
Don't miss out on our Question of the Week! You only have until tomorrow, October 23, 2025, 2PM ET to comment your answers and earn rewards! Click here for more details.
Uncover the power of data with the Analytics Hub —your ultimate guide to mastering NetSuite Saved Searches and Reports. Simplify the complex and unlock your organization's true potential. Dive into the Analytics Hub now and soar to new heights!
To help you get the most out of your 2025.2 Release Preview account, review the topics outlined in the 2025.2 Release Preview Guide.
Nominate Your Peers for NetSuite Support Community's Choice of the Quarter! Submit your nomination today.
Intelligent Payment Automation version 1.0.3 is now available in the SuiteApp Marketplace. The SuiteApp, powered by BILL, lets you automate payments, manage vendor details, and bank account information within NetSuite. Learn more
robots.txt - Crawl-Delay: 20
Anybody notice a new addition to their robots.txt?
NetSuite is adding:
Crawl-Delay: 20
This translates into requesting that any spider / bot which comes through to wait 20 seconds between requests.
Its not really a big deal, I would not want a bot going any faster than this anyway.
That said, the rogue bots are why you would do such a command but those rogue bots will not respect it anyway.
From what I have read, Google does not recognize the command. Yahoo and MSN apparently do. Anybody know the answer for Ask.com?
Does anyone recall a announcement that robots.txt was being modified?
0