How can I delay each fetch request?
I have source X which has one crawl thread set in the 'Crawling Parameters' page. The trouble is, this one thread is hammering the server causing it to hit is requests per minute limit. Because of this, the webserver throws a status 'Service unavailable'.
Ideally, I would like to specify a maximum number of requests to a given Source or site per minute. Is this possible?
Thanks for your time,
Is that really the only metric we can influence?
This is firing off a more than 1 thread per second which is many times more than the web servers allowed requests per minute.
Thanks for your response.
I'm not sure if there are any internal settings that would add a delay, I think there might possibly be but I'm not involved with the SES product any more.
You could try raising an SR with support.
When you say "This is firing off a more than 1 thread per second" do you mean more than one request per second?
That doesn't seem high - it would take a long time to crawl most datasources at one request per second. Can you not adjust the server so it allows more requests from the SES server IP address, while still limiting other addresses?