This content has been marked as final. Show 4 replies
All you can do is reduce the number of crawler threads, with a minimum of 1.1 person found this helpful
Is that really the only metric we can influence?
This is firing off a more than 1 thread per second which is many times more than the web servers allowed requests per minute.
Thanks for your response.
I'm not sure if there are any internal settings that would add a delay, I think there might possibly be but I'm not involved with the SES product any more.
You could try raising an SR with support.
When you say "This is firing off a more than 1 thread per second" do you mean more than one request per second?
That doesn't seem high - it would take a long time to crawl most datasources at one request per second. Can you not adjust the server so it allows more requests from the SES server IP address, while still limiting other addresses?
When you say "This is firing off a more than 1 thread per second" do you mean more than one request per second?Yes, sorry. That's exactly what I mean - more than one request to the webserver per second.
I will have to work around it, like you say, at the webserver end.