This discussion is archived
4 Replies Latest reply: Jul 12, 2012 3:20 AM by JAC2703 RSS

Delay between each Fetch

JAC2703 Newbie
Currently Being Moderated
Hi all,

How can I delay each fetch request?

I have source X which has one crawl thread set in the 'Crawling Parameters' page. The trouble is, this one thread is hammering the server causing it to hit is requests per minute limit. Because of this, the webserver throws a status 'Service unavailable'.

Ideally, I would like to specify a maximum number of requests to a given Source or site per minute. Is this possible?

Thanks for your time,

James.
  • 1. Re: Delay between each Fetch
    Roger Ford Expert
    Currently Being Moderated
    All you can do is reduce the number of crawler threads, with a minimum of 1.
  • 2. Re: Delay between each Fetch
    JAC2703 Newbie
    Currently Being Moderated
    Thanks Roger,

    Is that really the only metric we can influence?

    This is firing off a more than 1 thread per second which is many times more than the web servers allowed requests per minute.

    Thanks for your response.

    James.
  • 3. Re: Delay between each Fetch
    Roger Ford Expert
    Currently Being Moderated
    I'm not sure if there are any internal settings that would add a delay, I think there might possibly be but I'm not involved with the SES product any more.

    You could try raising an SR with support.

    When you say "This is firing off a more than 1 thread per second" do you mean more than one request per second?

    That doesn't seem high - it would take a long time to crawl most datasources at one request per second. Can you not adjust the server so it allows more requests from the SES server IP address, while still limiting other addresses?
  • 4. Re: Delay between each Fetch
    JAC2703 Newbie
    Currently Being Moderated
    When you say "This is firing off a more than 1 thread per second" do you mean more than one request per second?
    Yes, sorry. That's exactly what I mean - more than one request to the webserver per second.

    I will have to work around it, like you say, at the webserver end.

    Thanks, Roger.

    James.

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points