4 Replies Latest reply: Jul 12, 2012 5:20 AM by JAC2703 RSS

    Delay between each Fetch

    JAC2703
      Hi all,

      How can I delay each fetch request?

      I have source X which has one crawl thread set in the 'Crawling Parameters' page. The trouble is, this one thread is hammering the server causing it to hit is requests per minute limit. Because of this, the webserver throws a status 'Service unavailable'.

      Ideally, I would like to specify a maximum number of requests to a given Source or site per minute. Is this possible?

      Thanks for your time,

      James.
        • 1. Re: Delay between each Fetch
          Roger Ford-Oracle
          All you can do is reduce the number of crawler threads, with a minimum of 1.
          • 2. Re: Delay between each Fetch
            JAC2703
            Thanks Roger,

            Is that really the only metric we can influence?

            This is firing off a more than 1 thread per second which is many times more than the web servers allowed requests per minute.

            Thanks for your response.

            James.
            • 3. Re: Delay between each Fetch
              Roger Ford-Oracle
              I'm not sure if there are any internal settings that would add a delay, I think there might possibly be but I'm not involved with the SES product any more.

              You could try raising an SR with support.

              When you say "This is firing off a more than 1 thread per second" do you mean more than one request per second?

              That doesn't seem high - it would take a long time to crawl most datasources at one request per second. Can you not adjust the server so it allows more requests from the SES server IP address, while still limiting other addresses?
              • 4. Re: Delay between each Fetch
                JAC2703
                When you say "This is firing off a more than 1 thread per second" do you mean more than one request per second?
                Yes, sorry. That's exactly what I mean - more than one request to the webserver per second.

                I will have to work around it, like you say, at the webserver end.

                Thanks, Roger.

                James.