This discussion is archived
8 Replies Latest reply: Apr 19, 2011 10:07 AM by jtahlborn RSS

Thread pool rejecting threads when I don't think it should, ideas?

852668 Newbie
Currently Being Moderated
Hi,
I have a server application in which I only want a specific number of simultaneous requests. If the server gets more then this number it is suppose to close the connection (sends an HTTP 503 error to the client). To do this I used a fix thread pool. When I start the server and submit the max number of requests I get the expected behavior. However if I resubmit the request (within a small period of time, e.g. 1-15 seconds after the first one) I get very odd behavior in that some of the requests are rejected. For example if I set the max to 100 the first set of requests will work fine (100 requests, 100 responses). I then submit again and a small number will be rejected (I've seen it range from 1 to 15 rejected)....

I made a small app which kind of duplicates this behavior (see below). Basically when I see is that the first time submitting requests works fine but the second time I get a rejected one. As best as I can tell none should be rejected....

Here is the code, I welcome your thoughts or if you see something I am doing wrong here...
-----
<pre>
import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicInteger;

public class ThreadPoolTest {
     static AtomicInteger count = new AtomicInteger();
     
     public static class threaded implements Runnable {

          @Override
          public void run() {
               System.out.println("In thread: " + Thread.currentThread().getId());
               try {
                    Thread.sleep(500);
               } catch (InterruptedException e) {
                    System.out.println("Thread: " + Thread.currentThread().getId()
                              + " interuptted");
               }
               System.out.println("Exiting run: " + Thread.currentThread().getId());
          }
     }
     
     private static int maxThreads = 3;
     private ThreadPoolExecutor pool;
     
     public ThreadPoolTest() {
          super();
          pool = new java.util.concurrent.ThreadPoolExecutor(
                    1, maxThreads - 1, 60L, TimeUnit.SECONDS,
                    new ArrayBlockingQueue<Runnable>(1));

     }

     public static void main(String[] args) throws InterruptedException {
          ThreadPoolTest object = new ThreadPoolTest();
          object.doThreads();

          Thread.sleep(3000);
          object.doThreads();
          
          object.pool.shutdown();
          try {
               object.pool.awaitTermination(60, TimeUnit.SECONDS);
          } catch (InterruptedException e) {
               // TODO Auto-generated catch block
               e.printStackTrace();
          }
     }
     
     private void doThreads() {
          int submitted = 0, rejected = 0;
          int counter = count.getAndIncrement();
          for (int x = 0; x < maxThreads ; x++) {
               try {
                    System.out.println("Run #: " + counter + " submitting " + x);
                    pool.execute(new threaded());
                    submitted++;
               }
               catch (RejectedExecutionException re) {
                    System.err.println("\tRun #: " + counter + ", submission " + x
                              + " was rejected");
                    System.err.println("\tQueue active: " + pool.getActiveCount());
                    System.err.println("\tQueue size: " + pool.getPoolSize());
                    rejected++;
               }
          }
          System.out.println("\n\n\tRun #: " + counter);
          System.out.println("\tSubmitted: " + (submitted + rejected));
          System.out.println("\tAccepted: " + submitted);
          System.out.println("\tRejected: " + rejected + "\n\n");
     }
}
</pre>
  • 1. Re: Thread pool rejecting threads when I don't think it should, ideas?
    jtahlborn Expert
    Currently Being Moderated
    this looks like one giant race condition to me. you do a bunch of stuff with random sleeps. sleep(), even in the best of times isn't exact (on normal systems your accuracy is usually around the 10ms mark). additionally, you've got delays between waits and notifies (queuing runnables and pulling them off of the queue). now, you throw a bunch of threads in there and you have the os scheduler to contend with. long story short, you have no guarantee that the tasks from the first doThreads() call have cleared out before you run the second batch.

    Edited by: jtahlborn on Apr 17, 2011 8:37 PM
  • 2. Re: Thread pool rejecting threads when I don't think it should, ideas?
    EJP Guru
    Currently Being Moderated
    I have a server application in which I only want a specific number of simultaneous requests.
    You are going to find that extremely difficult to control. Are you aware of the TCP/IP backlog queue? It permits the existence of typically hundreds of connections that have already been completed by the stack before the application even knows about them. That means that the client can connect and send data before your application ever gets around to accepting the connection.

    If you have a specific part of the application where you only want N simultaneous users I would have a look elsewhere in java.util.concurrent. Maybe a CountDownLatch.
  • 3. Re: Thread pool rejecting threads when I don't think it should, ideas?
    852668 Newbie
    Currently Being Moderated
    First thank you for taking the time to reply, I do appreciate it.


    jtahlborn - The code provided here is a contrived example trying to emulate the bigger app as best as I could. The actual program doesn't have any sleeps, the sleep in the secondary thread is to simulate the program doing some work & replying to a request. The sleep in the primary thread is to simulate a small delay between 'requests' to the pool. I can make this 1 second and up to (at least) 5 seconds with the same results. Additionally I can take out the sleep in the secondary thread and still see the a rejection.

    EJP - Yes I am aware of the TCP/IP queue, however; I don't see that as relevant to my question. The idea is not to prevent the connection but to respond to the client saying we can't process the request (send an "HTTP 503" error). So basically if we have, say, 100 threads running then the 101st, connection will get a 503 error and the connection will be closed.

    Also my test platform - Windows 7 64bit running Java 1.6.0_24-b07 (32bit) on an Intel core i7.

    -----

    It occurred to me that I did not show the output of the test program. As the output shows below, the first set of requests are all processed properly. The second set of requests is not. The pool should have 2 threads and 1 slot in the queue, so by the time the second "request" is made at least 2 of the requests from the first call should be done processing, so I could possibly understand run 1, submit #2 failing but not submit 1.

    <pre>
    Run #: 0 submitting 0
    Run #: 0 submitting 1
    Run #: 0 submitting 2
    In thread: 8
    In thread: 9
    Exiting run: 8
    Exiting run: 9

         Run #: 0
         Submitted: 3
         Accepted: 3
         Rejected: 0


    In thread: 8
    Exiting run: 8
    Run #: 1 submitting 0
    In thread: 9
    Run #: 1 submitting 1
         Run #: 1, submission 1 was rejected
         Queue active: 1
         Queue size: 2
    Run #: 1 submitting 2

         Run #: 1
         Submitted: 3
         Accepted: 2
         Rejected: 1


    In thread: 8
    Exiting run: 9
    Exiting run: 8
    </pre>
  • 4. Re: Thread pool rejecting threads when I don't think it should, ideas?
    jtahlborn Expert
    Currently Being Moderated
    JimM wrote:
    jtahlborn - The code provided here is a contrived example trying to emulate the bigger app as best as I could. The actual program doesn't have any sleeps, the sleep in the secondary thread is to simulate the program doing some work & replying to a request. The sleep in the primary thread is to simulate a small delay between 'requests' to the pool. I can make this 1 second and up to (at least) 5 seconds with the same results. Additionally I can take out the sleep in the secondary thread and still see the a rejection.
    my comments about race conditions are still relevant. without any explicit coordination, you can't make any assumptions about what threads are available at any given time.

    that said, i ran the test code on my system 10 times and didn't any rejections: java 1.6.0_22, ubuntu, core i7. only after running a parallel compilation on my box to add some system load was i able to get a rejection. this supports my theory that it's a race condition in your code.

    looking at the ThreadPoolExecutor impl, i think i know why this is happening. if the expected number of threads in the tpool are already started, every subsequent task submission goes through the queue. so, in the second run, since the max number of threads have been started, the first 2 tasks hit the queue and the second is rejected. the race condition being whether or not the first task can be pulled off the queue faster than the second task can be offered.

    Edited by: jtahlborn on Apr 18, 2011 10:59 AM
  • 5. Re: Thread pool rejecting threads when I don't think it should, ideas?
    852668 Newbie
    Currently Being Moderated
    I can understand what you are saying, however; in the real app there is no race condition. The test program we run makes 'x' requests and gets 'x' responses back, we then run it again and we end up getting several requests rejected. Since the test application is invoked separately for each run there are no requests on the server (e.g. no load).

    Interesting that you had trouble getting the rejected exception, I get it consistently.

    Thanks again
  • 6. Re: Thread pool rejecting threads when I don't think it should, ideas?
    jtahlborn Expert
    Currently Being Moderated
    did you see the last part of what i wrote? if you are making x requests and your ArrayBlockingQueue size limit is less than x, then you have a good chance of seeing a rejection on a subsequent pass, especially in a loaded system. it might make more sense for your scenario to use a SynchronousQueue instead of an ArrayBlockingQueue.

    Edited by: jtahlborn on Apr 18, 2011 3:53 PM
  • 7. Re: Thread pool rejecting threads when I don't think it should, ideas?
    852668 Newbie
    Currently Being Moderated
    OK, I think I see what you are saying. I reviewed the ThreadPoolExecutor code (again) and think I had a flaw in how I thought the flow went.

    I think what you are saying is that it is possible for 2 requests to come in simultaneously such that one of the requests is put in the queue and before it can be moved from the queue to an available thread the next requests tries to put a new job in the queue and as such the queue is 'full' and failure occurs, that makes sense.


    -jim
  • 8. Re: Thread pool rejecting threads when I don't think it should, ideas?
    jtahlborn Expert
    Currently Being Moderated
    JimM wrote:
    OK, I think I see what you are saying. I reviewed the ThreadPoolExecutor code (again) and think I had a flaw in how I thought the flow went.

    I think what you are saying is that it is possible for 2 requests to come in simultaneously such that one of the requests is put in the queue and before it can be moved from the queue to an available thread the next requests tries to put a new job in the queue and as such the queue is 'full' and failure occurs, that makes sense.
    exactly.

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points