0 Replies Latest reply on Oct 9, 2010 11:06 AM by 804086

    Re-dispatching tasks with the ProcessingPattern

      Hi All,

      We are using the Processing Pattern to dispatch work in a cluster. Generally, the work is submitted asynchronously, but there are some tasks that we need to submit synchronously, meaning - Task2 must execute right after Task1 has completed, Task3 must execute right after Task2 has completed, etc... The way we do this is to submit the task to the ProcessingSession and then get() the result from the SubmissionOutcome. After we've received a result, we submit synchronously Task2, and so on. Now here comes the catch. Let's say we submit Task2 from Node1 and Task2 starts executing on Node2. Now, if Node2 fails (meaning shuts down or gets disconnected from the network) during the execution of that task, the recovery mechanism will kick in, restore Task2 from the database, and re-submit it. Here we observe the following behavior:

      1. An exception like the following gets thrown:

      Exception in thread "EventQueueRecoveryThread" Portable(java.lang.IllegalStateException): An object with the key Identifier{0x0000012B878726EB0AE802E7148B05F30AC8B079F579FEDCC7821798F4B22746} already exists

      2. The thread on Node1 which waits to get() the result of the task stays like that forever. As a result, Task3 never gets executed.

      What we would like to achieve is the following behavior:

      We want somehow to resubmit the failed task in a way that the task will execute and the result will be returned back to the thread that submitted the task originally. This way the thread will continue executing Task3 (instead of freezing).

      The only other option we have now is to cancel the task first and then submit it again. This will solve partially both issues (but Task3 still won't get executed and the operation will be marked as failed), but we're looking for a way to fully recover from this situation.

      Any help will be greatly appreciated!


      Edited by: 801083 on Oct 9, 2010 4:04 AM