5 Replies Latest reply: Jan 20, 2012 9:12 AM by Gerhard Drasch (Oracle) RSS

    Reliable Messaging, Queue Retries and AIA Error Handling

    RobertMetcalf
      Hi,
      I would like some advice on how to best use the AIA Error Handling process in a process that uses queues.
      We have a composite that picks up messages from a queue and runs a particular process on each message. If the process errors, it preforms a rollback placing the message back on the queue. It will then retry 3 times, possibly succeeding in another attempt.
      My question is how should this process use the Error Handling Framework. We want to see one single Error Notification only when the final retry fails. We don’t want to be notified for every failure as we will end up with a lot of junk notifications.
      How should this be handled in AIA? Is there a flag that is set in the message when it is the final retry which the composite can check before sending out the error notification? Or is another method used?
      Robert
        • 1. Re: Reliable Messaging, Queue Retries and AIA Error Handling
          Gerhard Drasch (Oracle)
          Robert,
          the typical approach is to not let the consumer adapter framework retry multiple times, cause that would lead to the execution of the entire flow several times including sending the error notifications as you observe it.

          The situation is different with the outbound adapters. If these guys retry a few times and then are successful with some attempt, that would not show up as an error anywhere. It would only raise an error and then exactly one error notification if the final retry fails.

          Gerhard
          • 2. Re: Reliable Messaging, Queue Retries and AIA Error Handling
            RobertMetcalf
            Gerhard,

            Thanks for the response.
            My question relates to messages consumers. I am using the JMS adapter to consume a message.

            We have it setup so that if the message fails 3 retries then it is placed on the error queue. This is setup via the weblogic console.

            You are suggesting that we don’t use the consumer adapter framework for the retries. What method should we use to implement the retries?

            We need to have the message tried 3 times if it fails the final time an error should be sent and the message should be moved to the error queue. The retries may be up to 10 minutes apart so the message should also not block other messages from being tried in the mean time.

            Robert
            • 3. Re: Reliable Messaging, Queue Retries and AIA Error Handling
              Gerhard Drasch (Oracle)
              Robert,
              the thing is that with your current design, the retry mechanism causes the entire flow to retry, and not only the failing bit. As an example, let's assume your flow looks like this: JMS Consumer->Req ABCS->EBS->Prov ABCS -> DB Adapter. So every time the DB Adapter fails to process the message for whatever reason, the message is rolled back to the originating queue - including triggering the default EH code in the ABCSs and it starts over from there with each retry. Usually I would rather try to configure the DB adapter for a re-try rather than the whole flow. In that case AIA EH would be triggered only once if the delivery fails.

              Now the problem is that you are looking for long time intervals between your re-tries, so the approach above won't fit. Unfortunately, there is no way I could think of for the EH code to realize if it is triggered in a final attempt according to your current design.

              Clearly the best solution would be to work around the root cause of why you actually need a retry in your target app.

              Also there might be ways to customize the AIA EH in order to filter out multiple messages related to the same unique message ID, but that would really the last resort.

              Gerhard
              • 4. Re: Reliable Messaging, Queue Retries and AIA Error Handling
                RobertMetcalf
                Hi,

                We have implemented our first integration and it is in production. I am now in the process of learning from the experience and creating strategies for future integrations. This will involve re-visiting the current integration. I am trying to look at the SOA and AIA documentation, our environment, our requirements and our experience developing and running our first integration, then come up with standard ways we should best implement each type of integration. I am working on an Asycnrenous design which needs to be reliable and use the AIA Error handling where appropriate to send notifications to support staff.

                My current thoughts are initially splitting it into smaller chunks so it would look more like:
                OA Adapter (Dequeue) -> Transport Requestor -> ABCS Requestor -> JMS Queue
                JMS Queue -> EBS -> JMS Topic/Queue*
                JMS Topic/Queue* -> ABCS Provider -> Transport Provider -> OA Adapter (Run PLSQL)
                *Some situations will require a queue, others a topic.

                I am currently concentrating on how we should be designing the stages that take messages from a JMS Queue. I also need to consider that there may be many different types of error which may occur at any point during the process. As this needs to be reliable all errors need to be considered not just known frequently occurring ones.

                Your suggested solution of altering the AIA error handling logic to deal with the situation would be possible. It would count the matching error messages a particular instance has produced and only when it sees there are 3 messages would it let the final one through. This would be complicated by the fact the retry count can be different for every queue and can be changed. The custom logic would need to read the weblogic queue configuration.

                Another solution I can think of is having three JMS queues. A main Queue a Final Queue and an Error Queue. The main queue could be configured to retry 2 times and when it finally fails put the message in the final queue. The final queue could be configured with no retries and when it fails put the message in the Error queue. I could then design the composite with two JMS adapter dequeue activities connecting to mediators. These would dequeue from the Main and Final Queues and add a final message field to the payload. This is a bit convoluted but it would (I think) meet the requirements.

                It’s a shame I can’t read some queue or SOAP header variable and find out if it’s the final retry as that would be a much simpler solution

                Robert
                • 5. Re: Reliable Messaging, Queue Retries and AIA Error Handling
                  Gerhard Drasch (Oracle)
                  While I can't help you make these decisions, I'd like to make one final comment on this: you should also be careful not to over-engineer your flows. What you describe with queues before and after the EBS might make sense in some complex scenarios (and that's why you would find it the AIA Developer's Guide), but I doubt this is the right blueprint in every asynchronous integration use case. This design apparently introduces more options, but also more code, more middleware components like queues, hence significantly more complexity leading to harder maintenance overall.

                  The AIA PIPs use these intermediate queues as well - but only where the use case really demands it, e.g. where the orchestration of an order goes through various processing stages. In that scenario, you would want to be able to re-submit a failed message from each of these persistence points while guaranteeing delivery. For a one-way integration flow that basically only brings a message from application A to application B we wouldn't usually do that.

                  Gerhard