7 Replies Latest reply: Aug 21, 2014 3:35 PM by Vlad Dyuzhev RSS

    Work Manager configuration for Vendor Unresponsive

    Aashima Jain

      Hi All,

       

      We are considering configuring work managers in OSB to stop sending requests to a third party vendor in case the vendor is not responding within the read time out SLA. When a situation of this kind arises, we want our business service/proxy service to stop sending the requests to vendor and respond back to consumer immediately without waiting for entire timeout duration. Also, when we start receiving response from vendor, we start sending our requests in normal manner.

       

      Not sure if WM is really a solution for this or not. Please suggest what can be done.

        • 1. Re: Work Manager configuration for Vendor Unresponsive
          Vlad Dyuzhev

          WMs are useful, but not in this case.

           

          What you probably want to do is to configure the throttling on the business service. Throttling limits the number of concurrent in-progress requests to the backend service.

           

          Say you configure throttling as 12 and you have 4 managed servers. Then each managed server supports maximum 3 outgoing requests. If those requests take a long time (backend is slow), the 4th request coming into business service will be immediately rejected.

           

          See more details in Oracle documentation (of course) and also here http://genericparallel.com/2014/05/about-throttling-in-business-services/

           

          Vlad

          http://genericparallel.com

          • 2. Re: Work Manager configuration for Vendor Unresponsive
            Aashima Jain

            Thanks for you reply Vlad.

             

            I had one concern on this solution. How long will the first 3 outgoing requests wait for the response?

            Once the vendor goes down, and we realize it, we do not want to push more requests to it until it's back up again.

            • 3. Re: Work Manager configuration for Vendor Unresponsive
              Vlad Dyuzhev

              3 requests are waiting until ... well, depends on the nature of the backend issues.

               

              If the backend doesn't accept the connections anymore, the requests fail right away.

               

              If the backend doesn't accept the connections anymore but also doesn't reply with connection refused, the requests wait until the connect timeout expires (value set in Business service). Usually very short, 1-3 seconds.

               

              If the backend accepts the connections (TCP ACK), but doesn't reply with the response, the requests wait until the read timeout expires. This one is longer, say 5 to 30 seconds, depending on normal response time.

               

              Then there is that logic that takes the endpoint offline (in the Operational settings). That one we, in our team, do not use, because it seems unnecessary -- if the backend is really down, you get the connection refused immediately anyway. In your case though it may be useful if you're believing that every new request actually reaching the backend and adds a load to it.

               

              Managing Endpoint URIs for Business Services  47.2.1

               

              I would still go with throttling first. You see, throttling prevents the backend overload issue from happening, while offline uri deals with issue already happen. I prefer to prevent than troubleshoot.

               

              Vlad

              http://genericparallel.com

              • 4. Re: Work Manager configuration for Vendor Unresponsive
                Aashima Jain

                Vlad,

                 

                Here are 2 concerns:

                 

                1. With Offline Endpoint - The configuration does set the endpoint as Offline in case the URL is down but the system does not gets it back online once the URL becomes responsive again. We want both the things to be done.

                 

                2. Throttling - The initial requests that are waiting for response will timeout after the read time out value set in the service. Then, the next set of requests will reach the vendor. We don't want that to happen. We do not want the requests to go to vendor after the first set. They should only resume going to vendor once the vendor is up and responsive again.

                • 5. Re: Work Manager configuration for Vendor Unresponsive
                  Vlad Dyuzhev

                  Offline endpoint does come back online again, if after the retry interval it is accessible:

                   

                  When the retry interval has passed and this business service attempts to process a new request, it tries to access this endpoint URI. If this attempt is successful, then the endpoint URI is marked online again.

                   

                  With your requirements, it seems that taking the endpoint offline is the only viable option.

                   

                  Vlad

                  http://genericparalllel.com

                  • 6. Re: Work Manager configuration for Vendor Unresponsive
                    Aashima Jain

                    Even after the retry interval, request will reach the vendor to check whether the URL is up or not. We don't want that.

                     

                    Problem statement is, system should go down when vendor goes down and come up when vendor comes back up. During this downtime, no request should reach vendor.

                    • 7. Re: Work Manager configuration for Vendor Unresponsive
                      Vlad Dyuzhev

                      Even after the retry interval, request will reach the vendor to check whether the URL is up or not. We don't want that

                       

                      There is no magic. To know the backend is back up you must call it.

                       

                      The only alternative is manual shutdown (obviously, it is only useful for planned downtimes).

                       

                      Vlad

                      http://genericparallel.com