7 Replies Latest reply: Sep 16, 2010 1:24 PM by 807581 RSS

    Implementing Failover for OpenMQ C Client

    807581
      In earlier discussion, I have realized that the OpenMQ C client does not support auto-reconnection failover, but that this would be possible to implement manually in the application-level layer. To implement this, is it necessary to create a new layer of abstraction? Would it be possible to implement this failover in one of the layers that already exists?

      Also, how would I go about doing so?

      Thanks!
        • 1. Re: Implementing Failover for OpenMQ C Client
          807581
          At application layer, the application has better flexibility to coordinate application activities with the failover by using the C-API functions and callbacks,
          http://docs.sun.com/app/docs/doc/821-1795/aeldi?l=en&q=message+queue&a=view

          Yes, auto-reconnection failover can be implemented in the C-API library.
          • 2. Re: Implementing Failover for OpenMQ C Client
            807581
            Is the only way to implement auto-reconnection failover to set an exception listener to listen for connection exceptions, then when you detect one, to close the connection properly then re-connect by opening the connection again?

            Is there a fix in the event that a connection silently fails and an exception is not thrown immediately? For example, a way to ping the connection or have keep-alive messages to know when the connection goes down, then re-open the connection, using the given C-API? Or do we have to write our own abstract API on top of this?

            Also, will all of this work in a clustering capactiy with a composite URL?

            Thanks!
            • 3. Re: Implementing Failover for OpenMQ C Client
              807581
              s the only way to implement auto-reconnection failover to set an exception listener to listen for connection exceptions, then when you detect one, to close the connection properly then re-connect by opening the connection again?
              yes, that's the idea.
              Is there a fix in the event that a connection silently fails and an exception is not thrown immediately? For example, a way to ping the connection or have keep-alive messages to know when the connection goes down, then re-open the connection, using the given C-API? Or do we have to write our own abstract API on top of this?
              If the connection is broken the connection exception listener will be called; if the connection becomes half-open due to network problem, the C-API runtime automatic ping (if no network traffic otherwise in the client), will trigger underline TCP layer to close the TCP connection (which in turn causes connection listener to be called) after a system configurable period (TCP layer parameter).
              Also, will all of this work in a clustering capactiy with a composite URL?
              The C-API currently supports specifying broker host name and port number on creating a connection, so the application can keep a list of broker hostnames/ports and control which broker to connect on each MQCreateConnection call.
              • 4. Re: Implementing Failover for OpenMQ C Client
                807581
                in our application (uses C mq api) we've created separate "connection monitoring" thread which all it does creates MQCreateSession with shared connection handle, if it fails closes existing connection (object) and tries to create new one, then sleeps to repeat the same. All other threads using shared connection handle of course have to use mutexes/conditional variables to get proper access to it (connection handle). this method works like a charm for quite heavy "record loader" application 7x24 for quite some time now. as a side note, make sure you correctly free resources on C client side, esp. when using multiple threads doing simultaneous MQ connections, otherwise your MQ server will have memory leaks.

                /mareks
                • 5. Re: Implementing Failover for OpenMQ C Client
                  807581
                  Mareks,

                  So basically the thread you have created monitors the connection by attempting to create a session and re-creating the connection, if I understand correctly. About how often do you sleep to check again? And does this mean you have a large number of sessions created if the connection never fails, and is this an ok side effect to have? I don't know enough about what happens underneath to know if this is also a significant memory or speed drain.

                  Also, when you say to make sure you free all of the memory, are you referring to doing so manually, or just calling MQCloseConnection()- does this function take care of the problem?

                  Thanks.
                  • 6. Re: Implementing Failover for OpenMQ C Client
                    807581
                    Sleep interval is several seconds (qos dependable for this particular service), also important note is that this "ping" thread is not leaving session open - it runs MQCloseSession if MQCreateSession returns MQ_TRUE status, thus freeing resources on a server and not creating any overhead at all.

                    Our application is not running MQCloseConnection after each submission of message (multiple threads) - it keeps running and reuses existing connection handle, which is controlled by "ping" thread above. Creating connection compared to creating only a session is a heavy-weight process and we can't afford to recreate new connection for each individual worker thread. By recreating only session and reusing existing connection message load times/performance/number of messages received by the server is 20..30-fold increase compared to the case when each worker is creating it's own connection.

                    To free resources on a server it is not enough to MQCloseSession only, You have to also do explicit MQFreeMessage(messageHandle), MQCloseMessageProducer(producerHandle), MQCloseSession(sessionHandle) and MQFreeDestination(destinationHandle) - all of the resources You've created opening new session and producing message must be freed by hand otherwise server will run out of memory quite soon. When we created first version of C client for our application we used only MQCloseSession (I believe that MQ C guide says only to do MQCloseSession and all the rest will be done auto-magically), but soon realized that server is running out of resources quite soon, so we switched to freeing resources by hand and that solved a problem.

                    Of course, running MQCloseConnection is destroying all resources allocated on a server side, but as I wrote above, it is a significant performance hit when You have to do a lot of message delivery. Please note that MQCloseConnection is not freeing resources on a client side though, they have to be freed manually anyway.

                    /mareks
                    • 7. Re: Implementing Failover for OpenMQ C Client
                      807581
                      MQCloseSession will close producers/consumers created in that session and free memory allocated in the C-API runtime that are associated with the session. Destinations and message handles used in the session must be freed explicitly by the client application. This is documented in the C-API Developer Guide Reference MQFreeConnection
                      "You must free memory allocated for a message or a destination by explicitly calling the MQFreeMessage or the MQFreeDestination function."
                      http://docs.sun.com/app/docs/doc/821-1795/aelek?l=en&a=view

                      A doc CR6985459 has been filed to clarify that in MQCloseConnection and MQCloseSession reference as well.

                      and for MQCloseConnection and MQFreeConnection usage in release resources, please see
                      http://docs.sun.com/app/docs/doc/821-1795/aeldq?l=en&a=view