This discussion is archived
5 Replies Latest reply: Feb 25, 2011 1:09 PM by jschellSomeoneStoleMyAlias RSS

Distributed JMS based logging .. falling flat?

841826 Newbie
Currently Being Moderated
In our fancy ESB, logging of each request is done via a common infrastructure based on JMS based logging. Here is what happens in a nutshell:

service gets a request service
prepares some data in a LogData
object service calls database
time taken for db interaction is captured in LogData object
service is ready to send response
LogData object is sent to a messaging destination
service sends response
Very rosey! yes for paper architects. Here is the actual issue: The JMS service provider sometimes becomes unavailable - due to a system level error or the software crashes. Then the service waits at the step (step no. 6) where it has to make JMS connection to send LogData object. Resulting in delayed response, thus leading to bad performance and user experience.

So that is the biggest shortcoming of "Distributed logging using JMS" touted by a lot of developer websites. Also note that the presance of LogData is kind of critical non-functional requirement. That means the messages are sent in persistent mode, leading to a wait until the JMS provider confirms receipt of message to the sender (the service in this case) - what to be blamed? immature design? Are there any success stories of implementing something like this?
  • 1. Re: Distributed JMS based logging .. falling flat?
    jduprez Pro
    Currently Being Moderated
    I understand in your case that you have to live with an existing architecture...

    Theserverside.com had published a discussion about that something like 5 years ago.
    Distributed asynchronous logging has two stated goals:
    - distribution of logs to a.g. a central place.
    - performance (the service supposedly doesn't have to wait for the disk latency)

    The second one falls short: if the logs have any kind of guarantee of delivery requirement, and in your case they have, the client has either to wait for an acknowledgment (as you describe), or, if persisted queuing is implied, to wait for the persistence store to store the message (that is, wait for the disk, the problem that was supposed to be avoided).

    As far as the first go, depending on the requirements, an alternative is to log on the local disk, and asynchronously send or fetch the logs to a central place: the coalescing is then done "offline" from the main application (which doesn't mean it has no impact, as it incurs more disk activity).

    I have a "success story" of doing that latter (offline gathering/coalescing), at an e-commerce site where logs reliability was important for maintenance and support, but not "critical" (the logs had no legal value), but not of doing distributed logging.
  • 2. Re: Distributed JMS based logging .. falling flat?
    jschellSomeoneStoleMyAlias Expert
    Currently Being Moderated
    user13559021 wrote:
    In our fancy ESB, logging of each request is done via a common infrastructure based on JMS based logging. Here is what happens in a nutshell:

    service gets a request service
    prepares some data in a LogData
    object service calls database
    time taken for db interaction is captured in LogData object
    service is ready to send response
    LogData object is sent to a messaging destination
    service sends response
    Very rosey! yes for paper architects. Here is the actual issue: The JMS service provider sometimes becomes unavailable - due to a system level error or the software crashes. Then the service waits at the step (step no. 6) where it has to make JMS connection to send LogData object. Resulting in delayed response, thus leading to bad performance and user experience.

    So that is the biggest shortcoming of "Distributed logging using JMS" touted by a lot of developer websites. Also note that the presance of LogData is kind of critical non-functional requirement. That means the messages are sent in persistent mode, leading to a wait until the JMS provider confirms receipt of message to the sender (the service in this case) - what to be blamed? immature design? Are there any success stories of implementing something like this?
    I don't understand what this has to do with JMS.

    You can substitute 'database', 'web service' or 'jboss' (as examples) in the above and change nothing.

    If you have a remote server that is a required part of your processing system in a 24x7 system then you must come to some understanding of both what is allowed when the service is not available at all and what is allowed when there are intermittent outages.

    The first must be considered because something like a router going done can take quite a while to diagnose and replace.

    Besides the solution noted above I have also implemented a solution that cause my service to stop accepting any transactions at all if the secondary processors cannot respond in a timely manner. The reasoning there is that in planned for volume scenarios it is possible for several thousand transactions to be backed up even in the best circumstances in a couple of seconds, thus taking more in could result in a large number of failures.

    Additionally back end systems might run via a load balancer. That however can still have failure scenarios, again for example when a router goes out, or when one balanced server accepts a connection but fails to process it.
  • 3. Re: Distributed JMS based logging .. falling flat?
    841826 Newbie
    Currently Being Moderated
    Hey
    jschell

    Why I mentioned "JMS" because you will see 1000 articles on the web or otherwise from "paper architects" stating the benefits of so called "asynchronous JMS logging".
    Which as I see is like a knee jerk reaction, when someone is ignoring the asynchronous part and implementing it like a databse or a socket based logging.
    Thanks!
  • 4. Re: Distributed JMS based logging .. falling flat?
    jduprez Pro
    Currently Being Moderated
    Which as I see is like a knee jerk reaction, when someone is ignoring the asynchronous part and implementing it like a databse or a socket based logging.
    I feel the contrary (but maybe I misunderstood you): the "asynchronous" aspect is overrated (and I'm understated: I should merely say "is a filthy lie") if the logging must be guaranteed.

    Posting a message to a queue, sending a message on a socket, saving the request to a transaction log,... one or several of those are a required preliminary of anything remotely reliable, and these parts are synchronous (if you need the guarantee, such as, an acknowledgement that the queue has received the message (I'm not talking about acknowledging the processing of the message, just the reception ).

    Serial as in: the business transaction that wants to log must wait for this acknowledgement part before moving on: if the ack is delayed, the business transaction is delayed. If it's frozen, the business transaction freezes too.
    Asynchronous processing has its merits when the processing is an order of magnitude heavier than the post/acknowledgement part, but for a "simple" message logging, there's little to no gain.

    That's for the asynchronous bit. Now, the centralized bit may be a more compelling (in terms of business requirements) reason.
  • 5. Re: Distributed JMS based logging .. falling flat?
    jschellSomeoneStoleMyAlias Expert
    Currently Being Moderated
    user13559021 wrote:
    Hey
    jschell

    Why I mentioned "JMS" because you will see 1000 articles on the web or otherwise from "paper architects" stating the benefits of so called "asynchronous JMS logging".
    Which as I see is like a knee jerk reaction, when someone is ignoring the asynchronous part and implementing it like a databse or a socket based logging.
    Thanks!
    Yes but 'logging" has a rather indefinite definition as far as I am concerned.

    I use 'logging' to provide information relevant to development (me) in terms of debugging run time errors that have already occurred. This is collected in log files localized to each server instance. it originates with log statements throughout the code base that posts relevant state information about the application.

    On the other hand I have seen business requirements that specifically indicated that when a user actions on the system must be 'logged'.

    For me the first is a process that although useful should not stop the business functionality of the system if it fails. Thus if the server is set up with the wrong permissions such that the file cannot be written to the application will keep running.

    The second however is something where the system must not continue to function if user actions cannot be logged.

    The second is where my previous comments apply. If I cannot successfully log user actions then there must be a business driven decision as to what is acceptable.

    The JMS comments that you have seen could apply to one or the other or fail to differentiate or even indeed be ignorant of the considerations needed when using any external server.

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points