1 Reply Latest reply: Sep 12, 2013 3:06 PM by jschellSomeoneStoleMyAlias RSS

    How to analyze real time data for correlation ?.


      Hi All,


          I have been wondering on how to get a data set from production environment be analyzed in near real time for some specific patterns/data analysis, e.g  Please consider the following scenario


          An alert Generation systems which works for various devices on the network and generates nearly thousands of alerts each day, now at present we have a specialist  who analyzes these patterns manually  and       gets back to us with a possible correlation strategy,

          After the alerts are available in the given environment which is very time consuming process... also the correlation strategy are determined by a number of      

          factors which are dynamic and changing, so we should be able to do a lookup from a central data warehouse from time to time...


          Now we would like a methodology to achieve the same so that we become a predictive and preventive system rather then a reactive one.

          I am not sure if we have a tool suite or  some strategy that can be suggested to encapsulate this entire tedious process ?.

          All suggestions and expert advice on the same would be invaluable.


          * If more information is required please let me know.*





        • 1. Re: How to analyze real time data for correlation ?.

          Myself I would think that the predictive part is either very hard or impossible.


          However other than that you have described a rules engine.  You persist rules and then run the rules against a set of data, where the set is defined by each rule.  The analyst or even operators create, update and delete rules.  How you do that depends on how much time you want to invest in the system.  But ultimately you would have a GUI or even an IDE that would be used to create a rule.


          Other complexities involved are

          - Can you find an existing rules system or must you build your own.

          - How complex do you want to allow the rules to become.  The more complex they are the harder it is for a new person to learn how to use it.

          - Will you and must you be prepared to build plug in rule modules perhaps either for performance or complexity needs.  If you must then how will the process for that proceed.


          With only thousands of alarms in a day a very fast system might not be a requirement but if it does then that also becomes a factor in the above.