Forum Stats

  • 3,855,354 Users
  • 2,264,499 Discussions
  • 7,905,978 Comments

Discussions

Query for missing events/heart beat

922144
922144 Member Posts: 17
edited Apr 2, 2012 9:20AM in Complex Event Processing
Hello Everyone,

I am trying to monitor network packets I send to a certain specific ip address and also check if I have sent packets for that particular ip at regular intervals(something like a heartbeat). I am forwarding the network output from wireshark to a csv file and reading it at OCEP using csv adapter.

In order to check for missing events/heart beat (this is used to check if a server responds to ping once every hour) I followed this link
4326322 and
[http://docs.oracle.com/cd/E21764_01/doc.1111/e14476/examples.htm]
I have configured the OCEP. This is the query I am firing

SELECT Packets.ipdst AS ipdst,Packets.ipsrc AS ipsrc,Packets.framelen AS framelen,"Error-Missing Event" AS alertType
from FilterNetworkPackets
MATCH_RECOGNIZE (
PARTITION BY ipdst
MEASURES Ipdst_Packets.ipdst AS ipdst,
Ipdst_Packets.ipsrc AS ipsrc,
Ipdst_Packets.framelen AS framelen
ALL MATCHES
include timer events
PATTERN( Ipdst_Packets*)
DURATION 10 DEFINE Ipdst_Packets AS ipdst = "xx.xx.xx.xx")
AS Packets

I have configured heartbeat(10000) on the input channel as well.

I get output in which some packets are repeated. I dont know how to get notification for missing events after the duration has elapsed, please help me.
Thanks in advance
Shilpa

Best Answer

  • Unmesh
    Unmesh Member Posts: 16
    edited Mar 30, 2012 3:27AM Answer ✓
    Hi,

    The output you are getting is infact the notification you receive when there is no event with 173.194.5.140 for 10 seconds.

    e.g. The first output.
    OutputBean:onEvent() + eventType=RequiredMailFields object=RequiredMailFields ki
    nd=null time=696477423597 ipsrc=192.168.0.10, ipdst=173.194.5.140,
    elt_time=686477423597, framelen=5, alertType=Error-Missing Event isTotalOrderGuarantee=false

    Here elt_time is the time at which the event arrived and time is the time associated with the output. If you subtract elt_time from time, you will notice that for each of the output the difference is 10 * 10^9 nanos = 10 seconds. So the above output should be interpreted as 'no event with ipdst=173.194.5.140 arrived for 10 seconds after the event with framelen=5' and hence the notification is generated.

    You don't see any output event that has framelen=3 because after the framelen = 3 input arrived, the next input (with framelen=5) for that partition arrived before the 10 seconds duration expired.

    As per your requirement, it appears that you need some sort of an indication in both the cases :
    1) whenever an event arrives with that ip address - will mark start of green colored line
    2) when no event arrives for 10 seconds for that ip address after an event for that ip address has been received - will mark start of red colored line

    I think you can model this as follows:

    view normalEvents:

    select ipdst as ipdst, ipsrc as ipsrc , framelen as framelen, "Normal Event" as alertType, ELEMENT_TIME as elt_time from FilterNetworkPackets

    view missedEvents: (the existing query - have added MULTIPLES OF and ALL MATCHES clause so that you will see output every 10 seconds till the next event arrives for that partition)

    SELECT Packets.ipdst AS ipdst,Packets.ipsrc AS ipsrc,Packets.framelen AS framelen,"Error-Missing Event" AS alertType,Packets.elt_time AS elt_time
    from FilterNetworkPackets
    MATCH_RECOGNIZE
    (
    PARTITION BY ipdst
    MEASURES
    Ipdst_Packets.ipdst AS ipdst,
    Ipdst_Packets.ipsrc AS ipsrc,
    Ipdst_Packets.framelen AS framelen,
    Ipdst_Packets.ELEMENT_TIME AS elt_time
    ALL MATCHES
    include timer events
    PATTERN( Ipdst_Packets) DURATION MULTIPLES OF 10
    DEFINE Ipdst_Packets AS ipdst = "173.194.5.140") AS Packets

    Query allEvents:
    select * from (normalEvents union missedEvents)

    On output of the query 'allEvents', based on the 'alertType' of the event you can trigger appropriate processing in the output bean.

    Regards

    Edited by: Unmesh on Mar 30, 2012 12:26 AM

Answers

  • Unmesh
    Unmesh Member Posts: 16
    Hi,

    If you have configured heartbeat correctly for the input channel, I think "pattern(Ipdst_Packets) duration 10" should give you an output if you don't get any further event on that partition for the next 10 seconds after an event arrives. No need to include * quantifier in the pattern clause. Also it appears that ALL MATCHES clause is not needed here.

    In any case, it will be helpful in understanding the scenario if you could send a sample input and expected output for your use-case.

    Regards
    Unmesh
  • 922144
    922144 Member Posts: 17
    edited Mar 29, 2012 11:08AM
    Thanks a lot for the reply Unmesh. Really appreciate it.

    ipsrc ipdst framelen/seq
    192.168.0.11 173.194.5.140 3
    192.168.0.9 239.255.255.250 4
    192.168.0.10 173.194.5.140 5
    192.168.0.9 239.255.255.250 6
    192.168.0.9 173.194.41.161 7
    192.168.0.9 173.194.41.161 8
    192.168.0.9 173.194.41.161 9
    192.168.0.9 239.255.255.25 10
    192.168.0.9 90.192.176.67 11
    192.168.0.9 87.112.197.250 12
    192.168.0.9 173.194.41.166 13
    192.168.0.9 173.194.41.166 14
    192.168.0.9 173.194.41.181 15
    192.168.0.11 173.194.5.140 16
    192.168.0.9 173.194.41.181 17
    192.168.0.9 90.192.176.67 18
    192.168.0.9 173.194.41.181 19
    192.168.0.19 173.194.5.140 20
    192.168.0.9 173.194.41.181 21
    192.168.0.9 194.168.8.100 22
    192.168.0.9 173.194.41.181 23

    This is part of the test data which consists of ipsrc,ipdst,framelen. Currently I have set framelen to sequence number for checking. I run the loadgen.cmd on this csv file. I need to check if a packet with ipdst = "173.194.5.140" arrives once every 10 sec. Give some warning if it does not arrive.
    My config file is:

    <query id="FilterCriteria"><![CDATA[
    SELECT Packets.ipdst AS ipdst, Packets.ipsrc AS ipsrc, Packets.framelen AS framelen, "Error-Missing Event" AS alertType
    from FilterNetworkPackets
    MATCH_RECOGNIZE ( PARTITION BY ipdst
    MEASURES Ipdst_Packets.ipdst AS ipdst, Ipdst_Packets.ipsrc AS ipsrc, Ipdst_Packets.framelen AS framelen
    include timer events
    PATTERN( Ipdst_Packets) DURATION 10
    DEFINE Ipdst_Packets AS ipdst = "173.194.5.140") AS Packets
    ]]></query>
    </rules>
    </processor>

    <channel>
    <name>FilterNetworkPackets</name>
    <heartbeat>100000000</heartbeat>
    </channel>

    <http-pub-sub-adapter>
    <name>alertadapter</name>
    <server-context-path>/pubsub</server-context-path>
    <channel>/packetalert</channel>
    </http-pub-sub-adapter>


    If I maintain the entry for channel, I get error from the server
    Error encountered while initializing configuration object. The configuration name, FilterNetworkPackets, is used multiple times, but configuration names must be unique within the same configuration file or application.>

    I set the heartbeat value via the visualiser. Is this wrong?

    This is the output

    {"ipsrc":"192.168.0.10","alertType":"Error-Missing Event","framelen":5,"ipdst":"173.194.5.140"}
    {"ipsrc":"192.168.0.10","alertType":"Error-Missing Event","framelen":5,"ipdst":"173.194.5.140"}
    {"ipsrc":"192.168.0.10","alertType":"Error-Missing Event","framelen":5,"ipdst":"173.194.5.140"}
    {"ipsrc":"192.168.0.19","alertType":"Error-Missing Event","framelen":20,"ipdst":"173.194.5.140"}
    {"ipsrc":"192.168.0.19","alertType":"Error-Missing Event","framelen":20,"ipdst":"173.194.5.140"}
    {"ipsrc":"192.168.0.19","alertType":"Error-Missing Event","framelen":20,"ipdst":"173.194.5.140"}
    {"ipsrc":"192.168.0.9","alertType":"Error-Missing Event","framelen":64,"ipdst":"173.194.5.140"}
    {"ipsrc":"192.168.0.9","alertType":"Error-Missing Event","framelen":64,"ipdst":"173.194.5.140"}
    {"ipsrc":"192.168.0.9","alertType":"Error-Missing Event","framelen":64,"ipdst":"173.194.5.140"}

    Sometimes it repeats the packets and sometimes it does not.

    It did not display packet 3,16 which also follows the criteria. Also for the next iterations of ten seconds when the criteria fails it does not do anything. Is it possible to perform some action or notification that there was no event within that time interval?

    Thanks
    Shilpa

    Edited by: 919141 on 29-Mar-2012 07:28

    Edited by: 919141 on 29-Mar-2012 08:07
  • Unmesh
    Unmesh Member Posts: 16
    Hi,

    I think you are not seeing any output for 3 and 16 because the next event (5 and 20 respectively) are arriving before the 10 seconds duration expires.
    PATTERN(A) DURATION X will output a match when there is only one event that matches A and no further event arrives till duration X expires. So I think that the case for which you see output are actually the notifications that you are looking for when no event comes till the duration expires.

    In any case, I think things will be clearer if you can -

    1. Write a custom adapter for understanding the working of the query wherein you can control the time difference between successive events. e.g. You could send first event with that particular Ipdst and then wait for 5 seconds and then send the next with the same Ipdst.. Here you won't get output since 5 < 10 (duration). Then you can send the next event for that Ipdst after 12 seconds..here the gap in successive events is > 10 seconds so you should see some output.
    2. Add ELEMENT_TIME to measures clause and project it out in the SELECT list so as to see the timestamp of the event. Also print the timestamp of event received in your output bean. These can also be helpful in understanding the query output.

    Regarding configuring heartbeat on the channel, as the error suggests, please ensure that the channel configuration is not happening at multiple places. I think setting it via visualizer should be fine.

    Regards
    Unmesh
  • 922144
    922144 Member Posts: 17
    Hello,

    Thanks again Unmesh for the reply. I am new to cep and your comments are very helpful.
    I have changed my query to include ELEMENT_TIME as
    SELECT Packets.ipdst AS ipdst,Packets.ipsrc AS ipsrc,Packets.framelen AS framelen,"Error-Missing Event" AS alertType,Packets.elt_time AS elt_time
    from FilterNetworkPackets
    MATCH_RECOGNIZE ( PARTITION BY ipdst
    MEASURES Ipdst_Packets.ipdst AS ipdst,
    Ipdst_Packets.ipsrc AS ipsrc,
    Ipdst_Packets.framelen AS framelen,
    Ipdst_Packets.ELEMENT_TIME AS elt_time
    include timer events
    PATTERN( Ipdst_Packets) DURATION 10
    DEFINE Ipdst_Packets AS ipdst = "173.194.5.140") AS Packets

    I have set the data such that:
    173.194.5.140 3
    173.194.5.140 5
    173.194.5.140 16
    173.194.5.140 20
    173.194.5.140 64
    173.194.5.140 90
    173.194.5.140 108
    173.194.5.140 134
    173.194.5.140 290
    173.194.5.140 293
    173.194.5.140 294
    173.194.5.140 295
    173.194.5.140 296

    The data rate is around 1 packet/sec. It displays
    OutputBean:onEvent() + eventType=RequiredMailFields object=RequiredMailFields ki
    nd=null time=696477423597 ipsrc=192.168.0.10, ipdst=173.194.5.140, elt_time=6864
    77423597, framelen=5, alertType=Error-Missing Event isTotalOrderGuarantee=false
    OutputBean:onEvent() + eventType=RequiredMailFields object=RequiredMailFields ki
    nd=null time=711477053997 ipsrc=192.168.0.19, ipdst=173.194.5.140, elt_time=7014
    77053997, framelen=20, alertType=Error-Missing Event isTotalOrderGuarantee=false

    OutputBean:onEvent() + eventType=RequiredMailFields object=RequiredMailFields ki
    nd=null time=755477515927 ipsrc=192.168.0.9, ipdst=173.194.5.140, elt_time=74547
    7515927, framelen=64, alertType=Error-Missing Event isTotalOrderGuarantee=false
    OutputBean:onEvent() + eventType=RequiredMailFields object=RequiredMailFields ki
    nd=null time=781480311114 ipsrc=192.168.0.9, ipdst=173.194.5.140, elt_time=77148
    0311114, framelen=90, alertType=Error-Missing Event isTotalOrderGuarantee=false
    OutputBean:onEvent() + eventType=RequiredMailFields object=RequiredMailFields ki
    nd=null time=799477103444 ipsrc=192.168.0.9, ipdst=173.194.5.140, elt_time=78947
    7103444, framelen=108, alertType=Error-Missing Event isTotalOrderGuarantee=false

    OutputBean:onEvent() + eventType=RequiredMailFields object=RequiredMailFields ki
    nd=null time=825476560777 ipsrc=192.168.0.9, ipdst=173.194.5.140, elt_time=81547
    6560777, framelen=134, alertType=Error-Missing Event isTotalOrderGuarantee=false

    Packet Connection Closed
    OutputBean:onEvent() + eventType=RequiredMailFields object=RequiredMailFields ki
    nd=null time=987476943158 ipsrc=192.168.0.9, ipdst=173.194.5.140, elt_time=97747
    6943158, framelen=296, alertType=Error-Missing Event isTotalOrderGuarantee=false

    I wanted to make sure that it triggers something when there is no packet present in a 10sec time duration. I need to depict the output graphically.Green line when there are packets existing in 10 sec duration and red line when duration expires. I am not totally sure how I should go about this. Please help me with this.

    Thanks and Regards
    Shilpa
  • Unmesh
    Unmesh Member Posts: 16
    edited Mar 30, 2012 3:27AM Answer ✓
    Hi,

    The output you are getting is infact the notification you receive when there is no event with 173.194.5.140 for 10 seconds.

    e.g. The first output.
    OutputBean:onEvent() + eventType=RequiredMailFields object=RequiredMailFields ki
    nd=null time=696477423597 ipsrc=192.168.0.10, ipdst=173.194.5.140,
    elt_time=686477423597, framelen=5, alertType=Error-Missing Event isTotalOrderGuarantee=false

    Here elt_time is the time at which the event arrived and time is the time associated with the output. If you subtract elt_time from time, you will notice that for each of the output the difference is 10 * 10^9 nanos = 10 seconds. So the above output should be interpreted as 'no event with ipdst=173.194.5.140 arrived for 10 seconds after the event with framelen=5' and hence the notification is generated.

    You don't see any output event that has framelen=3 because after the framelen = 3 input arrived, the next input (with framelen=5) for that partition arrived before the 10 seconds duration expired.

    As per your requirement, it appears that you need some sort of an indication in both the cases :
    1) whenever an event arrives with that ip address - will mark start of green colored line
    2) when no event arrives for 10 seconds for that ip address after an event for that ip address has been received - will mark start of red colored line

    I think you can model this as follows:

    view normalEvents:

    select ipdst as ipdst, ipsrc as ipsrc , framelen as framelen, "Normal Event" as alertType, ELEMENT_TIME as elt_time from FilterNetworkPackets

    view missedEvents: (the existing query - have added MULTIPLES OF and ALL MATCHES clause so that you will see output every 10 seconds till the next event arrives for that partition)

    SELECT Packets.ipdst AS ipdst,Packets.ipsrc AS ipsrc,Packets.framelen AS framelen,"Error-Missing Event" AS alertType,Packets.elt_time AS elt_time
    from FilterNetworkPackets
    MATCH_RECOGNIZE
    (
    PARTITION BY ipdst
    MEASURES
    Ipdst_Packets.ipdst AS ipdst,
    Ipdst_Packets.ipsrc AS ipsrc,
    Ipdst_Packets.framelen AS framelen,
    Ipdst_Packets.ELEMENT_TIME AS elt_time
    ALL MATCHES
    include timer events
    PATTERN( Ipdst_Packets) DURATION MULTIPLES OF 10
    DEFINE Ipdst_Packets AS ipdst = "173.194.5.140") AS Packets

    Query allEvents:
    select * from (normalEvents union missedEvents)

    On output of the query 'allEvents', based on the 'alertType' of the event you can trigger appropriate processing in the output bean.

    Regards

    Edited by: Unmesh on Mar 30, 2012 12:26 AM
  • 922144
    922144 Member Posts: 17
    Hello,

    Thanks a lot Unmesh for the reply. I will check the solution once I get back home but it definitely makes lot of sense now. I am sure it will work fine. I was unaware of lot of cep concepts. Thanks a lot for helping me understand.

    Regards
    Shilpa
  • 922144
    922144 Member Posts: 17
    Hello,

    Thanks a lot Unmesh for the help. I tried out the Missed event query. It works as expected.
    I am unable to union the output of both the views though it says that the stream should be either a relation stream or it should return something of type relation. I was reading the documentation available on this
    [http://docs.oracle.com/cd/E14571_01/doc.1111/e14302.pdf |http://docs.oracle.com/cd/E14571_01/doc.1111/e14302.pdf]
    [http://docs.oracle.com/cd/E17904_01/apirefs.1111/e12048/queries.htm#autoId11]
    I do not think that making a channel as Rstream will solve the prob. I do not understand why cant we simply do union of two streams of data? Am I missing some basic knowledge about cep streams?

    Thanks in advance
    Shilpa
  • Unmesh
    Unmesh Member Posts: 16
    Hi,

    Union in CEP deletes the duplicates so it is not allowed on two streams since duplicate elimination will require maintaining the entire history (stream) right from the first event.
    However, there is a variant UNION ALL which can be used here. It does not remove duplicates so it can be used over streams.
    http://docs.oracle.com/cd/E17904_01/apirefs.1111/e12048/cqlstatements.htm#BABJFGFI

    Regards
    Unmesh
  • 922144
    922144 Member Posts: 17
    Hello,

    Thanks a lot for the reply Unmesh. I was able to grasp the UNION and UNION ALL concept now.

    Thanks again
    Shilpa
This discussion has been closed.