This discussion is archived
1 2 Previous Next 15 Replies Latest reply: Dec 27, 2012 8:25 AM by AntonZ RSS

PofExtractor with ExtractorEventTransformer

AntonZ Newbie
Currently Being Moderated
Hi.

When trying to use PofExtractors along with ExtractorEventTransformer I get an error. If ReflectionExtractor used - no problem occurs. (Using coherence 3.7.1)

Is it possible at the stage of Event Transformer still avoid pof serialization currently? And if not is something like this coming to coherence any time soon?

Thank you.
  • 1. Re: PofExtractor with ExtractorEventTransformer
    robvarga Oracle ACE
    Currently Being Moderated
    AntonZ wrote:
    Hi.

    When trying to use PofExtractors along with ExtractorEventTransformer I get an error. If ReflectionExtractor used - no problem occurs. (Using coherence 3.7.1)

    Is it possible at the stage of Event Transformer still avoid pof serialization currently? And if not is something like this coming to coherence any time soon?

    Thank you.
    Hi Anton,

    there is no point using POF with EventTransformers. Events are generated in object form, then optionally transformed and only the transformation result (or the original if there wasn't any transformation) is serialized. I.e. transformation takes place on the event producer side before serialization.

    Best regards,

    Robert
  • 2. Re: PofExtractor with ExtractorEventTransformer
    AntonZ Newbie
    Currently Being Moderated
    Ok. So do you see the possibility that events can be raised without conversion from pof to object? So than one can still transform event via pof?
  • 3. Re: PofExtractor with ExtractorEventTransformer
    robvarga Oracle ACE
    Currently Being Moderated
    AntonZ wrote:
    Ok. So do you see the possibility that events can be raised without conversion from pof to object? So than one can still transform event via pof?
    I see the usefulness of being able to get stuff out of the POF binaries stored within the Event without deserializing the binary in a MapEventTransformer and use that stuff to construct the surrogate event, but I don't see how or why you would use specifically the PofExtractor abstraction (which needs a BinaryEntry) to extract stuff from. It is much simpler to use PofValueParser to give you a PofValue representing the POF hierarchy and start from there.

    As for transforming the event object itself, POF has nothing to do with it, you never encounter the binary representation of the event object itself, it is always serialized only after user code had its way with it and is always deserialized when receiving it from the network before passing it on to user code.

    Best regards,

    Robert
  • 4. Re: PofExtractor with ExtractorEventTransformer
    AntonZ Newbie
    Currently Being Moderated
    robvarga wrote:
    I see the usefulness of being able to get stuff out of the POF binaries stored within the Event without deserializing the binary in a MapEventTransformer and use that stuff to construct the surrogate event, but I don't see how or why you would use specifically the PofExtractor abstraction (which needs a BinaryEntry) to extract stuff from. It is much simpler to use PofValueParser to give you a PofValue representing the POF hierarchy and start from there.
    That is essentially what I'm looking to get. What would be currently an implementation (some example if exists) of that? Essentially transforming (or sending surrogate) event without pof deserialization.
  • 5. Re: PofExtractor with ExtractorEventTransformer
    user738616 Pro
    Currently Being Moderated
    Hi,

    You need to listen to BackingMap Events and directly transform the backing MapListener events returned from distributed caches. In order to read these events, you would require to implement AbstractMultiplexingBackingMapListener class which is implemented in Coherence Common project. More details can be found here: http://docs.oracle.com/cd/E18686_01/coh.37/e18677/api_events.htm (section 21.11)

    HTH

    Cheers,
    _NJ                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
  • 6. Re: PofExtractor with ExtractorEventTransformer
    robvarga Oracle ACE
    Currently Being Moderated
    user738616 wrote:
    Hi,

    You need to listen to BackingMap Events and directly transform the backing MapListener events returned from distributed caches. In order to read these events, you would require to implement AbstractMultiplexingBackingMapListener class which is implemented in Coherence Common project. More details can be found here: http://docs.oracle.com/cd/E18686_01/coh.37/e18677/api_events.htm (section 21.11)

    HTH

    Cheers,
    _NJ
    This is not really correct. It is not worth transforming backing map events. Whatever you would do there, you can do it in the backing map listener, too, without any difference in cost.

    The main purpose of event transformations are for reducing the amount of network traffic to the cache listeners by reducing the amount of information in an event. Another purpose is that it gives you another chance to selectively discard events for specific subscribers, although MapEventFilters give you that, too.

    Best regards,

    Robert
  • 7. Re: PofExtractor with ExtractorEventTransformer
    robvarga Oracle ACE
    Currently Being Moderated
    AntonZ wrote:
    robvarga wrote:
    I see the usefulness of being able to get stuff out of the POF binaries stored within the Event without deserializing the binary in a MapEventTransformer and use that stuff to construct the surrogate event, but I don't see how or why you would use specifically the PofExtractor abstraction (which needs a BinaryEntry) to extract stuff from. It is much simpler to use PofValueParser to give you a PofValue representing the POF hierarchy and start from there.
    That is essentially what I'm looking to get. What would be currently an implementation (some example if exists) of that? Essentially transforming (or sending surrogate) event without pof deserialization.
    Hi Anton,

    the following code snippet illustrates how to get the binary values out of the map event in the transformer, and how to get a PofValue representation of it:
    public class MyTransformer implements MapEventTransformer, PortableObject {
        @Override
        public MapEvent transform(MapEvent event) {
            ConverterMapEvent converterEvent = (ConverterMapEvent)event;
            MapEvent eventWithBinaries = converterEvent.getMapEvent();
            Binary newValue = (Binary)eventWithBinaries.getNewValue();
            PofValue pofValue = PofValueParser.parse(newValue, (PofContext)converterEvent.getContext().getCacheService().getSerializer());
            System.out.println("PofValue: "+pofValue);
    
            // extract whatever you want
    
            // return surrogate event
            return new MapEvent(event.getMap(), event.getId(), event.getKey(), yourOldValueAsObjectOrNull, yourNewValueAsObjectOrNull); // or return null to not send the event to the listener the transformer belongs to
        }
    
        // readExternal and writeExternal also needs to be implemented
    }
    Of course you would need to register this class in POF configuration.

    If the transformer properly implements equals() and hashCode(), and multiple listeners register with an equalling transformer, then Coherence is going to notice it and run transformation only once for all the listeners with equalling transformer instance, not once per listener.

    Best regards,

    Robert
  • 8. Re: PofExtractor with ExtractorEventTransformer
    AntonZ Newbie
    Currently Being Moderated
    Thanks, Rob.

    Am I getting this right, though, in the code above you're converting desirialized object back to pof? Or there isn't going to be any serialization/desirialization, so I should theoretically gain speed on coherence side by using this custom transformer along with pofextractors vs cacheeventtransformer with reflectionextractors? (given I have quite big objects, with lots of properties and nested collections)
  • 9. Re: PofExtractor with ExtractorEventTransformer
    robvarga Oracle ACE
    Currently Being Moderated
    AntonZ wrote:
    Thanks, Rob.

    Am I getting this right, though, in the code above you're converting desirialized object back to pof?
    No, I am not deserializing the POF binary at all. I just create a PofValue which lets you extract part of the POF binary without deserializing the entire binary.

    Afterwards, I just instantiate a new MapEvent which is going to be dispatched to the specific listener(s) belonging to the transformer. That event gets serialized by Coherence code. It may be possible by some hacking to create an event which contains the old/new values in binary form (extracting binary format out of the PofValue with methods in AbstractPofValue), but I did not experiment with that part now...

    Or there isn't going to be any serialization/desirialization, so I should theoretically gain speed on coherence side by using this custom transformer along with pofextractors vs cacheeventtransformer with reflectionextractors? (given I have quite big objects, with lots of properties and nested collections)
    What you should gain here is that you do not generate that much garbage and not spend so much time with deserializing the full value, and by not sending that much data on the network by reducing the information the event has.

    Best regards,

    Robert
  • 10. Re: PofExtractor with ExtractorEventTransformer
    AntonZ Newbie
    Currently Being Moderated
    robvarga wrote:
    That event gets serialized by Coherence code.
    You mean coherence code on the receiving end (whoever attached such listener)?
    What you should gain here is that you do not generate that much garbage and not spend so much time with deserializing the full value, and by not sending that much data on the network by reducing the information the event has.
    I am already gaining reduced network load and serialization time on the client side. What I thought I can save in addition is time it takes on coherence side in the cluster to desirialize (reflection extractor bit) then serialize object (to send back pof to whoever attached according listener). So that's what you're calling the garbage part, right?
  • 11. Re: PofExtractor with ExtractorEventTransformer
    robvarga Oracle ACE
    Currently Being Moderated
    AntonZ wrote:
    robvarga wrote:
    That event gets serialized by Coherence code.
    You mean coherence code on the receiving end (whoever attached such listener)?
    No, on the storage node which generates the event as this is where the event transformer runs.
    What you should gain here is that you do not generate that much garbage and not spend so much time with deserializing the full value, and by not sending that much data on the network by reducing the information the event has.
    I am already gaining reduced network load and serialization time on the client side. What I thought I can save in addition is time it takes on coherence side in the cluster to desirialize (reflection extractor bit) then serialize object (to send back pof to whoever attached according listener). So that's what you're calling the garbage part, right?
    This is what I call the "not generate that much garbage and not spend so much time with deserializing the full value" part.

    The serialization on the sending side actually is a bit more complex. Coherence still has a reference to the old and new binary value, therefore based on that information if it sees that the old/new values are the same objects as the old and new binary values, then it could theoretically optimize away the serialization step as it already has the serialized form. Whether this happens or not, I am not sure.

    But if yes, then, I think, it should be possible to create a surrogate event with Binary data extracted out of the original Binary objects, but let's not open that can of worms now...

    Best regards,

    Robert
  • 12. Re: PofExtractor with ExtractorEventTransformer
    AntonZ Newbie
    Currently Being Moderated
    Thanks, Rob. I will try using your snippet and then revert back.

    I'm still though a bit confused by following:
    robvarga wrote:
    AntonZ wrote:
    robvarga wrote:
    That event gets serialized by Coherence code.
    You mean coherence code on the receiving end (whoever attached such listener)?
    No, on the storage node which generates the event as this is where the event transformer runs.
    If serialization is still going to occur, what am I winning then in terms of performance compared to using ReflectionExtractor with ExtractorEventTransformer? What bit I gonna safe time on by using this and what is still an overhead?

    I would expect no serialization/deserialization to occur if PofExctrator based Transformer is used, while when ReflectionExtractor with ExtractorEventTransformer is used, I would expect that object gets fully deserialized before extractor is used, and then serialized back to pof to return the event to whoever was listening for it. Correct me if any part sounds wrong to you.

    Thanks a lot.

    (Just trying have full clarity on this)
  • 13. Re: PofExtractor with ExtractorEventTransformer
    AntonZ Newbie
    Currently Being Moderated
    Oh, I guess you meant that this bit in your snippet:

    return new MapEvent(event.getMap(), event.getId(), event.getKey(), yourOldValueAsObjectOrNull, yourNewValueAsObjectOrNull);

    Still requires Object format so serialization and deserialization at that point is going to occur and thus it is not exactly avoiding all pof conversions.

    If that is so, then the only time I'm going to gain anything is when I decide not to send the event entirely (or send only new or old object). In my use case though, it isn't useful, since I prefilter using PofExctractor in the first place and then basically expect to get a transformed object from the event every time.

    Do you see anything coming or existing implementation that can completely get away from any serialization?
  • 14. Re: PofExtractor with ExtractorEventTransformer
    robvarga Oracle ACE
    Currently Being Moderated
    AntonZ wrote:
    Oh, I guess you meant that this bit in your snippet:

    return new MapEvent(event.getMap(), event.getId(), event.getKey(), yourOldValueAsObjectOrNull, yourNewValueAsObjectOrNull);

    Still requires Object format so serialization and deserialization at that point is going to occur and thus it is not exactly avoiding all pof conversions.

    If that is so, then the only time I'm going to gain anything is when I decide not to send the event entirely (or send only new or old object). In my use case though, it isn't useful, since I prefilter using PofExctractor in the first place and then basically expect to get a transformed object from the event every time.
    No. You can put an arbitrary object or null for old and new value, so you can just create a new object with an arbitrary subset of the original data extracted via PofValues, or just use a single object extracted with a PofValue. Extracting via PofValues does not deserialize the entire object so it is much cheaper than a full deserialization.

    Do you see anything coming or existing implementation that can completely get away from any serialization?
    See above.

    Best regards,

    Robert
1 2 Previous Next

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points