This discussion is archived
1 2 Previous Next 18 Replies Latest reply: Jun 10, 2013 6:12 AM by greybird RSS

Use case for triggers, is there a better way?

731051 Newbie
Currently Being Moderated
Hi,

I'm using DB to store data, but for performance reasons I'm also keeping fast, concurrent data structures, which are updated, when I am updating DB contents.
This strategy works well for one DB, but it will not work for replicated environment, because my data structures will not be updated any more (on slaves).
It seems, that triggers might be the answer - if they were invoked on data changes on master and on slaves.
I noticed, that there already is DatabaseTrigger class, which is used for secondary databases. Are those triggers fired also during replication?
Is there a better way, than using trigger in my scenario?

Best regards,
Jarek
  • 1. Re: Use case for triggers, is there a better way?
    Linda Lee Journeyer
    Currently Being Moderated
    Jarek,

    As you say, while it's easy to tie application code to database updates in a single, non replicated JE environment, it's hard to do that with replication. You do need a trigger to be able to invoke the same application action on the replica. The DatabaseTrigger class you see is an internal class, and is not replication aware.

    We are actually working on a trigger feature right now that will work in a replicated environment. We are not permitted to say when it will be released. If would be great to know more about your use case! For example, we'd like to know:

    -are the concurrent data structures you're using entirely in memory, and therefore temporary?

    -do you use transactions in your JE operations? How does that interact with the fast data structures? For example, if a transaction aborts, do you want to rollback the changes to your data structure?

    -do you have any kind of initialization that you do to the data structures at startup? Do you load them in some way at startup?

    -are you using replication, and how many nodes do you deploy?

    Regards,

    Linda
  • 2. Re: Use case for triggers, is there a better way?
    731051 Newbie
    Currently Being Moderated
    Hi,

    Great to know, that you are working on replication aware triggers. As to my use case:
    - Yes they are entirely in memory and temporary i.e. after DB is closed, they re gone, but it is possible to recreate them from DB. To be more specific, I need to have very quick access to few characteristics of entries in DB e.g. what is the highest value stored in DB with key equal or less some threshold.
    - I am using transactions and interaction with in memory structure is tricky. Currently, because there is no triggers in place in DB I took different road: I completely abandoned the in memory structures and I'm keeping those values in DB using transactions. However, I'd like to go back to the idea of memory data structures for performance reasons. For my use case I would have to be able to roll back the memory structure's changes when DB transaction is aborted. It would be great if your new triggers functionality allow hooking up in transactional fashion e.g. on modification of the entry in db, on commit of this modification and on abort of this modification. It would be also great if there were 3 types of triggers: on create entry, on update, on delete.
    - When DB is opened, before it is first time used I'm populating data structures from the contents of DB.
    - Application is in development, I'm testing it using just two nodes.


    Looking forward to see the triggers functionality. Can't you give any indication when it might be available e.g. maybe this year?

    Best regards,
    Jarek
  • 3. Re: Use case for triggers, is there a better way?
    Linda Lee Journeyer
    Currently Being Moderated
    Jarek,
    For my use case I would have to be able to roll back the memory structure's changes when DB >transaction is aborted. It would be great if your new triggers functionality allow hooking up in
    transactional fashion e.g. on modification of the entry in db, on commit of
    his modification and on abort of this modification. It would be also great if there were
    3 types of triggers: on create entry, on update, >on delete.
    Yes, the new trigger functionality will have entry points for transaction aborts and commits as well as data operations.
    Looking forward to see the triggers functionality. Can't you give any indication when it might be
    available e.g. maybe this year?
    I'm afraid that we are prohibited from announcing release plans in advance. However, we'd like to make sure our test suite covers the kind of usage you describe.

    In particular, there are some interesting issues that come up when a trigger has to do initialization in a replicated system. In a non-replicated environment, or on the master node of a replicated system, the application is aware of when write operations are issued, and can coordinate the trigger initialization with the writes. On a replica in a replicated system, the application isn't aware of how its read operations interact with the background write operations carried out by the replication system, so there's some work to make sure that the initialization happens at an appropriate time. Your use case needs to initialize your in-memory data structure as part of the trigger, so it's a good exercise of that area.

    If you'd like, contact us via email at linda dot q dot lee at o... dot com. When we have a test program that covers your kind of use case, we could pass it by you to see if you've got any feedback on it.

    Regards,

    Linda
  • 4. Re: Use case for triggers, is there a better way?
    731051 Newbie
    Currently Being Moderated
    Has there been any progress on triggers in replicated environment?
  • 5. Re: Use case for triggers, is there a better way?
    greybird Expert
    Currently Being Moderated
    Hi Jarek. Yes, there has been some progress on this internally, but we don't have a date for releasing the trigger functionality yet. Sorry I can't be any more specific than that.

    --mark                                                                                                                                                                                                                                                                                                                                                                                               
  • 6. Re: Use case for triggers, is there a better way?
    985313 Newbie
    Currently Being Moderated
    How about progress now? I'm bumping up against an issue where this would save us a ton of frustration if there's a way to run triggers off a replication db.
  • 7. Re: Use case for triggers, is there a better way?
    greybird Expert
    Currently Being Moderated
    I'm sorry but you should not count on this functionality being available soon. I only mentioned it so that you know that we realize it's important. I cannot give you any guarantee or estimate for when it might be available, per Oracle policy.

    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
  • 8. Re: Use case for triggers, is there a better way?
    Andrius Juozapaitis Newbie
    Currently Being Moderated
    Hey,

    Sorry to revive a thread this old, but I have quite similar requirements as the original poster. I would like to integrate elasticsearch (www.elasticsearch.org) with BDB/JE:
    -Service layer would use DPL API to store/modify/delete data in JE (in transactional context)
    -Elasticsearch would be mostly used to query the stored data in Near-Real-Time
    -Elasticsearch has a neat java API which allows it to insert the documents in bulk, as well as a "river" concept, which essentially publishes all the document inserts, updates, and deletes since a given timestamp.
    -I am also planning on testing it out in a distributed environment, with a single JE master and a backup node(s), although only the master would be publishing the updates to ES.

    Now, the obvious choice to generate the data for the ES would be upon the transaction commit, provided such a trigger exists - I saw something similar used in replication code in JE. Do you have any pointers, code examples, or general suggestions on how to implement this? Worst case, I could just implement this using aspects, but I'd prefer to use standard API if at all possible.


    regards,
    Andrius Juozapaitis
  • 9. Re: Use case for triggers, is there a better way?
    greybird Expert
    Currently Being Moderated
    -I am also planning on testing it out in a distributed environment, with a single JE master and a backup node(s), although only the master would be publishing the updates to ES.
    If I understand correctly, you will not be publishing based on updates to a JE replica node. Therefore, you don't need a trigger capability to do this. Triggers are needed on replicas to do this sort of thing, because it is JE that is doing the writes on the replica. On a JE master node (or a JE standalone node), it is the application that invokes the writes, and therefore the application can fairly easily publish the information to other entities as well.

    I'm not saying that triggers are not desirable for many other situations. I'm only pointing out that the most critical need for triggers is for use on a JE replica node.

    As I mentioned earlier, we at Oracle can't discuss dates or commitments for features in future releases.

    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
  • 10. Re: Use case for triggers, is there a better way?
    Andrius Juozapaitis Newbie
    Currently Being Moderated
    For starters, I created a trigger which intercepts put/delete by implementing a Trigger interface, and I maintains a collection of the affected documents within the transaction. Then, on a TransactionTrigger, I would like to either commit the dataset as a whole, or discard it if the transaction has been aborted. At this moment, I found something weird in the code:


    TriggerManager.java:218
    final List<Trigger> triggers = dbImpl.getTriggers();
    ...
    for (Trigger trigger : triggers ) {
    Trigger dbt = trigger;
    invoker.run(triggerTransaction, dbt );
    }


    Extra variable inside the loop is redundant, but the real issue is:
    TriggerManager.java:229
    public void run(Transaction triggerTransaction,
    Trigger dbt) {
    *if (dbt instanceof TransactionTrigger) {*
    ((TransactionTrigger)dbt).commit(triggerTransaction);
    ...

    But, TransactionTrigger.java interface does not extend a Trigger interface:

    *public interface TransactionTrigger {*

    So transaction triggers would never fire with the 5.0.73 JE, unless I am missing something obvious?


    regards,
    Andrius
  • 11. Re: Use case for triggers, is there a better way?
    greybird Expert
    Currently Being Moderated
    You're using methods that are not part of the product (not published in the javadoc), not fully implemented, not fully tested, and will be removed in the future.

    --mark                                                                                                                                                                                                                                                                                                                                                       
  • 12. Re: Use case for triggers, is there a better way?
    Andrius Juozapaitis Newbie
    Currently Being Moderated
    To be honest, I am very new to BDB/JE, so I may be missing the context here. Documentation is split among a number of forum posts, html pages, pdf docs, javadocs and few topics on stackexchange, and no offense, but I've seen plenty of indie products that had it way better organized. In such a situations, I usually follow the code. The trigger functionality is present in the code, the documentation is in the class headers, no @Deprecated warnings or anything. And it actually works - to some extent.

    Triggers is just a simple callback mechanism, which I thought suited my situation quite well - if it's going away, I'll try something else. All I expect from JE is that it allows me to conveniently access the data that has been committed within a transaction, triggers or aspects or whatever else is available. Any other suggestions?

    regards,
    Andrius
  • 13. Re: Use case for triggers, is there a better way?
    greybird Expert
    Currently Being Moderated
    To be honest, I am very new to BDB/JE, so I may be missing the context here.
    This forum thread is discussing triggers as a possible future feature.
    The trigger functionality is present in the code, the documentation is in the class headers, no @Deprecated warnings or anything.
    The same is true of many internal methods in JE, but you really shouldn't use them. Please only use the methods in the javadoc.
    Triggers is just a simple callback mechanism, which I thought suited my situation quite well - if it's going away, I'll try something else. All I expect from JE is that it allows me to conveniently access the data that has been committed within a transaction, triggers or aspects or whatever else is available. Any other suggestions?
    To access the data that has been committed, you'll need to query it using the Database and Cursor APIs, or the DPL (persist package) if you prefer. Triggers are not currently available.

    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
  • 14. Re: Use case for triggers, is there a better way?
    greybird Expert
    Currently Being Moderated
    >

    -I am also planning on testing it out in a distributed environment, with a single JE master and a backup node(s), although only the master would be publishing the updates to ES.

    If I understand correctly, you will not be publishing based on updates to a JE replica node. Therefore, you don't need a trigger capability to do this. Triggers are needed on replicas to do this sort of thing, because it is JE that is doing the writes on the replica. On a JE master node (or a JE standalone node), it is the application that invokes the writes, and therefore the application can fairly easily publish the information to other entities as well.

    >

    What I was trying to say here is that your application can publish data to elasticsearch at the same time it writes it to JE. So you would only need to query it (as I mentioned earlier), if you are doing the publish step separately. Does that make sense?

    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
1 2 Previous Next

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points