1 2 Previous Next 16 Replies Latest reply: Sep 23, 2010 4:30 AM by 799756 RSS

    question about quorum model

    677539
      Hi,

      I have 3 replicas: A,B, C. A is master, B is down, C is initializing(sync) with A. A has LSN=10, B has LSN=10, and C has LSN=1. I am using QUORUM ack model.

      In this case, a writing operation in A can be successful?

      I expect it should not be successful, otherwise it will cause data loss.
      For example, based on the above scenario, after a new write which make A's lsn reach 15, A is down and B comes back. Right now, A has lsn=15 but dead, B has lsn=10, C has lsn=1. No matter B or C become master, data in A (lsn 11- lsn 15) will be lost.


      I need an engineer from Oracle to confirm this.

      Thanks,

      -hp
        • 1. Re: question about quorum model
          629233
          There is a related SR 3-1939228011. Can you please answer the question there as well.
          • 2. Re: question about quorum model
            524722
            Ritesh, a similar answer, for the SR has also been provided to support to this same question.

            In the case shown, when A writes LSNs 11-15, it will not receive an ack from the
            QUORUM policy from C. Therefore, A will receive the DB_EVENT_REP_PERM_FAILED
            event indicating that data is not known to be durable among the members of the group
            and is at risk. The last known HA durable data is at LSN 10.

            So, yes, there is a period where data is vulnerable and the app is being told
            that the data is at risk. A poorly timed crash during that period can result in data loss
            on that one site. It is up to the application to decide what it wants to do with the
            PERM_FAILED event.

            If master leases were being used, during that same situation, site A would panic if its
            attempts to get a lease grant during the txn_commit were unsuccessful.

            Sue LoVerso
            Oracle
            • 3. Re: question about quorum model
              629233
              Thank you Sue.

              So if we use master leases, the master will not be permitted to commit a transaction until it has quorum. Thus, the replication layer guarantees that (if configured with Quorum) each committed txn is present with quorum which in a 3 way replication should avoid loss of transactions.

              The problem with the PERM failure is that the transaction is committed successfully and the PERM failure is sent as a callback. If instead the PERM failure was sent sync in the call itself, the app can skip sending a successful response or even abort/undo the transaction which prevents "perceived" data loss.
              In the worst case we wrote data and did not inform which would be rectified on the subsequent read. Also, the app can then continue waiting for meeting quorum before resuming updates.

              So your recommendation is to use master leases and sync data to avoid this situation. If we do get into a situation where quorum is not met for commit the txn will fail and the DB layer will PANIC forcing a restart and a re-election.

              In the documentation often the term is "total replication group failure"referenced without specifying it's definition. Which conditions would constitute a "total replication group failure"

              Durability from rollback: A guarantee that the data being written or read by the application is permanent across a majority of client sites and will never be rolled back.
              The rollback guarantee also depends on the DB_TXN_NOSYNC flag. The guarantee is effective as long as there isn't total replication group failure while clients have granted leases but are holding the updates in their cache. The application must weigh the performance impact of synchronous transactions against the risk of total replication group failure. If clients grant a lease while holding updated data in cache, and total failure occurs, then the data is no longer present on the clients and rollback can occur if the master also crashes.
              • 4. Re: question about quorum model
                524722
                Ritesh S, wrote:
                The problem with the PERM failure is that the transaction is committed successfully and the PERM failure is sent as a callback. If instead the PERM failure was sent sync in the call itself, the app can skip sending a successful response or even abort/undo the transaction which prevents "perceived" data loss.
                I am not quite sure what you mean here. You are correct that the txn_commit
                API call returns success even if the event callback is made, or not.
                But the PERM_FAILED event callback is made with the thread that has called txn_commit,
                before txn_commit returns to the user. So, a thread, using some kind of thread-local storage,
                could set some kind of "is_my_txn_durable" bit to indicate the failure and then
                when the txn_commit call returns, check that bit to indicate a successful response
                to the user/caller or not. Since a PERM_FAILED occurs after the txn is committed
                on the master site, you cannot undo or abort at that time. It is committed locally.
                In the documentation often the term is "total replication group failure"referenced without specifying it's definition. Which conditions would constitute a "total replication group failure"

                Durability from rollback: A guarantee that the data being written or read by the application is permanent across a majority of client sites and will never be rolled back.
                The rollback guarantee also depends on the DB_TXN_NOSYNC flag. The guarantee is effective as long as there isn't total replication group failure while clients have granted leases but are holding the updates in their cache. The application must weigh the performance impact of synchronous transactions against the risk of total replication group failure. If clients grant a lease while holding updated data in cache, and total failure occurs, then the data is no longer present on the clients and rollback can occur if the master also crashes.
                Here the term refers to all sites in the group crashing. If all sites only have the data in
                cache/memory, and all sites crash and the memory is cleared, there will not be any
                site that retains a record of that data.

                Sue LoVerso
                Oracle
                • 5. Re: question about quorum model
                  629233
                  Posting a question asked in the SR.

                  Hi,

                  I get the error message "DB_ENV->rep_elect: nsites must be zero if leases configured" on all of sites when enable master lease.

                  We are using quorum ack;

                  have the code below on all of sites:
                  repEnv->rep_set_nsites(3);
                  repEnv->rep_set_config(DB_REP_CONF_LEASE, 1);
                  repEnv->rep_set_timeout(DB_REP_LEASE_TIMEOUT, 15000000);

                  Thanks,
                  -Yuan
                  • 6. Re: question about quorum model
                    524761
                    I gather you're using the replication Base API, rather than Replication Manager; correct?

                    What this message is trying to say is that when using leases, the call to rep_elect() must pass a 0 for the nsites argument.

                    Alan Bram
                    Oracle
                    • 7. Re: question about quorum model
                      629233
                      No we are using replication manager.
                      • 8. Re: question about quorum model
                        524761
                        Are you using an older version of Berkeley DB then?

                        There was a bug (#16494) in Replication Manager when using leases that looked like this. But it was fixed in DB version 4.8.

                        Alan Bram
                        Oracle
                        • 9. Re: question about quorum model
                          629233
                          We are using 4.7.25
                          • 10. Re: question about quorum model
                            629233
                            Can you please provide us with the set of known patches against 4.7.25 for Master leases.
                            • 11. Re: question about quorum model
                              524761
                              We would encourage you to upgrade to the current release of Berkeley DB.

                              Alan Bram
                              Oracle
                              • 12. Re: question about quorum model
                                629233
                                We do plan to do that but we need to support master leases with 4.7.25.
                                Do you recommend that it is possible?
                                • 13. Re: question about quorum model
                                  629233
                                  DB_REP_CONF_2SITE_STRICT
                                  Does this guarantee no txn will be committed on the master if the slave is down? or only that an election will not elect a master when one out of the two sites is down?
                                  We can delay the master lease if DB_REP_CONF_2SITE_STRICT insures that on site failure the master will not be able to commit any transaction.
                                  • 14. Re: question about quorum model
                                    524761
                                    The 2SITE_STRICT setting affects elections.

                                    Have you considered Sue's suggestion? That is, during a commit if the
                                    master does not receive acknowledgements from a quorum of clients, it
                                    fires the PERM_FAILED event. You can use that to decide if the
                                    transaction is successfully replicated to enough sites to guarantee
                                    durability. (If you don't get a PERM_FAILED event, you know the
                                    transaction has been replicated durably.)

                                    Alan Bram
                                    Oracle
                                    1 2 Previous Next