7 Replies Latest reply: Dec 16, 2011 1:19 AM by 857786 RSS

    TXN_SYNC understanding

    857786
      Hi,

      The documentation states the following regarding the documentation of function txn_begin() and the flag DB_TXN_SYNC:

      DB_TXN_SYNC

      Synchronously flush the log when this transaction commits or prepares. This means the transaction will exhibit all of the ACID (atomicity, consistency, isolation, and durability) properties.

      This behavior is the default for Berkeley DB environments unless the DB_TXN_NOSYNC flag was specified to the DbEnv::set_flags() method. Any value specified to this method overrides that setting.

      I have an ambiguity regarding the last sentence:
      "Any value specified to this method overrides that setting."

      What method? The just aforementioned set_flags of DbEnv or txn_begin() itself.

      In other terms if I specify this in the environment (DB_INIT_REP | DB_TXN_NOSYNC); as the flags(equivalent to set_flags) of it but then when creating a transaction with this environment using txn_begin() i specify DB_TXN_SYNC, which one will be taken into account? Will the transaction be SYNC or NOSYNC

      I am using BDB 5.1.19.

      Thx in advance.
        • 1. Re: TXN_SYNC understanding
          "Oracle, Sandra Whitman-Oracle"
          Hello,

          If I understand your question correctly you want to know if
          you invoke DB_ENV->txn_begin() with the DB_TXN_SYNC flag
          and open the environment DB_ENV->open() with the DB_TXN_NOSYNC
          flag will the transaction synchronously flush the log when
          the transaction commits or prepares.

          In this case the transaction will flush the log synchronously
          because DB_TXN_NOSYNC was not specified to DB_ENV->set_flags().
          It is actually the case that DB_ENV->open() does not make use
          of the DB_TXN_NOSYNC flag. If you look at the documentation
          on DB_ENV->open() you will notice DB_TXN_NOSYNC is not mentioned:
          http://docs.oracle.com/cd/E17076_02/html/api_reference/C/envopen.html
          The value of DB_TXN_NOSYNC from <platform>/db.h is:
          #define DB_TXN_NOSYNC 0x00000001
          This is the same as DB_CREATE:
          #define DB_CREATE 0x00000001

          So setting DB_TXN_NOSYNC is the same as using DB_CREATE in
          DB_ENV->open(). It will not influence the way the transaction flushes
          the log on commit or prepare.


          Thank you,
          Sandra
          • 2. Re: TXN_SYNC understanding
            857786
            Thx Sandra, that's helpful.

            If i call explicitly to set_flags on the environment with TXN_NOSYNC, and create the transaction tn_begin() with TXN_SYNC, which one will be taken into account in this case when committing the transaction? Will it be flushed to log synchronously(like the transaction) or asynchronoulsy(like the environment)?

            Also I am using BDB 5.1.19, and i actually use the replication manager. Do i understand correctly that the difference between TXN_SYNC and TXN_NOSYNC as following?

            TXN_SYNC: this means that when the thread calls the function commit() on a given transaction, the call will be blocked(and hence the thread) until the data of the transaction is written to disk and sent to the replica and ack is received. Is this understanding accurate?

            TXN_NOSYNC: this means that when the thread calls the function commit() on a given transaction, the writin to disk + sending and receiving ack on the transaction will be done asynchronously which i believe is using another thread to get the async behavior. Is that understanding correct?

            Our application is very sensitive to latency and throughput and it's critical for us to get an accurate understanding of things work. Th in advance for your explanations.
            • 3. Re: TXN_SYNC understanding
              857786
              If i open a ticket in the support will the question be answered faster?
              Thx again for cooperation.
              • 4. Re: TXN_SYNC understanding
                Paula B-Oracle
                If i call explicitly to set_flags on the environment with TXN_NOSYNC, and create the transaction tn_begin() with TXN_SYNC, which one will be taken into account in this case when committing the transaction? Will it be flushed to log synchronously(like the transaction) or asynchronoulsy(like the environment)?
                The setting on the transaction is the one that is used and it overrides the setting on the environment.
                Also I am using BDB 5.1.19, and i actually use the replication manager. Do i understand correctly that the difference between TXN_SYNC and TXN_NOSYNC as following?

                TXN_SYNC: this means that when the thread calls the function commit() on a given transaction, the call will be blocked(and hence the thread) until the data of the transaction is written to disk and sent to the replica and ack is received. Is this understanding accurate?
                Basically you are correct with a few additional points.

                The commit thread waits up to the DB_REP_ACK_TIMEOUT for its acks rather than waiting forever.

                Also, there are some cases where we can determine in advance that we cannot possibly get enough acks to satisfy the ack policy. For example, if you have specified the ACKS_ALL policy but one site is not currently connected, we don't have to wait for the acks to know that we won't get enough. In these cases, we do not wait, but instead return immediately.
                TXN_NOSYNC: this means that when the thread calls the function commit() on a given transaction, the writin to disk + sending and receiving ack on the transaction will be done asynchronously which i believe is using another thread to get the async behavior. Is that understanding correct?
                No, this is not correct. Notice that our terminology is SYNC/ NOSYNC rather than SYNC /ASYNC.

                For TXN_NOSYNC, the commit of a transaction will not write its changes to the log and will not flush the log. Writing the changes to the log and flushing the log are the "SYNC" that we avoid doing if TXN_NOSYNC is set.

                Setting TXN_NOSYNC has no effect on whether Replication Manager waits for acks. The commit thread still waits for acks, subject to the exceptions I mentioned above.

                Paula Bingham
                Oracle
                • 5. Re: TXN_SYNC understanding
                  857786
                  Thx a lot that is very helpful.

                  A few other quesitons:
                  You say:
                  "For TXN_NOSYNC, the commit of a transaction will not write its changes to the log and will not flush the log. Writing the changes to the log and flushing the log are the "SYNC" that we avoid doing if TXN_NOSYNC is set."

                  I am confused about the TXN_NOSYNC, are you saying that when using TXN_NOSYNC, all the db-put of this transactions will not be written to the disk at all?
                  If this is true does that mean if the slave is not there for 1 hour, and the master box reboots, does that mean that the all the transactions of the last hour will be lost?

                  This makes me re-think my understanding of how BDB works.
                  Is the following correct?
                  A typical application like the one we develop is basically writing to database(db_put() for ex) and then committing transactions.
                  My understanding is:
                  - Every db->put will never write to disk but to the shared mem buffer of BDB. Is it correct or db->put could trigger write to disk too?
                  - At commit time, all of the changes in the transaction will be put to log file as well as sent to the slave. Is it correct?

                  Are you saying that at commit time with the transaction is TX_NOSYNC it will be just sent to the slave but not written to disk? Or is the NOSYNC saying just that the data is not written to the log immediately when performing the commit() call but later using some kind of thread?

                  Thx in advance.

                  Edited by: 854783 on Dec 15, 2011 12:27 AM
                  • 6. Re: TXN_SYNC understanding
                    Paula B-Oracle
                    You say:
                    "For TXN_NOSYNC, the commit of a transaction will not write its changes to the log and will not flush the log. Writing the changes to the log and flushing the log are the "SYNC" that we avoid doing if TXN_NOSYNC is set."

                    I am confused about the TXN_NOSYNC, are you saying that when using TXN_NOSYNC, all the db-put of this transactions will not be written to the disk at all?
                    No, the write of the changes to the log file and the log flush will happen at the earlier of the following two events: 1. your next checkpoint or 2. when the in-memory log buffer fills up and needs more space.

                    You get a performance benefit by not doing separate disk I/Os for each transaction commit. These disk I/O's are instead batched up and done together at one of the above two times.
                    If this is true does that mean if the slave is not there for 1 hour, and the master box reboots, does that mean that the all the transactions of the last hour will be lost?
                    If you haven't done a checkpoint or if the in-memory log buffer hasn't happened to fill up in the last hour, then yes, those transactions are lost.

                    The checkpoints are under your control. Use of TXN_NOSYNC needs to be paired with a checkpoint strategy that reflects the maximum amount of time that you want to risk transaction durability.


                    >
                    This makes me re-think my understanding of how BDB works.
                    Is the following correct?
                    A typical application like the one we develop is basically writing to database(db_put() for ex) and then committing transactions.
                    My understanding is:
                    - Every db->put will never write to disk but to the shared mem buffer of BDB. Is it correct or db->put could trigger write to disk too?
                    No, db_put does not cause the log to be written to disk.
                    - At commit time, all of the changes in the transaction will be put to log file as well as sent to the slave. Is it correct?
                    If you do not use TXN_NOSYNC, the changes are written to the log file at commit time.

                    The changes are sent to the slave as they occur. At commit time, a commit log record is sent to the slave. The replication master waits for acks only on a commit record (not all the other changes) if required by your ack policy.

                    >
                    Are you saying that at commit time with the transaction is TX_NOSYNC it will be just sent to the slave but not written to disk? Or is the NOSYNC saying just that the data is not written to the log immediately when performing the commit() call but later using some kind of thread?
                    Regardless of TXN_NOSYNC, changes are sent to the slave as they occur.

                    When using TXN_NOSYNC, changes are written to the in-memory log buffer as they occur. The log buffer is written to the log disk file and the log disk file is flushed either at your next checkpoint or when the log buffer fills up. The write to the log disk file is not done in another thread, but rather in the same thread at one of these two times.

                    If you anticipate failures at replicated sites, you can also add more replicated sites to improve the chances that a committed transaction is durably stored somewhere.

                    Paula Bingham
                    Oracle
                    • 7. Re: TXN_SYNC understanding
                      857786
                      Thx a lot. That is helpful.