This discussion is archived
6 Replies Latest reply: Apr 7, 2013 7:39 AM by 998836 RSS

Log files not being removed.

998836 Newbie
Currently Being Moderated
Hello,

I've upgraded an application from BerkeleyDB 5.1.25 to 5.3.21, and after that, log files are no more automatically removed. This is the only change in the application. It's an application written in C.

The environment of the application is created with the flag DB_LOG_AUTO_REMOVE
dbenv->log_set_config(dbenv, DB_LOG_AUTO_REMOVE, TRUE).

The application has a thread to periodically checkpoint the data
dbenv->txn_checkpoint(dbenv, 0, 0, 0)

So far, so good, with version 5.1.25, this was enough to remove unused log files (I don't need to be able to do catastrophic recovery). But this doesn't work anymore with version 5.3.21.

I I run db_archive (no options), it shows nothing, suggesting that all log files are still needed. But if I run db_hot_backup on the database, all but the last logfiles are removed (on the backup) as wanted.

Rem : Usually, I don't want to run db_archive or any external tool, to remove unused log files. I hope what is inside the application is enough to remove unused log files.

Is this something known, something changed or can you suggest me something to look for ?

Thanks for your help

José-Marcio

Edited by: user564597 on Mar 24, 2013 6:35 PM

Edited by: user564597 on Mar 24, 2013 6:38 PM

Edited by: user564597 on Mar 25, 2013 8:57 AM
  • 1. Re: Log files not being removed.
    userBDBDMS Guru Moderator
    Currently Being Moderated
    We will take a look at this and see if we can reproduce the behavior. If we find something, I will post it to this thread.


    thanks
    mike
  • 2. Re: Log files not being removed.
    userBDBDMS Guru Moderator
    Currently Being Moderated
    We did some testing in house and it all worked for us. We are not getting the behavior characteristics that you are seeing. If you could post some code that represents a reproducible test case, then we can take another look.

    thanks
    mike
  • 3. Re: Log files not being removed.
    998836 Newbie
    Currently Being Moderated
    Thanks for your answer.

    I have this problem with Solaris 10. I'll try it under Linux. And I'm writing a piece of code doing just database updates, using the database abstraction code of my application to try on both OSs. If it doesn't work, or if I discover something, I'll post it here.

    Regards,

    José-Marcio
  • 4. Re: Log files not being removed.
    998836 Newbie
    Currently Being Moderated
    Hello,

    I checked again and I have this problem with both Solaris 10 and Linux (tested on Fedora 18). Log files are autoremoved with BDB 5.1.25 but not BDB 5.3.21.

    I've write a small test program, with the way I use it.
    * j-db.c is the interface between my application and Berkeley DB.
    * do-test.c is the test program which adds new records during 15 minutes. You can launch it as many times you want. The database/environment will be create inside the directory "tmp" below.

    As long as I can't attach something here, I uploaded it at :

    http://www.j-chkmail.org/users/bdb/test-log-remove.tar.gz

    The tar file contains a directory named test-log-remove. You shall put it inside Berkeley DB distribution, at the same level of build_unix directory, and launch make after building Berkeley DB library, which was configured as "../dist/configure --enable-static --disable-shared".

    Best regards and thanks for your help

    As

    Edited by: user564597 on Apr 3, 2013 2:02 PM
  • 5. Re: Log files not being removed.
    userBDBDMS Guru Moderator
    Currently Being Moderated
    thank you for giving us a test program. This helped tremendously to fully understand what you are doing. In 5.3 we fixed a bug dealing with the way log files are archived in an HA environment. What you are running into is the consequences of that bug fix. In the test program you are using DB_INIT_REP. This is the key to use that you want an HA environment. With HA, there is a master and some number of read only clients. By default we treat the initiating database as the master. This is what is happening in our case. In an HA (replicated) environment, we cannot archive log files until we can be assured that the clients have applied the contents of that log file. Our belief is that you are not really running in an HA environment and you do not need the DB_INIT_REP flag. In our initial testing where we said it worked for us, this was because we did not use the DB_INIT_REP flag, as there was no mention of replication being needed in the post.

    Recommendation: Please remove the use of the DB_INIT_REP flag or properly set up an HA environment (details in our docs).


    thanks
    mike
  • 6. Re: Log files not being removed.
    998836 Newbie
    Currently Being Moderated
    Hello,
    Yes ! That solved my problem.
    Thank you very much for the efficient support.
    Regards,
    José-Marcio

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points