I've upgraded an application from BerkeleyDB 5.1.25 to 5.3.21, and after that, log files are no more automatically removed. This is the only change in the application. It's an application written in C.
The environment of the application is created with the flag DB_LOG_AUTO_REMOVE
dbenv->log_set_config(dbenv, DB_LOG_AUTO_REMOVE, TRUE).
The application has a thread to periodically checkpoint the data
dbenv->txn_checkpoint(dbenv, 0, 0, 0)
So far, so good, with version 5.1.25, this was enough to remove unused log files (I don't need to be able to do catastrophic recovery). But this doesn't work anymore with version 5.3.21.
I I run db_archive (no options), it shows nothing, suggesting that all log files are still needed. But if I run db_hot_backup on the database, all but the last logfiles are removed (on the backup) as wanted.
Rem : Usually, I don't want to run db_archive or any external tool, to remove unused log files. I hope what is inside the application is enough to remove unused log files.
Is this something known, something changed or can you suggest me something to look for ?
We did some testing in house and it all worked for us. We are not getting the behavior characteristics that you are seeing. If you could post some code that represents a reproducible test case, then we can take another look.
I have this problem with Solaris 10. I'll try it under Linux. And I'm writing a piece of code doing just database updates, using the database abstraction code of my application to try on both OSs. If it doesn't work, or if I discover something, I'll post it here.
I checked again and I have this problem with both Solaris 10 and Linux (tested on Fedora 18). Log files are autoremoved with BDB 5.1.25 but not BDB 5.3.21.
I've write a small test program, with the way I use it.
* j-db.c is the interface between my application and Berkeley DB.
* do-test.c is the test program which adds new records during 15 minutes. You can launch it as many times you want. The database/environment will be create inside the directory "tmp" below.
As long as I can't attach something here, I uploaded it at :
The tar file contains a directory named test-log-remove. You shall put it inside Berkeley DB distribution, at the same level of build_unix directory, and launch make after building Berkeley DB library, which was configured as "../dist/configure --enable-static --disable-shared".
thank you for giving us a test program. This helped tremendously to fully understand what you are doing. In 5.3 we fixed a bug dealing with the way log files are archived in an HA environment. What you are running into is the consequences of that bug fix. In the test program you are using DB_INIT_REP. This is the key to use that you want an HA environment. With HA, there is a master and some number of read only clients. By default we treat the initiating database as the master. This is what is happening in our case. In an HA (replicated) environment, we cannot archive log files until we can be assured that the clients have applied the contents of that log file. Our belief is that you are not really running in an HA environment and you do not need the DB_INIT_REP flag. In our initial testing where we said it worked for us, this was because we did not use the DB_INIT_REP flag, as there was no mention of replication being needed in the post.
Recommendation: Please remove the use of the DB_INIT_REP flag or properly set up an HA environment (details in our docs).