5 Replies Latest reply on Apr 5, 2013 10:03 AM by 928755

    After upgrade 4.3 -> 5.3: file unknown has LSN ..., past end of log at ...

    928755
      We have been using a BerkeleyDB for about six years in production, storing ~100 GB of data in ~500 db files in one environment. The application uses the Perl API to access the BerkeleyDB.

      After a week of testing, we upgraded our production server from BerkeleyDB 4.3.29 to 5.3.21 yesterday, running db_upgrade on the db files. While the upgrade ran smoothly, since then, we have the following issue:
      When accessing some database files for writing for the first time, db_put fails with the error message:

      file unknown has LSN ..., past end of log at ...
      Commonly caused by moving a database from one database environment
      to another without clearing the database LSNs, or by removing all of
      the log files from a database environment

      Since we did not move the environment or any of the database files, the "common cause" is not the issue here.
      db_verify is happy with the database files in question, db_recover has also been tried without success.

      The only way to get the database working (we found so far) is a dump-load cycle.

      Do you have any idea what is causing these issues?
      The BerkeleyDB is stored on vxfs mounted with rw,delaylog,largefiles,ioerror=mwdisable options.
      We open the environment with these flags:

      DB_INIT_TXN | DB_INIT_LOCK | DB_INIT_LOG | DB_INIT_MPOOL | DB_DSYNC_DB
      and set DB_LOG_DSYNC, just to be on the save side...

      The application is running on Red Hat Enterprise Linux Server release 5.6 (Tikanga)

      Thanks for your help...
        • 2. Re: After upgrade 4.3 -> 5.3: file unknown has LSN ..., past end of log at ...
          Bogdan Coman-Oracle
          Hi, Did you follow the upgrade procedure as specified in the documentation?

          http://docs.oracle.com/cd/E17076_02/html/upgrading/upgrade_process.html

          The upgrade procedure for your application is as follows:
          1. Shut down the old version of the application.

          2. Still using the old version of Berkeley DB, run recovery on the database environment using the DB_ENV->open() method, or the db_recover utility.

          3. If you used the DB_ENV->open() method to run recovery, make sure that the Berkeley DB environment is removed using the DB_ENV->remove() method or an appropriate system utility.

          4. Archive the database environment for catastrophic recovery using the db_archive utility as described in the Database and log file archival section in the Berkeley DB Programmer's Reference Guide.

          5. Recompile and install the new version of the application.

          6. Upgrade the application's databases. See Database upgrade for more information.

          7. Archive the database for catastrophic recovery again (using different media than before, of course). Note: This archival is not strictly necessary. However, if you have to perform catastrophic recovery after restarting the application, that recovery must be done based on the last archive you have made. If you make this second archive, you can use it as the basis of that catastrophic recovery. If you do not make this second archive, you have to use the archive you made in step 4 as the basis of your recovery, and you have to do a full upgrade on it before you can apply log files created after the upgrade to it.

          8. Force a checkpoint using the DB_ENV->txn_checkpoint() method or the db_checkpoint utility. If you use the db_checkpoint utility, make sure to use the new version of the utility; that is, the version that came with the release of Berkeley DB to which you are upgrading.

          Note that forcing a checkpoint might result in warning messages about log files that are being skipped. This is normal, and can be safely ignored.

          9. Remove unnecessary log files from the environment using the -d option on the db_archive utility, or from an application which calls the DB_ENV->log_archive() method with the DB_ARCH_REMOVE flag.

          Note that removing log files in this way might result in warning messages about log files that are being skipped. This is normal, and can be safely ignored.

          Note that if you are upgrading a replicated application, then you should not perform this step until all of the replication sites have been upgraded to the current release level. If you run this site before all your sites are upgraded, then errors can occur in your replication activities because important version information might be lost.

          10. Restart the application.

          What db_upgrade does, it does only the 6th step. You need to follow the other steps too.

          Thanks,
          Bogdan
          1 person found this helpful
          • 3. Re: After upgrade 4.3 -> 5.3: file unknown has LSN ..., past end of log at ...
            928755
            Dear Bogdan,

            Instead of steps 8 and 9 we removed the __db.* environment files and the log files, intending to start a new environment from scratch.
            • 4. Re: After upgrade 4.3 -> 5.3: file unknown has LSN ..., past end of log at ...
              Bogdan Coman-Oracle
              925752 wrote:
              Dear Bogdan,

              Instead of steps 8 and 9 we removed the __db.* environment files and the log files, intending to start a new environment from scratch.
              I see, that's okay, you can do it, but you'd have to reset the databases' log sequence numbers (LSNs), so you disconnect the databases from the old log files and start with fresh ones.
              In order to do this, I would run db_upgrade on the databases and after that reset the LSNs. You have two options to reset the LSNs, either you do it programmatically by using the DB_ENV->lsn_reset() method (documented at http://docs.oracle.com/cd/E17076_02/html/api_reference/C/envlsn_reset.html) or, even better, you use the 'db_load -r lsn' utility (documented at http://docs.oracle.com/cd/E17076_02/html/api_reference/C/db_load.html) which does the same thing. Once you reset the LSNs, replace the old databases, remove the region and log files, start your application and way you go.

              Let me know if you still have problems.

              Bogdan Coman
              • 5. Re: After upgrade 4.3 -> 5.3: file unknown has LSN ..., past end of log at ...
                928755
                Bogdan,
                thanks a lot, we could verify your answer: After running "db_load -r lsn", writing to the databases is again possible without running into the mentioned error!

                Kind regards,
                Philipp