5 Replies Latest reply: Jan 22, 2014 1:55 AM by 3c09370d-b4ea-4ce1-9672-25481141d6c4 RSS

LSN "past end of log" error

3c09370d-b4ea-4ce1-9672-25481141d6c4 Newbie
Currently Being Moderated

Hello,

 

My application has shutdown unexpectedly and during shutdown procedure got an error 9 from env_close.

From then on every attempt to open the DB gives an error like this:

     file user.db has LSN 19/569276, past end of log at 19/9564

     Commonly caused by moving a database from one database environment

     to another without clearing the database LSNs, or by removing all of

     the log files from a database environment

     __db_meta_setup: user.db: unexpected file type or format

 

The environment is opened with DB_INIT_LOG | DB_INIT_MPOOL | DB_INIT_LOCK | DB_INIT_TXN from multiple processes.

DB_THREAD is not used since access to the api from these processes is serialized.

The database file is opened with DB_CREATE | DB_AUTO_COMMIT.

There is a watchdog process that opens the environmen twith DB_CREATE | DB_INIT_LOG | DB_INIT_MPOOL | DB_INIT_LOCK | DB_INIT_TXN | DB_THREAD| DB_RECOVER .

The watchdog runs first to recover the DB at each app start. During normal operation it has threads the take checkpoints , flush and remove old logs.

 

 

When I do a db_verify I get the same LSN error.

When I do a db_recover it seems to work since the db_verify then succeeds and i am able to db_dump.

Put when start again my application I get the same LSN error again.

I did not move the database and only remove logs that are returned from log_archive with DB_ARCH_ABS.

 

What could be going on here?

 

The application is using BDB 4.8.(its a legacy system)

 

Thanks,

  • 1. Re: LSN "past end of log" error
    3c09370d-b4ea-4ce1-9672-25481141d6c4 Newbie
    Currently Being Moderated

    Hi,

     

    Let me clarify my question.

    In this case of LSN past end of log:

    How can the DB be recovered in this situation?

    What could of caused this to happen? (aside from what the error messages suggested)

     

    Any help would be appreciated

  • 2. Re: LSN "past end of log" error
    userBDBDMS Guru Moderator
    Currently Being Moderated

    From your description:

    When I do a db_verify I get the same LSN error.

    When I do a db_recover it seems to work since the db_verify then succeeds and i am able to db_dump.

    Put when start again my application I get the same LSN error again.

     

    if your database was in a known good state after the recovery, and you get the error after you start your application, then it is possible there is something in the application that is stomping on the log records before they are written out to disk.   There could be some buffer over flow or something else in that space.  The best way to recover is to use catastrophic recovery and apply logs up to the point where lsn is corrupted.  

     

    This is a fairly complicated problem and it is going to be very difficult to debug over the forum.   Do you have a support contract with Oracle?  If so, have you filed an SR?  You have already listed the common cause for this error and ruled that out.    Other than the common case, this comes up when there has been some data corruption of which a common one there is corrupt sectors on a disk.

     

    thanks

    mike 

  • 3. Re: LSN "past end of log" error
    3c09370d-b4ea-4ce1-9672-25481141d6c4 Newbie
    Currently Being Moderated

    Hi,

     

    Thanks for the reply, I just now noticed it.

    This does seem like a complicated problem yet unfortunately we do not have a support contract.

     

     

    I do not think it was caused by a hardware failure since we are running on virtual machine.

    The possibility of a buffer over flow would be much more likely, although I can't see how this could the LSNs of the DB and logs to be mismatching.

     

    (Could log_archive have caused anything if the env was open by another process?)

     

    One thing maybe easier to answer is the  report from the db_verify tool.

    Why does it not detect the LSN mismatch after the db_recover was run?

    The output of db_printlog shows that the DB LSN is ahead of the log file so why does db_verify return success?

     

    Thanks again,

  • 4. Re: LSN "past end of log" error
    userBDBDMS Guru Moderator
    Currently Being Moderated

    purpose of db_verify is to verify the integrity of the database files themselves.

    It checks for many different things dealing with the structural integrity of

    the files themselves.    There is also another tool to verify the integrity

    of the log files.   I would recommend you run db_log_verify and see

    what that gives you.    It seems that you log file may have gotten

    corrupted.  You can dump the data in the db after it is recovered

    and then rebuild a new db from scratch, then load it with the

    data the dumped.   Recovery has to touch both the database files

    and the log files because it needs to handle roll forward/backward

    issues.    We have 2 verify tools each focused in on their specific

    area, one for logs and one for db.

     

    thanks

    mike

  • 5. Re: LSN "past end of log" error
    3c09370d-b4ea-4ce1-9672-25481141d6c4 Newbie
    Currently Being Moderated

    Thanks,

    Unfortunately I don't have the db_log_verify utility since I am on 4.8 ...

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points