This discussion is archived
6 Replies Latest reply: Jun 14, 2013 1:56 PM by greybird RSS

com.sleepycat.je.EnvironmentFailureException: (JE 5.0.73) com.sleepycat.je.log.ChecksumException

1005812 Newbie
Currently Being Moderated

We are seeing the following exceptions after it was advised that upgrading to Berkley DB  5.0.73 would fix the issues… (je-5.0.73.jar)

The Platform we are running this is  Linux

 

1)       Linux njqemsaux04 2.6.32-358.0.1.el6.x86_64 #1 SMP Wed Feb 20 11:05:23 EST 2013 x86_64 x86_64 x86_64 GNU/Linux

 

 

We only set the following to 90 M

EnvironmentConfig envConfig = new EnvironmentConfig();

           

2)                     envConfig.setConfigParam(EnvironmentConfig.LOG_FILE_MAX, "90000000"); //90M

 

 

 

3)       These are the contents of  the ENV HOME DIR  ( where the DB files are stored)

 

total 179672

-rw-rw-r--   1 fxall fxall        0 Jun 10 17:21 je.info.0.lck

-rw-rw-r--   1 fxall fxall        0 Jun 10 17:21 je.lck

drwxrwxr-x 266 fxall fxall    49152 Jun 10 17:21 ..

-rw-rw-r--   1 fxall fxall 89909152 Jun 11 02:10 00000000.jdb

-rw-rw-r--   1 fxall fxall 89920384 Jun 11 02:16 00000001.jdb

drwxr-xr-x   2 fxall fxall     4096 Jun 11 02:16 .

-rw-rw-r--   1 fxall fxall  4076773 Jun 11 02:16 00000002.jdb

-rw-rw-r--   1 fxall fxall     5563 Jun 11 02:16 je.info.0

 

 

 

4)       These were the exceptions in PRODN

 

 

Jun 11, 2013 2:16:58 AM com.sleepycat.je.utilint.DaemonThread run

SEVERE: <DaemonThread name="Cleaner-1"/> caught exception, com.sleepycat.je.EnvironmentFailureException: (JE 5.0.73) /opt/fxall/logs/TG_REGION com.sleepycat.je.log.ChecksumException: Read invalid log entry type: 102 LOG_CHECKSUM: Checksum invalid on read, log is likely invalid. Environment is invalid and must be closed. fetchTarget of 0x0/0x9bc934 parent IN=5 IN class=com.sleepycat.je.tree.BIN lastFullVersion=0x1/0x430c18e lastLoggedVersion=0x2/0x55d46 parent.getDirty()=true state=0 Exiting

  1. com.sleepycat.je.EnvironmentFailureException: (JE 5.0.73) /opt/fxall/logs/TG_REGION com.sleepycat.je.log.ChecksumException: Read invalid log entry type: 102 LOG_CHECKSUM: Checksum invalid on read, log is likely invalid. Environment is invalid and must be closed. fetchTarget of 0x0/0x9bc934 parent IN=5 IN class=com.sleepycat.je.tree.BIN lastFullVersion=0x1/0x430c18e lastLoggedVersion=0x2/0x55d46 parent.getDirty()=true state=0

        at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:851)

        at com.sleepycat.je.log.LogManager.getLogEntryAllowInvisibleAtRecovery(LogManager.java:809)

        at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1412)

        at com.sleepycat.je.dbi.CursorImpl.getCurrentAlreadyLatched(CursorImpl.java:1466)

        at com.sleepycat.je.dbi.CursorImpl.getNext(CursorImpl.java:1595)

        at com.sleepycat.je.cleaner.UtilizationProfile.getObsoleteDetail(UtilizationProfile.java:632)

        at com.sleepycat.je.cleaner.FileProcessor.processFile(FileProcessor.java:439)

        at com.sleepycat.je.cleaner.FileProcessor.doClean(FileProcessor.java:289)

        at com.sleepycat.je.cleaner.FileProcessor.onWakeup(FileProcessor.java:151)

        at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:163)

        at java.lang.Thread.run(Thread.java:662)

Caused by: com.sleepycat.je.log.ChecksumException: Read invalid log entry type: 102

        at com.sleepycat.je.log.LogEntryHeader.<init>(LogEntryHeader.java:161)

        at com.sleepycat.je.log.LogManager.getLogEntryFromLogSource(LogManager.java:928)

        at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:848)

        at com.sleepycat.je.log.LogManager.getLogEntryAllowInvisibleAtRecovery(LogManager.java:809)

        at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1416)

        ... 8 more

 

 

 

 

  1. com.sleepycat.je.EnvironmentFailureException: (JE 5.0.73) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 5.0.73) /opt/fxall/logs/TG_REGION com.sleepycat.je.log.ChecksumException: Read invalid log entry type: 102 LOG_CHECKSUM: Checksum invalid on read, log is likely invalid. Environment is invalid and must be closed. fetchTarget of 0x0/0x9bc934 parent IN=5 IN class=com.sleepycat.je.tree.BIN lastFullVersion=0x1/0x430c18e lastLoggedVersion=0x2/0x55d46 parent.getDirty()=true state=0

        at com.sleepycat.je.EnvironmentFailureException.wrapSelf(EnvironmentFailureException.java:196)

        at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1512)

        at com.sleepycat.je.Database.checkEnv(Database.java:1884)

        at com.sleepycat.je.Database.openCursor(Database.java:676)

        at com.sleepycat.collections.CurrentTransaction.openCursor(CurrentTransaction.java:415)

        at com.sleepycat.collections.MyRangeCursor.openCursor(MyRangeCursor.java:53)

        at com.sleepycat.collections.MyRangeCursor.<init>(MyRangeCursor.java:29)

        at com.sleepycat.collections.DataCursor.init(DataCursor.java:170)

        at com.sleepycat.collections.DataCursor.<init>(DataCursor.java:58)

        at com.sleepycat.collections.StoredContainer.getValue(StoredContainer.java:300)

        at com.sleepycat.collections.StoredMap.get(StoredMap.java:240)

        at com.fxall.ems.berkleyDB.BerkleyDBInterface.getObjectFromCache(BerkleyDBInterface.java:353)

        at com.fxall.ems.persistance.BerkleyDBPersist.getChildOrder(BerkleyDBPersist.java:354)

        at com.fxall.ems.sor.meclient.MEClient.handleApiCancelOrderReport(MEClient.java:734)

        at com.fxall.ems.sor.meclient.MEOutHandler.onOrderCancelled(MEOutHandler.java:132)

 

 

 

  • 1. Re: com.sleepycat.je.EnvironmentFailureException: (JE 5.0.73) com.sleepycat.je.log.ChecksumException
    Andrei Costache, Oracle Journeyer
    Currently Being Moderated

    Hi Dipin,

     

    You mentioned:

    > We are seeing the following exceptions after it was advised that upgrading to Berkley DB  5.0.73 would fix the issues… (je-5.0.73.jar)

    Were you advised to do this here on the forum?  If so, can you point me to the thread containing the discussion?


    What was the JE version previously used?  How did you perform the upgrade exactly?


    Please try to run DbVerifyLog utility over the logs to determine whether we're faced with a transient or persistent checksum error.  It's probably persistent so DbVerifyLog should complain.

    If so, and if there's no sensitive data in the logs, you can send them to me by e-mail so that I can have look locally.

     

    Regards,

    Andrei

  • 2. Re: com.sleepycat.je.EnvironmentFailureException: (JE 5.0.73) com.sleepycat.je.log.ChecksumException
    1005812 Newbie
    Currently Being Moderated

    Andrei, if you search for the below post. you will find it

    BDB  Exceptions when saving large amounts of Data using je 5.0.58.jar

     

    1) The version previously used was je 5.0.58. jar and we upgraded the jar to 5.0.73 in prodn

    ok, i will email you the logs tomorrow..from the INtegratoin env

  • 3. Re: com.sleepycat.je.EnvironmentFailureException: (JE 5.0.73) com.sleepycat.je.log.ChecksumException
    1005812 Newbie
    Currently Being Moderated

    Hello Andrei, After trying to run the same test in INT, we seee a different exception ...

     

     

    Caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (

    JE 5.0.73) /opt/fxall/Aggie-1.10.103/TradingGateway_1.10.103/bin/../../../logs/TG_REGION fetchTarget of 0x0/0

    x3f327521 parent IN=289 IN class="com".sleepycat.je.tree.BIN lastFullVersion=0x1/0x317f1c61 lastLoggedVersion=0

    x1/0x317f1c61 parent.getDirty()=false state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. En

    vironment is invalid and must be closed.

            at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1429)

            at com.sleepycat.je.tree.BIN.fetchTarget(BIN.java:1251)

            at com.sleepycat.je.dbi.CursorImpl.fetchCurrent(CursorImpl.java:2261)

            at com.sleepycat.je.dbi.CursorImpl.getCurrentAlreadyLatched(CursorImpl.java:1466)

            at com.sleepycat.je.dbi.CursorImpl.getNext(CursorImpl.java:1595)

            at com.sleepycat.je.cleaner.UtilizationProfile.getObsoleteDetail(UtilizationProfile.java:632)

            at com.sleepycat.je.cleaner.FileProcessor.processFile(FileProcessor.java:439)

            at com.sleepycat.je.cleaner.FileProcessor.doClean(FileProcessor.java:289)

            at com.sleepycat.je.cleaner.FileProcessor.onWakeup(FileProcessor.java:151)

            at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:163)

            ... 1 more

    Caused by: java.io.FileNotFoundException: /opt/fxall/Aggie-1.10.103/TradingGateway_1.10.103/bin/../../../logs

    /TG_REGION/00000000.jdb (No such file or directory)

            at java.io.RandomAccessFile.open(Native Method)

            at java.io.RandomAccessFile.<init>(RandomAccessFile.java:216)

            at java.io.RandomAccessFile.<init>(RandomAccessFile.java:101)

            at com.sleepycat.je.log.FileManager$DefaultRandomAccessFile.<init>(FileManager.java:3209)

            at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:163)

            ... 1 more

    Caused by: java.io.FileNotFoundException: /opt/fxall/Aggie-1.10.103/TradingGateway_1.10.103/bin/../../../logs

    /TG_REGION/00000000.jdb (No such file or directory)

            at java.io.RandomAccessFile.open(Native Method)

            at java.io.RandomAccessFile.<init>(RandomAccessFile.java:216)

            at java.io.RandomAccessFile.<init>(RandomAccessFile.java:101)

            at com.sleepycat.je.log.FileManager$DefaultRandomAccessFile.<init>(FileManager.java:3209)

            at com.sleepycat.je.log.FileManager$6.createFile(FileManager.java:3237)

            at com.sleepycat.je.log.FileManager.openFileHandle(FileManager.java:1280)

            at com.sleepycat.je.log.FileManager.getFileHandle(FileManager.java:1152)

            at com.sleepycat.je.log.LogManager.getLogSource(LogManager.java:1141)

            at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:845)

            at com.sleepycat.je.log.LogManager.getLogEntryAllowInvisibleAtRecovery(LogManager.java:809)

            at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1412)

            at com.sleepycat.je.tree.BIN.fetchTarget(BIN.java:1251)

            at com.sleepycat.je.dbi.CursorImpl.fetchCurrent(CursorImpl.java:2261)

            at com.sleepycat.je.dbi.CursorImpl.getCurrentAlreadyLatched(CursorImpl.java:1466)

            at com.sleepycat.je.dbi.CursorImpl.getNext(CursorImpl.java:1593)

            at com.sleepycat.je.cleaner.UtilizationProfile.getObsoleteDetail(UtilizationProfile.java:632)

            at com.sleepycat.je.cleaner.FileProcessor.processFile(FileProcessor.java:439)

            at com.sleepycat.je.cleaner.FileProcessor.doClean(FileProcessor.java:289)

            at com.sleepycat.je.cleaner.FileProcessor.onWakeup(FileProcessor.java:148)

            ... 2 more

     

    It seems that Berkley DB is looking for a 00000000.jdb and that file doesnt seem to exist.. in the HOME ENV Directory

    ---------------------------------------------------------------------------------------------------------------------------------------------------------------------

     

    [09:56:21][fxall@nyatint03 ~/logs/TG_REGION] ls -ltra

    total 1045892

    drwxr-xr-x 85 fxall fxall     892928 Jun 13 18:01 ..

    -rw-rw-r--  1 fxall fxall 1069999349 Jun 13 19:11 00000001.jdb

    drwxr-xr-x  2 fxall fxall       4096 Jun 13 19:11 .

    -rw-rw-r--  1 fxall fxall      80633 Jun 13 19:11 00000002.jdb

     

     

     

    Please let us know how to proceed further..

  • 4. Re: com.sleepycat.je.EnvironmentFailureException: (JE 5.0.73) com.sleepycat.je.log.ChecksumException
    Andrei Costache, Oracle Journeyer
    Currently Being Moderated

    Hi Dipin,

     

    I see that you posted the same stack / issue in the other OTN thread, BDB Exceptions when saving large amounts of Data using je 5.0.58.jar

    So, in PRODN you got into LOG_CHECKSUM exceptions, whereas in INT you are now seeing LOG_FILE_NOT_FOUND.

     

    Can you clarify what does "running the same test in INT" means? Did you copy the log files from PRODN (they way they were after the checksum exceptions) over to INT and continued?  Or did you start from scratch in INT with an empty environment?

     

    Also, you mentioned you would send me the logs in e-mail.  Can you send both sets, from PRODN and INT?

     

    Thanks,

    Andrei

  • 5. Re: com.sleepycat.je.EnvironmentFailureException: (JE 5.0.73) com.sleepycat.je.log.ChecksumException
    1005812 Newbie
    Currently Being Moderated


    Could you send us a list of  params that we could disable to workaround the errors  at the EnvironmentConfig level ?

    such as

      envConfig.setConfigParam(EnvironmentConfig.LOG_VERIFY_CHECKSUMS,"false") etc

     

     

  • 6. Re: com.sleepycat.je.EnvironmentFailureException: (JE 5.0.73) com.sleepycat.je.log.ChecksumException
    greybird Expert
    Currently Being Moderated

    Since we don't know what the problem is yet, that's not possible.  If you could send us your .jdb files and the other information we've requested over email, we can try to determine what the problem is.

     

    --mark

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points