6 Replies Latest reply on Jun 14, 2013 8:56 PM by Greybird-Oracle

    com.sleepycat.je.EnvironmentFailureException: (JE 5.0.73) com.sleepycat.je.log.ChecksumException

    1005812

      We are seeing the following exceptions after it was advised that upgrading to Berkley DB  5.0.73 would fix the issues… (je-5.0.73.jar)

      The Platform we are running this is  Linux

       

      1)       Linux njqemsaux04 2.6.32-358.0.1.el6.x86_64 #1 SMP Wed Feb 20 11:05:23 EST 2013 x86_64 x86_64 x86_64 GNU/Linux

       

       

      We only set the following to 90 M

      EnvironmentConfig envConfig = new EnvironmentConfig();

                 

      2)                     envConfig.setConfigParam(EnvironmentConfig.LOG_FILE_MAX, "90000000"); //90M

       

       

       

      3)       These are the contents of  the ENV HOME DIR  ( where the DB files are stored)

       

      total 179672

      -rw-rw-r--   1 fxall fxall        0 Jun 10 17:21 je.info.0.lck

      -rw-rw-r--   1 fxall fxall        0 Jun 10 17:21 je.lck

      drwxrwxr-x 266 fxall fxall    49152 Jun 10 17:21 ..

      -rw-rw-r--   1 fxall fxall 89909152 Jun 11 02:10 00000000.jdb

      -rw-rw-r--   1 fxall fxall 89920384 Jun 11 02:16 00000001.jdb

      drwxr-xr-x   2 fxall fxall     4096 Jun 11 02:16 .

      -rw-rw-r--   1 fxall fxall  4076773 Jun 11 02:16 00000002.jdb

      -rw-rw-r--   1 fxall fxall     5563 Jun 11 02:16 je.info.0

       

       

       

      4)       These were the exceptions in PRODN

       

       

      Jun 11, 2013 2:16:58 AM com.sleepycat.je.utilint.DaemonThread run

      SEVERE: <DaemonThread name="Cleaner-1"/> caught exception, com.sleepycat.je.EnvironmentFailureException: (JE 5.0.73) /opt/fxall/logs/TG_REGION com.sleepycat.je.log.ChecksumException: Read invalid log entry type: 102 LOG_CHECKSUM: Checksum invalid on read, log is likely invalid. Environment is invalid and must be closed. fetchTarget of 0x0/0x9bc934 parent IN=5 IN class=com.sleepycat.je.tree.BIN lastFullVersion=0x1/0x430c18e lastLoggedVersion=0x2/0x55d46 parent.getDirty()=true state=0 Exiting

      1. com.sleepycat.je.EnvironmentFailureException: (JE 5.0.73) /opt/fxall/logs/TG_REGION com.sleepycat.je.log.ChecksumException: Read invalid log entry type: 102 LOG_CHECKSUM: Checksum invalid on read, log is likely invalid. Environment is invalid and must be closed. fetchTarget of 0x0/0x9bc934 parent IN=5 IN class=com.sleepycat.je.tree.BIN lastFullVersion=0x1/0x430c18e lastLoggedVersion=0x2/0x55d46 parent.getDirty()=true state=0

              at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:851)

              at com.sleepycat.je.log.LogManager.getLogEntryAllowInvisibleAtRecovery(LogManager.java:809)

              at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1412)

              at com.sleepycat.je.dbi.CursorImpl.getCurrentAlreadyLatched(CursorImpl.java:1466)

              at com.sleepycat.je.dbi.CursorImpl.getNext(CursorImpl.java:1595)

              at com.sleepycat.je.cleaner.UtilizationProfile.getObsoleteDetail(UtilizationProfile.java:632)

              at com.sleepycat.je.cleaner.FileProcessor.processFile(FileProcessor.java:439)

              at com.sleepycat.je.cleaner.FileProcessor.doClean(FileProcessor.java:289)

              at com.sleepycat.je.cleaner.FileProcessor.onWakeup(FileProcessor.java:151)

              at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:163)

              at java.lang.Thread.run(Thread.java:662)

      Caused by: com.sleepycat.je.log.ChecksumException: Read invalid log entry type: 102

              at com.sleepycat.je.log.LogEntryHeader.<init>(LogEntryHeader.java:161)

              at com.sleepycat.je.log.LogManager.getLogEntryFromLogSource(LogManager.java:928)

              at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:848)

              at com.sleepycat.je.log.LogManager.getLogEntryAllowInvisibleAtRecovery(LogManager.java:809)

              at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1416)

              ... 8 more

       

       

       

       

      1. com.sleepycat.je.EnvironmentFailureException: (JE 5.0.73) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 5.0.73) /opt/fxall/logs/TG_REGION com.sleepycat.je.log.ChecksumException: Read invalid log entry type: 102 LOG_CHECKSUM: Checksum invalid on read, log is likely invalid. Environment is invalid and must be closed. fetchTarget of 0x0/0x9bc934 parent IN=5 IN class=com.sleepycat.je.tree.BIN lastFullVersion=0x1/0x430c18e lastLoggedVersion=0x2/0x55d46 parent.getDirty()=true state=0

              at com.sleepycat.je.EnvironmentFailureException.wrapSelf(EnvironmentFailureException.java:196)

              at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1512)

              at com.sleepycat.je.Database.checkEnv(Database.java:1884)

              at com.sleepycat.je.Database.openCursor(Database.java:676)

              at com.sleepycat.collections.CurrentTransaction.openCursor(CurrentTransaction.java:415)

              at com.sleepycat.collections.MyRangeCursor.openCursor(MyRangeCursor.java:53)

              at com.sleepycat.collections.MyRangeCursor.<init>(MyRangeCursor.java:29)

              at com.sleepycat.collections.DataCursor.init(DataCursor.java:170)

              at com.sleepycat.collections.DataCursor.<init>(DataCursor.java:58)

              at com.sleepycat.collections.StoredContainer.getValue(StoredContainer.java:300)

              at com.sleepycat.collections.StoredMap.get(StoredMap.java:240)

              at com.fxall.ems.berkleyDB.BerkleyDBInterface.getObjectFromCache(BerkleyDBInterface.java:353)

              at com.fxall.ems.persistance.BerkleyDBPersist.getChildOrder(BerkleyDBPersist.java:354)

              at com.fxall.ems.sor.meclient.MEClient.handleApiCancelOrderReport(MEClient.java:734)

              at com.fxall.ems.sor.meclient.MEOutHandler.onOrderCancelled(MEOutHandler.java:132)

       

       

       

        • 1. Re: com.sleepycat.je.EnvironmentFailureException: (JE 5.0.73) com.sleepycat.je.log.ChecksumException
          &quot;Andrei Costache, Oracle-Oracle&quot;

          Hi Dipin,

           

          You mentioned:

          > We are seeing the following exceptions after it was advised that upgrading to Berkley DB  5.0.73 would fix the issues… (je-5.0.73.jar)

          Were you advised to do this here on the forum?  If so, can you point me to the thread containing the discussion?


          What was the JE version previously used?  How did you perform the upgrade exactly?


          Please try to run DbVerifyLog utility over the logs to determine whether we're faced with a transient or persistent checksum error.  It's probably persistent so DbVerifyLog should complain.

          If so, and if there's no sensitive data in the logs, you can send them to me by e-mail so that I can have look locally.

           

          Regards,

          Andrei

          • 2. Re: com.sleepycat.je.EnvironmentFailureException: (JE 5.0.73) com.sleepycat.je.log.ChecksumException
            1005812

            Andrei, if you search for the below post. you will find it

            BDB  Exceptions when saving large amounts of Data using je 5.0.58.jar

             

            1) The version previously used was je 5.0.58. jar and we upgraded the jar to 5.0.73 in prodn

            ok, i will email you the logs tomorrow..from the INtegratoin env

            • 3. Re: com.sleepycat.je.EnvironmentFailureException: (JE 5.0.73) com.sleepycat.je.log.ChecksumException
              1005812

              Hello Andrei, After trying to run the same test in INT, we seee a different exception ...

               

               

              Caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (

              JE 5.0.73) /opt/fxall/Aggie-1.10.103/TradingGateway_1.10.103/bin/../../../logs/TG_REGION fetchTarget of 0x0/0

              x3f327521 parent IN=289 IN class="com".sleepycat.je.tree.BIN lastFullVersion=0x1/0x317f1c61 lastLoggedVersion=0

              x1/0x317f1c61 parent.getDirty()=false state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. En

              vironment is invalid and must be closed.

                      at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1429)

                      at com.sleepycat.je.tree.BIN.fetchTarget(BIN.java:1251)

                      at com.sleepycat.je.dbi.CursorImpl.fetchCurrent(CursorImpl.java:2261)

                      at com.sleepycat.je.dbi.CursorImpl.getCurrentAlreadyLatched(CursorImpl.java:1466)

                      at com.sleepycat.je.dbi.CursorImpl.getNext(CursorImpl.java:1595)

                      at com.sleepycat.je.cleaner.UtilizationProfile.getObsoleteDetail(UtilizationProfile.java:632)

                      at com.sleepycat.je.cleaner.FileProcessor.processFile(FileProcessor.java:439)

                      at com.sleepycat.je.cleaner.FileProcessor.doClean(FileProcessor.java:289)

                      at com.sleepycat.je.cleaner.FileProcessor.onWakeup(FileProcessor.java:151)

                      at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:163)

                      ... 1 more

              Caused by: java.io.FileNotFoundException: /opt/fxall/Aggie-1.10.103/TradingGateway_1.10.103/bin/../../../logs

              /TG_REGION/00000000.jdb (No such file or directory)

                      at java.io.RandomAccessFile.open(Native Method)

                      at java.io.RandomAccessFile.<init>(RandomAccessFile.java:216)

                      at java.io.RandomAccessFile.<init>(RandomAccessFile.java:101)

                      at com.sleepycat.je.log.FileManager$DefaultRandomAccessFile.<init>(FileManager.java:3209)

                      at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:163)

                      ... 1 more

              Caused by: java.io.FileNotFoundException: /opt/fxall/Aggie-1.10.103/TradingGateway_1.10.103/bin/../../../logs

              /TG_REGION/00000000.jdb (No such file or directory)

                      at java.io.RandomAccessFile.open(Native Method)

                      at java.io.RandomAccessFile.<init>(RandomAccessFile.java:216)

                      at java.io.RandomAccessFile.<init>(RandomAccessFile.java:101)

                      at com.sleepycat.je.log.FileManager$DefaultRandomAccessFile.<init>(FileManager.java:3209)

                      at com.sleepycat.je.log.FileManager$6.createFile(FileManager.java:3237)

                      at com.sleepycat.je.log.FileManager.openFileHandle(FileManager.java:1280)

                      at com.sleepycat.je.log.FileManager.getFileHandle(FileManager.java:1152)

                      at com.sleepycat.je.log.LogManager.getLogSource(LogManager.java:1141)

                      at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:845)

                      at com.sleepycat.je.log.LogManager.getLogEntryAllowInvisibleAtRecovery(LogManager.java:809)

                      at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1412)

                      at com.sleepycat.je.tree.BIN.fetchTarget(BIN.java:1251)

                      at com.sleepycat.je.dbi.CursorImpl.fetchCurrent(CursorImpl.java:2261)

                      at com.sleepycat.je.dbi.CursorImpl.getCurrentAlreadyLatched(CursorImpl.java:1466)

                      at com.sleepycat.je.dbi.CursorImpl.getNext(CursorImpl.java:1593)

                      at com.sleepycat.je.cleaner.UtilizationProfile.getObsoleteDetail(UtilizationProfile.java:632)

                      at com.sleepycat.je.cleaner.FileProcessor.processFile(FileProcessor.java:439)

                      at com.sleepycat.je.cleaner.FileProcessor.doClean(FileProcessor.java:289)

                      at com.sleepycat.je.cleaner.FileProcessor.onWakeup(FileProcessor.java:148)

                      ... 2 more

               

              It seems that Berkley DB is looking for a 00000000.jdb and that file doesnt seem to exist.. in the HOME ENV Directory

              ---------------------------------------------------------------------------------------------------------------------------------------------------------------------

               

              [09:56:21][fxall@nyatint03 ~/logs/TG_REGION] ls -ltra

              total 1045892

              drwxr-xr-x 85 fxall fxall     892928 Jun 13 18:01 ..

              -rw-rw-r--  1 fxall fxall 1069999349 Jun 13 19:11 00000001.jdb

              drwxr-xr-x  2 fxall fxall       4096 Jun 13 19:11 .

              -rw-rw-r--  1 fxall fxall      80633 Jun 13 19:11 00000002.jdb

               

               

               

              Please let us know how to proceed further..

              • 4. Re: com.sleepycat.je.EnvironmentFailureException: (JE 5.0.73) com.sleepycat.je.log.ChecksumException
                &quot;Andrei Costache, Oracle-Oracle&quot;

                Hi Dipin,

                 

                I see that you posted the same stack / issue in the other OTN thread, BDB Exceptions when saving large amounts of Data using je 5.0.58.jar

                So, in PRODN you got into LOG_CHECKSUM exceptions, whereas in INT you are now seeing LOG_FILE_NOT_FOUND.

                 

                Can you clarify what does "running the same test in INT" means? Did you copy the log files from PRODN (they way they were after the checksum exceptions) over to INT and continued?  Or did you start from scratch in INT with an empty environment?

                 

                Also, you mentioned you would send me the logs in e-mail.  Can you send both sets, from PRODN and INT?

                 

                Thanks,

                Andrei

                • 5. Re: com.sleepycat.je.EnvironmentFailureException: (JE 5.0.73) com.sleepycat.je.log.ChecksumException
                  1005812


                  Could you send us a list of  params that we could disable to workaround the errors  at the EnvironmentConfig level ?

                  such as

                    envConfig.setConfigParam(EnvironmentConfig.LOG_VERIFY_CHECKSUMS,"false") etc

                   

                   

                  • 6. Re: com.sleepycat.je.EnvironmentFailureException: (JE 5.0.73) com.sleepycat.je.log.ChecksumException
                    Greybird-Oracle

                    Since we don't know what the problem is yet, that's not possible.  If you could send us your .jdb files and the other information we've requested over email, we can try to determine what the problem is.

                     

                    --mark