- 197.1K All Categories
- 2.5K Data
- 546 Big Data Appliance
- 1.9K Data Science
- 450.7K Databases
- 221.9K General Database Discussions
- 31 Multilingual Engine
- 552 MySQL Community Space
- 479 NoSQL Database
- 7.9K Oracle Database Express Edition (XE)
- 3.1K ORDS, SODA & JSON in the Database
- 555 SQLcl
- 4K SQL Developer Data Modeler
- 187.2K SQL & PL/SQL
- 21.4K SQL Developer
- 296.3K Development
- 17 Developer Projects
- 139 Programming Languages
- 293K Development Tools
- 110 DevOps
- 3.1K QA/Testing
- 646.1K Java
- 28 Java Learning Subscription
- 37K Database Connectivity
- 158 Java Community Process
- 105 Java 25
- 22.1K Java APIs
- 138.2K Java Development Tools
- 165.3K Java EE (Java Enterprise Edition)
- 19 Java Essentials
- 162 Java 8 Questions
- 86K Java Programming
- 81 Java Puzzle Ball
- 65.1K New To Java
- 1.7K Training / Learning / Certification
- 13.8K Java HotSpot Virtual Machine
- 94.3K Java SE
- 13.8K Java Security
- 205 Java User Groups
- 468 LiveLabs
- 39 Workshops
- 10.2K Software
- 6.7K Berkeley DB Family
- 3.5K JHeadstart
- 5.7K Other Languages
- 2.3K Chinese
- 175 Deutsche Oracle Community
- 1.1K Español
- 1.9K Japanese
- 233 Portuguese
Over 4GB read-only mmap B-tree fails to find records
Systems: Windows 10, RHLE, HP-UX, always a 64-bit compile.
BDB: 5.x, 6.x, 18.x
This problem exists on every system I have tried, with every version of BDB I can get my hands on.
- Create a b-tree with a size just over 4GB. I use a 64-bit unsigned key and a 24-byte record, which takes about 67,200,300 records to get over 4GB (4,341,264,384). The same key/record setup with 66,200,300 records is just under 4GB and does NOT exhibit the problem.
- After generation, reopen the database "read-only" (DB_RDONLY) with an environment, cache, and mmap setting to allow BDB to mmap the file. I use a 5GB cache and set the mmap limit to 16GB.
- Start looking up the records. Very quickly db->get will fail with BDB0073 DB_NOTFOUND. The lib also produces a BDB3008 message: "dirty flag set for readonly file page". If you keep calling db->get, then the lib will panic.
- Reopen the database, but this time pass the DB_NOMMAP to the db->open call. Notice how all the keys are found.
- Generate a database with 66,200,300 records of the same type as above. Notice that the file is just under 4GB (4,247,216,128).
- Reopen the under 4GB database the same as step #2, i.e. read-only, cache and mmap set to allow BDB to mmap the file. Notice how all the keys are found.
- Use db_stat to see there is nothing wrong found with either db file.
- Use db_dump to view the first 100 or so records and notice that the key that failed to be found above, is indeed in the database file.
Using a different OS, compiler, or system has not made any difference with this problem, neither has 4096 vs 8192 size pages, and making sure the page size matches the filesystem.
Something else that is strange. If you generate the database, sync it, close it, then reopen it read-only, all in the same run of a program within the same db environment, then the database over 4GB checks out fine, i.e. all the keys are found. However, exit the program and reopen the database read-only for checking, and the same program / code-path fails as above.
Any insight, questions, or suggestion would be greatly appreciated.