opening database from other OS on Intel Mac

529931

I can open a database created on Windows on my PPC Mac.
I can open a database created on a PPC Mac on Windows or Intel Linux
But I don't seem to be able to open a database created on a PPC Mac on an Intel Mac.
Is there a known bug about this ?
Am I missing something ?
I've taken care of all endianness issues I'm aware of and it works between PPC OS X and Windows or Intel Linux.

Thanks.

Andi..

529931

I forgot to say what the actual error was:
(-30974, 'DB_RUNRECOVERY: Fatal error, run database recovery')

524300

Hi Andi,

The DB_RUNRECOVERY error usually indicates a problem with the environment (roughly, the __db.* files), rather than with a database. You can't copy those files from one architecture to another, only the databases (usually *.db).

Would this explain what you are seeing?

If not, can you please enable error output (with DB_ENV->set_errfile), and let us know what you see before the error is returned?

Regards,
Michael.

529931

Yes, I know the env files can't come across. The error I first reported was bogus, I had left the PPC log file in dbHome. Since the endianness is different it can't come across. Still, I can't open the database files, I get this error: (I open with recovery, the env opens fine, the databases don't open):
DBInvalidArgError: (22, 'Invalid argument')

and in the errfile I set, I get this:

file unknown (meta pgno = 0) has LSN [9][6163696].
end of log is [1][28]
./__repository__/__items.db: unexpected file type or format

Regards,

Andi..

524300

Hi Andi,

If you take database files from one transactional environment to another, you will need to call DB_ENV->lsn_reset on each database file in the new environment before opening. Without this check, the database would be unrecoverable in the new environment, because the log sequence numbers (LSNs) on pages would be inconsistent with the log files.

Alternatively, just do a dump in the old environment and load the dump into the new environment.

Regards,
Michael.

529931

Hi Michael,

For clarification: I'm not moving database files from one transactional env to another. I'm doing a backup of database files and log files (no env files) and taking that to another platform where a new environment is going to be created during env->open() with DB_RECOVER_FATAL and DB_CREATE. When the endianness of the target platform is different I don't copy the log files to the target platform because the docs say that the log files are endianness dependent.

Does this operation count as 'moving a database from one transactional env to another' ?
I understand it as a 'env gets trashed and recovered' operation with a platform change in between.

I didn't know about env->lsn_reset() and never used it before. For a long time now, I've been able to copy database backups from OS to OS such as Windows to PPC Mac OS X, PPC Mac OS X to Intel Linux or vice-versa. It's only failing, consistently, when copying to Intel Mac OS X.

If I understand the env->lsn_reset() docs correctly, this function needs to be called at the time of backup, before the copy is made. I'm going to try this out in the meantime.

Thanks for the info.

Andi..

529931

Ok, I made some progress but it still doesn't work. Here is what I learned:
- Restoring my backups works as long as the original log file is there. I can only reuse the log file
if the endianness of both platforms are the same and I verified that this worked indeed.
- When I don't copy the log file, I get the error reported earlier. The problem I reported earlier is not specific to Intel Mac OS X but specific to restoring database files without their log file.

I changed the code to call env->lsn_reset() on the db files as they are copied during a backup. Here are the steps that I take to make a backup:
- create a destination dir
- run env->txn_checkpoint() with DB_FORCE in the source dir
- run env->log_archive(DB_ARCH_REMOVE) at which point I'm left with one log file only
- each dbFile is copied by:
+ copy dbFile to dbFile.lsn
+ env->lsn_reset(dbFile.lsn)
+ move dbFile.lsn to destDir/dbFile

I'm now unable to even open these backed up database files later locally, on the same machine.
After restoring the files, the env is re-created with DB_RECOVER_FATAL | DB_CREATE, and that seems to work, but the db->open() statements on the restored db files fail.

The errfile says this:
file unknown (meta pgno = 0) has LSN [3][2067729].
end of log is [1][28]
./__repository__/__items.db: unexpected file type or format

I did verify that lsn_reset() did something on __items.db, the src file and dst files are different.

What am I missing now ?

Thanks

Andi..

529931

I'm kind of stuck with this problem. Has someone had a chance to look at my previous reply ?
Thanks !

Andi..

524300

Hi Andi,

That error suggests that the call to lsn_reset has not worked properly, or the database has been modified after the call to lsn_reset. The effect of that call is to set the LSN on all pages to [0][1], but your database still has LSN [3][2067729]

Can you confirm that lsn_reset returned zero, and that the database was not opened at the time in another thread, or modified after the call to lsn_reset? Was the original database file open at the time or afterwards (before the copy to the new environment)?

There are caveats around taking a copy of a database file within an environment as well -- Berkeley DB maintains a file ID, which needs to be reset (with DB_ENV->fileid_reset) if you make a copy of a file within an environment.

Rather than copying, resetting ID or LSNs, then copying again into the new directory, is there any reason why you can't just dump from the old environment (with db_dump), and load into the new one (with db_load)?

Regards,
Michael.

529931

Hi Michael,

lsn_reset() did return 0, no error was reported.
I added a call to fileid_reset() and tried it both before the call to lsn_reset() and after and it made no difference. The db.err file still reports the same error:
file unknown (meta pgno = 0) has LSN [3][1963241].
end of log is [1][28]
./__repository__/__items.db: unexpected file type or format

Here are the steps executed and the state of database files when the copying occurs:
- __foo.db is copied to __foo.db.lsn (while __foo.db is open) (I was hoping to be able to do a database backup without having to close the databases, and this works great when the log file is included)
- lsn_reset() and fileid_reset() are called on __foo.db.lsn (__foo.db.lsn is closed, it was never opened)
- __foo.db.lsn is moved to __foo.db in another directory (the backup dir) that has no env files
- later, during restore, these files are copied into a fresh, empty dbHome dir
- an env is created with DB_RECOVER_FATAL|DB_CREATE
- the database file copies are opened, and this fails currently

You asked why I wasn't using dump and load. Well, the docs advertise that copying database files around can work provided the platform endianness differences are taken care of. I'd think that copying database files around would be vastly faster than dump and load but I haven't tried it yet.

Regards

Andi..

529931

Hi (again) Michael,

I took a look at the db_hotbackup sources and it seems to be doing pretty much what I'm trying to do myself. It copies the log file as well. I can restore from a db_hotbackup directory just as well as when I do a backup myself with the log file.
The problem this thread is all about is when I don't copy the log file because the target platform is going to be of a different endianness. db_hotbackup doesn't seem to support that at all since it always copies the log file.

Can I do a hot backup without backing up the log file ? your answers about lsn_reset() and fileid_reset() seem to imply that (as well as the docs about db file portability) but there's still a bug or a misunderstanding somewhere.

I also took a look at db_dump and db_load. There don't seem to be simple dump and load APIs that are documented. I'm a trying not to have to depend on the utilities. Yes, I could copy the db_dump and db_load sources into my app but that feels a little fishy.

Thanks !

Andi..