JE uses it's data log (the .jdb files) to log transactions, which is a technique similar to other databases. Records in a transaction are logged prior to commit, and if the transaction aborts they are undone. This is the standard approach.
If you create a very large transaction, however, you will risk running out of memory due to locks. Locks are held for each record in JE. Large transactions should be avoided with any database for many reasons, but in particular with databases that perform per-record locking like JE.