What's wrong?

coolnight · Jul 26, 2006 11:16PM

I want to write a small program to test how fast je is.
but the result is disappointing.
it's very slow.If I enable database transaction, it'll use 20s to write 1000records.
if I don't enable database transaction, it'll be much faster,but not stable.
maybe something is wrong,here is the code, it needs jdk5.0!
the first copy is for reading, the second copy is used to copy and paste into java ide to run it!
Thanks!

package&nbsp;tests;


import&nbsp;java.io.File;


import&nbsp;com.sleepycat.bind.EntryBinding;

import&nbsp;com.sleepycat.bind.tuple.TupleBinding;

import&nbsp;com.sleepycat.je.Database;

import&nbsp;com.sleepycat.je.DatabaseConfig;

import&nbsp;com.sleepycat.je.DatabaseEntry;

import&nbsp;com.sleepycat.je.DatabaseException;

import&nbsp;com.sleepycat.je.Environment;

import&nbsp;com.sleepycat.je.EnvironmentConfig;

import&nbsp;com.sleepycat.je.EnvironmentStats;

import&nbsp;com.sleepycat.je.StatsConfig;


public&nbsp;class&nbsp;BdbTest&nbsp;{


&nbsp;&nbsp;&nbsp;&nbsp;private&nbsp;Environment&nbsp;myEnv;

&nbsp;&nbsp;&nbsp;&nbsp;private&nbsp;Database&nbsp;&nbsp;&nbsp;&nbsp;simpleDb;


&nbsp;&nbsp;&nbsp;&nbsp;private&nbsp;int&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;num&nbsp;=&nbsp;10000;


&nbsp;&nbsp;&nbsp;&nbsp;/**

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;*&nbsp;@param&amp;nbsp;args

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;*/

&nbsp;&nbsp;&nbsp;&nbsp;public&nbsp;static&nbsp;void&nbsp;main(String[]&nbsp;args)&nbsp;throws&nbsp;DatabaseException&nbsp;{

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;int&nbsp;num&nbsp;=&nbsp;10000;

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if&nbsp;(args.length&nbsp;&gt;&nbsp;0)&nbsp;num&nbsp;=&nbsp;Integer.parseInt(args[0]);

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;BdbTest&nbsp;bm&nbsp;=&nbsp;new&nbsp;BdbTest();

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;bm.num&nbsp;=&nbsp;num;

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;bm.init();

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;bm.write();

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;bm.close();

&nbsp;&nbsp;&nbsp;&nbsp;}


&nbsp;&nbsp;&nbsp;&nbsp;public&nbsp;void&nbsp;write()&nbsp;throws&nbsp;DatabaseException&nbsp;{

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;EntryBinding&nbsp;myBinding&nbsp;=&nbsp;TupleBinding.getPrimitiveBinding(Long.class);

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;DatabaseEntry&nbsp;theKey&nbsp;=&nbsp;new&nbsp;DatabaseEntry();

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;DatabaseEntry&nbsp;theData&nbsp;=&nbsp;new&nbsp;DatabaseEntry();


&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;final&nbsp;int&nbsp;cap&nbsp;=&nbsp;1000;

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;//long&nbsp;total_counts&nbsp;=&nbsp;0;

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;long&nbsp;start&nbsp;=&nbsp;System.nanoTime();

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;long&nbsp;lastEnd&nbsp;=&nbsp;start;


&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;for&nbsp;(long&nbsp;i&nbsp;=&nbsp;0;&nbsp;i&nbsp;&lt;&nbsp;num;&nbsp;i++)&nbsp;{

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;//

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Long&nbsp;l&nbsp;=&nbsp;Long.valueOf(i);

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;//&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;theKey&nbsp;=&nbsp;new&nbsp;DatabaseEntry();

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;//&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;theData&nbsp;=&nbsp;new&nbsp;DatabaseEntry();


&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;myBinding.objectToEntry(l,&nbsp;theKey);

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;myBinding.objectToEntry(l,&nbsp;theData);


&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;simpleDb.put(null,&nbsp;theKey,&nbsp;theData);


&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if&nbsp;(i&nbsp;%&nbsp;cap&nbsp;==&nbsp;0)&nbsp;{

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;long&nbsp;end1&nbsp;=&nbsp;System.nanoTime();

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;System.out.println(String.format("it&nbsp;takes&nbsp;[%d]&nbsp;us&nbsp;to&nbsp;put&nbsp;[%d]&nbsp;records!",&nbsp;(end1&nbsp;-&nbsp;lastEnd)&nbsp;/&nbsp;1000,&nbsp;cap));

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;lastEnd&nbsp;=&nbsp;end1;

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;long&nbsp;end&nbsp;=&nbsp;System.nanoTime();

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;System.out.println(String.format("it&nbsp;takes&nbsp;[%d]&nbsp;us&nbsp;to&nbsp;write&nbsp;[%d]&nbsp;records!",&nbsp;(end&nbsp;-&nbsp;start)&nbsp;/&nbsp;1000,&nbsp;num));

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;StatsConfig&nbsp;config&nbsp;=&nbsp;new&nbsp;StatsConfig();

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;config.setClear(true);

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;config.setFast(true);

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;config.setShowProgressInterval(1000);

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;EnvironmentStats&nbsp;stats&nbsp;=&nbsp;myEnv.getStats(config);


&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;System.out.println("stats="&nbsp;+&nbsp;stats);

&nbsp;&nbsp;&nbsp;&nbsp;}


&nbsp;&nbsp;&nbsp;&nbsp;public&nbsp;void&nbsp;init()&nbsp;throws&nbsp;DatabaseException&nbsp;{

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;File&nbsp;envHome&nbsp;=&nbsp;new&nbsp;File(".");

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;boolean&nbsp;readOnly&nbsp;=&nbsp;false;

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;EnvironmentConfig&nbsp;myEnvConfig&nbsp;=&nbsp;new&nbsp;EnvironmentConfig();

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;DatabaseConfig&nbsp;myDbConfig&nbsp;=&nbsp;new&nbsp;DatabaseConfig();


&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;//&nbsp;If&nbsp;the&nbsp;environment&nbsp;is&nbsp;read-only,&nbsp;then

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;//&nbsp;make&nbsp;the&nbsp;databases&nbsp;read-only&nbsp;too.

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;myEnvConfig.setReadOnly(readOnly);

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;myDbConfig.setReadOnly(readOnly);


&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;//&nbsp;If&nbsp;the&nbsp;environment&nbsp;is&nbsp;opened&nbsp;for&nbsp;write,&nbsp;then&nbsp;we&nbsp;want&nbsp;to&nbsp;be&nbsp;

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;//&nbsp;able&nbsp;to&nbsp;create&nbsp;the&nbsp;environment&nbsp;and&nbsp;databases&nbsp;if&nbsp;

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;//&nbsp;they&nbsp;do&nbsp;not&nbsp;exist.

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;myEnvConfig.setAllowCreate(!readOnly);

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;myDbConfig.setAllowCreate(!readOnly);


&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;//&nbsp;Allow&nbsp;transactions&nbsp;if&nbsp;we&nbsp;are&nbsp;writing&nbsp;to&nbsp;the&nbsp;database

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;myEnvConfig.setTransactional(true);

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;myDbConfig.setTransactional(false);


&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;//&nbsp;Open&nbsp;the&nbsp;environment

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;myEnv&nbsp;=&nbsp;new&nbsp;Environment(envHome,&nbsp;myEnvConfig);


&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;System.out.println("cachesize="&nbsp;+&nbsp;myEnv.getConfig().getCacheSize());

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;System.out.println("config="&nbsp;+&nbsp;myEnv.getConfig());

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;System.out.println("cache&nbsp;percent="&nbsp;+&nbsp;myEnv.getConfig().getCachePercent());

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;//&nbsp;Now&nbsp;open,&nbsp;or&nbsp;create&nbsp;and&nbsp;open,&nbsp;our&nbsp;databases

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;//&nbsp;Open&nbsp;the&nbsp;vendors&nbsp;and&nbsp;inventory&nbsp;databases

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;simpleDb&nbsp;=&nbsp;myEnv.openDatabase(null,&nbsp;"simpleDB",&nbsp;myDbConfig);

&nbsp;&nbsp;&nbsp;&nbsp;}


&nbsp;&nbsp;&nbsp;&nbsp;public&nbsp;void&nbsp;close()&nbsp;{

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if&nbsp;(myEnv&nbsp;!=&nbsp;null)&nbsp;{

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;try&nbsp;{

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;//Close&nbsp;the&nbsp;secondary&nbsp;before&nbsp;closing&nbsp;the&nbsp;primaries

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;simpleDb.close();

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;//&nbsp;Finally,&nbsp;close&nbsp;the&nbsp;environment.

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;myEnv.close();

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}&nbsp;catch&nbsp;(DatabaseException&nbsp;dbe)&nbsp;{

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;System.err.println("Error&nbsp;closing&nbsp;MyDbEnv:&nbsp;"&nbsp;+&nbsp;dbe.toString());

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;System.exit(-1);

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}

&nbsp;&nbsp;&nbsp;&nbsp;}


}

package tests;
import java.io.File;
import com.sleepycat.bind.EntryBinding;
import com.sleepycat.bind.tuple.TupleBinding;
import com.sleepycat.je.Database;
import com.sleepycat.je.DatabaseConfig;
import com.sleepycat.je.DatabaseEntry;
import com.sleepycat.je.DatabaseException;
import com.sleepycat.je.Environment;
import com.sleepycat.je.EnvironmentConfig;
import com.sleepycat.je.EnvironmentStats;
import com.sleepycat.je.StatsConfig;
public class BdbTest {
private Environment myEnv;
private Database simpleDb;
private int num = 10000;
/**
* @param args
*/
public static void main(String[] args) throws DatabaseException {
int num = 10000;
if (args.length > 0) num = Integer.parseInt(args[0]);
BdbTest bm = new BdbTest();
bm.num = num;
bm.init();
bm.write();
bm.close();
}
public void write() throws DatabaseException {
EntryBinding myBinding = TupleBinding.getPrimitiveBinding(Long.class);
DatabaseEntry theKey = new DatabaseEntry();
DatabaseEntry theData = new DatabaseEntry();
final int cap = 1000;
//long total_counts = 0;
long start = System.nanoTime();
long lastEnd = start;
for (long i = 0; i < num; i++) {
//
Long l = Long.valueOf(i);
// theKey = new DatabaseEntry();
// theData = new DatabaseEntry();
myBinding.objectToEntry(l, theKey);
myBinding.objectToEntry(l, theData);
simpleDb.put(null, theKey, theData);
if (i % cap == 0) {
long end1 = System.nanoTime();
System.out.println(String.format("it takes [%d] us to put [%d] records!", (end1 - lastEnd) / 1000, cap));
lastEnd = end1;
}
}
long end = System.nanoTime();
System.out.println(String.format("it takes [%d] us to write [%d] records!", (end - start) / 1000, num));
StatsConfig config = new StatsConfig();
config.setClear(true);
config.setFast(true);
config.setShowProgressInterval(1000);
EnvironmentStats stats = myEnv.getStats(config);
System.out.println("stats=" + stats);
}
public void init() throws DatabaseException {
File envHome = new File(".");
boolean readOnly = false;
EnvironmentConfig myEnvConfig = new EnvironmentConfig();
DatabaseConfig myDbConfig = new DatabaseConfig();
// If the environment is read-only, then
// make the databases read-only too.
myEnvConfig.setReadOnly(readOnly);
myDbConfig.setReadOnly(readOnly);
// If the environment is opened for write, then we want to be
// able to create the environment and databases if
// they do not exist.
myEnvConfig.setAllowCreate(!readOnly);
myDbConfig.setAllowCreate(!readOnly);
// Allow transactions if we are writing to the database
myEnvConfig.setTransactional(true);
myDbConfig.setTransactional(false);
// Open the environment
myEnv = new Environment(envHome, myEnvConfig);
System.out.println("cachesize=" + myEnv.getConfig().getCacheSize());
System.out.println("config=" + myEnv.getConfig());
System.out.println("cache percent=" + myEnv.getConfig().getCachePercent());
// Now open, or create and open, our databases
// Open the vendors and inventory databases
simpleDb = myEnv.openDatabase(null, "simpleDB", myDbConfig);
}
public void close() {
if (myEnv != null) {
try {
//Close the secondary before closing the primaries
simpleDb.close();
// Finally, close the environment.
myEnv.close();
} catch (DatabaseException dbe) {
System.err.println("Error closing MyDbEnv: " + dbe.toString());
System.exit(-1);
}
}
}
}

coolnight · Jul 26, 2006 11:24PM

my computer is Celeron(R) 2.66G/1G ram

512799 · Jul 27, 2006 10:50AM

Hi,

I have reviewed your test program and first have a question:
What is it you want to accomplish? Are you trying to look at loading a large batch of records at once? If so why do you need transactions at all?

Assuming you do want transactions you have some other parameters that can help this run faster depending on your requirements for durability. If you run with transactions and take the default settings (as I believe you have) you will end up flushing the log buffer for each put to achieve the highest level of durability. There are other sync options that you can consider. This is described in more detail in the following FAQ entry:

http://dev.sleepycat.com/resources/faq_show.html?id=39

I would recommend trying the test with the other 2 settings, (commitWriteNoSync and commitNoSync).

If you are running with CommitSync , each put (with auto commit) will result in the log buffer being flushed all the way to disk. Your elapsed time will be very much dependent on the speed of the I/O to disk on your Celeron. If I/O to disk on your system is slow, then this will be noticed since you are flushing to disk for each record.

Try the different settings and see what type of numbers you get.
What is your cache size? Another reference you may find useful can be found here:

http://www.sleepycat.com/jedocs/GettingStartedGuide/cachesize.html

I hope this helps, let us know if you have further questions.

Regards,

Ron Cohen
Berkeley DB Java Edition
Oracle Corporation

512799 · Jul 27, 2006 12:04PM

Hi again,

I have one more suggestion:

Depending on your requirements you may want to consider a
Deferred Write Database:

http://www.sleepycat.com/jedocs/GettingStartedGuide/DB.html#dwdatabase

Ron

Forum Stats

Quick Links

Categories

What's wrong?

Comments

Data

Databases

Development

Development Tools

Open Source

Programs

Forum Stats

Categories

What's wrong?

Comments