I am new to Timesten database.
I gone through the articles of Timesten and i feel that it would meet our conditions.
At present we are reading the a huge file from memory,so we are looking for alternate solution by keeping the file in database and read the same,all these things should happen with fraction of second not even millisecond micro second.so i think Timesten database will help us.
But the thing is what will happen when there is a power off/crash how it will react.
How quick the data's will be written to disk.
I am having much more questions based on the current conditions where we are working.
I need to know whether Timesten database will help us in solving the problem
Although Timesten is an in-memory database it provides full persistence, durability and recoverability for data stored within it by means of a combination of checkpointing and transaction logging. TimesTen provides all the ACID properties you would expect from any database. Being a database, changes to data become persistent when the transaction that makes those changes is committed by the application.
TimesTen supports two kinds of commit; non-durable (perhaps more accurately called 'delayed durability') and durable (or 'immediate durability'). The type of commit used is controlled by a connection parameter called DurableCommits which defaults to 0 (delayed durability0 but which can be changed to 1 (immediate durability) as required. Applications can also force a specific transaction to be committed durably by calling the built in procedure ttDurableCommit within the transaction prior to committing it.
With delayed durability, a commit initially ocurs only in memory and the log records for the change are posted to the in memory log buffer. A background flusher thread will then write batches of changes to the log on disk. Once the log records have reached disk the changes made by the transaction become fully persistent and recoverable. In a typical configuration the delay involved between commit and full persistence is normally of the order of milliseconds but it can be more and it is variable; no specific time can be guaranteed. The advantage of non-durable commit is that it delivers very high write performance; tens or hundreds of thousands of TPS on commodity hardware. The disadvantage is that there is a (small) window of vulnerability to data loss if there is some kind of system failure.
If an application needs absolute assurance that a particular transaction's changes are fully durable then it can use durable commit. With a durable commit, the commit operation forces an immediate, synchronous flush of all not yet flushed records from the transction log buffer to disk. Only when the O/S acknowledges that the records have been fully written to the disk storage media does the commit return success. The advantage of durable commit is that it provides 100% assurance (as long as the disk storage medium does not fail - but you should use RAID to guard against that). The disadvantage is that write performance is limited by the performance of the underlying storage and so typical TPS rates are much lower.
As I mentioned, TimesTen allows applications to selectively choose durable commit on a per transaction basis and this feature can often be used to balance the conflicting demands of performance and durability. It is also worth mentioning that TimesTen supports very high performance data replication and offers a fully synchronous mode (RETURN TWOSAFE). This provides 100% assurance by using dual memory commit across two machines and in most configurations provides significantly better performance than durable commit to disk.
The choice of commit mode and/or replciation mode has no impact on query operations; they always execute at full in-memory speed.
I hope that helps. if you have further questions please post them here and we will do our best to answer them.
Hey Chris thanks for your immediate response on this!
I have few more questions.The project which i am working is new to me at present the data load is happened from memory and read/writes are also happen in memory only.In order to have much detailed information we would like to go with the database,i have googled and found Timesten database,I am not even aware of it how it is ,how it will work and so on... Only aim for us is what ever we are writing in memory we need to write to database with same speed at present it reacts,so our aim is not to have any performance degrade when we choose Timesten database will it be a right choice for us or whether any other options are there?
As i said early our records are less micro sec level thousands of records will be written at a stretch to memory and process takes place from memory so when we implement database it should not effect our process. Since it is public forum i cant able to disclose my business needs here.
Our ultimate aim is to have write datas to database instead of writing to memory
Whether Timesten database is the only choice or we can go for some other database as well.
As I don't know the exact deytails of your current solution I cannot make exact performance comparisons. If you are simply copying data to/from memory in a single threaded process then that will be incredibly fast and no in-memory database will be able to match it for speed. An IMDB like TimesTen has a lot of functionality (table and index based access mechanism, SQL parser and execution, concurrency control, API layer, persitence mechanism) which all add 'cost' to both queries and write operations.
To put things into perspective, on modern Intel hardware we see key based query operations (look up and retrieve a single row from a table based on its primary key value) execute in around 2-3 microseconds and we see an update transaction (identify a row based on its key value, update the row and then commit) take around 10 microseconds. This is with delayed durability (DurableCommits=0). With immediate durability you will need to add in several milliseconds for the synchronous commit to disk (this is a hardware effect and the exact time depends on your disk hardware). Synchronous commit to another TimesTen accros the network may be as low as a couple of milliseconds. It is extremely unlikely that you will find any other in-memory relational database that provides persistence that can do any better than this and many are significantly slower. As TimesTen scales well across multiple CPu cores we see very high throughput on a multi-core machine with a multi-threaded application. Depending on hardware we can see several million TPS.
Maybe this will help you asses if Timesten may be a fit for your requirements. If you would prefer a private e-mail conversation then if you can provide an e-mai laddress I will contact you directly.