This discussion is archived
10 Replies Latest reply: Jun 11, 2012 2:32 PM by Linda Lee RSS

On enabling HA on existing store the Replica node store arrives corrupted

940250 Newbie
Currently Being Moderated
Using je-5.0.48 we enabled replication on existing non-ha store created with je-4.1.17.

java -cp je-5.0.48.jar com.sleepycat.je.rep.util.DbEnableReplication -h ha/ha-work/Node-5001/bdbstore -groupName TestClusterGroup -nodeName Node-5001 -nodeHostPort <removed hostname>:5001

Then we started the 2-node cluster with je-5.0.48.

First node was started with the migrated store and became the Master.
Our application has an application specific upgrade steps to do which includes:
tables renaming
data moving

So, we changed store at this point.

Then we started a second node with an empty store directory.
The InsufficientLogException was thrown on startup and NetworkRestore was used to copy the Master logs into Replica.
The logs were copied successfully but one of the databases in the store appeared corrupted on Replica.

DbDump utility with list option(-l) showed the database (MESSAGE_METADATA) in the list of available databases but running it with -s option for the corrupted database generated DatabaseNotFoundException


$ java -cp je-5.0.48.jar com.sleepycat.je.util.DbDump -h ha-work/Node-5002/bdbstore -l
BRIDGES
CONFIGURED_OBJECTS
DB_VERSION
LINKS
MESSAGE_CONTENT
MESSAGE_METADATA
QUEUE_ENTRIES
XIDS
$ java -cp je-5.0.48.jar com.sleepycat.je.util.DbDump -h ha-work/Node-5002/bdbstore -s MESSAGE_METADATA
com.sleepycat.je.DatabaseNotFoundException: (JE 5.0.48) Database MESSAGE_METADATA not found.
at com.sleepycat.je.Environment.setupDatabase(Environment.java:797)
at com.sleepycat.je.Environment.openDatabase(Environment.java:607)
at com.sleepycat.je.util.DbDump.dump(DbDump.java:356)
at com.sleepycat.je.util.DbDump.main(DbDump.java:204)

Repeating the DbDump command on the first node (Master) did not show the problem: database content was displayed successfully

We restarted the cluster by starting the nodes in the same order: First node , Second node. On re-start, surprisingly, the second Node became the Master and the first node became the Replica

On trying to insert data into a second node (Master store) the NullPointerException was thrown on a first Node(Replica)

120529 11:27:48:287 SEVERE [Node-5001] Replica unexpected exception java.lang.NullPointerException java.lang.NullPointerException
at com.sleepycat.je.rep.impl.node.DbCache.get(DbCache.java:124)
at com.sleepycat.je.rep.impl.node.Replay.applyLN(Replay.java:767)
at com.sleepycat.je.rep.impl.node.Replay.replayEntry(Replay.java:479)
at com.sleepycat.je.rep.impl.node.Replica.doRunReplicaLoopInternalWork(Replica.java:465)
at com.sleepycat.je.rep.impl.node.Replica.runReplicaLoopInternal(Replica.java:374)
at com.sleepycat.je.rep.impl.node.Replica.runReplicaLoop(Replica.java:309)
at com.sleepycat.je.rep.impl.node.RepNode.run(RepNode.java:1312)

120529 11:27:48:288 WARNING [Node-5001] Exiting inner Replica loop with exception java.lang.NullPointerException
java.lang.NullPointerException
at com.sleepycat.je.rep.impl.node.DbCache.get(DbCache.java:124)
at com.sleepycat.je.rep.impl.node.Replay.applyLN(Replay.java:767)
at com.sleepycat.je.rep.impl.node.Replay.replayEntry(Replay.java:479)
at com.sleepycat.je.rep.impl.node.Replica.doRunReplicaLoopInternalWork(Replica.java:465)
at com.sleepycat.je.rep.impl.node.Replica.runReplicaLoopInternal(Replica.java:374)
at com.sleepycat.je.rep.impl.node.Replica.runReplicaLoop(Replica.java:309)
at com.sleepycat.je.rep.impl.node.RepNode.run(RepNode.java:1312)

120529 11:27:48:288 INFO [Node-5001] RepNode main thread shutting down.
120529 11:27:48:290 INFO [Node-5001] RepNode shutdown exception:
nullNode-5001(1)[REPLICA]
No feeders.
GlobalCBVLSN=-1
Group info [TestClusterGroup] f8e8d1f9-b6e2-4d77-996f-898abf987d24
Representation version: 2
Change version: 2
Max rep node ID: 2
Node:Node-5001 <removed host name>:5001 (is member) changeVersion:1 LocalCBVLSN:2 at:Tue May 29 11:01:08 BST 2012
Node:Node-5002 <removed host name>:5002 (is member) changeVersion:2 LocalCBVLSN:2 at:Tue May 29 11:01:15 BST 2012
vlsnRange=first=2 last=96 sync=96 txnEnd=96
lastReplayedTxn= VLSN: 96 masterTxnEndTime=Tue May 29 11:26:49 BST 2012 lastReplayedVLSN=97 numActiveReplayTxns=1

120529 11:27:48:291 INFO [Node-5001] Shutting down node Node-5001(1)
120529 11:27:48:292 INFO [Node-5001] Refreshed 0 monitors.
120529 11:27:48:292 INFO [Node-5001] Elections shutdown initiated
120529 11:27:48:293 INFO [Node-5001] Elections shutdown completed
120529 11:27:48:294 INFO [Node-5001] RepNode main thread: REPLICA Node-5001(1) exited.
120529 11:27:48:294 INFO [Node-5001] ServiceDispatcher shutdown starting. HostPort=<removed host name>:5001 Registered services: []
120529 11:27:48:295 INFO [Node-5001] ServiceDispatcher shutdown completed. HostPort=<removed host name>:5001
120529 11:27:48:295 INFO [Node-5001] Node-5001(1) shutdown completed.
120529 11:27:48:296 SEVERE [Node-5001] Node-5001(1): ha-work/Node-5001/bdbstore:DETACHED Node-5001(1) exited unexpectedly with exception java.lang.NullPointerExceptionjava.lang.NullPointerException
at com.sleepycat.je.rep.impl.node.DbCache.release(DbCache.java:100)
at com.sleepycat.je.rep.impl.node.DbCache.clear(DbCache.java:173)
at com.sleepycat.je.rep.impl.node.Replica.clearDbTreeCache(Replica.java:676)
at com.sleepycat.je.rep.impl.node.Replica.loopExitCleanup(Replica.java:657)
at com.sleepycat.je.rep.impl.node.Replica.runReplicaLoopInternal(Replica.java:427)
at com.sleepycat.je.rep.impl.node.Replica.runReplicaLoop(Replica.java:309)
at com.sleepycat.je.rep.impl.node.RepNode.run(RepNode.java:1312)

It seems that the last was a consequence of database corruption on first run but might be useful for investigation.

Is it a know bug? Is any way to work around it?

Kind Regards,
Alex
  • 1. Re: On enabling HA on existing store the Replica node store arrives corrupted
    Linda Lee Journeyer
    Currently Being Moderated
    Alex,

    This problem isn't expected, but it may be that the sequence of events you've done has triggered something we didn't expect or handle correctly. I do agree that the second error seem related to the first one. Somehow, the network restore did not acquire all the metadata (our metadata) about the databases that exist.

    Did you review the directions on upgrade from JE 4.1 to JE 5.0, as described here: http://docs.oracle.com/cd/E17277_02/html/changelog.html, and did you do those steps on the initial, standalone (non-ha) store? I think that you did not. I am not sure that this is the source of the problem, but it's not what we expect, so it would help to know that.

    We'll wait to hear your answer, and then we'll look at the code to see if we see any possible issues. It may be that we ask you to repeat the steps you've gone through, adding additional steps to explicitly run the upgrade utility before the DbEnableReplication call, and then do some DbDumping of the master before the second node comes up, but I'll see if I see anything obvious.

    Linda
  • 2. Re: On enabling HA on existing store the Replica node store arrives corrupted
    Linda Lee Journeyer
    Currently Being Moderated
    Alex,

    I did take a look at the affected code, and I don't see anything obvious. In addition to my question about whether you had run the upgrade utility, can I also ask if
    (a) all databases were created in the non-HA store, before replication started?
    (b) if you can tell which database was accessed by your application -- whether it was the missing metadata database? I understand that may not be obvious, since the error occurs when the data operation gets sent over to the replica.

    At the moment, it does seem like some kind of initialization problem, due to something that happened during the conversion or upgrade. My guess is that the best approach for either getting your deployment working, or debugging the problem if it still exists, is to repeat the conversion.
    - first convert the existing, non-HA store to JE 5.0, by running the upgrade procedure. Check that all databases and contents exist
    - then enable replication on a single node, again using DbDump to list databases and dump contents
    - then finally add the second node.

    Linda

    Edited by: Linda Lee on May 29, 2012 12:15 PM
  • 3. Re: On enabling HA on existing store the Replica node store arrives corrupted
    940250 Newbie
    Currently Being Moderated
    Hi Linda,

    Thanks a lot for your help.

    We ran DbPreUpgrade_4_1 on original non-ha store and it did not help to resolve the issue.
    Also, we could reproduce the same issue on store created with je-5.0.34.

    Our application on start-up tries to upgrade the original database schema by creating new databases, renaming existing databases and moving data between old and new databases.
    So, answer on your question whether all databases were created in the non-HA store, before replication started is No.

    Only one database MESSAGE_METADATA was corrupted during schema upgrade.
    That database contained entries and was renamed from messageMetaDataDb_v5 into MESSAGE_METADATA by the Master node on start-up

    environment.renameDatabase(null, oldName, newName);

    The Replica node tried to open that database but could not find it in logs copied from Master with NetworkRestore.

    We can successfully upgrade the store schema with non-ha Environment using the same code.
    Also, we can start cluster successfully after enabling replication on upgraded non-ha store.

    I followed the steps suggested in your previous post to upgrade the store. Please, find below the results.
    At the moment, it does seem like some kind of initialization problem, due to something that happened during the conversion or upgrade. My guess is that the best approach for either getting your deployment working, or debugging the problem if it still exists, is to repeat the conversion.
    - first convert the existing, non-HA store to JE 5.0, by running the upgrade procedure. Check that all databases and contents exist
    DbPreUpgrade_4_1 finished successfully, all databases in store and their content can be retrieved with DbDump
    - then enable replication on a single node, again using DbDump to list databases and dump contents
    DbEnableReplication could successfully enable replication. DbDump listed all database and their content
    The Single HA node was started and successfully upgraded schema. I stopped the node in order to run DbDump. DbDump listed all database and their content.
    - then finally add the second node.
    Second node started and logs are copied with NetworkRestore as part of start-up.
    It starts successfully only if master is re-started after upgrade. Otherwise, MESSAGE_METADATA database became corrupted.

    I'll try to write a sample java application to reproduce the issue.

    Kind Regards,
    Alex
  • 4. Re: On enabling HA on existing store the Replica node store arrives corrupted
    Linda Lee Journeyer
    Currently Being Moderated
    Alex,

    I think we've used the word "upgrade" in two senses: (a) upgrade from JE 4.1 -> JE 5 and (b) upgrade of your application schema, by explicitly doing database and data movement operations. And I think you've ruled out (a).

    In theory, database operations (create, rename, delete, truncate) are exactly like the operations that apply to data records, and get replicated in the same way. It would indeed be very helpful if you could make a small test program. We have existing unit tests that exercise the database creation, rename, delete, etc, operations in a replication group, and they all seem to work, so it must be something about the order of actions, or the specific combination, which is triggering a problem. It sounds like perhaps you have a workaround for now, which is that it seems to work if you do your application upgrade before enabling replication, but it would be very helpful to have a test case to use to find the problem.

    Thank you,

    Linda
  • 5. Re: On enabling HA on existing store the Replica node store arrives corrupted
    940250 Newbie
    Currently Being Moderated
    Hi Linda,

    As promised, here is the junit test reproducing the problem.
    On running the test case below a ReplicaWriteException is thrown when Replica tries to open corrupted TEST1 database (In fact Replica actually tries to create this database as it thinks TEST1 database does not exist inspite of the fact that list returned by Environment#getDatabaseNames() contains TEST1 database).
    The test fails for me on je-5.0.48 with the following Exception:

    com.sleepycat.je.rep.ReplicaWriteException: (JE 5.0.48) The current state is:REPLICA. The node transitioned to this state at:Tue May 29 22:15:46 BST 2012
         at com.sleepycat.je.rep.txn.ReadonlyTxn.disallowReplicaWrite(ReadonlyTxn.java:83)
         at com.sleepycat.je.dbi.DbTree.checkReplicaWrite(DbTree.java:818)
         at com.sleepycat.je.dbi.DbTree.doCreateDb(DbTree.java:565)
         at com.sleepycat.je.dbi.DbTree.createDb(DbTree.java:456)
         at com.sleepycat.je.Database.initNew(Database.java:124)
         at com.sleepycat.je.Environment.setupDatabase(Environment.java:807)
         at com.sleepycat.je.Environment.openDatabase(Environment.java:607)
         at ReplicationTest.assertTestDatabaseContent(ReplicationTest.java:110)
         at ReplicationTest.testReplication(ReplicationTest.java:97)
         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
         at java.lang.reflect.Method.invoke(Method.java:597)
         at junit.framework.TestCase.runTest(TestCase.java:154)
         at junit.framework.TestCase.runBare(TestCase.java:127)
         at junit.framework.TestResult$1.protect(TestResult.java:106)
         at junit.framework.TestResult.runProtected(TestResult.java:124)
         at junit.framework.TestResult.run(TestResult.java:109)
         at junit.framework.TestCase.run(TestCase.java:118)
         at junit.framework.TestSuite.runTest(TestSuite.java:208)
         at junit.framework.TestSuite.run(TestSuite.java:203)
         at org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(JUnit3TestReference.java:130)
         at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
         at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
         at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
         at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
         at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)


    import java.io.File;
    import java.io.IOException;
    import java.net.InetAddress;
    import java.net.ServerSocket;
    import java.util.Arrays;
    import java.util.HashSet;
    import java.util.List;
    import java.util.Set;
    
    import com.sleepycat.bind.tuple.LongBinding;
    import com.sleepycat.bind.tuple.StringBinding;
    import com.sleepycat.je.Database;
    import com.sleepycat.je.DatabaseConfig;
    import com.sleepycat.je.DatabaseEntry;
    import com.sleepycat.je.DatabaseException;
    import com.sleepycat.je.Durability;
    import com.sleepycat.je.Environment;
    import com.sleepycat.je.EnvironmentConfig;
    import com.sleepycat.je.LockMode;
    import com.sleepycat.je.Transaction;
    import com.sleepycat.je.rep.InsufficientLogException;
    import com.sleepycat.je.rep.NetworkRestore;
    import com.sleepycat.je.rep.NetworkRestoreConfig;
    import com.sleepycat.je.rep.ReplicatedEnvironment;
    import com.sleepycat.je.rep.ReplicationConfig;
    import com.sleepycat.je.rep.util.DbEnableReplication;
    
    import junit.framework.TestCase;
    
    public class ReplicationTest extends TestCase
    {
        private static final String TEST_REPLICATION_GROUP = "TestReplicationGroup";
        private static final String REPLICATION_POLICY = "SYNC,SYNC,ALL";
        private static final String NODE2 = "Node2";
        private static final String NODE1 = "Node1";
        private static final String STORE_FOLDER = System.getProperty("java.io.tmpdir") + File.separator + "test";
        private static final File STORE_LOCATION = new File(STORE_FOLDER);
        private static final Set<String> DATABASES = new HashSet<String>(Arrays.asList("TEST1", "SECOND1"));
    
        private Environment _firstNode;
        private Environment _secondNode;
    
        public void setUp() throws Exception
        {
            if (STORE_LOCATION.exists())
            {
                deleteFile(STORE_LOCATION);
            }
            createNonHaStore(STORE_LOCATION);
        }
    
        public void tearDown()
        {
            if (_secondNode != null)
            {
                _secondNode.close();
            }
            if (_firstNode != null)
            {
                _firstNode.close();
            }
        }
    
        public void testReplication() throws Exception
        {
            int node1Port = findFreePort();
            String hostname = InetAddress.getLocalHost().getHostName();
            String node1HostPort = hostname + ":" + node1Port;
            File workDir = new File(STORE_LOCATION, NODE1);
    
            // enable replication
            DbEnableReplication dbEnableReplication = new DbEnableReplication(workDir, TEST_REPLICATION_GROUP, NODE1, node1HostPort);
            dbEnableReplication.convert();
    
            // start first node
            _firstNode = createReplicationEnvironment(workDir, TEST_REPLICATION_GROUP, NODE1, node1HostPort, node1HostPort);
    
            // upgrade schema
            _firstNode.removeDatabase(null, "third");
            _firstNode.renameDatabase(null, "TEST", "TEST1");
            _firstNode.renameDatabase(null, "second", "SECOND1");
    
            // assert master data
            List<String> databases = _firstNode.getDatabaseNames();
            assertEquals("Unexpected databases on Master:" + databases, DATABASES, new HashSet<String>(databases));
            assertTestDatabaseContent(_firstNode);
    
            // start replica
            File workDir2 = new File(STORE_LOCATION, NODE2);
            workDir2.mkdirs();
            int node2Port = findFreePort();
            _secondNode = createReplicationEnvironment(workDir2, TEST_REPLICATION_GROUP, NODE2, hostname + ":" + node2Port, node1HostPort);
            
            // assert replica data
            List<String> databases2 = _secondNode.getDatabaseNames();
            assertEquals("Unexpected databases on Replica:" + databases2, DATABASES, new HashSet<String>(databases2));
            assertTestDatabaseContent(_secondNode);
        }
    
        private void assertTestDatabaseContent(Environment env)
        {
            DatabaseConfig dbConfig = new DatabaseConfig();
            dbConfig.setTransactional(true);
            dbConfig.setAllowCreate(true);
    
            // check entries
            Database db = null;
            try
            {
                db = env.openDatabase(null, "TEST1", dbConfig);
                for (int i = 0; i < db.count(); i++)
                {
                    DatabaseEntry keyEntry = new DatabaseEntry();
                    DatabaseEntry valueEntry = new DatabaseEntry();
                    LongBinding.longToEntry(i, keyEntry);
                    db.get(null, keyEntry, valueEntry, LockMode.DEFAULT);
                    String val = StringBinding.entryToString(valueEntry);
                    assertEquals("Unexpected entry at " + i, "Test" + i, val);
                }
            }
            finally
            {
                if (db != null)
                {
                    db.close();
                }
            }
        }
    
        private EnvironmentConfig createEnvironmentConfig()
        {
            EnvironmentConfig envConfig = new EnvironmentConfig();
            envConfig.setAllowCreate(true);
            envConfig.setTransactional(true);
            envConfig.setConfigParam(EnvironmentConfig.LOCK_N_LOCK_TABLES, "7");
            return envConfig;
        }
    
        private Environment createReplicationEnvironment(File environmentPath, String groupName, String nodeName,
                String nodeHostPort, String helperHostPort) throws DatabaseException
        {
            Durability replicationDurability = Durability.parse(REPLICATION_POLICY);
            ReplicationConfig replicationConfig = new ReplicationConfig(groupName, nodeName, nodeHostPort);
            replicationConfig.setHelperHosts(helperHostPort);
            EnvironmentConfig envConfig = createEnvironmentConfig();
            envConfig.setDurability(replicationDurability);
            ReplicatedEnvironment replicatedEnvironment = null;
            try
            {
                replicatedEnvironment = new ReplicatedEnvironment(environmentPath, replicationConfig, envConfig);
            }
            catch (final InsufficientLogException ile)
            {
                NetworkRestore restore = new NetworkRestore();
                NetworkRestoreConfig config = new NetworkRestoreConfig();
                config.setRetainLogFiles(false);
                restore.execute(ile, config);
                replicatedEnvironment = new ReplicatedEnvironment(environmentPath, replicationConfig, envConfig);
            }
            return replicatedEnvironment;
        }
    
        private int findFreePort() throws IOException
        {
            int port = -1;
            ServerSocket s = null;
            try
            {
                s = new ServerSocket(0);
                port = s.getLocalPort();
            }
            finally
            {
                if (s != null)
                {
                    s.close();
                }
            }
            return port;
        }
    
        private void createNonHaStore(File storeLocation)
        {
            File workDir = new File(storeLocation, NODE1);
            workDir.mkdirs();
    
            // create non HA environment
            EnvironmentConfig envConfig = createEnvironmentConfig();
            Environment env = new Environment(workDir, envConfig);
    
            // create database
            DatabaseConfig dbConfig = new DatabaseConfig();
            dbConfig.setTransactional(true);
            dbConfig.setAllowCreate(true);
            Database db = env.openDatabase(null, "TEST", dbConfig);
    
            int numberOfEntries = 10;
    
            // insert test entries
            for (int i = 0; i < numberOfEntries; i++)
            {
                Transaction t = env.beginTransaction(null, null);
                DatabaseEntry keyEntry = new DatabaseEntry();
                DatabaseEntry valueEntry = new DatabaseEntry();
                LongBinding.longToEntry(i, keyEntry);
                StringBinding.stringToEntry("Test" + i, valueEntry);
                db.put(t, keyEntry, valueEntry);
                t.commit();
            }
            db.close();
    
            // create more databases
            env.openDatabase(null, "second", dbConfig).close();
            env.openDatabase(null, "third", dbConfig).close();
    
            // close non ha environment
            env.close();
        }
    
        private void deleteFile(File temp)
        {
            if (temp.isDirectory())
            {
                File[] files = temp.listFiles();
                if (files == null || files.length == 0)
                {
                    temp.delete();
                }
                else
                {
                    for (int i = 0; i < files.length; i++)
                    {
                        deleteFile(files);
    }
    }
    }
    else
    {
    temp.delete();
    }
    }

    }
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
  • 6. Re: On enabling HA on existing store the Replica node store arrives corrupted
    Linda Lee Journeyer
    Currently Being Moderated
    Alex,

    Thanks very much for generating the test case. I was able to reproduce the problem. It will take us a few days to reply back to you with our findings, and we'll post that here.

    Thanks!

    Linda
  • 7. Re: On enabling HA on existing store the Replica node store arrives corrupted
    524761 Journeyer
    Currently Being Moderated
    We have found and fixed the bug that caused this problem. The fix will be in the next release of JE (our internal SR #21537).

    Another way to work around the bug is to start the new replicas (having them join the group) before renaming the database.
  • 8. Re: On enabling HA on existing store the Replica node store arrives corrupted
    greybird Expert
    Currently Being Moderated
    Alan and I discussed this, and another safer, workaround is to do a checkpoint after renaming the database and before converting to a replicated environment.

    --mark                                                                                                                                                                                                                                                                                                                                                   
  • 9. Re: On enabling HA on existing store the Replica node store arrives corrupted
    greybird Expert
    Currently Being Moderated
    Sorry if I confused this issue -- it doesn't matter whether you do the checkpoint before or after conversion to a replicated environment. The important thing is to do the checkpoint after the rename, and before other actions (like starting a replica).

    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
  • 10. Re: On enabling HA on existing store the Replica node store arrives corrupted
    Linda Lee Journeyer
    Currently Being Moderated
    Alex,

    Thank you for reporting this bug and taking the time to construct a test case. You'll see that this has been fixed as SR [#21537] in the release of JE 5.0.55 that was made available today. Your help made it much easier to fix this critical bug.

    Thanks,

    The JE team

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points