14 Replies Latest reply: Mar 21, 2013 9:35 AM by 997517 RSS

    Error 8191: This store (MASTER2 on host2.example.com) is not involved

    997517
      Hi,
      I have 3 hosts, host-1.example.com, host-2.example.com, host-3.example.com and want to setup active-standby replication on those.
      host1 is the master, host-2 is standby, host3 is subscriber. hostname command output on all 3 hosts gives FQDN of their own hostname, no other entries in /etc/hosts file.
      Here is the /var/TimesTen/sys.odbc.ini on host1
      [HOST1]
      Driver=/db/TimesTen/53397/lib/libtten.so
      # datastore is the directory PLUS filename
      DataStore=/db/TimesTen/53397/data/MASTER1
      AutoCreate=1
      DatabaseCharacterSet=AL32UTF8
      ConnectionCharacterSet=AL32UTF8
      PermSize=20480
      TempSize=100

      [HOST2]
      Driver = /db/TimesTen/53397/lib/libtten.so
      Hostname = host-2.example.com
      TTC_SERVER = host-2.example.com
      TTC_SERVER_DSN = HOST2
      DatabaseCharacterSet=AL32UTF8
      ConnectionCharacterSet=AL32UTF8

      [HOST3]
      Driver = /db/TimesTen/53397/lib/libtten.so
      Hostname = host-3.example.com
      TTC_Server = host-2=3.example.com
      TTC_Server_DSN = HOST3
      DatabaseCharacterSet=AL32UTF8
      ConnectionCharacterSet=AL32UTF8

      Here is /var/TimesTen/sys.odbc.ini on host-2

      [HOST2]
      Driver = /db/TimesTen/53397/lib/libtten.so
      # datastore is the directory PLUS filename
      DataStore = /db/TimesTen/53397/data/MASTER2
      #AutoCreate = 1
      Overwrite = 1
      DatabaseCharacterSet = AL32UTF8
      ConnectionCharacterSet = AL32UTF8
      PermSize = 20480
      TempSize = 100

      [HOST1]
      Driver = /db/TimesTen/53397/lib/libtten.so
      TTC_SERVER = host-1.example.com
      TTC_SERVER_DSN = HOST1
      DatabaseCharacterSet = AL32UTF8
      ConnectionCharacterSet = AL32UTF8

      [HOST3]
      Driver = /db/TimesTen/53397/lib/libtten.so
      TTC_Server = host-3.example.com
      TTC_Server_DSN = HOST3
      DatabaseCharacterSet = AL32UTF8
      ConnectionCharacterSet = AL32UTF8

      Now, I am able to setup and start rep on host-1. Also, I can use ttRepAdmin -duplicate to duplicate MASTER1 datastore on host-1.example.com to host-2.example.com with dns="HOST2"
      # ttRepAdmin -duplicate -from MASTER1 -host host-1.example.com -uid terry -pwd terry "dsn=HOST2"

      However, when I try to "CALL ttRepStart;" on host-2 (logged in using ttIsql HOST2 on host-2.example.com), I am getting
      Command> call ttRepStart;
      8191: This store (MASTER2 on HOST-2.EXAMPLE.COM) is not involved in a replication scheme

      Using tcpdump, I can see host-1 and host-2 are talking back and forth all the time on the server port (53396)

      On host-2, I can use ttIsqlCS "dsn=HOST1;uid=terry" to login instance on host-1, which I guess verifies the odbc entry is correct.

      ttRepAdmin on host-1 gives output
      [root@host-1 ~]# ttRepAdmin -showconfig "dsn=HOST1"

      Self host "HOST-1.EXAMPLE.COM", port auto, name "MASTER1", LSN 0/14939096, timeout 120, threshold 0


      List of subscribers
      -------------------

      Peer name Host name Port State Proto Track
      ---------------- ------------------------ ------ ------- ----- -----
      SUBSCRIBER1 HOST-3.EXAMPLE.COM Auto Start 35 0

      Last Msg Sent Last Msg Recv Latency TPS RecordsPS
      ------------- ------------- ------- ------- ---------
      - - -1.00 -1 -1


      Peer name Host name Port State Proto Track
      ---------------- ------------------------ ------ ------- ----- -----
      MASTER2 HOST-2.EXAMPLE.COM Auto Start 35 0

      Last Msg Sent Last Msg Recv Latency TPS RecordsPS
      ------------- ------------- ------- ------- ---------
      - - -1.00 -1 -1




      List of objects and subscriptions
      ---------------------------------


      Table details
      -------------
      Table : ROOT.ENTRY_TAB Timestamp updates : -

      Master Name Subscriber name
      ----------- ---------------
      MASTER1 MASTER2
      MASTER1 SUBSCRIBER1

      Table details
      -------------
      Table : ROOT.ENTRY_TAB Timestamp updates : -

      Master Name Subscriber name
      ----------- ---------------
      MASTER2 MASTER1
      MASTER2 SUBSCRIBER1

      Table details
      -------------
      Table : ROOT.ZLU Timestamp updates : -

      Master Name Subscriber name
      ----------- ---------------
      MASTER1 MASTER2
      MASTER1 SUBSCRIBER1

      Table details
      -------------
      Table : ROOT.ZLU Timestamp updates : -

      Master Name Subscriber name
      ----------- ---------------
      MASTER2 MASTER1
      MASTER2 SUBSCRIBER1

      Datastore details
      -----------------
      Master Name Subscriber name
      ----------- ---------------
      MASTER1 MASTER2
      MASTER1 SUBSCRIBER1

      Datastore details
      -----------------
      Master Name Subscriber name
      ----------- ---------------
      MASTER2 MASTER1
      MASTER2 SUBSCRIBER1

      Anybody can help?

      Thanks,
        • 1. Re: Error 8191: This store (MASTER2 on host2.example.com) is not involved
          Chrisjenkins-Oracle
          Hi,

          Firstly an aside. These client DSN definitions are not correct:

          [HOST2]
          Driver = /db/TimesTen/53397/lib/libtten.so
          Hostname = host-2.example.com
          TTC_SERVER = host-2.example.com
          TTC_SERVER_DSN = HOST2
          DatabaseCharacterSet=AL32UTF8
          ConnectionCharacterSet=AL32UTF8

          In a client definition the dDriver directive should have a different value, you cannot specify DatabaseCharacterSet and Hostname is not a valid attibute. The correct client DSN definition would look like this:

          [HOST2]
          Drive =/db/TimesTen/53397/lib/libttclient.so
          TTC_SERVER=host-2.example.com
          TTC_SERVER_DSN=HOST2
          ConnectionCharacterSet=AL32UTF8

          I suspect you may have received some warnings when you connected to this DSN (depending on what tool you used). You should correct these ASAP, though they are not related to the replication issue.

          The issue you are having is because TimesTen is unable to figure out an association between the hostname of the local machine and a hostname used in the replication scheme. Can you please provide the following info:

          1. Output of the 'hostname' command from each machine.

          2. Is host name resolution being done via DNS or /etc/hosts file?

          3. Copy of /etc/hosts file from each machine

          4. Output of the /sbin/ifconfig command from each machine

          5. The full CREATE ACTIVE STANDBY PAIR statement that you use to create replication on the initial machine (HOST1).

          Also note that actual hostnames (official names) should not be FQDN names but plain names )i.e. the hostname for HOST1 should be to 'host-1' not 'host-1.example.com'.

          Chris
          • 2. Re: Error 8191: This store (MASTER2 on host2.example.com) is not involved
            997517
            Thanks Chris for the quick reply.

            What did you mean by "the actual hostname should NOT be FQDN"? Where is this plain name (short name) used?
            By reading all other replication related threads, I had a feeling that this error is hostname related, but just could not figure out where and which.

            Thanks,
            • 3. Re: Error 8191: This store (MASTER2 on host2.example.com) is not involved
              Chrisjenkins-Oracle
              What I meant was that when setting up a machine and assigning it it's hostname that hostname used should always be just the plain name (e.g. host-1) and not a FQDN (host-1.example.com). This is standard O/S best practice. Using a FQDN as the hostname can lead to complications in some cases and is not recommended. This is not really anything specific to TimesTen, just general best practice.

              Chris
              • 4. Re: Error 8191: This store (MASTER2 on host2.example.com) is not involved
                997517
                Here are the requested outputs (only from host-1.example.com and host-2.example.com to reduce the amount of text)
                This is after /var/TimesTen/sys.odbc.ini been changed per your corrections.

                [root@host-1 ~]# hostname
                host-1
                [root@dbslab-m03 ~]# cat /etc/hosts
                # Do not remove the following line, or various programs
                # that require network functionality will fail.
                127.0.0.1 localhost.localdomain localhost
                172.26.176.48 host-1
                172.26.176.49 host-2
                172.26.175.43 host-3
                [root@host-1 ~]# ifconfig eth0
                eth0 Link encap:Ethernet HWaddr 00:16:3E:20:AC:E4
                inet addr:172.26.176.48 Bcast:172.26.176.255 Mask:255.255.255.0
                UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
                RX packets:5219972 errors:0 dropped:0 overruns:0 frame:0
                TX packets:2325738 errors:0 dropped:0 overruns:0 carrier:0
                collisions:0 txqueuelen:1000
                RX bytes:325724007 (310.6 MiB) TX bytes:842881723 (803.8 MiB)
                Interrupt:23


                [root@host-2 ~]# hostname
                host-2
                [root@host-2 ~]# cat /etc/hosts
                # Do not remove the following line, or various programs
                # that require network functionality will fail.
                127.0.0.1 localhost.localdomain localhost
                172.26.176.49 host-2
                172.26.176.48 host-1
                172.26.175.43 host-3
                [root@host-2 ~]# ifconfig eth0
                eth0 Link encap:Ethernet HWaddr 00:16:3E:5C:F8:0D
                inet addr:172.26.176.49 Bcast:172.26.176.255 Mask:255.255.255.0
                UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
                RX packets:8666151 errors:0 dropped:0 overruns:0 frame:0
                TX packets:1297955 errors:0 dropped:0 overruns:0 carrier:0
                collisions:0 txqueuelen:1000
                RX bytes:1787405405 (1.6 GiB) TX bytes:184011773 (175.4 MiB)
                Interrupt:23


                [root@dbslab-m03 ~]# ttIsql HOST1

                Copyright (c) 1996-2011, Oracle. All rights reserved.
                Type ? or "help" for help, type "exit" to quit ttIsql.



                connect "DSN=HOST1";
                Connection successful: DSN=HOST1;UID=root;DataStore=/db/TimesTen/53397/data/MASTER1;DatabaseCharacterSet=AL32UTF8;ConnectionCharacterSet=AL32UTF8;DRIVER=/db/TimesTen/53397/lib/libtten.so;PermSize=20480;TempSize=100;TypeMode=0;
                (Default setting AutoCommit=1)
                Command> create table zlu ( a number not null, b date, primary key (a));
                Command> CREATE ACTIVE STANDBY PAIR MASTER1, MASTER2 ON "host-2" SUBSCRIBER SUBSCRIBER1 ON "host-3";
                Command> CALL ttRepStart;
                Command> CALL ttRepStateSet('ACTIVE');
                Command> CALL ttRepStateGet;
                < ACTIVE, NO GRID >
                1 row found.
                Command> CREATE USER mbs_rep identified by mbs_pwd;

                User created.

                Command> GRANT ADMIN to mbs_rep;


                [root@host-2 ~]# ttRepAdmin -duplicate -from MASTER1 -host host-1 -uid mbs_rep -pwd mbs_pwd -verbosity 2 "dsn=HOST2"
                12:17:38 Contacting remote main daemon at 172.26.176.48 port 53396
                12:17:38 Contacting the replication agent for MASTER1 ON HOST1 (172.26.176.48) port 43935
                12:17:38 Beginning transfer from MASTER1 ON HOST-1 to MASTER2 ON HOST-2
                12:17:41 Checkpoint transfer 100 percent complete
                12:17:41 Checkpoint transfer phase complete
                12:17:42 Log transfer 100 percent complete
                12:17:42 Log transfer phase complete
                12:17:42 Transfer complete
                12:17:44 Duplicate Operation Ends

                [root@host-2 ~]# ttIsql HOST2

                Copyright (c) 1996-2011, Oracle. All rights reserved.
                Type ? or "help" for help, type "exit" to quit ttIsql.



                connect "DSN=HOST2";
                Connection successful: DSN=DBSLABM04;UID=root;DataStore=/db/TimesTen/53397/data/MASTER2;DatabaseCharacterSet=AL32UTF8;ConnectionCharacterSet=AL32UTF8;Overwrite=1;DRIVER=/db/TimesTen/53397/lib/libtten.so;PermSize=20480;TempSize=100;TypeMode=0;
                (Default setting AutoCommit=1)
                Command> call ttRepStart;
                8191: This store (MASTER2 on HOST-2) is not involved in a replication scheme
                The command failed.
                Command>
                • 5. Re: Error 8191: This store (MASTER2 on host2.example.com) is not involved
                  Chrisjenkins-Oracle
                  The problem, I believe, lies here:

                  CREATE ACTIVE STANDBY PAIR MASTER1, MASTER2 ON "host-2" SUBSCRIBER SUBSCRIBER1 ON "host-3";

                  This should be:

                  CREATE ACTIVE STANDBY PAIR MASTER1 ON "host-1", MASTER2 ON "host-2" SUBSCRIBER SUBSCRIBER1 ON "host-3";

                  Please try that and see if it resolves the problem or changes the behaviour.

                  Thanks,

                  Chris
                  • 6. Re: Error 8191: This store (MASTER2 on host2.example.com) is not involved
                    997517
                    Still got the same error.

                    Following is from tterror.log on host-1 when I run CREATE ACTIVE STANDBY PAIR on host-1 and ttRepAdmin -duplicate on host-2


                    08:24:12.95 Warn: REP: 23975: [139846631651072, 0, noAwt] MASTER1:repagent.c(1227): TT16025: Thread 'REPHOLD' (context 0x7f308c0008c0) starting
                    08:24:12.95 Warn: REP: 23975: [139846625351424, 0, noAwt] MASTER1:repagent.c(1227): TT16025: Thread 'XLA_PARENT' (context 0x7f2b700008c0) starting
                    08:24:12.96 Warn: REP: 23975: [139846628501248, 0, noAwt] MASTER1:repagent.c(1227): TT16025: Thread 'FAILOVER' (context 0x7f2b780008c0) starting
                    08:24:12.97 Warn: REP: 23975: [139824750921472, 0, noAwt] MASTER1:repagent.c(1227): TT16025: Thread 'REPLISTENER' (context 0x7f2b740008c0) starting
                    08:24:13.01 Warn: REP: 23975: [139846622201600, 0, noAwt] MASTER1:repagent.c(1227): TT16025: Thread 'LOGFORCE' (context 0x7f3088000c50) starting
                    08:24:13.02 Warn: REP: 23975: [139824747771648, 0, noAwt] MASTER1:repagent.c(1227): TT16025: Thread 'TRANSMITTER(M)' (context 0x7f2b680008c0) starting
                    08:24:13.02 Warn: REP: 23975: [139824747771648, 0, noAwt] MASTER1:transmitter.c(1112): TT16285: Replication transmitter with local store id 15440354107024781670, remote store id 17223408502688007975, track id 0 got repSlave object 0x6806d9a0
                    08:24:13.02 Warn: REP: 23975: [139824747771648, 0, noAwt] MASTER1:transmitter.c(1434): TT16999: Transmitter connecting to receiver
                    08:24:13.02 Warn: REP: 23975: [139824744621824, 0, noAwt] MASTER1:repagent.c(1227): TT16025: Thread 'TRANSMITTER(M)' (context 0x7f2b6c0008c0) starting
                    08:24:13.02 Warn: REP: 23975: [139824744621824, 0, noAwt] MASTER1:transmitter.c(1112): TT16285: Replication transmitter with local store id 15440354107024781670, remote store id 16233787989021148383, track id 0 got repSlave object 0x6c06d9a0
                    08:24:13.03 Warn: REP: 23975: [139824744621824, 0, noAwt] MASTER1:transmitter.c(1434): TT16999: Transmitter connecting to receiver
                    08:24:55.06 Warn: REP: 23975: [139824750921472, 0, noAwt] MASTER1:receiver.c(2263): TT16999: RepListner succesfully done spawning a receiver
                    08:24:55.06 Warn: REP: 23975: [139824326489856, 0, noAwt] MASTER1:repagent.c(1227): TT16025: Thread 'RECEIVER' (context 0x7f2b5c0008c0) starting
                    08:25:00.17 Warn: REP: 23975: [139824326489856, 0, noAwt] MASTER1:repagent.c(1276): TT16999: Receiver exiting isAwt (false)

                    When trying to start rep agent on host-2, I got the following
                    [root@host-2 ~]# ttIsql HOST2

                    Copyright (c) 1996-2011, Oracle. All rights reserved.
                    Type ? or "help" for help, type "exit" to quit ttIsql.



                    connect "DSN=HOST2";
                    Connection successful: DSN=HOST2;UID=root;DataStore=/db/TimesTen/53397/data/MASTER2;DatabaseCharacterSet=AL32UTF8;ConnectionCharacterSet=AL32UTF8;Overwrite=1;DRIVER=/db/TimesTen/53397/lib/libtten.so;PermSize=20480;TempSize=100;TypeMode=0;
                    (Default setting AutoCommit=1)
                    Command> call ttRepStart;
                    8191: This store (MASTER2 on HOST-2) is not involved in a replication scheme
                    The command failed.


                    On host-2, the following message block continuously written to ttmesg.log at about 2 seconds interval.

                    08:31:04.38 Info: : 20750: maind got #220.670, hello: pid=23975 type=utility payload=%00%00%00%00 protocolID=TimesTen 11.2.2.4.1.53397 remote ident=%00%00%00%00
                    08:31:04.38 Info: : 20750: Got hello from pid 23975, type utility
                    08:31:04.38 Info: : 20750: Accepting incoming message from 172.26.176.48 with remote protocol (we are TimesTen 11.2.2.4.1.53397, they are TimesTen 11.2.2.4.1.53397 remote)
                    08:31:04.38 Info: : 20750: 23975 ------------------: Utility program registering
                    08:31:04.38 Info: : 20750: maind: done with request #220.670
                    08:31:04.38 Info: : 20750: maind 220: socket closed, calling recovery (last cmd was 671)
                    08:31:04.38 Info: : 20750: Starting daRecovery for 23975
                    08:31:04.38 Info: : 20750: Finished daRecovery for pid 23975.


                    On host-2, after duplicate succeeded, when I log on the instance and run queries, it looks like nothing been duplicated from host-1 (empty TTREP.TTSTORES table, no mbs_rep user in SYS.USER$ table etc).
                    Command> select * from TTREP.TTSTORES;
                    0 rows found.
                    Command> select * from TTREP.REPSUBSCRIPTIONS;
                    0 rows found.
                    Command> select * from TTREP.REPPEERS
                    > ;
                    0 rows found.
                    Command> select * from SYS.USER$;
                    < 0, SYS , 1, 0, <NULL>, 2013-03-20 08:36:42.768919, <NULL>, <NULL>, 2695088, 0, 0, 0000, 00000000000000000000000000000000 >
                    < 1, PUBLIC , 0, 0, <NULL>, 2013-03-20 08:36:42.768919, <NULL>, <NULL>, 2695104, 0, 0, 0000, 00000000000000000000000000000000 >
                    < 2, TTREP , 1, 0, <NULL>, 2013-03-20 08:36:42.768919, <NULL>, <NULL>, 2695120, 0, 0, 0000, 00000000000000000000000000000000 >
                    < 3, SYSTEM , 1, 0, <NULL>, 2013-03-20 08:36:42.768919, <NULL>, <NULL>, 2695136, 0, 0, 0000, 00000000000000000000000000000000 >
                    < 4, GRID , 1, 0, <NULL>, 2013-03-20 08:36:42.768919, <NULL>, <NULL>, 2695152, 0, 0, 0000, 00000000000000000000000000000000 >
                    < 10, ROOT , 1, 0, <NULL>, 2013-03-20 08:36:42.768919, <NULL>, <NULL>, 2695168, 1, 0, 0000, 00000000000000000000000000000000 >
                    6 rows found.

                    Edited by: 994514 on Mar 20, 2013 6:12 AM


                    Right after ttRepAdmin -duplicate succeeded on host-2, if I do strings MASTER2.* |grep HOST, I can see hosts (HOST-1, HOST-2, HOST-3) from the output. But they disappear shortly after.

                    Edited by: 994514 on Mar 20, 2013 6:50 AM
                    • 7. Re: Error 8191: This store (MASTER2 on host2.example.com) is not involved
                      Chrisjenkins-Oracle
                      Okay, can you please do the following:

                      1. On 'host-1', check that replication is still setup and 'active' (call ttrepstateget). If it is, in ttIsql please do SELECT * FROM TTREP.TTSTORES and capture the output (spool repout.txt).

                      2. On 'host-2', destroy the datastore created by duplicate (ttDestroy) and re-create it by just connecting from ttIsql. Then create the exact same replication scheme as on 'host-1' (please use exactly the same SQL statement) and then run SELECT * FROM TTREP.TTSTORES and capture the output.

                      Please post both sets of output.

                      Thanks,

                      Chris
                      • 8. Re: Error 8191: This store (MASTER2 on host2.example.com) is not involved
                        997517
                        I guess I have to create a table on host-2 before I can create the rep pair.

                        Following is from host-1
                        Command> CALL ttRepStateGet;
                        < ACTIVE, NO GRID >
                        1 row found.
                        Command> select * from TTREP.TTSTORES;
                        < -3006389966684769946, HOST-1, MASTER1, 01, 0, 0, 5, 0, N, 0, 0, 00, N, 10, <NULL>, <NULL>, Y, A, <NULL>, 5857400248515502287, <NULL>, <NULL>, <NULL>, E, 0 >
                        < -2212956084688403233, HOST-3, SUBSCRIBER1, 00, 0, 0, -1, 0, N, 0, 0, 00, N, 10, <NULL>, <NULL>, N, <NULL>, <NULL>, <NULL>, <NULL>, <NULL>, <NULL>, E, 1 >
                        < -1223335571021543641, HOST-2, MASTER2, 00, 0, 0, -1, 0, N, 0, 0, 00, N, 10, <NULL>, <NULL>, Y, S, <NULL>, 5857400248515502287, <NULL>, <NULL>, <NULL>, E, 2 >
                        3 rows found.

                        Then on host-2,
                        # ttDestroy /db/TimesTen/53397/data/MASTER2

                        # ttIsql HOST2
                        Command> create table zlu ( a number not null, b date, primary key (a));
                        Command> CREATE ACTIVE STANDBY PAIR MASTER1 ON "dbslab-m03", MASTER2 ON "dbslab-m04" SUBSCRIBER SUBSCRIBER1 ON "dbslab-m05";
                        Command> select * from TTREP.TTSTORES;
                        < -3006389966684769946, HOST-1, MASTER1, 00, 0, 0, -1, 0, N, 0, 0, 00, N, 10, <NULL>, <NULL>, Y, <NULL>, <NULL>, <NULL>, <NULL>, <NULL>, <NULL>, E, 0 >
                        < -2212956084688403233, HOST-3, SUBSCRIBER1, 00, 0, 0, -1, 0, N, 0, 0, 00, N, 10, <NULL>, <NULL>, N, <NULL>, <NULL>, <NULL>, <NULL>, <NULL>, <NULL>, E, 1 >
                        < -1223335571021543641, HOST-2, MASTER2, 01, 0, 0, 5, 0, N, 0, 0, 00, N, 10, <NULL>, <NULL>, Y, <NULL>, <NULL>, <NULL>, <NULL>, <NULL>, <NULL>, E, 2 >
                        3 rows found.
                        Command> call ttRepStart;
                        Command> CALL ttRepStateGet;
                        < ACTIVE, NO GRID >
                        1 row found.


                        After that, on host-1 in tterror.log
                        12:40:46.33 Warn: REP: 475: [140364391810816, 0, noAwt] MASTER1:repagent.c(1227): TT16025: Thread 'TRANSMITTER(M)' (context 0x7fa3f40008c0) starting
                        12:40:46.33 Warn: REP: 475: [140364391810816, 0, noAwt] MASTER1:transmitter.c(1112): TT16285: Replication transmitter with local store id 15440354107024781670, remote store id 17223408502688007975, track id 0 got repSlave object 0xf406d9a0
                        12:40:46.35 Warn: REP: 475: [140364391810816, 0, noAwt] MASTER1:transmitter.c(1434): TT16999: Transmitter connecting to receiver
                        12:40:47.03 Warn: REP: 475: [140364391810816, 0, noAwt] MASTER1:transmitter.c(3763): TT16999: Neither Standby nor Active: Cannot deal with this locally generated transaction txn nowtxn->ctn = 1363786580.15 txn->fctn = 0.0
                        12:40:47.03 Warn: REP: 475: [140364391810816, 0, noAwt] MASTER1:transmitter.c(13496): TT16293: Replication thread transmitter is exiting (exitLine 12856)
                        12:40:47.03 Warn: REP: 475: [140364391810816, 0, noAwt] MASTER1:transmitter.c(8461): TT16290: Replication transmitters are stopping.
                        12:40:47.03 Warn: REP: 475: [140364391810816, 0, noAwt] MASTER1:repagent.c(1276): TT16999: Transmitter exiting isAwt (false)
                        12:40:47.80 Warn: REP: 475: [140364398110464, 0, noAwt] MASTER1:receiver.c(2263): TT16999: RepListner succesfully done spawning a receiver
                        12:40:47.81 Warn: REP: 475: [140364385511168, 0, noAwt] MASTER1:repagent.c(1227): TT16025: Thread 'RECEIVER' (context 0x7fa3ec0008c0) starting
                        12:40:47.82 Warn: REP: 475: [140364385511168, -1, notKnown] MASTER1:receiver.c(4350): TT16999: stopRcvrThreads (track 0) set to 0
                        *12:40:47.82 Err : REP: 475: [140364385511168, 0, noAwt] MASTER1:receiver.c(3810): TT16227: Standby store has replicated transactions not present on the active. Local CTN=1363786580.15, Backup CTN=0.0, Received CTN=0.0.*
                        12:40:47.82 Warn: REP: 475: [140364385511168, -1, notKnown] MASTER1:receiver.c(1829): TT16999: stopRcvrThreads (track 0) set to 1
                        12:40:47.82 Warn: REP: 475: [140364385511168, -1, notKnown] MASTER1:receiver.c(1870): TT16999: stopRcvrThreads (track 0) set to 0
                        12:40:47.83 Warn: REP: 475: [140364385511168, 0, noAwt] MASTER1:repagent.c(1276): TT16999: Receiver exiting isAwt (false)

                        And in tterror.log on host-2
                        12:41:44.83 Warn: REP: 28587: [140093521921792, 0, noAwt] MASTER2:receiver.c(2263): TT16999: RepListner succesfully done spawning a receiver
                        12:41:44.83 Warn: REP: 28587: [140115408209664, 0, noAwt] MASTER2:repagent.c(1227): TT16025: Thread 'RECEIVER' (context 0x7f69f00008c0) starting
                        12:41:44.84 Warn: REP: 28587: [140115408209664, -1, notKnown] MASTER2:receiver.c(4350): TT16999: stopRcvrThreads (track 0) set to 0
                        12:41:44.89 Warn: REP: 28587: [140115408209664, -1, notKnown] MASTER2:receiver.c(3112): TT16999: stopRcvrThreads (track 0) set to 1
                        12:41:44.89 Warn: REP: 28587: [140115408209664, 0, noAwt] MASTER2:receiver.c(3113): TT16060: Failed to read data from the network. select() timed out isAwt (false) parallelism (1) trackId (0) socket (17)
                        12:41:44.89 Warn: REP: 28587: [140115408209664, -1, notKnown] MASTER2:receiver.c(1829): TT16999: stopRcvrThreads (track 0) set to 1
                        12:41:44.89 Warn: REP: 28587: [140115408209664, -1, notKnown] MASTER2:receiver.c(1870): TT16999: stopRcvrThreads (track 0) set to 0
                        12:41:44.90 Warn: REP: 28587: [140115408209664, 0, noAwt] MASTER2:repagent.c(1276): TT16999: Receiver exiting isAwt (false)

                        12:41:45.10 Warn: REP: 28587: [140115414509312, 0, noAwt] MASTER2:transmitter.c(7160): TT16060: Failed to read data from the network. select() timed out isAwt (false) parallelism (1) trackId (0) socket (15)
                        12:41:45.10 Warn: REP: 28587: [140115414509312, 0, noAwt] MASTER2:transmitter.c(13496): TT16293: Replication thread transmitter is exiting (exitLine 12605)
                        12:41:45.11 Warn: REP: 28587: [140115414509312, 0, noAwt] MASTER2:repagent.c(1276): TT16999: Transmitter exiting isAwt (false)

                        12:41:48.11 Warn: REP: 28587: [140115414509312, 0, noAwt] MASTER2:repagent.c(1227): TT16025: Thread 'TRANSMITTER(M)' (context 0x7f6a000008c0) starting
                        12:41:48.11 Warn: REP: 28587: [140115414509312, 0, noAwt] MASTER2:transmitter.c(1112): TT16285: Replication transmitter with local store id 17223408502688007975, remote store id 15440354107024781670, track id 0 got repSlave object 0x6d9a0
                        12:41:48.14 Warn: REP: 28587: [140115414509312, 0, noAwt] MASTER2:transmitter.c(1434): TT16999: Transmitter connecting to receiver
                        12:41:48.54 Warn: REP: 28587: [140093521921792, 0, noAwt] MASTER2:receiver.c(2263): TT16999: RepListner succesfully done spawning a receiver
                        12:41:48.54 Warn: REP: 28587: [140115408209664, 0, noAwt] MASTER2:repagent.c(1227): TT16025: Thread 'RECEIVER' (context 0x7f69f00008c0) starting
                        12:41:48.57 Warn: REP: 28587: [140115408209664, -1, notKnown] MASTER2:receiver.c(4350): TT16999: stopRcvrThreads (track 0) set to 0
                        12:41:48.62 Warn: REP: 28587: [140115408209664, -1, notKnown] MASTER2:receiver.c(3112): TT16999: stopRcvrThreads (track 0) set to 1
                        12:41:48.62 Warn: REP: 28587: [140115408209664, 0, noAwt] MASTER2:receiver.c(3113): TT16060: Failed to read data from the network. select() timed out isAwt (false) parallelism (1) trackId (0) socket (17)
                        12:41:48.62 Warn: REP: 28587: [140115408209664, -1, notKnown] MASTER2:receiver.c(1829): TT16999: stopRcvrThreads (track 0) set to 1
                        12:41:48.62 Warn: REP: 28587: [140115408209664, -1, notKnown] MASTER2:receiver.c(1870): TT16999: stopRcvrThreads (track 0) set to 0
                        12:41:48.62 Warn: REP: 28587: [140115408209664, 0, noAwt] MASTER2:repagent.c(1276): TT16999: Receiver exiting isAwt (false)
                        12:41:48.81 Warn: REP: 28587: [140115414509312, 0, noAwt] MASTER2:transmitter.c(7160): TT16060: Failed to read data from the network. select() timed out isAwt (false) parallelism (1) trackId (0) socket (15)
                        12:41:48.81 Warn: REP: 28587: [140115414509312, 0, noAwt] MASTER2:transmitter.c(13496): TT16293: Replication thread transmitter is exiting (exitLine 12605)
                        12:41:48.82 Warn: REP: 28587: [140115414509312, 0, noAwt] MASTER2:repagent.c(1276): TT16999: Transmitter exiting isAwt (false)

                        Edited by: 994514 on Mar 20, 2013 1:30 PM
                        • 9. Re: Error 8191: This store (MASTER2 on host2.example.com) is not involved
                          Chrisjenkins-Oracle
                          I'm afraid I am very confused now. In all your examples to date you have been referring to 'host-1' and 'host-2' but now here you are using DBSLAB-M03, DBSLAB-M04 and DBSLAB-M05.

                          Please explain...

                          Chris
                          • 10. Re: Error 8191: This store (MASTER2 on host2.example.com) is not involved
                            997517
                            Sorry, I suppose to change the host names from real name.
                            Updated previous post.

                            Also, after I destroy the store on host-2, run duplicate again, the tterror.log on host-2 prints
                            15:58:41.39 Warn: : 5475: 5482/0x1264460: Recovery started
                            15:58:42.67 Warn: : 5475: 5482/(nil): Another checkpoint is in progress. The final checkpoint will continue when the other checkpoint finishes.
                            16:29:35.66 Warn: : 5475: 5482/(nil): Another checkpoint is in progress. The final checkpoint will continue when the other checkpoint finishes.

                            Wondering if that can lead to the real problem?

                            As I mentioned before, at this time strings data/MASTER2.* |grep HOST prints HOST-1, HOST-2, etc. and ttRepAdmin prints following
                            [root@host-2 53397]# ttRepAdmin -self -list HOST2
                            Self host "HOST-2", port auto, name "MASTER2", LSN 0/20349192
                            Operation successful

                            [root@host-2 53397]# ttRepAdmin -receiver -list HOST2
                            Peer name Host name Port State Proto Track
                            ---------------- ------------------------ ------ ------- ----- -----
                            MASTER1 HOST-1 Auto Start 35 0

                            Last Msg Sent Last Msg Recv Latency TPS RecordsPS Logs
                            ------------- ------------- ------- ------- --------- ----
                            - - -1.00 -1 -1 1

                            Now, after logging in to HOST2 and run call ttRepStart (failed with same error), strings data/MASTER2.* |grep HOST prints nothing. ttRepAdmin prints nothing neither.
                            [root@host-2 53397]# ttRepAdmin -receiver -list HOST2
                            [root@host-2 53397]#

                            Sorry for the confusion.

                            Zhang
                            • 11. Re: Error 8191: This store (MASTER2 on host2.example.com) is not involved
                              Chrisjenkins-Oracle
                              Well, all this obfuscation of host names (not sure for what purpose) has certainly confused things when trying to diagnose this. As far as I can see from the information presented everything looks in order. But c;learly there is some subtle problem relating to hostname matching. Can you please confirm the exact TimesTen version (output of ttVersion command) and the exact OS version.

                              I think that to resolve this you will need to open a support case with Oracle Support.

                              Chris
                              • 12. Re: Error 8191: This store (MASTER2 on host2.example.com) is not involved
                                jspalmer
                                There are notes in MOS which detail possible causes of TT8191 e.g Note 887589.1, Note 1537425.1. I agree with Chris - you need to log this as an SR because in all likelihood this is down to incorrect configuration of networking files which TimesTen needs to use. Note 887589.1 provides an example of this. If you need to disguise hostnames in a public forum, then we need to look at this through the confidentiality afforded by an SR.
                                • 13. Re: Error 8191: This store (MASTER2 on host2.example.com) is not involved
                                  997517
                                  All hosts are configured exactly the same. Following is the output from host-2.

                                  [root@host-2 53397]# ttVersion
                                  TimesTen Release 11.2.2.4.1 (64 bit Linux/x86_64) (53397:53396) 2012-10-30T07:17:51Z
                                  Instance admin: root
                                  Instance home directory: /db/TimesTen/53397
                                  Group owner: root
                                  Daemon home directory: /var/TimesTen/53397
                                  PL/SQL enabled.
                                  [root@host-2 53397]# uname -a
                                  Linux host-2 2.6.32-220.el6.x86_64 #1 SMP Tue Dec 6 19:48:22 GMT 2011 x86_64 x86_64 x86_64 GNU/Linux
                                  [root@host-2 53397]# cat /etc/redhat-release
                                  CentOS release 6.2 (Final)

                                  I read those 2 mentioned notes, checked my /etc/hosts file and looks like everything is in order.
                                  So I destroyed the store on host-2 again, run duplicate. This time, instead of ttIsql HOST2 and call ttRepStart, I ran utility ttAdmin -RepStart HOST2 and got following output.
                                  [root@host-2 ~]# ttAdmin -RepStart HOST2
                                  RAM Residence Policy : inUse
                                  Replication Agent Policy : manual
                                  Replication Manually Started : True
                                  Cache Agent Policy : manual
                                  Cache Agent Manually Started : False

                                  Same time, host-1 tterror.log starts complaining clock skew (5528, which I guess is in ms?) on MASTER2, so I started ntp service on both to sync the time, the error on host-1 stopped after a while.
                                  The following is from tterror.log on host-2 after repagent been started using ttAdmin
                                  09:21:21.21 Warn: REP: 23907: [140048246884096, 0, noAwt] MASTER2:repagent.c(1227): TT16025: Thread 'REPHOLD' (context 0x7f5f7c0008c0) starting
                                  09:21:21.21 Warn: REP: 23907: [140048243734272, 0, noAwt] MASTER2:repagent.c(1227): TT16025: Thread 'FAILOVER' (context 0x7f5a680008c0) starting
                                  09:21:21.22 Warn: REP: 23907: [140026345948928, 0, noAwt] MASTER2:repagent.c(1227): TT16025: Thread 'REPLISTENER' (context 0x7f5a640008c0) starting
                                  09:21:21.23 Warn: REP: 23907: [140048240584448, 0, noAwt] MASTER2:repagent.c(1227): TT16025: Thread 'XLA_PARENT' (context 0x7f5a600008c0) starting
                                  09:21:21.26 Warn: REP: 23907: [140048237434624, 0, noAwt] MASTER2:repagent.c(1227): TT16025: Thread 'LOGFORCE' (context 0x7f5f78000c50) starting
                                  09:21:21.27 Warn: REP: 23907: [140048234284800, 0, noAwt] MASTER2:repagent.c(1227): TT16025: Thread 'TRANSMITTER(M)' (context 0x7f5a580008c0) starting
                                  09:21:21.27 Warn: REP: 23907: [140048234284800, 0, noAwt] MASTER2:transmitter.c(1112): TT16285: Replication transmitter with local store id 17223408502688007975, remote store id 15440354107024781670, track id 0 got repSlave object 0x5806d9a0
                                  09:21:21.27 Warn: REP: 23907: [140048234284800, 0, noAwt] MASTER2:transmitter.c(1434): TT16999: Transmitter connecting to receiver
                                  09:21:21.56 Warn: REP: 23907: [140026345948928, 0, noAwt] MASTER2:receiver.c(2263): TT16999: RepListner succesfully done spawning a receiver
                                  09:21:21.57 Warn: REP: 23907: [140048231134976, 0, noAwt] MASTER2:repagent.c(1227): TT16025: Thread 'RECEIVER' (context 0x7f5a540008c0) starting
                                  *09:21:21.57 Err : REP: 23907: [140048231134976, 0, noAwt] MASTER2:meta.c(3134): TT16999: receiver slot alloc: 17223408502688007975 15440354107024781670*
                                  09:21:21.57 Warn: REP: 23907: [140048231134976, -1, notKnown] MASTER2:receiver.c(4350): TT16999: stopRcvrThreads (track 0) set to 0

                                  Now, I ran "ttRepAdmin -showconfig" and "ttRepAdmin -dsn <DSN> -receiver -list" on both hosts, seems like the replication is working. If I inserted one record on MASTER1, the LSN on both MASTER1 and MASTER2 are increasing.

                                  One more problem is that I cannot ttIsql HOST2 after repagent is started.
                                  [root@host-2 ~]# ttIsql HOST2

                                  Copyright (c) 1996-2011, Oracle. All rights reserved.
                                  Type ? or "help" for help, type "exit" to quit ttIsql.



                                  connect "DSN=HOST2";
                                  839: Cannot access data store because it is in use. A data store may be considered to be in use due to its RAM Policy setting, even though there are no active connections to it.
                                  830: Cannot create data store file. OS-detected error: Could not destroy previous data store
                                  The command failed.
                                  Done.

                                  The following is from tterror.log
                                  09:37:35.53 Err : : 5475: TT14007: TimesTen daemon destroy failed: destroy by 24627, datastore has 18 users (11 connections by subdaemon)

                                  Edited by: 994514 on Mar 21, 2013 6:24 AM
                                  • 14. Re: Error 8191: This store (MASTER2 on host2.example.com) is not involved
                                    997517
                                    I think I know what was wrong.

                                    On host-2, I had
                                    Overwrite = 1
                                    in sys.odbc.ini for store MASTER2.
                                    So every time when I use ttIsql HOST2 to connect, it tries to destroy MASTER2 but since it is used by replication (started from command line by ttAdmin -RepStart), it cannot destroy and won't let me log in.
                                    After commented out that line, I am able to get in MASTER2 and see replicated data from MASTER1.

                                    Same, if I try to start rep from ttIsql, as soon as I connect to MASTER2, the duplicated store was destroyed and an new one got created. That's why it kept telling me the store is not involved in any replication scheme.

                                    Everything should be good now. Thanks