7 Replies Latest reply: May 8, 2012 11:05 AM by Gennady Sigalaev RSS

    TimesTen Cache Grid Setup Issues on Clusterware

    Thomascong-Oracle
      Dear Experts,

      I would like to set up two TimesTen A/S pairs on Clusterware. Here's the A/S pairs status after the initial setup. I was able to load cache groups into the pairs.


      sss6202/u01/app/TimesTen/tt1122/info> ttcwadmin -status
      TimesTen Cluster status report as of Wed Apr 25 21:48:35 2012

      ====================================================================
      TimesTen daemon monitors:
      Host:SSS6202 Status: online
      Host:SSS6203 Status: online

      ====================================================================
      ====================================================================
      TimesTen Cluster agents
      Host:SSS6202 Status: online
      Host:SSS6203 Status: online

      ====================================================================


      Status of Cluster related to DSN ST_0.0.0.1:
      ====================================================================
      1. Status of Cluster monitoring components:
      Monitor Process for Active datastore:RUNNING on Host sss6203
      Monitor Process for Standby datastore:RUNNING on Host sss6202
      Monitor Process for Master Datastore 1 on Host sss6202: RUNNING
      Monitor Process for Master Datastore 2 on Host sss6203: RUNNING
      Monitor for Application DM_TT_0.0.0.1: RUNNING on Host sss6203

      2.Status of Datastores comprising the cluster
      Master Datastore 1:
      Host:sss6202
      Status:AVAILABLE
      State:STANDBY
      Grid:AVAILABLE
      Master Datastore 2:
      Host:sss6203
      Status:AVAILABLE
      State:ACTIVE
      Grid:AVAILABLE
      ====================================================================
      The cluster containing the replicated DSN is online


      Status of Cluster related to DSN ST_0.1.0.1:
      ====================================================================
      1. Status of Cluster monitoring components:
      Monitor Process for Active datastore:RUNNING on Host sss6203
      Monitor Process for Standby datastore:RUNNING on Host sss6203
      Monitor Process for Master Datastore 1 on Host sss6202: RUNNING
      Monitor Process for Master Datastore 2 on Host sss6203: RUNNING
      Monitor for Application DM_TT_0.1.0.1: RUNNING on Host sss6203

      2.Status of Datastores comprising the cluster
      Master Datastore 1:
      Host:sss6202
      Status:AVAILABLE
      State:IDLE
      Grid:NO GRID
      Master Datastore 2:
      Host:sss6203
      Status:AVAILABLE
      State:ACTIVE
      Grid:AVAILABLE
      ====================================================================
      The cluster containing the replicated DSN is online



      When I attemp to start my application connecting to TimesTen A/S pairs it failed with the following errors. Seems like TT grid just has gone bad for unknown reason.


      M Thu Apr 26 13:41:53 2012 sss6202 dm:10236 dm_search.c(106):3243 1:sss6202:cm:29186:1:0:1335469313:0
      in update size is 1000 (default)
      E Thu Apr 26 13:41:53 2012 sss6202 dm:10235 dm_subr.c(154):1904 1:sss6202:cm:29186:1:0:1335469313:0
      ORACLE error: do_sql_select: PINStmtExecute: code 57000, op 0
      =ORA-57000: TT3331: Failed to send a message to member UNKNOWN -- file "cacheGrid.c", lineno 28552, procedure "sbCGGridCompile"
      E Thu Apr 26 13:41:53 2012 sss6202 dm:10235 dm_ops.c(238):6823 1:sss6202:cm:29186:1:0:1335469313:0
      op_search_and_bulk_act: do_sql_select of search: "select template, flags from pin.search_t where poid_id0 = :1", x=1, id 500
      E Thu Apr 26 13:41:53 2012 sss6202 dm:10235 dm_back.c(31):1440 1:sss6202:cm:29186:1:0:1335469313:0
      DMbe #6: process_op: op 7(PCM_OP_SEARCH), err 43(PIN_ERR_STORAGE)
      E Thu Apr 26 13:41:53 2012 sss6202 dm:10236 dm_subr.c(154):1904 1:sss6202:cm:29186:1:0:1335469313:0
      ORACLE error: do_sql_select: PINStmtExecute: code 57000, op 0
      =ORA-57000: TT3331: Failed to send a message to member UNKNOWN -- file "cacheGrid.c", lineno 28552, procedure "sbCGGridCompile"
      E Thu Apr 26 13:41:53 2012 sss6202 dm:10236 dm_ops.c(238):6823 1:sss6202:cm:29186:1:0:1335469313:0
      op_search_and_bulk_act: do_sql_select of search: "select template, flags from pin.search_t where poid_id0 = :1", x=1, id 500
      E Thu Apr 26 13:41:53 2012 sss6202 dm:10236 dm_back.c(31):1440 1:sss6202:cm:29186:1:0:1335469313:0
      DMbe #7: process_op: op 7(PCM_OP_SEARCH), err 43(PIN_ERR_STORAGE)


      So I try to drop and recreate the A/S pairs, but the call ttcwadmin –create –dsn st_0.0.0.1 now errs out with the following errors in ttcwerrors.log. It’s the same process I ran previously to set up the pairs. Did I miss anything in the process?


      2012-04-26 20:06:04.22 Err : : 15926: (ttCRSdaemon:) ttctl.c(5801): TT16032: Call to send() failed. System Error: 134
      2012-04-26 20:07:01.28 Err : : 1685: (ttClusterAgent:) ttctl.c(7764): S1T00:[TimesTen][TimesTen 11.2.2.1.0 ODBC Driver]
      [TimesTen]TT6003: Lock request denied because of time-out Details: Tran 24.49 (pid 1690) wants IXn lock on table SYS.CACHE_
      GROUP. But tran 23.59 (pid 1690) has it in Un (request was Un). Holder SQL (call ttrepstateset('active')) -- file "cache.c"
      , lineno 7173, procedure "sbCacheGetDDLLocks()"
      2012-04-26 20:07:01.28 Err : : 1685: (ttClusterAgent:) crsagent.c 2086: Failed to set state of autorefresh cache groups
      to paused
      2012-04-26 20:07:01.29 Err : : 15935: (ttCWAdmin:) cwutils.c(1998): TT16032: Call to recv() failed. System Error: -1
      2012-04-26 20:07:01.29 Err : : 15935: (ttCWAdmin:) cwutils.c(1998): TT16032: Call to recv() failed. System Error: -1
      2012-04-26 20:07:01.29 Err : : 15935: (ttCWAdmin:) crsctl.c(19818): TT48013: Failed to create ACTIVE STANDBY PAIR scheme
      for DSN ST_0.0.0.1 on host sss6202.
      2012-04-26 20:07:38.40 Err : : 15935: (ttCWAdmin:) cwutils.c(1941): TT16032: Call to recv() failed. System Error: 131

      It's TimesTen 11.2.2 on Clusterware 11g2.

      Many thanks for your advices / insights!
        • 1. Re: TimesTen Cache Grid Setup Issues on Clusterware
          Thomascong-Oracle
          I find it's odd that ttRepStateGet still returns the following status for my first datastore.

          Command> call ttRepStateGet;
          < ACTIVE, IN TRANSITION>

          Command> call ttGridDetach;
          48115: Not allowed since this store (ST_0.0.0.1 on sss6202) is managed by Clusterware.


          And for the second datastore

          Command> call ttRepStateGet;
          < IDLE, NO GRID>
          • 2. Re: TimesTen Cache Grid Setup Issues on Clusterware
            Gennady Sigalaev
            Dear user10240285,

            You create two AS pairs replications (two DNS ST_0.0.0.1 and ST_0.1.0.1). In both cases you use sss6203 for Active and sss6202 for Standby purpose. Is that correct?

            For DSN ST_0.0.0.1 it looks fine.
            Master Datastore 1:
            Host:sss6202
            Status:AVAILABLE
            State:STANDBY
            Grid:AVAILABLE
            Master Datastore 2:
            Host:sss6203
            Status:AVAILABLE
            State:ACTIVE
            Grid:AVAILABLE
            But for DSN ST_0.1.0.1 it doesnt.
            Master Datastore 1:
            Host:sss6202
            Status:AVAILABLE
            State:IDLE
            Grid:NO GRID
            Master Datastore 2:
            Host:sss6203
            Status:AVAILABLE
            State:ACTIVE
            Grid:AVAILABLE
            Why have you decided to recreate st_0.0.0.1 dns? Could you please post your code, crs_stat -t output and ./ttCWAdmin -status output and call ttGridNodeStatus; from both hosts (sss6202, sss6203).

            Best regards,
            Gena

            Edited by: Gennady Sigalaev on 27-Apr-2012 10:46
            • 3. Re: TimesTen Cache Grid Setup Issues on Clusterware
              Thomascong-Oracle
              Hi Gena,

              Thank you for your reply. My intention is to set up both active datastores on sss6202, and standby on sss6203.

              The reason I dropped them because of the application error shown in the original post and ttcwadmin -status at that time showed NO GRID

              Now I am facing a problem with recreating the A/S pairs. The errors in ttcwerrors.log are shown below. It seems to me the drop command did not drop the A/S pairs cleanly.

              2012-04-27 11:49:32.19 Err : : 3126: (ttClusterAgent:) ttctl.c(7764): S1T00:[TimesTen][TimesTen 11.2.2.1.0 ODBC Driver][TimesTen]TT6003: Lock request denied because of time-out Details: Tran 23.69 (pid 3131) wants IXn lock on table SYS.CACHE_GROUP. But tran 1.109139 (pid 3131) has it in Un (request was Un). Holder SQL (call ttrepstateset('active')) -- file "cache.c", lineno 7173, procedure "sbCacheGetDDLLocks()"
              2012-04-27 11:49:32.19 Err : : 3126: (ttClusterAgent:) crsagent.c 2086: Failed to set state of autorefresh cache groups to paused
              2012-04-27 11:49:32.19 Err : : 4559: (ttCWAdmin:) cwutils.c(1998): TT16032: Call to recv() failed. System Error: -1
              2012-04-27 11:49:32.19 Err : : 4559: (ttCWAdmin:) cwutils.c(1998): TT16032: Call to recv() failed. System Error: -1
              2012-04-27 11:49:32.19 Err : : 4559: (ttCWAdmin:) crsctl.c(19818): TT48013: Failed to create ACTIVE STANDBY PAIR scheme for DSN ST_0.0.0.1 on host sss6202.
              2012-04-27 11:50:25.26 Err : : 4559: (ttCWAdmin:) crsctl.c(19818): TT48013: Failed to create ACTIVE STANDBY PAIR scheme for DSN ST_0.0.0.1 on host sss6203.

              Here's crs_stat -t output

              sss6202/u01/app/TimesTen/tt1122/info> crs_stat -t
              Name Type Target State Host
              ------------------------------------------------------------
              TT_A...0.0.0.1 application OFFLINE OFFLINE
              TT_A...SSS6202 application ONLINE ONLINE sss6202
              TT_A...SSS6203 application ONLINE ONLINE sss6203
              TT_A...1_DMTT1 application OFFLINE OFFLINE
              TT_D...SSS6202 application OFFLINE OFFLINE
              TT_D...SSS6203 application OFFLINE OFFLINE
              TT_M...0.0.1_0 application OFFLINE OFFLINE
              TT_M...0.0.1_1 application OFFLINE OFFLINE
              TT_S...0.0.0.1 application OFFLINE OFFLINE
              ora....ER.lsnr ora....er.type ONLINE ONLINE sss6202
              ora....N1.lsnr ora....er.type ONLINE ONLINE sss6202
              ora.asm ora.asm.type OFFLINE OFFLINE
              ora.cvu ora.cvu.type ONLINE ONLINE sss6202
              ora.gsd ora.gsd.type OFFLINE OFFLINE
              ora....network ora....rk.type ONLINE ONLINE sss6202
              ora.oc4j ora.oc4j.type ONLINE ONLINE sss6202
              ora.ons ora.ons.type ONLINE ONLINE sss6202
              ora....ry.acfs ora....fs.type OFFLINE OFFLINE
              ora.scan1.vip ora....ip.type ONLINE ONLINE sss6202
              ora....SM1.asm application OFFLINE OFFLINE
              ora....02.lsnr application ONLINE ONLINE sss6202
              ora....202.gsd application OFFLINE OFFLINE
              ora....202.ons application ONLINE ONLINE sss6202
              ora....202.vip ora....t1.type ONLINE ONLINE sss6202
              ora....SM2.asm application OFFLINE OFFLINE
              ora....03.lsnr application ONLINE ONLINE sss6203
              ora....203.gsd application OFFLINE OFFLINE
              ora....203.ons application ONLINE ONLINE sss6203
              ora....203.vip ora....t1.type ONLINE ONLINE sss6203

              My concern is the grid status has changed from AVAILABLE to NO GRID causing the client application error, then subsequently the error in recreating the A/S pair.

              Many thanks!
              Thomas Cong
              • 4. Re: TimesTen Cache Grid Setup Issues on Clusterware
                Gennady Sigalaev
                Hi Thomas,

                For dropping configuration use these commands

                ttCWAdmin -stop -dns ST_0.0.0.1
                ttCWAdmin -stop -dns ST_0.1.0.1
                ttCWAdmin -drop -dns ST_0.0.0.1
                ttCWAdmin -drop -dns ST_0.1.0.1

                Before you create an AS pairs, can you create a GRID on local node (sss6203)?
                Could you post call ttGridInfo; output.

                regards,
                Gena

                Edited by: Gennady Sigalaev on 27-Apr-2012 12:07
                • 5. Re: TimesTen Cache Grid Setup Issues on Clusterware
                  Gennady Sigalaev
                  Tomas,

                  I forgot to write about this:
                  ora.asm ora.asm.type OFFLINE OFFLINE
                  ora....SM1.asm application OFFLINE OFFLINE
                  ora....SM2.asm application OFFLINE OFFLINE
                  Your ASM doesnt work. Have you placed voting disk and OCR on ASM?

                  regards,
                  Gena
                  • 6. Re: TimesTen Cache Grid Setup Issues on Clusterware
                    user8745679
                    Gena, thanks for your replies.

                    This issue has been resolved by completely stopping clusterware, ttdaemonadmin and unloading the DSN using ttAdmin -ramPolicy manual -ramUnload MYDSN. Then I was able to destroy the old DSN and setup a new one. The sequence of actions seems to be important. I am currently doing failover testing but still need to figure out the correct way to trigger failover.
                    • 7. Re: TimesTen Cache Grid Setup Issues on Clusterware
                      Gennady Sigalaev
                      Hi user8745679,

                      I'm glad that your configuration works fine now :)

                      I've tested the AS pair fuctionality and it's worked fine (http://www.ruoug.org/library/9/index.html). But it was 11.2.1 version. Now it is much difficult, especially with introduction of cache grid.

                      If you need to handle TimesTen failover on application you have two options:

                      - Synchronous detection of automatic client failover (error handling)
                      - Asynchronous detection of automatic client failover (client failover listener)

                      (http://docs.oracle.com/cd/E21901_01/doc/timesten.1122/e21638/writing_app.htm#BABDIJGC).


                      Best regards,
                      Gena