13 Replies Latest reply: May 1, 2013 2:10 PM by Orna Weisman RSS

    unable to disable a thread htat belongs to a long gone RAC node

    Orna Weisman
      I have a two RAC instance , that one upon a time used to have 3 nodes.
      thread#3 is marked as 'CLOSED' on v$thread.

      there are two log files associated with that thread.
      I want to disable that thread and get rid of these two log files ( this is needed as preparation to a switchover yo my dataguard )

      However , the logfiles are long gone , so when I try to disable the thread I get :

      SQL> ALTER DATABASE disable THREAD 3;
      ALTER DATABASE disable THREAD 3
      *
      ERROR at line 1:
      ORA-00313: open failed for members of log group 30 of thread 3
      ORA-00312: online log 30 thread 3: '/q01/oradata/timst/redo30a.log'
      ORA-17503: ksfdopn:4 Failed to open file /q01/oradata/timst/redo30a.log
      ORA-17500: ODM err:File does not exist


      How can I get rid of that thread and its redo log files ?

      version is 10.2.0.4 on SPARC Solaris
        • 1. Re: unable to disable a thread htat belongs to a long gone RAC node
          FreddieEssex
          Have you tried dropping the redo log groups that are part of this thread.
          • 2. Re: unable to disable a thread htat belongs to a long gone RAC node
            Orna Weisman
            yep.

            I had 3 logs.
            one was dropped the other two would not :

            SQL> select group#,status from v$log where thread# = 3;

            GROUP# STATUS
            ---------- ----------------
            30 ACTIVE
            33 INACTIVE

            SQL> ALTER DATABASE DROP LOGFILE GROUP 30;
            ALTER DATABASE DROP LOGFILE GROUP 30
            *
            ERROR at line 1:
            ORA-01623: log 30 is current log for instance timst3 (thread 3) - cannot drop
            ORA-00312: online log 30 thread 3: '/q01/oradata/timst/redo30a.log'


            SQL> ALTER DATABASE DROP LOGFILE GROUP 33;
            ALTER DATABASE DROP LOGFILE GROUP 33
            *
            ERROR at line 1:
            ORA-01567: dropping log 33 would leave less than 2 log files for instance
            timst3 (thread 3)
            ORA-00312: online log 33 thread 3: '/q01/oradata/timst/redo33a.log'


            SQL>
            • 3. Re: unable to disable a thread htat belongs to a long gone RAC node
              FreddieEssex
              so you're node 3 has long gone, but was it removed properly from the cluster configuration?
              olsnodes -n
              http://docs.oracle.com/cd/B19306_01/rac.102/b14197/adddelunix.htm
              • 4. Re: unable to disable a thread htat belongs to a long gone RAC node
                Orna Weisman
                oracle@racbur02:~:timst2:>$CRS_HOME/bin/olsnodes -n
                racbur01 0
                racbur02 1
                oracle@racbur02:~:timst2:>

                looks like it was removed correctly, though obviously something went wrong somewhere...
                • 5. Re: unable to disable a thread htat belongs to a long gone RAC node
                  FreddieEssex
                  Does cluvfy show anything??
                  cluvfy stage -post crsinst -n all -verbose
                  • 6. Re: unable to disable a thread htat belongs to a long gone RAC node
                    Mahir M. Quluzade
                    Hi,

                    Your thread 3 is using 30,33 log groups. Did you remove log files from OS manually?
                    If remove node normally, then thread 3 is must disable automatically.

                    Did you try restart database?

                    Mahir
                    • 7. Re: unable to disable a thread htat belongs to a long gone RAC node
                      Orna Weisman
                      no issues with cluster verify :

                      Performing post-checks for cluster services setup

                      Checking node reachability...

                      Check: Node reachability from node "racbur01"
                      Destination Node Reachable?
                      ------------------------------------ ------------------------
                      racbur01 yes
                      racbur02 yes
                      Result: Node reachability check passed from node "racbur01".


                      Checking user equivalence...

                      Check: User equivalence for user "oracle"
                      Node Name Comment
                      ------------------------------------ ------------------------
                      racbur02 passed
                      racbur01 passed
                      Result: User equivalence check passed for user "oracle".

                      Checking Cluster manager integrity...


                      Checking CSS daemon...
                      Node Name Status
                      ------------------------------------ ------------------------
                      racbur02 running
                      racbur01 running
                      Result: Daemon status check passed for "CSS daemon".

                      Cluster manager integrity check passed.

                      Checking cluster integrity...

                      Node Name
                      ------------------------------------
                      racbur01
                      racbur02

                      Cluster integrity check passed


                      Checking OCR integrity...

                      Checking the absence of a non-clustered configuration...
                      All nodes free of non-clustered, local-only configurations.

                      Uniqueness check for OCR device passed.

                      Checking the version of OCR...
                      OCR of correct Version "2" exists.

                      Checking data integrity of OCR...
                      Data integrity check for OCR passed.

                      OCR integrity check passed.

                      Checking CRS integrity...

                      Checking daemon liveness...

                      Check: Liveness for "CRS daemon"
                      Node Name Running
                      ------------------------------------ ------------------------
                      racbur02 yes
                      racbur01 yes
                      Result: Liveness check passed for "CRS daemon".

                      Checking daemon liveness...

                      Check: Liveness for "CSS daemon"
                      Node Name Running
                      ------------------------------------ ------------------------
                      racbur02 yes
                      racbur01 yes
                      Result: Liveness check passed for "CSS daemon".

                      Checking daemon liveness...

                      Check: Liveness for "EVM daemon"
                      Node Name Running
                      ------------------------------------ ------------------------
                      racbur02 yes
                      racbur01 yes
                      Result: Liveness check passed for "EVM daemon".

                      Liveness of all the daemons
                      Node Name CRS daemon CSS daemon EVM daemon
                      ------------ ------------------------ ------------------------ ----------
                      racbur02 yes yes yes
                      racbur01 yes yes yes

                      Checking CRS health...

                      Check: Health of CRS
                      Node Name CRS OK?
                      ------------------------------------ ------------------------
                      racbur02 yes
                      racbur01 yes
                      Result: CRS health check passed.

                      CRS integrity check passed.

                      Checking node application existence...


                      Checking existence of VIP node application
                      Node Name Required Status Comment
                      ------------ ------------------------ ------------------------ ----------
                      racbur02 yes exists passed
                      racbur01 yes exists passed
                      Result: Check passed.

                      Checking existence of ONS node application
                      Node Name Required Status Comment
                      ------------ ------------------------ ------------------------ ----------
                      racbur02 no exists passed
                      racbur01 no exists passed
                      Result: Check passed.

                      Checking existence of GSD node application
                      Node Name Required Status Comment
                      ------------ ------------------------ ------------------------ ----------
                      racbur02 no exists passed
                      racbur01 no exists passed
                      Result: Check passed.

                      Post-check for cluster services setup was successful.
                      • 8. Re: unable to disable a thread htat belongs to a long gone RAC node
                        Orna Weisman
                        Hi,

                        Your thread 3 is using 30,33 log groups. Did you remove log files from OS manually?

                        ==> not me , it was done before my time , but yes , the logfiles are long gone...

                        If remove node normally, then thread 3 is must disable automatically.

                        ==> it did not though

                        Did you try restart database?

                        ==> oh yes, hte database was restarted multiple times since the removal

                        Mahir
                        • 9. Re: unable to disable a thread htat belongs to a long gone RAC node
                          FreddieEssex
                          I guess it's time to open an SR then.
                          • 10. Re: unable to disable a thread htat belongs to a long gone RAC node
                            Orna Weisman
                            agreed :-)
                            I already opened one yesterday but these guys are not half as quick as you are ..

                            thanks for your help in any case
                            I will update this thread with any solution
                            • 11. Re: unable to disable a thread htat belongs to a long gone RAC node
                              FreddieEssex
                              I will update this thread with any solution
                              That would be good.....doesn't seem to happen very often on these forums.
                              • 12. Re: unable to disable a thread htat belongs to a long gone RAC node
                                Orna Weisman
                                I think that they will probably recommand to recreate the control file without thread #3 info and logfiles 30 and 33...

                                We'll see...
                                • 13. Re: unable to disable a thread htat belongs to a long gone RAC node
                                  Orna Weisman
                                  YESS !!

                                  that did the trick :

                                  alter database clear logfile group 30;
                                  alter database clear logfile group 33;
                                  ALTER DATABASE DROP LOGFILE GROUP 33;
                                  ALTER DATABASE DROP LOGFILE GROUP 30
                                  ALTER DATABASE disableTHREAD 3


                                  problem solved