11 Replies Latest reply: Dec 18, 2012 10:33 PM by GauravAhuja RSS

    unable to start CRS after system reboot

    poorna
      Hi,

      Oracle version :11.2.0.1.
      Operating system:CentOs

      After rebooting the server asm disk has disappeared for /dev/oraclasm/disks/OCR_VOTE04 which was mapped for VOTING disk after adding the same disk for OCR file.

      Below is the error what i am getting.
      [root@rac1 bin]# ./crsctl stat res -t
      CRS-4535: Cannot communicate with Cluster Ready Services
      CRS-4000: Command Status failed, or completed with errors.
      
      [root@rac1 bin]# ./ocrcheck
      PROT-602: Failed to retrieve data from the cluster registry
      PROC-26: Error while accessing the physical storage ASM error [SLOS: cat=8, opn=kgfolclcpi1, dep=210, loc=kgfokge
      AMDU-00210: No disks found in diskgroup OCR_VOTE_MIRROR
      AMDU-00210: No disks found in diskgroup OCR_VOTE_MIRROR
      ] [8]
      
      [root@rac1 bin]# ./ocrcheck  -config
      Oracle Cluster Registry configuration is :
               Device/File Name         :  +OCR_VOTE
               Device/File Name         : +OCR_VOTE_MIRROR
      
      [root@rac1 bin]# ./crsctl query css votedisk
      Unable to communicate with the Cluster Synchronization Services daemon.
      [root@rac1 bin]#
      Here the same asm disk was mounted to +OCR_VOTE_MIRROR(/dev/oracleasm/disks/OCR_VOTE04) diskgroup in asm instance for adding another OCR File.
      Please help me how to solve this problem.
        • 1. Re: unable to start CRS after system reboot
          poorna
          Hi Gurus,

          Can any one please help me on this .

          Thanks & Regards,
          Poorna Prasad.S
          • 2. Re: unable to start CRS after system reboot
            585179
            Hi,

            can you post the disk that map to /dev/oraclasm/disks/OCR_VOTE04?
            Cheers
            • 3. Re: unable to start CRS after system reboot
              poorna
              Hi FZheng ,

              After rebooting both the nodes the disk ORA_VOTE04 did not appear in the path /dev/oracleasm/disks/

              Thanks & Regards,
              Poorna Prasad.S
              • 4. Re: unable to start CRS after system reboot
                585179
                Hi,


                I saw your other thread, seems like your /dev/oracleasm/disks/ORA_VOTE04 map to /dev/emcpowera12 ? can you check if this /dev/emcpowera12 exists?

                and try run /etc/int.d/oracleasm querydisk /dev/emcpowera12


                Cheers
                • 5. Re: unable to start CRS after system reboot
                  poorna
                  Hi FZheng ,


                  Below is the out put for the command .
                  [root@rac1 dev]# /etc/init.d/oracleasm querydisk /dev/emcpowera12
                  Device "/dev/emcpowera12" is not marked as an ASM disk
                  Thanks & Regards,
                  Pooran Prasad.S

                  Edited by: SIDDABATHUNI on Jun 20, 2011 10:07 AM
                  • 6. Re: unable to start CRS after system reboot
                    585179
                    Hi,

                    That is the problem, your disk that map to VOTE04 is missing, get formatted/erased, etc


                    Cheers
                    • 7. Re: unable to start CRS after system reboot
                      poorna
                      Hi FZheng ,

                      I only dropped that disk and recreated after the disk was missed now there is no way to recover my clusterware because i have replaced my voting disk to that drive previously.

                      Here i have to re-create RAC setup once again.

                      Thanks & Regards,
                      Pooran Prasad.S

                      Edited by: SIDDABATHUNI on Jun 20, 2011 11:30 AM
                      • 8. Re: unable to start CRS after system reboot
                        Sebastian Solbach -Dba Community-Oracle
                        Hi,

                        if this is 11.2 you don't have to recreate the whole RAC setup.
                        You should have an OCR backup and with this you are able to recreate OCR and Voting disks:

                        http://download.oracle.com/docs/cd/E11882_01/rac.112/e16794/votocr.htm#CHDIICJB
                        http://download.oracle.com/docs/cd/E11882_01/rac.112/e16794/votocr.htm#BGBCHCBI

                        Regards
                        Sebastian
                        • 9. Re: unable to start CRS after system reboot
                          poorna
                          Hi ssolbach ,

                          I am getting the below error when restoring OCR file for the backup
                          [root@rac1 bin]# ./ocrconfig -showbackup
                          PROT-26: Oracle Cluster Registry backup locations were retrieved from a local copy
                          
                          rac1     2011/06/17 11:22:33     /u01/11.2.0/grid/cdata/racdb/backup00.ocr
                          
                          rac1     2011/06/17 07:22:33     /u01/11.2.0/grid/cdata/racdb/backup01.ocr
                          
                          rac1     2011/06/17 03:22:33     /u01/11.2.0/grid/cdata/racdb/backup02.ocr
                          
                          rac1     2011/06/16 11:22:30     /u01/11.2.0/grid/cdata/racdb/day.ocr
                          
                          rac1     2011/06/07 07:22:01     /u01/11.2.0/grid/cdata/racdb/week.ocr
                          
                          rac1     2011/06/16 13:01:14     /u01/11.2.0/grid/cdata/racdb/backup_20110616_130114.ocr
                          
                          rac1     2011/06/16 11:22:12     /u01/11.2.0/grid/cdata/racdb/backup_20110616_112212.ocr
                          [root@rac1 bin]# ./ocrconfig -restore /u01/11.2.0/grid/cdata/racdb/day.ocr
                          PROT-16: Internal Error
                          [root@rac1 bin]#
                          Please help me how to restore.

                          Thanks & Regards,
                          Poorna Prasad.S
                          • 10. Re: unable to start CRS after system reboot
                            poorna
                            Hi,

                            Here i came to notice to find that in my node 1 /etc/oracle/ocr.loc file the mirror ocr disk was updated but in my node 2 file it was not updated.

                            Node 1: /etc/oracle/ocr.loc
                            [root@rac1 bin]# cat /etc/oracle/ocr.loc
                            #Device/file  getting replaced by device +OCR_VOTE_MIRROR
                            ocrconfig_loc=+OCR_VOTE
                            #ocrmirrorconfig_loc=+OCR_VOTE_MIRROR
                            local_only=false
                            Node 2: /etc/oracle/ocr.loc
                            [root@rac2 ~]#  cat /etc/oracle/ocr.loc
                            ocrconfig_loc=+OCR_VOTE
                            local_only=FALSE
                            [root@rac2 ~]#
                            Here when i try to start my cluster on both the nodes it gave me the following error on both the nodes

                            Node 1:
                            [root@rac1 bin]# ./ocrcheck
                            Status of Oracle Cluster Registry is as follows :
                                     Version                  :          3
                                     Total space (kbytes)     :     262120
                                     Used space (kbytes)      :       3204
                                     Available space (kbytes) :     258916
                                     ID                       : 1090453280
                                     Device/File Name         :  +OCR_VOTE
                                                                Device/File integrity check succeeded
                            
                                                                Device/File not configured
                            
                                                                Device/File not configured
                            
                                                                Device/File not configured
                            
                                                                Device/File not configured
                            
                                     Cluster registry integrity check succeeded
                            
                                     Logical corruption check succeeded
                            
                            [root@rac2 bin]# ./olsnodes
                            PRCO-19: Failure retrieving list of nodes in the cluster
                            PRCO-2: Unable to communicate with the clusterware
                            
                            [root@rac1 bin]# ./crsctl start crs
                            CRS-4640: Oracle High Availability Services is already active
                            CRS-4000: Command Start failed, or completed with errors.
                            [root@rac1 bin]# ./olsnodes
                            PRCO-19: Failure retrieving list of nodes in the cluster
                            PRCO-2: Unable to communicate with the clusterware
                            [root@rac1 bin]# ./crsctl start cluster
                            
                            
                            CRS-5702: Resource 'ora.evmd' is already running on 'rac1'
                            CRS-4000: Command Start failed, or completed with errors.
                            [root@rac1 bin]#
                            [root@rac1 bin]#
                            [root@rac1 bin]# ./crsctl query css votedisk
                            Unable to communicate with the Cluster Synchronization Services daemon.
                            [root@rac1 bin]# ./crsctl start cluster
                            CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
                            CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'
                            CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded
                            
                            
                            
                            
                            
                            CRS-2674: Start of 'ora.cssd' on 'rac1' failed
                            CRS-2679: Attempting to clean 'ora.cssd' on 'rac1'
                            CRS-2681: Clean of 'ora.cssd' on 'rac1' succeeded
                            CRS-4000: Command Start failed, or completed with errors.
                            [root@rac1 bin]#
                            [root@rac1 bin]#
                            [root@rac1 bin]#
                            [root@rac1 bin]#
                            [root@rac1 bin]#
                            [root@rac1 bin]# ./crsctl query css votedisk
                            Unable to communicate with the Cluster Synchronization Services daemon.
                            [root@rac1 bin]# cat /etc/oracle/ocr.loc
                            #Device/file  getting replaced by device +OCR_VOTE_MIRROR
                            ocrconfig_loc=+OCR_VOTE
                            #ocrmirrorconfig_loc=+OCR_VOTE_MIRROR
                            local_only=false
                            [root@rac1 bin]#
                            Node:2
                            [root@rac2 bin]# ./ocrcheck
                            Status of Oracle Cluster Registry is as follows :
                                     Version                  :          3
                                     Total space (kbytes)     :     262120
                                     Used space (kbytes)      :       3204
                                     Available space (kbytes) :     258916
                                     ID                       : 1090453280
                                     Device/File Name         :  +OCR_VOTE
                                                                Device/File integrity check succeeded
                            
                                                                Device/File not configured
                            
                                                                Device/File not configured
                            
                                                                Device/File not configured
                            
                                                                Device/File not configured
                            
                                     Cluster registry integrity check succeeded
                            
                                     Logical corruption check succeeded
                            
                            [root@rac2 bin]# ./crsctl stat res -t
                            CRS-4535: Cannot communicate with Cluster Ready Services
                            CRS-4000: Command Status failed, or completed with errors.
                            [root@rac2 bin]# ./crsctl start crs
                            CRS-4640: Oracle High Availability Services is already active
                            CRS-4000: Command Start failed, or completed with errors.
                            [root@rac2 bin]# ./crsctl start cluster
                            CRS-2672: Attempting to start 'ora.cssd' on 'rac2'
                            CRS-2672: Attempting to start 'ora.diskmon' on 'rac2'
                            CRS-2676: Start of 'ora.diskmon' on 'rac2' succeeded
                            CRS-2674: Start of 'ora.cssd' on 'rac2' failed
                            CRS-2679: Attempting to clean 'ora.cssd' on 'rac2'
                            CRS-2681: Clean of 'ora.cssd' on 'rac2' succeeded
                            CRS-4000: Command Start failed, or completed with errors.
                            [root@rac2 bin]#
                            Please help me how to proceed to start my cluster ware with out any problems.

                            Thanks & Regards,
                            Pooran Prasad.S
                            • 11. Re: unable to start CRS after system reboot
                              GauravAhuja
                              Hi,


                              There is an article on MOS to recreate the ocr and votedisk group from scratch if existing Diskgroup is Unavailable.

                              Try to read that article, Hope will helps you.


                              Regards
                              Gaurav