This discussion is archived
11 Replies Latest reply: Dec 18, 2012 8:33 PM by GauravAhuja RSS

unable to start CRS after system reboot

poorna Newbie
Currently Being Moderated
Hi,

Oracle version :11.2.0.1.
Operating system:CentOs

After rebooting the server asm disk has disappeared for /dev/oraclasm/disks/OCR_VOTE04 which was mapped for VOTING disk after adding the same disk for OCR file.

Below is the error what i am getting.
[root@rac1 bin]# ./crsctl stat res -t
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.

[root@rac1 bin]# ./ocrcheck
PROT-602: Failed to retrieve data from the cluster registry
PROC-26: Error while accessing the physical storage ASM error [SLOS: cat=8, opn=kgfolclcpi1, dep=210, loc=kgfokge
AMDU-00210: No disks found in diskgroup OCR_VOTE_MIRROR
AMDU-00210: No disks found in diskgroup OCR_VOTE_MIRROR
] [8]

[root@rac1 bin]# ./ocrcheck  -config
Oracle Cluster Registry configuration is :
         Device/File Name         :  +OCR_VOTE
         Device/File Name         : +OCR_VOTE_MIRROR

[root@rac1 bin]# ./crsctl query css votedisk
Unable to communicate with the Cluster Synchronization Services daemon.
[root@rac1 bin]#
Here the same asm disk was mounted to +OCR_VOTE_MIRROR(/dev/oracleasm/disks/OCR_VOTE04) diskgroup in asm instance for adding another OCR File.
Please help me how to solve this problem.
  • 1. Re: unable to start CRS after system reboot
    poorna Newbie
    Currently Being Moderated
    Hi Gurus,

    Can any one please help me on this .

    Thanks & Regards,
    Poorna Prasad.S
  • 2. Re: unable to start CRS after system reboot
    585179 Expert
    Currently Being Moderated
    Hi,

    can you post the disk that map to /dev/oraclasm/disks/OCR_VOTE04?
    Cheers
  • 3. Re: unable to start CRS after system reboot
    poorna Newbie
    Currently Being Moderated
    Hi FZheng ,

    After rebooting both the nodes the disk ORA_VOTE04 did not appear in the path /dev/oracleasm/disks/

    Thanks & Regards,
    Poorna Prasad.S
  • 4. Re: unable to start CRS after system reboot
    585179 Expert
    Currently Being Moderated
    Hi,


    I saw your other thread, seems like your /dev/oracleasm/disks/ORA_VOTE04 map to /dev/emcpowera12 ? can you check if this /dev/emcpowera12 exists?

    and try run /etc/int.d/oracleasm querydisk /dev/emcpowera12


    Cheers
  • 5. Re: unable to start CRS after system reboot
    poorna Newbie
    Currently Being Moderated
    Hi FZheng ,


    Below is the out put for the command .
    [root@rac1 dev]# /etc/init.d/oracleasm querydisk /dev/emcpowera12
    Device "/dev/emcpowera12" is not marked as an ASM disk
    Thanks & Regards,
    Pooran Prasad.S

    Edited by: SIDDABATHUNI on Jun 20, 2011 10:07 AM
  • 6. Re: unable to start CRS after system reboot
    585179 Expert
    Currently Being Moderated
    Hi,

    That is the problem, your disk that map to VOTE04 is missing, get formatted/erased, etc


    Cheers
  • 7. Re: unable to start CRS after system reboot
    poorna Newbie
    Currently Being Moderated
    Hi FZheng ,

    I only dropped that disk and recreated after the disk was missed now there is no way to recover my clusterware because i have replaced my voting disk to that drive previously.

    Here i have to re-create RAC setup once again.

    Thanks & Regards,
    Pooran Prasad.S

    Edited by: SIDDABATHUNI on Jun 20, 2011 11:30 AM
  • 8. Re: unable to start CRS after system reboot
    Sebastian Solbach (DBA Community) Guru
    Currently Being Moderated
    Hi,

    if this is 11.2 you don't have to recreate the whole RAC setup.
    You should have an OCR backup and with this you are able to recreate OCR and Voting disks:

    http://download.oracle.com/docs/cd/E11882_01/rac.112/e16794/votocr.htm#CHDIICJB
    http://download.oracle.com/docs/cd/E11882_01/rac.112/e16794/votocr.htm#BGBCHCBI

    Regards
    Sebastian
  • 9. Re: unable to start CRS after system reboot
    poorna Newbie
    Currently Being Moderated
    Hi ssolbach ,

    I am getting the below error when restoring OCR file for the backup
    [root@rac1 bin]# ./ocrconfig -showbackup
    PROT-26: Oracle Cluster Registry backup locations were retrieved from a local copy
    
    rac1     2011/06/17 11:22:33     /u01/11.2.0/grid/cdata/racdb/backup00.ocr
    
    rac1     2011/06/17 07:22:33     /u01/11.2.0/grid/cdata/racdb/backup01.ocr
    
    rac1     2011/06/17 03:22:33     /u01/11.2.0/grid/cdata/racdb/backup02.ocr
    
    rac1     2011/06/16 11:22:30     /u01/11.2.0/grid/cdata/racdb/day.ocr
    
    rac1     2011/06/07 07:22:01     /u01/11.2.0/grid/cdata/racdb/week.ocr
    
    rac1     2011/06/16 13:01:14     /u01/11.2.0/grid/cdata/racdb/backup_20110616_130114.ocr
    
    rac1     2011/06/16 11:22:12     /u01/11.2.0/grid/cdata/racdb/backup_20110616_112212.ocr
    [root@rac1 bin]# ./ocrconfig -restore /u01/11.2.0/grid/cdata/racdb/day.ocr
    PROT-16: Internal Error
    [root@rac1 bin]#
    Please help me how to restore.

    Thanks & Regards,
    Poorna Prasad.S
  • 10. Re: unable to start CRS after system reboot
    poorna Newbie
    Currently Being Moderated
    Hi,

    Here i came to notice to find that in my node 1 /etc/oracle/ocr.loc file the mirror ocr disk was updated but in my node 2 file it was not updated.

    Node 1: /etc/oracle/ocr.loc
    [root@rac1 bin]# cat /etc/oracle/ocr.loc
    #Device/file  getting replaced by device +OCR_VOTE_MIRROR
    ocrconfig_loc=+OCR_VOTE
    #ocrmirrorconfig_loc=+OCR_VOTE_MIRROR
    local_only=false
    Node 2: /etc/oracle/ocr.loc
    [root@rac2 ~]#  cat /etc/oracle/ocr.loc
    ocrconfig_loc=+OCR_VOTE
    local_only=FALSE
    [root@rac2 ~]#
    Here when i try to start my cluster on both the nodes it gave me the following error on both the nodes

    Node 1:
    [root@rac1 bin]# ./ocrcheck
    Status of Oracle Cluster Registry is as follows :
             Version                  :          3
             Total space (kbytes)     :     262120
             Used space (kbytes)      :       3204
             Available space (kbytes) :     258916
             ID                       : 1090453280
             Device/File Name         :  +OCR_VOTE
                                        Device/File integrity check succeeded
    
                                        Device/File not configured
    
                                        Device/File not configured
    
                                        Device/File not configured
    
                                        Device/File not configured
    
             Cluster registry integrity check succeeded
    
             Logical corruption check succeeded
    
    [root@rac2 bin]# ./olsnodes
    PRCO-19: Failure retrieving list of nodes in the cluster
    PRCO-2: Unable to communicate with the clusterware
    
    [root@rac1 bin]# ./crsctl start crs
    CRS-4640: Oracle High Availability Services is already active
    CRS-4000: Command Start failed, or completed with errors.
    [root@rac1 bin]# ./olsnodes
    PRCO-19: Failure retrieving list of nodes in the cluster
    PRCO-2: Unable to communicate with the clusterware
    [root@rac1 bin]# ./crsctl start cluster
    
    
    CRS-5702: Resource 'ora.evmd' is already running on 'rac1'
    CRS-4000: Command Start failed, or completed with errors.
    [root@rac1 bin]#
    [root@rac1 bin]#
    [root@rac1 bin]# ./crsctl query css votedisk
    Unable to communicate with the Cluster Synchronization Services daemon.
    [root@rac1 bin]# ./crsctl start cluster
    CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
    CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'
    CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded
    
    
    
    
    
    CRS-2674: Start of 'ora.cssd' on 'rac1' failed
    CRS-2679: Attempting to clean 'ora.cssd' on 'rac1'
    CRS-2681: Clean of 'ora.cssd' on 'rac1' succeeded
    CRS-4000: Command Start failed, or completed with errors.
    [root@rac1 bin]#
    [root@rac1 bin]#
    [root@rac1 bin]#
    [root@rac1 bin]#
    [root@rac1 bin]#
    [root@rac1 bin]# ./crsctl query css votedisk
    Unable to communicate with the Cluster Synchronization Services daemon.
    [root@rac1 bin]# cat /etc/oracle/ocr.loc
    #Device/file  getting replaced by device +OCR_VOTE_MIRROR
    ocrconfig_loc=+OCR_VOTE
    #ocrmirrorconfig_loc=+OCR_VOTE_MIRROR
    local_only=false
    [root@rac1 bin]#
    Node:2
    [root@rac2 bin]# ./ocrcheck
    Status of Oracle Cluster Registry is as follows :
             Version                  :          3
             Total space (kbytes)     :     262120
             Used space (kbytes)      :       3204
             Available space (kbytes) :     258916
             ID                       : 1090453280
             Device/File Name         :  +OCR_VOTE
                                        Device/File integrity check succeeded
    
                                        Device/File not configured
    
                                        Device/File not configured
    
                                        Device/File not configured
    
                                        Device/File not configured
    
             Cluster registry integrity check succeeded
    
             Logical corruption check succeeded
    
    [root@rac2 bin]# ./crsctl stat res -t
    CRS-4535: Cannot communicate with Cluster Ready Services
    CRS-4000: Command Status failed, or completed with errors.
    [root@rac2 bin]# ./crsctl start crs
    CRS-4640: Oracle High Availability Services is already active
    CRS-4000: Command Start failed, or completed with errors.
    [root@rac2 bin]# ./crsctl start cluster
    CRS-2672: Attempting to start 'ora.cssd' on 'rac2'
    CRS-2672: Attempting to start 'ora.diskmon' on 'rac2'
    CRS-2676: Start of 'ora.diskmon' on 'rac2' succeeded
    CRS-2674: Start of 'ora.cssd' on 'rac2' failed
    CRS-2679: Attempting to clean 'ora.cssd' on 'rac2'
    CRS-2681: Clean of 'ora.cssd' on 'rac2' succeeded
    CRS-4000: Command Start failed, or completed with errors.
    [root@rac2 bin]#
    Please help me how to proceed to start my cluster ware with out any problems.

    Thanks & Regards,
    Pooran Prasad.S
  • 11. Re: unable to start CRS after system reboot
    GauravAhuja Newbie
    Currently Being Moderated
    Hi,


    There is an article on MOS to recreate the ocr and votedisk group from scratch if existing Diskgroup is Unavailable.

    Try to read that article, Hope will helps you.


    Regards
    Gaurav

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points