5 Replies Latest reply on Aug 4, 2011 9:14 PM by debrucer1

    Unable to find ASM DISKS after nodes restart for EMC powerpath

    poorna
      Hi,

      Oracle Version:11gR2
      Operating System:Cent Os

      we are unable to start the CRS after rebooting the server and finally came to know that ASM DISKS are not visible after nodes reboot here we are using power path for storage accessing what setup do we need to put in the /etc/sysconfig/oracleasm

      here is the content of my /etc/sysconfig/oracleasm
      [root@rac2 disks]# cat /etc/sysconfig/oracleasm
      #
      # This is a configuration file for automatic loading of the Oracle
      # Automatic Storage Management library kernel driver.  It is generated
      # By running /etc/init.d/oracleasm configure.  Please use that method
      # to modify this file
      #
      
      # ORACLEASM_ENABELED: 'true' means to load the driver on boot.
      ORACLEASM_ENABLED=true
      
      # ORACLEASM_UID: Default user owning the /dev/oracleasm mount point.
      ORACLEASM_UID=grid
      
      # ORACLEASM_GID: Default group owning the /dev/oracleasm mount point.
      ORACLEASM_GID=asmdba
      
      # ORACLEASM_SCANBOOT: 'true' means scan for ASM disks on boot.
      ORACLEASM_SCANBOOT=true
      
      # ORACLEASM_SCANORDER: Matching patterns to order disk scanning
      ORACLEASM_SCANORDER=""
      
      # ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan
      ORACLEASM_SCANEXCLUDE=""
      Please help me how to solve this problem.

      Thanks & Regards,
      Poorna Prasad.S
        • 1. Re: Unable to find ASM DISKS after nodes restart for EMC powerpath
          Sebastian Solbach -Database Community-Oracle
          Hi,

          emc powerpath should actually be up, when the cluster stack starts ASM, because that is very late in stack...

          2 questions, that I can give you the correct answer:
          a.) Does your cluster starts up successfully, if you disable the startup stack (crsctl disable crs) and start it manually after waiting some minutes (crsctl start crs)?
          b.) Does asmlib see the disks after startup (oracleasm listdisks)? Or do you have to do a scandisks that they are discovered?

          However what you should always set is the
          ORACLEASM_SCANORDER="" and ORACLEASM_SCANEXCLUDE=""
          that the correct devices are scanned.

          I am not sure, how powerpath presents the devices. But put these ones into the SCANORDER, and put the standard device names (like /dev/sd*) into the SCANEXCLUDE string:

          ORACLEASM_SCANORDER="XXX"
          ORACLEASM_SCANEXCLUDE="SD"

          Regards
          Sebastian
          • 2. Re: Unable to find ASM DISKS after nodes restart for EMC powerpath
            poorna
            Hi ssolbach ,

            Here is the output for your first question it is throwing errors
            [root@rac1 bin]# ./crsctl disable crs
            CRS-4621: Oracle High Availability Services autostart is disabled.
            [root@rac1 bin]# ./crsctl start crs
            CRS-4640: Oracle High Availability Services is already active
            CRS-4000: Command Start failed, or completed with errors.
            Even it is showing nothing for the command oracleasm listdiskd and in the path /dev/oracleasm/disks/ there it is showing empty folder
            and not showing and ASMDISK which we created yesterday.

            here is the output for cluvfy command
            [grid@rac1 bin]$ ./cluvfy stage -post crsinst -n  rac1,rac2 -verbose
            
            Performing post-checks for cluster services setup
            
            Checking node reachability...
            
            Check: Node reachability from node "rac1"
              Destination Node                      Reachable?
              ------------------------------------  ------------------------
              rac2                                  yes
              rac1                                  yes
            Result: Node reachability check passed from node "rac1"
            
            
            Checking user equivalence...
            
            Check: User equivalence for user "grid"
              Node Name                             Comment
              ------------------------------------  ------------------------
              rac2                                  passed
              rac1                                  passed
            Result: User equivalence check passed for user "grid"
            Checking time zone consistency...
            Time zone consistency check passed.
            
            ERROR:
            Cluster manager integrity check failed
            PRVF-5434 : Cannot identify the current CRS software version
            
            UDev attributes check for OCR locations started...
            Result: UDev attributes check passed for OCR locations
            
            
            UDev attributes check for Voting Disk locations started...
            
            ERROR:
            PRVF-5197 : Failed to retrieve voting disk locations
            Result: UDev attributes check failed for Voting Disk locations
            
            
            Check default user file creation mask
              Node Name     Available                 Required                  Comment
              ------------  ------------------------  ------------------------  ----------
              rac2          0022                      0022                      passed
              rac1          0022                      0022                      passed
            Result: Default user file creation mask check passed
            
            Checking cluster integrity...
            
            
            Cluster integrity check failed This check did not run on the following node(s):
                    rac2,rac1
            
            
            Checking OCR integrity...
            
            Checking the absence of a non-clustered configuration...
            All nodes free of non-clustered, local-only configurations
            
            
            ERROR:
            PRVF-5300 : Failed to retrieve active version for CRS on this node
            
            OCR integrity check failed
            
            Checking CRS integrity...
            
            ERROR:
            PRVF-5300 : Failed to retrieve active version for CRS on this node
            
            CRS integrity check failed
            OCR detected on ASM. Running ACFS Integrity checks...
            
            Starting check to see if ASM is running on all cluster nodes...
            PRVF-5137 : Failure while checking ASM status on node "rac2"
            PRVF-5137 : Failure while checking ASM status on node "rac1"
            
            Starting Disk Groups check to see if at least one Disk Group configured...
            PRVF-5112 : An Exception occurred while checking for Disk Groups
            PRVF-5114 : Disk Group check failed. No Disk Groups configured
            
            Task ACFS Integrity check failed
            
            Checking Oracle Cluster Voting Disk configuration...
            
            ERROR:
            PRVF-5434 : Cannot identify the current CRS software version
            
            PRVF-5431 : Oracle Cluster Voting Disk configuration check failed
            
            Checking to make sure user "grid" is not in "root" group
              Node Name     Status                    Comment
              ------------  ------------------------  ------------------------
              rac2          does not exist            passed
              rac1          does not exist            passed
            Result: User "grid" is not part of "root" group. Check passed
            
            Post-check for cluster services setup was unsuccessful on all the nodes.
            Thanks & Regards,
            Poorna Prasad.S

            Edited by: SIDDABATHUNI on Apr 29, 2011 1:15 PM
            • 3. Re: Unable to find ASM DISKS after nodes restart for EMC powerpath
              Sebastian Solbach -Database Community-Oracle
              Hi,

              after you disabled crs, please restart the server, that I can see the correct errors.
              Because it could be that there is a timing issue with powerpath/oraclasmlib and crs clusterstack.

              Regards
              Sebastian
              • 4. Re: Unable to find ASM DISKS after nodes restart for EMC powerpath
                poorna
                Hi ssolbach ,

                After disabling the CRS i restated the node and again try to restart the CRS and it is giving like this
                [root@rac1 bin]# ./crsctl start crs
                CRS-4123: Oracle High Availability Services has been started.
                [root@rac1 bin]#
                But ASM disks are not showing in /dev/oracleasm/disks/

                Regards,
                Poorna Prasad.S
                • 5. Re: Unable to find ASM DISKS after nodes restart for EMC powerpath
                  debrucer1
                  It would be helpful to know how you resolved this. When using EMC (Symmetric) and raw devices the oracleasm components is not required. Our installation used it for the voting disk only... the other 230 LUNs in the SAN are mounted by EMC (possibly using UDEV, not sure). The /dev/oracleasm/disks folder is also empty.