10 Replies Latest reply on Aug 23, 2010 9:55 AM by Riaz

    Shared Storage Check

    Riaz
      Hi all,

      We are planning to add a node to our existing RAC deployment (Database: 10gr2 and Sun Solaris 5.9 OS). Currently the shared storage is IBM SAN.

      When i run shared storage check using cluvfy, it fails to detect any shared storage. Given that i can ignore this error message (since cluvfy doesn't work wth SAN i beleive), how can i check whether the storage is shared or not?

      Note
      When i see partition table from both servers, it looks same (for the SAN drive, of course) but the name/label of the storages are different (For example: In existing node it show c6t0d0 but in the new node, which is to be added, it shows something different. Is it ok?).

      regards,

      Muhammad Riaz
        • 1. Re: Shared Storage Check
          736258
          runcluvfy.sh stage –post hwos –n host1,host2 -verbose
          • 2. Re: Shared Storage Check
            Riaz
            I get below error message:
            Shared storage check failed on nodes "node1,node2".
            
            Verification of shared storage accessibility was unsuccessful on all the nodes
            • 3. Re: Shared Storage Check
              Sebastian Solbach -Database Community-Oracle
              Hi Riaz,
              shared storage check is pretty limited:
              MySupport Note 316817.1: CLUSTER VERIFICATION UTILITY (CLUVFY) FAQ

              Easiest way to check sharedness would be (on an empty device) to DD as oracle/grid user to the device and read this with dd from the other node:

              Node1: dd if=<anytextdocument> of=<device> bs=1M count=10
              Node2: dd if=<device> of=/tmp/text.txt bs=1M count=10

              If the contents can be successfully retrieved, then the device is shared.
              Please don't do this with a device with data on!

              You could also check if the diskheader at least is recognized, whith simply reading the header contents with dd...

              Regards
              Sebastian
              1 person found this helpful
              • 4. Re: Shared Storage Check
                733204
                Thanks for input. Disk is raw, how can i check the sahredness?
                • 5. Re: Shared Storage Check
                  Riaz
                  The above post is by me. I posted it while logged in at metalink; hence OTN used the same user :)
                  • 6. Re: Shared Storage Check
                    rajeysh
                    you can use metalink username for OTN Forum also.

                    check the notes in metalink: hope this will helps you.
                    Shared disk check with the Cluster Verification Utility [ID 372358.1]

                    5852975 SHARED STORAGE CHECK DOESN'T DISCOVER SHARED DISKS

                    How To check that OS storage device files pointed to the same storage in both nodes [ID
                    821588.1]

                    check:
                    $ cluvfy comp ssa -n rac1,rac2

                    rajesh
                    http://oracleinstance.blogspot.com
                    1 person found this helpful
                    • 7. Re: Shared Storage Check
                      Sebastian Solbach -Database Community-Oracle
                      Hi,

                      since you already logged in to mysupport, see here:

                      Note 372358.1: Shared disk check with the Cluster Verification Utility

                      which describes how cluvfy tries to do it, and how you can do it by hand.

                      However your disks (rdsk) (especially for OCR and Voting) should be the same, otherwise the cluster has problems identifying the disks.
                      It does not matter so much for ASM....

                      Regards

                      Sebastian
                      1 person found this helpful
                      • 8. Re: Shared Storage Check
                        692600
                        When you add a node, what you'll be doing is extending the clusterware and oracle homes from one of the already existing nodes using the script addNode.sh. So it is important that the OCR and voting disks are accessible using the same path/soft link (which ever you are using) from the new node and that they have the right permissions.

                        It would suffice to confirm if the disks are shared using the dd command from the new node. for ex:-
                        dd if=/dev/rdsk/c1t500A098286E80F61d9s0 of=/dev/null bs=1048576 count=100

                        For ASM disks, you can afford to have different path/links for the disks, since once you have extended your ASM home to the new node, you can change the asm_diskstring in the spfile on new node and it'll scan and figure out the disks.

                        Don't worry much about cluvfy, it's not the smartest utility anyways.

                        Cheers.
                        • 9. Re: Shared Storage Check
                          Riaz
                          Dear All,

                          Thanks for the useful notes. After checking metalink note for cluvfy, i found the reason why it is not able to get the shared disk check passed: It can't get the serial# of SAN disk.

                          Could you guys please tell me how the method of using dd command works? I tried with different disks and everytime i get the same output:

                          0+1 records in
                          0+1 records out

                          regards,

                          Muhammamd Riaz
                          • 10. Re: Shared Storage Check
                            Riaz
                            Never mind. I found solution from http://www.idevelopment.info.

                            (1) Create following directory structure on second node (same as first node) with the same permissions on existins node:

                            /asmdisks

                            - crs

                            -disk1

                            -disk2

                            - vote

                            (2) use ls -lL /dev/rdsk/<Disk> to find out major and minor ids of shared disk and attach those ids to relveant direcotries above using mknod command:
                            # ls -lL /dev/rdsk/c4t0d0*
                            crw-r-----   1 root     sys       32,256 Aug  1 11:16 /dev/rdsk/c4t0d0s0
                            crw-r-----   1 root     sys       32,257 Aug  1 11:16 /dev/rdsk/c4t0d0s1
                            crw-r-----   1 root     sys       32,258 Aug  1 11:16 /dev/rdsk/c4t0d0s2
                            crw-r-----   1 root     sys       32,259 Aug  1 11:16 /dev/rdsk/c4t0d0s3
                            crw-r-----   1 root     sys       32,260 Aug  1 11:16 /dev/rdsk/c4t0d0s4
                            crw-r-----   1 root     sys       32,261 Aug  1 11:16 /dev/rdsk/c4t0d0s5
                            crw-r-----   1 root     sys       32,262 Aug  1 11:16 /dev/rdsk/c4t0d0s6
                            crw-r-----   1 root     sys       32,263 Aug  1 11:16 /dev/rdsk/c4t0d0s7
                            
                            mknod /asmdisks/crs      c 32 257
                            mknod /asmdisks/disk1      c 32 260
                            mknod /asmdisks/disk2      c 32 261
                            mknod /asmdisks/vote      c 32 259
                            
                            # ls -lL /asmdisks
                            total 0
                            crw-r--r--   1 root     oinstall  32,257 Aug  3 09:07 crs
                            crw-r--r--   1 oracle   dba       32,260 Aug  3 09:08 disk1
                            crw-r--r--   1 oracle   dba       32,261 Aug  3 09:08 disk2
                            crw-r--r--   1 oracle   oinstall  32,259 Aug  3 09:08 vote