1 2 Previous Next 23 Replies Latest reply: Oct 25, 2012 12:40 AM by 970402 RSS

    OUI cannot list candidate disks for ASM Disk Group during Grid Infra instln

    user-1221
      Hi,

      We are trying to install 2 node Oracle RAC 11gR2 on HP-UX 11.31, all the prerequisites are done, the CVU test is successfull for both the nodes, except the "-x" slewing option for NTP on node 2.

      During ASM Disk Group creation, the OUI cannot see the disks;

      provided grid:asmadmin ownership on the disks with 660 file permissions, even created the similar names for the shared disks on the both nodes.
      _Node 1:_
      
      */dev/rdisk*
      crw-rw----   1 grid       asmadmin    22 0x000010 Jul  9 10:33 ocr1
      crw-rw----   1 grid       asmadmin    22 0x000003 Jul  9 10:33 ocr2
      crw-rw----   1 grid       asmadmin    22 0x00000d Jul  9 10:33 ocr3
      crw-rw----   1 grid       asmadmin    22 0x00000e Jul  9 10:33 vote1
      crw-rw----   1 grid       asmadmin    22 0x00000c Jul  9 10:33 vote2
      crw-rw----   1 grid       asmadmin    22 0x00000b Jul  9 10:33 vote3
      crw-rw----   1 grid       asmadmin    22 0x00000f Jul  9 10:33 data1
      crw-rw----   1 grid       asmadmin    22 0x000002 Jul  9 10:33 arch1
      
      */dev/rdsk*
      crw-rw----   1 grid       asmadmin   188 0x030200 Jul  9 09:21 ocr1
      crw-rw----   1 grid       asmadmin   188 0x1b0200 Jul  9 09:21 ocr2
      crw-rw----   1 grid       asmadmin   188 0x2d0100 Jul  9 09:21 ocr3
      crw-rw----   1 grid       asmadmin   188 0x2f0200 Jul  9 09:21 vote1
      crw-rw----   1 grid       asmadmin   188 0x110300 Jul  9 09:21 vote2
      crw-rw----   1 grid       asmadmin   188 0x0b0300 Jul  9 09:21 vote3
      crw-rw----   1 grid       asmadmin   188 0x150100 Jul  9 09:21 data1
      crw-rw----   1 grid       asmadmin   188 0x030100 Jul  9 09:21 arch1
      
      _Node 2:_
      
      */dev/rdisk*
      crw-rw----   1 grid       asmadmin    22 0x000014 Jul  9 10:33 ocr2
      crw-rw----   1 grid       asmadmin    22 0x000012 Jul  9 10:33 ocr1
      crw-rw----   1 grid       asmadmin    22 0x00000e Jul  9 10:33 ocr3
      crw-rw----   1 grid       asmadmin    22 0x000010 Jul  9 10:33 vote1
      crw-rw----   1 grid       asmadmin    22 0x000011 Jul  9 10:33 vote2
      crw-rw----   1 grid       asmadmin    22 0x000013 Jul  9 10:33 vote3
      crw-rw----   1 grid       asmadmin    22 0x00000c Jul  9 10:33 arch1
      crw-rw----   1 grid       asmadmin    22 0x00000f Jul  9 10:33 data1
      
      */dev/rdsk/*
      crw-rw----   1 grid       asmadmin   188 0x1a0200 Jul  9 09:28 ocr1
      crw-rw----   1 grid       asmadmin   188 0x1c0200 Jul  9 09:28 ocr2
      crw-rw----   1 grid       asmadmin   188 0x2f0100 Jul  9 09:28 ocr3
      crw-rw----   1 grid       asmadmin   188 0x180300 Jul  9 09:28 vote1
      crw-rw----   1 grid       asmadmin   188 0x1c0300 Jul  9 09:28 vote2
      crw-rw----   1 grid       asmadmin   188 0x2d0200 Jul  9 09:28 vote3
      crw-rw----   1 grid       asmadmin   188 0x220100 Jul  9 09:29 arch1
      crw-rw----   1 grid       asmadmin   188 0x180100 Jul  9 09:29 data1
      
      Even clean the disk header with dd...
      dd if=/dev/zero of=/dev/rdisk/ocr1 bs=8192 count=10240
      dd if=/dev/zero of=/dev/rdisk/ocr2 bs=8192 count=10240
      dd if=/dev/zero of=/dev/rdisk/ocr3 bs=8192 count=10240
      dd if=/dev/zero of=/dev/rdisk/vote1 bs=8192 count=10240
      dd if=/dev/zero of=/dev/rdisk/vote2 bs=8192 count=10240
      dd if=/dev/zero of=/dev/rdisk/vote3 bs=8192 count=10240
      dd if=/dev/zero of=/dev/rdisk/arch1 bs=8192 count=10240
      dd if=/dev/zero of=/dev/rdisk/data1 bs=8192 count=10240
      Node1_

      root@node1:/dev/rdisk# id grid
      uid=1100(grid) gid=1000(oinstall) groups=1020(asmadmin),1021(asmdba)
      root@node1:/dev/rdisk# id oracle
      uid=1101(oracle) gid=1000(oinstall) groups=1021(asmdba),1031(dba),1032(oper)


      Node2_

      root@node2:/dev/rdsk# id grid
      uid=1100(grid) gid=1000(oinstall) groups=1020(asmadmin),1021(asmdba)
      root@node2:/dev/rdsk# id oracle
      uid=1101(oracle) gid=1000(oinstall) groups=1031(dba),503(oper),1021(asmdba)

      tried to list the disks with both the Disk Discovery Paths, "/dev/rdsk/*" and "/dev/rdisk/*" but still the disks are not shown in the OUI Utility.

      Please let me know if I am missing something.

      Thanks in Advance!
        • 1. Re: OUI cannot list candidate disks for ASM Disk Group during Grid Infra instln
          585179
          HI,


          Are you using LVM disk device? if so can you please post

          #> ls -l /dev/*disk/ocr1


          Cheers
          • 2. Re: OUI cannot list candidate disks for ASM Disk Group during Grid Infra instln
            user-1221
            Hi,

            Thanks for your reply;

            NODE1:

            root@node1:/dev/rdisk# ls -l /dev/*disk/ocr*
            crw-rw---- 1 grid asmadmin 22 0x00000d Jul 9 10:33 /dev/rdisk/ocr1
            crw-rw---- 1 grid asmadmin 22 0x000010 Jul 9 10:33 /dev/rdisk/ocr2
            crw-rw---- 1 grid asmadmin 22 0x000003 Jul 9 10:33 /dev/rdisk/ocr3

            root@node1:/dev/rdisk# ls -l /dev/*dsk/ocr*
            crw-rw---- 1 grid asmadmin 188 0x2d0100 Jul 9 09:21 /dev/rdsk/ocr1
            crw-rw---- 1 grid asmadmin 188 0x030200 Jul 9 09:21 /dev/rdsk/ocr2
            crw-rw---- 1 grid asmadmin 188 0x1b0200 Jul 9 09:21 /dev/rdsk/ocr3

            NODE2:

            root@node2:/dev/rdisk# ls -l /dev/*disk/*ocr
            crw-rw---- 1 grid asmadmin 22 0x00000e Jul 9 10:33 /dev/rdisk/ocr3
            crw-rw---- 1 grid asmadmin 22 0x000014 Jul 9 10:33 /dev/rdisk/ocr2
            crw-rw---- 1 grid asmadmin 22 0x000012 Jul 9 10:33 /dev/rdisk/ocr1

            root@node2:/dev/rdisk# ls -l /dev/*dsk/*ocr
            crw-rw---- 1 grid asmadmin 188 0x2f0100 Jul 9 09:28 /dev/rdsk/ocr3
            crw-rw---- 1 grid asmadmin 188 0x1c0200 Jul 9 09:28 /dev/rdsk/ocr2
            crw-rw---- 1 grid asmadmin 188 0x1a0200 Jul 9 09:28 /dev/rdsk/ocr1
            • 3. Re: OUI cannot list candidate disks for ASM Disk Group during Grid Infra instln
              585179
              Hi,


              You need give read permission to the corresponding block device as well of each disk you will use

              e.g

              #> chmod 644 [block device location]


              Cheers
              • 4. Re: OUI cannot list candidate disks for ASM Disk Group during Grid Infra instln
                user-1221
                hi,

                The Installation documentation says the permissions should be

                # chown grid:asmadmin /dev/rdsk/cxtydz
                # chmod 660 /dev/rdsk/cxtydz

                http://download.oracle.com/docs/cd/E11882_01/install.112/e17211/storage.htm#CDEEDHIJ
                • 5. Re: OUI cannot list candidate disks for ASM Disk Group during Grid Infra instln
                  585179
                  user-1221 wrote:
                  hi,

                  The Installation documentation says the permissions should be

                  # chown grid:asmadmin /dev/rdsk/cxtydz
                  # chmod 660 /dev/rdsk/cxtydz

                  http://download.oracle.com/docs/cd/E11882_01/install.112/e17211/storage.htm#CDEEDHIJ
                  Hi,

                  See Unable to discover or use raw devices for ASM in HP-UX Itanium in 11.2.0.2 [ID 1276913.1] for details


                  CHeers
                  • 6. Re: OUI cannot list candidate disks for ASM Disk Group during Grid Infra instln
                    user-1221
                    Hi,

                    Thanks for the info.

                    I changed the permissions for the corresponding block devices an still the disks does not list in the OUI.


                    root@node1:/dev/disk# ll -rt
                    total 0
                    brw-r--r--   1 bin        sys          3 0x000002 Jul  6 00:07 disk21
                    brw-r----- 1 bin sys 3 0x000009 Jul 6 00:07 disk23_p3
                    brw-r----- 1 bin sys 3 0x000006 Jul 6 00:07 disk25
                    brw-r----- 1 bin sys 3 0x000005 Jul 6 00:07 disk24
                    brw-r----- 1 bin sys 3 0x000008 Jul 6 00:07 disk23_p2
                    brw-r----- 1 bin sys 3 0x000004 Jul 6 00:07 disk23
                    brw-r--r--   1 bin        sys          3 0x000003 Jul  6 00:07 disk22
                    brw-r----- 1 bin sys 3 0x000007 Jul 6 16:51 disk23_p1
                    brw-r--r--   1 bin        sys          3 0x00000f Jul  6 18:26 disk78
                    brw-r--r--   1 bin        sys          3 0x00000e Jul  6 18:26 disk77
                    brw-r--r--   1 bin        sys          3 0x00000c Jul  6 18:26 disk75
                    brw-r--r--   1 bin        sys          3 0x00000d Jul  6 18:26 disk76
                    brw-r--r--   1 bin        sys          3 0x00000b Jul  6 18:26 disk74
                    brw-r--r--   1 bin        sys          3 0x000010 Jul  6 18:26 disk79

                    root@node1:/dev/disk# ls -l /dev/*disk/disk21
                    brw-r--r-- 1 bin sys 3 *0x000002* Jul 6 00:07 /dev/disk/disk21
                    crw-r----- 1 bin sys 22 0x000002 Jul 6 00:07 /dev/rdisk/disk21
                    root@node1:/dev/disk# ls -l /dev/rdisk/data1
                    crw-rw---- 1 grid asmadmin 22 *0x000002* Jul 9 10:33 /dev/rdisk/data1

                    I earlier created an alias for /dev/rdisk/disk21 as /dev/rdisk/data1 and gave grid:asmadmin ownership along with 660 permissions.
                    the metalink note 1276913.1 has grid:oinstall ownership? do we need to change it to oinstall from asmadmin?
                    • 7. Re: OUI cannot list candidate disks for ASM Disk Group during Grid Infra instln
                      user-1221
                      Hi,

                      I installed the patch PHCO_41479 recommended in ID 1276913.1.
                      still I cannot see the devices


                      root@node1:/# swlist | grep PHCO_41479
                      PHCO_41479 1.0 diskowner(1M) cumulative patch

                      root@node2:/# swlist | grep PHCO_41479
                      PHCO_41479 1.0 diskowner(1M) cumulative patch
                      • 8. Re: OUI cannot list candidate disks for ASM Disk Group during Grid Infra instln
                        Hungry DBA
                        Hi friend,

                        For 11gR2 RAC Implementation, you need to create specail Character Device files in Hp-UX 11.31 or older.

                        Character Device: The Oracle GI, ASM & Oracle Database Homes binaries are established on the local disk of each of the RAC nodes. The OCR & Vote Disks required by Oracle GI must be on Character Devices.
                        • Make sure that the disks are available, enter the following command on every node:

                        # ioscan –fnNkCdisk
                        # ioscan –C disk -fun

                        •If the ioscan command does not display device name information for a device that you want to use, enter the following command to install the special device files for any new devices:
                        # insf –e

                        •The disk names for the same disk can be different on each node. A disk can be identified as the same one via the WWID. The WWID of a disk can be checked via the following command:
                        # scsimgr lun_map -D /dev/rdisk/disk25 | grep WWID

                        • At HP-UX, LUNs name could be different for same LUN ID, hence compare both LUNs WWID & map to the one Character Device File.


                        • I recommend to create a special Oracle device directory, and to use mknod to create device paths in this special Oracle folder. This has the advantage, that you get same names for OCR and Voting files across all nodes in the cluster. In addition, it ensures that the permissions of these Oracle device files remain untouched of 'insf −e'.

                        # mkdir /dev/oracle (on all RAC Nodes)

                        # ll /dev/rdisk/disk25
                        crw-r----- 1 bin sys *23 0x000019* Jan 16 12:16 /dev/rdisk/disk25

                        # mknod /dev/oracle/asmdisk1 c *23 0x000019*

                        Later during set-up, for the ASM instance, set the ASM_DISKSTRING parameter to /dev/oracle/*
                        Now when 'insf -e' is run, it only touches 'standard' DSF, rather than this special ones.

                        • Modify the owner, group, and permissions on the character raw device files (ASM and Voting Disks) on all nodes:

                        # chown grid:asmadmin /dev/oracle/*
                        # chmod 660 /dev/oracle/*


                        That's it. Now you should able to see above LUNs for storing OCR & Voting Disks in ASM.

                        I had just completed five 3 Nodes & 2 Nodes 11gR2 RAC Setup on HP-UX 11.31 & didn't find any issues for Shared Storage.


                        Regards,
                        Manish

                        Mail: manish.nashikkar@hotmail.com
                        Visit: http://manishn.wordpress.com
                        • 9. Re: OUI cannot list candidate disks for ASM Disk Group during Grid Infra instln
                          user-1221
                          Hi,

                          I ran the CVU and got to knwo that the user grid is missing "dba" membership.
                          Check: Membership of user "grid" in group "dba"
                            Node Name         User Exists   Group Exists  User in Group  Comment
                            ---------------- 
                            *node2           yes           yes           no            failed*
                            *node1           yes           yes           no            failed*
                          *Result: Membership check for user "grid" in group "dba" failed*
                          
                          
                          
                          Check default user file creation mask
                            Node Name     Available                 Required                  Comment
                            ------------  
                            *node2       unknown                   0022                      failed*
                            node1       022                       0022                      passed
                          Result: Default user file creation mask check failed
                          
                          Starting Clock synchronization checks using Network Time Protocol(NTP)...
                          
                          NTP Configuration file check started...
                          The NTP configuration file "/etc/ntp.conf" is available on all nodes
                          NTP Configuration file check passed
                          
                          Checking daemon liveness...
                          
                          Check: Liveness for "xntpd"
                            Node Name                             Running?
                            ------------------- 
                            node2                               yes
                            node1                               yes
                          Result: Liveness check passed for "xntpd"
                          
                          Checking NTP daemon command line for slewing option "-x"
                          Check: NTP daemon command line
                            Node Name                             Slewing Option Set?
                            ----------------
                            *node2                               no*
                            node1                               yes
                          Result:
                          NTP daemon slewing option check failed on some nodes
                          PRVF-5436 : The NTP daemon running on one or more nodes lacks the slewing option "-x"
                          Result: Clock synchronization check using Network Time Protocol(NTP) failed
                          
                          Pre-check for cluster services setup was unsuccessful on all the nodes.
                          Node1
                          root@node1:/dev/rdisk# id grid
                          uid=1100(grid) gid=1000(oinstall) groups=1020(asmadmin),1021(asmdba)
                          root@node1:/dev/rdisk# id oracle
                          uid=1101(oracle) gid=1000(oinstall) groups=1021(asmdba),1031(dba),1032(oper)

                          Node2:
                          root@node2:/dev/disk# id grid
                          uid=1100(grid) gid=1000(oinstall) groups=1020(asmadmin),1021(asmdba)
                          root@node2:/dev/disk# id oracle
                          uid=1101(oracle) gid=1000(oinstall) groups=1031(dba),503(oper),1021(asmdba)
                          • 10. Re: OUI cannot list candidate disks for ASM Disk Group during Grid Infra instln
                            user-1221
                            Hi,

                            Added grid user to dba group
                            Check: Membership of user "grid" in group "dba"
                              Node Name         User Exists   Group Exists  User in Group  Comment
                              ------------- 
                              arcdbs2           yes           yes           yes           passed
                              arcdbs1           yes           yes           yes           passed
                            Result: Membership check for user "grid" in group "dba" passed
                            root@node1:/dev/rdisk# id grid
                            uid=1100(grid) gid=1000(oinstall) groups=1020(asmadmin),1021(asmdba),1031(dba)


                            Still the disks are not listed in the OUI.
                            • 11. Re: OUI cannot list candidate disks for ASM Disk Group during Grid Infra instln
                              585179
                              Hi,

                              Can you please post the output from

                              # > ioscan -funN -C disk
                              # > /sbin/pvdisplay /dev/disk/disk21


                              Cheers
                              • 12. Re: OUI cannot list candidate disks for ASM Disk Group during Grid Infra instln
                                user-1221
                                Hi,
                                root@node1:/#  ioscan -funN -C disk
                                Class     I  H/W Path  Driver         S/W State   H/W Type     Description
                                ===========================================================================
                                disk      0  0/0/2/1.0x0.0x10   UsbScsiAdaptor   CLAIMED     LUN_PATH     USB SCSI Stack Adaptor
                                disk     21  64000/0xfa00/0x2   esdisk           CLAIMED     DEVICE       HP      HSV210
                                                      /dev/disk/disk21     /dev/rdisk/data1
                                                      /dev/rdisk/disk21
                                disk     22  64000/0xfa00/0x3   esdisk           CLAIMED     DEVICE       HP      HSV210
                                                      /dev/disk/disk22    /dev/rdisk/disk22   /dev/rdisk/ocr1
                                disk     23  64000/0xfa00/0x4   esdisk           CLAIMED     DEVICE       HP      IR Volume
                                                      /dev/disk/disk23      /dev/rdisk/disk23
                                                      /dev/disk/disk23_p1   /dev/rdisk/disk23_p1
                                                      /dev/disk/disk23_p2   /dev/rdisk/disk23_p2
                                                      /dev/disk/disk23_p3   /dev/rdisk/disk23_p3
                                disk     24  64000/0xfa00/0x5   esdisk           CLAIMED     DEVICE       HP      DG146ABAB4
                                                      /dev/disk/disk24   /dev/rdisk/disk24
                                disk     25  64000/0xfa00/0x6   esdisk           CLAIMED     DEVICE       HP      DG146ABAB4
                                                      /dev/disk/disk25   /dev/rdisk/disk25
                                disk     74  64000/0xfa00/0xb   esdisk           CLAIMED     DEVICE       HP      HSV210
                                                      /dev/disk/disk74     /dev/rdisk/vote3
                                                      /dev/rdisk/disk74
                                disk     75  64000/0xfa00/0xc   esdisk           CLAIMED     DEVICE       HP      HSV210
                                                      /dev/disk/disk75     /dev/rdisk/vote2
                                                      /dev/rdisk/disk75
                                disk     76  64000/0xfa00/0xd   esdisk           CLAIMED     DEVICE       HP      HSV210
                                                      /dev/disk/disk76    /dev/rdisk/disk76   /dev/rdisk/ocr3
                                disk     77  64000/0xfa00/0xe   esdisk           CLAIMED     DEVICE       HP      HSV210
                                                      /dev/disk/disk77     /dev/rdisk/vote1
                                                      /dev/rdisk/disk77
                                disk     78  64000/0xfa00/0xf   esdisk           CLAIMED     DEVICE       HP      HSV210
                                                      /dev/disk/disk78     /dev/rdisk/arch1
                                                      /dev/rdisk/disk78
                                disk     79  64000/0xfa00/0x10  esdisk           CLAIMED     DEVICE       HP      HSV210
                                                      /dev/disk/disk79    /dev/rdisk/disk79   /dev/rdisk/ocr2
                                root@node1:/# /sbin/pvdisplay /dev/disk/disk21
                                pvdisplay: Couldn't find the volume group to which
                                physical volume "/dev/disk/disk21" belongs.
                                pvdisplay: Cannot display physical volume "/dev/disk/disk21".
                                • 13. Re: OUI cannot list candidate disks for ASM Disk Group during Grid Infra instln
                                  user-1221
                                  Hi Manish,

                                  Thanks for the detailed explanation,

                                  I performed everything what you have mentioned in the post, even I performed the action mentioned in the metalink ID 1276913.1, but still the disks are not getting identified by the OUI.

                                  Please share the Grid Infrastructure installation procedure.

                                  Please Help me in resolving this issue, the deadline has already passed for this activity.

                                  Thanks in Advance!
                                  • 14. Re: OUI cannot list candidate disks for ASM Disk Group during Grid Infra instln
                                    Hungry DBA
                                    Hi friend,

                                    You are not seeing LUNs in GUI that means there could be issue with LUNs, which are not persistent across all the Nodes. Find it out correct LUN ID by above mentioned command & map to the Character Special file like /dev/oracle/asmdisk1, so that Oracle would be use the same disk even after reboot. Even check that proper permissions & ownerships you have given to these special files.


                                    Give me your mail ID, so that I shall be provided you the document for implementing 11gR2 RAC on HP-UX 11.31 (which I followed)


                                    Regards,
                                    Manish
                                    1 2 Previous Next