4 Replies Latest reply: Apr 9, 2013 6:23 AM by Billy~Verreynne RSS

    Why does lun show different size on different rac nodes?

    D Tej
      Hi all,

      We are trying to increase ASM disk space and with respect to it when we are trying to allocate more space this question came across my mind. Now this was previously configured by my previous SA.

      [root@oracledbtest1 ~]# /etc/init.d/oracleasm querydisk -d `/etc/init.d/oracl
      cut -f2,10,11 -d" " | perl -pe 's/"(.*)".*\[(.*), *(.*)\]/$1 $2 $3/g;' |
      while read v_asmdisk v_minor v_major
      v_device=`ls -la /dev | grep " $v_minor, *$v_major " | awk '{print $10}'`
      echo "ASM disk $v_asmdisk based on /dev/$v_device [$v_minor, $v_major]"
      **ASM disk ASM01 based on /dev/dm-15 [253, 15]**
      **ASM disk ASM02 based on /dev/dm-14 [253, 14]**

      node 2:

      [root@oracledbtest1 ~]# /etc/init.d/oracleasm querydisk -d `/etc/init.d/oracleasm listdisks -d` |
      cut -f2,10,11 -d" " | perl -pe 's/"(.*)".*\[(.*), *(.*)\]/$1 $2 $3/g;' |
      while read v_asmdisk v_minor v_major
      v_device=`ls -la /dev | grep " $v_minor, *$v_major " | awk '{print $10}'`
      echo "ASM disk $v_asmdisk based on /dev/$v_device [$v_minor, $v_major]"
      **ASM disk ASM01 based on /dev/dm-13 [253, 13]**
      **ASM disk ASM02 based on /dev/dm-14 [253, 14]**

      Now, can any one help me in telling why are my LUN's showing different sizes on RAC and also the best way to allocate the space to the disks in the above scenario.

      P.S I am fairly new in posting to the OTN. Please excuse if I am not clear with my question.

      Edited by: 902932 on Apr 8, 2013 8:25 AM
        • 1. Re: Why does lun show different size on different rac nodes?
          Moderator Action:
          Thread moved from the Generic Linux forum to the ASM forum.
          Your question deals with ASM. You just happen to have a Linux OS.
          • 2. Re: Why does lun show different size on different rac nodes?
            why you think the space is different between the 2 nodes?

            The devices are different, that's ok, dm- is the device mapper name, and several devices will create dm-* like entries, so it's ok to have them different.

            When you add a disk, you should follow an approach like:

            Present the disk on the storage
            Refresh the luns on all the nodes
            Create the oracleasm disk on one node
            refresh the oracleasm disks on all the nodes
            verify the new oracleasm disk is present on all nodes in /dev/oracleasm/disks/*

            At this point, you can use asmca or sqlplus in the +ASMn instance to make your asm diskgroup to grow, or create a new disk group                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
            • 3. Re: Why does lun show different size on different rac nodes?
              The other thing is that you really need to understand ASM and shared storage. Do yourself a favor and go through the concepts manual or search Amazon for "Automatic Storage Management - Under-the-Hood" It will serve you well.
              • 4. Re: Why does lun show different size on different rac nodes?
                To add to what was said.

                The kernel discovers devices at boot time. Devices can be enumerated in different orders, depending on a number of factors at that point in time. Server1 can see storage LUN0 as scsi device sdk, whereas server2 sees it as device sdg.

                This is expected.

                Many storage layers also provide multiple I/O paths to a LUN (e.g. dual fibre channels). So not only will LUN0 be seen as different scsi devices between server1 and server2 - server1 (and server2) will see the very same physical LUN0 on the storage layer as multiple different scsi devices. E.g. server1 can see LUN0 as device sdk and device sdm.

                To get a sane and consistent view, a logical device is needed - where this device is called for example data1, and will always refer to LUN0, irrespective of what the scsi device names happen to be at the time.

                This is done via WWIDs - identifier strings that uniquely identifies scsi devices sdk and sdm as being storage device LUN0.

                What is also needed is the logical device, data1 in our example, to support all I/O paths to LUN0. So it should use both scsi devices sdk and sdm - and when one fails (e.g. fibre channel cable breaks), be able to fall all I/O over from the broken path (e.g. scsi sdk) to the working path (e.g. scsi sdm).

                ASM diskgroups should be created using the logical and consistently named device - e.g. device data1 in our example.

                In your case, ASMLib created logical devices ASM01 and ASM02. These logical devices provide you with a consistent interface to the actual storage LUNs, irrespective of what the kernel happens to call the scsi devices it creates for these LUNs.

                I prefer not using ASMLib, but udev and multipath instead. These are standard Open Source drivers for Linux, and originated from, and are used in, some of the largest computing clusters in the world, for addressing storage layers with PBs of storage.