This discussion is archived
8 Replies Latest reply: Dec 12, 2011 8:30 AM by decEpticOn RSS

Install Oracle 11gr2 2Node RAC on ASM

decEpticOn Newbie
Currently Being Moderated
Hi,

Not sure what all questions i have are in correct thread.

We have to install Oracle 11gr2 RAC on 2 nodes unix machine, m doing it for first time. We are using storage disk and configured following LUN's on both the machine.

/dev/sde - appdata,
/dev/sdf - oradisk1,
/dev/sdg - oradisk2,
/dev/sdh - oradata1,
/dev/sdi - oradata2,
/dev/sdj - oradata3,
/dev/sdk - oradflt,
/dev/sdl - oratemp,
/dev/sdm - oraundo1,
/dev/sdn - oraundo2,
/dev/sdo - oraredo,
/dev/sdp - votingdisk1,
/dev/sdq - votingdisk2

Now, considering machine are named as Node1 and Node2.

Created partitions on all the disks from Node1 using (fdisk /dev/sde) - it created new files like
/dev/sde1, /dev/sdf1, ............. /dev/sdq1

then created ASM disk on all drives from node1 only..

/etc/init.d/oracleasm createdisk ASM_APPDAT /dev/sde1 ........ for all the drives..
/etc/init.d/oracleasm createdisk ASM_ORADISK1 /dev/sdf1
.
.
.
/etc/init.d/oracleasm createdisk ASM_VOTING2 /dev/sdq1

Now, the question is, what I need to do in Node2 ?? It shows /dev/sde, /dev/sef .... /dev/sdq, and It doesn't alloe to create diskgroup for same on Node2.

Please suggest how do i proceed?


Question 2> We are Installing Oracle 11gr2 Grid, do we need to install oracle database prior to it? or Grid automatically installs one?


Please suggest, and do let me know if anything i wrote is unclear...


Thanks!
  • 1. Re: Install Oracle 11gr2 2Node RAC on ASM
    585179 Expert
    Currently Being Moderated
    HI,


    For Q1, you just need to run oracleasm scandisks on the other node, it will scan all available disks and run oracleasm listdisks to see if it show correct disks

    For Q2, Install grid infrastructure first then Oracle DB software


    Cheers
  • 2. Re: Install Oracle 11gr2 2Node RAC on ASM
    decEpticOn Newbie
    Currently Being Moderated
    Thanks Fiedi!

    That was really helpful.

    I did that on both machines, so not i have /dev/sde ........ /dev/sdq and /dev/sde1 .... /dev/sdq1 sets on both nodes.

    I changed der permission, ownership to oracle user (m invoking grid using this user).

    But in 11gr2 grid installation step 6 "Create ASM disk group", I am not able to see any disk groups in "Add disk" section.

    Please suggest, how can i view diskgroups der?
  • 3. Re: Install Oracle 11gr2 2Node RAC on ASM
    585179 Expert
    Currently Being Moderated
    bLaK wrote:
    Thanks Fiedi!

    That was really helpful.

    I did that on both machines, so not i have /dev/sde ........ /dev/sdq and /dev/sde1 .... /dev/sdq1 sets on both nodes.

    I changed der permission, ownership to oracle user (m invoking grid using this user).

    But in 11gr2 grid installation step 6 "Create ASM disk group", I am not able to see any disk groups in "Add disk" section.

    Please suggest, how can i view diskgroups der?
    Hi,

    Post the file /etc/sysconfig/oracleasm

    and post ls -al /dev/sde*


    cheers
  • 4. Re: Install Oracle 11gr2 2Node RAC on ASM
    decEpticOn Newbie
    Currently Being Moderated
    Hi,

    Please find the file contents:


    [oracle@NODE1 grid]$ cat /etc/sysconfig/oracleasm
    #
    # This is a configuration file for automatic loading of the Oracle
    # Automatic Storage Management library kernel driver. It is generated
    # By running /etc/init.d/oracleasm configure. Please use that method
    # to modify this file
    #

    # ORACLEASM_ENABELED: 'true' means to load the driver on boot.
    ORACLEASM_ENABLED=true

    # ORACLEASM_UID: Default user owning the /dev/oracleasm mount point.
    ORACLEASM_UID=oracle

    # ORACLEASM_GID: Default group owning the /dev/oracleasm mount point.
    ORACLEASM_GID=dba

    # ORACLEASM_SCANBOOT: 'true' means scan for ASM disks on boot.
    ORACLEASM_SCANBOOT=true

    # ORACLEASM_SCANORDER: Matching patterns to order disk scanning
    ORACLEASM_SCANORDER=""

    # ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan
    ORACLEASM_SCANEXCLUDE=""

    [oracle@NODE1 grid]$ ls -al /dev/sde*
    brwxrwxrwx 1 oracle dba 8, 64 Dec 9 17:51 /dev/sde
    brwxrwxrwx 1 oracle dba 8, 65 Dec 9 22:25 /dev/sde1


    Thanks!
  • 5. Re: Install Oracle 11gr2 2Node RAC on ASM
    BillyVerreynne Oracle ACE
    Currently Being Moderated
    bLaK wrote:

    [oracle@NODE1 grid]$ ls -al /dev/sde*
    brwxrwxrwx 1 oracle dba 8, 64 Dec 9 17:51 /dev/sde
    brwxrwxrwx 1 oracle dba 8, 65 Dec 9 22:25 /dev/sde1
    Dangerous permissions. Why provide every single user on that platform full and unrestricted access to these devices? A user can run a format on that device. Or reset the device content to zeros (using dd). Etc.

    All these actions will destroy your cluster and/or database.

    Also you have incorrect ownership for Oracle 11g when installed as per Oracle recommendations. Oracle Grid Infrastructure should be installed as the grid user and default group should be oinstall.

    It is also not a good idea to use the +/dev/xxx+ devices directly. Several problems with that. The discovery string will likely include local devices that are not suppose to be seen by ASM - and how do you prevent the dba/admin user from not accidentally use the wrong device in ASM?

    The +/dev/xxx+ devices that are external (e.g. LUNs via fibre channel), will be named differently each time around when booting. The kernel cannot guarantee using the same device name for an external device each time around, as it depends on the h/w probe and discovery and other factors what sequence external devices are seen and device entries created.

    If you are using fibre channel (or Infiniband) as the I/O fabric layer, you typically will have multiple I/O paths to the external device. A storage LUN will therefore be seen multiple times and each of these paths will result in a device entry to be created. You need a logical layer above this to provide path redundancy and load balancing across these multiple paths - and have ASM use this logical multipath device. Not a device for a single specific path.

    Personally - I recommend not using ASMlib at all. Use Multipath and set the ASM discovery string to +/dev/mpath/*+. Use udev rules to set the permissions of multipath devices to grid and oinstall - and create and use the grid o/s user for installing Grid Infrastructure.
  • 6. Re: Install Oracle 11gr2 2Node RAC on ASM
    decEpticOn Newbie
    Currently Being Moderated
    Thanks Billy!

    This permissions are currently given on dev environment as we are not able to discover disk's, we will change it accordingly once installation is done.

    Yes we are using LUNs via fibre channel, so if not /dev/xxx, what path can be used?

    In grid installation (11gr2), we are not able to view any asm disks. Not sure what's gone wrong. There is no such dev/mpath/*, we'r using Linux Server release 5.3

    [oracle by-id]$ cat /etc/*-release
    Enterprise Linux Enterprise Linux Server release 5.3 (Carthage)
    Red Hat Enterprise Linux Server release 5.3 (Tikanga)

    Please suggest what could have gone incorrect in congif? such that it is not showing up disks in Installer.

    Thanks again!
  • 7. Re: Install Oracle 11gr2 2Node RAC on ASM
    BillyVerreynne Oracle ACE
    Currently Being Moderated
    bLaK wrote:

    Yes we are using LUNs via fibre channel, so if not /dev/xxx, what path can be used?
    That depends on what you have installed, driver wise.

    Are you using EMC kit? They recommend their own kernel driver called Powerpath. We use Multipath and have been for many years now. It is Open Source. It does not taint the kernel like Powerpath does.

    The LUN is identified by its WWID. So Multipath can determine that, for example, +/dev/sdf+ and +/dev/sdk+ are in fact the same LUN.

    In +/etc/multipath.conf+ you blacklist and whitelist the WWIDs - and can for example configure a WWID to be logical device +/dev/mpath/DATA1_LUN1+.

    This device is managed by Multipath - and should the path to +/dev/sdf+ fail, the path to +/dev/sdk+ will be transparently used.

    You also point ASM to using +/dev/mpath/DATA1_LUN1+ - instead of using +/dev/sdf+ or +/dev/sdk+. ASM does not particularly like seeing the same ASM disk label via 2 (or more) distinctly different scsi devices.

    Powerpath works in a similar fashion. However, Multipath is used in 2000+ node superclusters and works just fine. And does not cost you a single cent to use as its Open Source and bundled with both RHEL and Oracle Linux.
    In grid installation (11gr2), we are not able to view any asm disks. Not sure what's gone wrong. There is no such dev/mpath/*, we'r using Linux Server release 5.3
    You need to configure +/etc/multipath.conf+. And then configure +/etc/udev/rules.d/40-multipath.rules+ to set the permissions.

    Use scsi_id to determine the WWID.

    For example:
    // WWID for a device
    [root@dev1 ~]# ll /dev/sdak
    brw-r----- 1 root disk 66, 64 Dec  1 12:39 /dev/sdak
    [root@dev1 ~]# /sbin/scsi_id -g -u -s /block/sdak
    23634316365393032
    Using that WWID to configure a multipath device in +/etc/multipath.conf+
        multipath {
            wwid  23634316365393032
            alias scst2_sdg
          }
    Multipath device is created:
    // use this device in ASM
    [root@dev1 ~]# ll /dev/mpath/scst2_sdg
    lrwxrwxrwx 1 root root 7 Dec  1 12:38 /dev/mpath/scst2_sdg -> ../dm-5
    Using lssci to see the actual scsi devices (multiple I/O paths) to the LUN:
    [root@dev1 ~]# lsscsi | grep scst2_sdg
    [5:0:0:5]    disk    SCST_BIO scst2_sdg         300  /dev/sdg
    [6:0:0:5]    disk    SCST_BIO scst2_sdg         300  /dev/sdq
    [8:0:0:5]    disk    SCST_BIO scst2_sdg         300  /dev/sdak
    [9:0:0:5]    disk    SCST_BIO scst2_sdg         300  /dev/sdau
    The multipath configuration:
    [root@dev1 ~]# multipath -ll scst2_sdg 
    scst2_sdg (23634316365393032) dm-5 SCST_BIO,scst2_sdg
    [size=932G][features=0][hwhandler=0][rw]
    \_ round-robin 0 [prio=0][active]
     \_ 5:0:0:5 sdg  8:96   [active][ready]
    \_ round-robin 0 [prio=0][enabled]
     \_ 6:0:0:5 sdq  65:0   [active][ready]
    \_ round-robin 0 [prio=0][enabled]
     \_ 8:0:0:5 sdak 66:64  [active][ready]
    \_ round-robin 0 [prio=0][enabled]
     \_ 9:0:0:5 sdau 66:224 [active][ready]
    It's not that difficult to configure. And it provides a robust and flexible and easy device layer for ASM to use and for the sysadmin to manage.
  • 8. Re: Install Oracle 11gr2 2Node RAC on ASM
    decEpticOn Newbie
    Currently Being Moderated
    Thanks a lot Billy for such a detailed explanation. This is very informative. I will try this and let you know the outcome. Thanks

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points