8 Replies Latest reply: Jun 6, 2013 11:06 AM by Levi Pereira RSS

    Clusterware with ASM on NFS / NAS

    Hemant K Chitale
      Are there notes / experiences on using an NFS / NAS that provides the "LUNs" or "disks" for ASM in Grid Infrastructure install ?

      Oracle GI would be installed on local nodes but the Shared Storage that I am looking for is NFS / NAS (e.g. NetApp). Presumably ASMLib might not be required (even if the nodes are Oracle / RedHat Linux) ?


      Hemant K Chitale
        • 1. Re: Clusterware with ASM on NFS / NAS
          585179
          Hemant K Chitale wrote:
          Are there notes / experiences on using an NFS / NAS that provides the "LUNs" or "disks" for ASM in Grid Infrastructure install ?

          Oracle GI would be installed on local nodes but the Shared Storage that I am looking for is NFS / NAS (e.g. NetApp). Presumably ASMLib might not be required (even if the nodes are Oracle / RedHat Linux) ?


          Hemant K Chitale
          Hi Hermant,


          This note might comes to your interest : How To Create ASM Diskgroups using NFS/NAS Files? [ID 731775.1]


          Cheers
          • 2. Re: Clusterware with ASM on NFS / NAS
            Hemant K Chitale
            Thank you. Yes, I'd seen that note just before posting here.

            I want to see if anyone can describe any test implementation.


            Hemant K Chitale
            • 3. Re: Clusterware with ASM on NFS / NAS
              Levi Pereira
              Hi Hemant,
              Are there notes / experiences on using an NFS / NAS that provides the "LUNs" or "disks" for ASM in Grid Infrastructure install ?
              Oracle GI would be installed on local nodes but the Shared Storage that I am looking for is NFS / NAS (e.g. NetApp).
              Is it for educational or production purpose?
              Production purpose:
              There are few documents about configuring ASM Disks using Files on NAS Devices. Because it is not so common use is the solution.

              Using NAS Devices
              http://download.oracle.com/docs/cd/E11882_01/install.112/e16763/app_nas.htm

              Creating Files on a NAS Device for Use with Oracle ASM
              http://download.oracle.com/docs/cd/E11882_01/install.112/e17212/storage.htm#CWLIN291

              NFS Storage Supported
              NFS file systems, including file systems on NAS devices, may not be able to guarantee that writes to a disk are completed successfully, and this may lead to possible data file corruption. Oracle recommends that you do not store files on NFS mounted file systems unless the storage vendor and storage device are supported.
              Oracle not certify Storage Vendors because it's must be supported by servers (hosts), but if you are using NFS following NFS storage vendors are supported: EMC, Fujitsu, HP, IBM, NetApp, Pillar Data, Sun NFS file servers do not require RAC certification. The NFS file server must be supported by the system and storage vendors.
              http://www.oracle.com/technetwork/database/clustering/tech-generic-linux-new-086754.html


              *Mount Options for Oracle files when used with NAS devices [ID 359515.1]*
              The mandatory mount options comprise the minimum set of mount options that you must use while mounting the NFS volumes. These mount options are essential to protect the integrity of the data and to prevent any database corruption. Failure to use these mount options may result in the generation of file access errors.

              *Oracle ADVM does not support NFS. [ID 946211.1]*
              Files used as ASM Disks are not supported by ADVM. For example on NetApp Filer only real disks or Netapp Devices using ISCSI, SCSI or Fibre Channel can be used.


              Educational purpose:
              I recommend you use OPENFILER with iSCSI. It's more simple and all features will work.

              Build Your Own Oracle RAC 11g Cluster on Oracle Enterprise Linux and iSCSI
              http://www.oracle.com/technetwork/articles/hunter-rac11gr2-iscsi-088677.html

              *Using Openfiler iSCSI with an Oracle RAC database on Linux [ID 371434.1]*
              Presumably ASMLib might not be required (even if the nodes are Oracle / RedHat Linux)
              Yes. It's not required using NAS/NFS or not.
              Without ASMLIB Linux 2.6 kernel and later, block device paths do not maintain permissions and path persistence unless you create a permissions or rules file on each cluster member node; block device paths that were /dev/sda can appear as /dev/sdb after a system restart. Adding new disks requires you to modify the udev file to provide permissions and path persistence for the new disk.
              http://download.oracle.com/docs/cd/E11882_01/install.112/e16763/pre_install.htm#BABGCBBG


              Hope that helps,
              Levi Pereira
              • 4. Re: Clusterware with ASM on NFS / NAS
                Hemant K Chitale
                Modifying ASM_DISKSTRING via an ALTER SYSTEM command isn't an option until and unless you already have ASM installed and running. 11gR2 Clusteware installation attempts to install and configure ASM (at least for OCR and Voting Disks).
                So it becomes a "chicken and egg situation" !

                With devices created (by "dd") on an NFS mountpoint, oracleasm doesn't CREATEDISK unless I use losetup. Fine, I had the same issue testing 10gR2 ASM (although I have been told that 11gR2 does not support losetup).

                Currently, I have cleaned up all my NFS setup and done a reinstall with iSCSI. I will want to re-attempt NFS later.

                Hemant K Chitale
                • 5. Re: Clusterware with ASM on NFS / NAS
                  Levi Pereira
                  Hi Hemant,
                  Hemant K Chitale wrote:
                  Modifying ASM_DISKSTRING via an ALTER SYSTEM command isn't an option until and unless you already have ASM installed and running. 11gR2 Clusteware installation attempts to install and configure ASM (at least for OCR and Voting Disks).
                  So it becomes a "chicken and egg situation" !
                  You can configure ASM_DISKSTRING using OUI at step "Create ASM Disk Group" on button "Change Discovery Path".
                  With devices created (by "dd") on an NFS mountpoint, oracleasm doesn't CREATEDISK unless I use losetup. Fine, I had the same issue testing 10gR2 ASM (although I have been told that 11gR2 does not support losetup).
                  I belive, we will not need use losetup anymore.


                  I installed Oracle Clusterware using 2 Hosts without Storage System using NFS. (it's for test only)

                  Creating zero-padded files.
                   dd if=/dev/zero of=/oracle/asm/devices/nfsdisk1 bs=8192 count=131072
                  131072+0 records in
                  131072+0 records out
                  1073741824 bytes (1.1 GB) copied, 27.9333 seconds, 38.4 MB/s
                  
                   dd if=/dev/zero of=/oracle/asm/devices/nfsdisk2 bs=8192 count=131072
                  131072+0 records in
                  131072+0 records out
                  1073741824 bytes (1.1 GB) copied, 19.9333 seconds, 50.4 MB/s
                  
                   dd if=/dev/zero of=/oracle/asm/devices/nfsdisk3 bs=8192 count=131072
                  131072+0 records in
                  131072+0 records out
                  1073741824 bytes (1.1 GB) copied, 17.9333 seconds, 52.4 MB/s
                  
                  #  chown oracle.asmadmin /oracle/asm/devices/nfsdisk*
                  #  chmod 0660 /oracle/asm/devices/nfsdisk*
                  Configuring NFS
                  ### NFS Server
                  # cat /etc/exports
                  /oracle/asm/devices     129.10.10.0/24(rw,sync,no_wdelay,insecure_locks,no_root_squash)
                  
                  ### Mounting NFS on both servers "/etc/fstab"
                  server-nas:/oracle/asm/devices  /oracle/asm/disks  nfs  rw,bg,hard,nointr,tcp,vers=3,timeo=600,rsize=32768,wsize=32768,actimeo=0,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0  0 0
                  At step Create ASM Disk Group I used "Redundancy Normal" and changed "Change Discovery Path" to "/oracle/asm/disks/*"


                  The pre-reqs of ASM Devices will fails, I checked "Ignore":
                   Error Message:PRVF-7039 : File system exists on location "/oracle/asm/disks/nfsdisk1"
                   Cause: Existing file system found on the specified location.
                   Action: Ensure that the specified location does not have an existing file system.
                   Error Message:PRVF-7039 : File system exists on location "/oracle/asm/disks/nfsdisk2"
                   Cause: Existing file system found on the specified location.
                   Action: Ensure that the specified location does not have an existing file system.
                   Error Message:PRVF-7039 : File system exists on location "/oracle/asm/disks/nfsdisk3"
                   Cause: Existing file system found on the specified location.
                   Action: Ensure that the specified location does not have an existing file system.
                  Output of root.sh.
                  # NODE1
                  
                  .....
                  ASM created and started successfully.
                  Disk Group DATA created successfully.
                  clscfg: -install mode specified
                  Successfully accumulated necessary OCR keys.
                  Creating OCR keys for user 'root', privgrp 'root'..
                  Operation successful.
                  CRS-4256: Updating the profile
                  Successful addition of voting disk 948b607b2b554ffebf5d89ea1e68beff.
                  Successful addition of voting disk 4ac78fefc7e94f78bf6749fa48edbe0e.
                  Successful addition of voting disk 1a1931d2fcff4fa4bf9c8c4817c86b79.
                  Successfully replaced voting disk group with +DATA.
                  CRS-4256: Updating the profile
                  CRS-4266: Voting file(s) successfully replaced
                  ##  STATE    File Universal Id                File Name Disk group
                  --  -----    -----------------                --------- ---------
                   1. ONLINE   948b607b2b554ffebf5d89ea1e68beff (/oracle/asm/disks/nfsdisk1) [DATA]
                   2. ONLINE   4ac78fefc7e94f78bf6749fa48edbe0e (/oracle/asm/disks/nfsdisk2) [DATA]
                   3. ONLINE   1a1931d2fcff4fa4bf9c8c4817c86b79 (/oracle/asm/disks/nfsdisk3) [DATA]
                  Located 3 voting disk(s).
                  
                  CRS-2672: Attempting to start 'ora.asm' on 'holanda'
                  CRS-2676: Start of 'ora.asm' on 'holanda' succeeded
                  CRS-2672: Attempting to start 'ora.DATA.dg' on 'holanda'
                  CRS-2676: Start of 'ora.DATA.dg' on 'holanda' succeeded
                  ACFS-9200: Supported
                  ACFS-9200: Supported
                  CRS-2672: Attempting to start 'ora.registry.acfs' on 'holanda'
                  CRS-2676: Start of 'ora.registry.acfs' on 'holanda' succeeded
                  Configure Oracle Grid Infrastructure for a Cluster ... succeeded
                  The installation ended sucessfull and cluster is up.

                  Regards,
                  Levi Pereira
                  • 6. Re: Clusterware with ASM on NFS / NAS
                    Hemant K Chitale
                    Thanks.

                    I might retry Grid Infrastructure Install with CRS / Voting on NFS a few weeks later.
                    (My test box consisting of 4 VMs on a Quad Core machine runs so slowly, it takes a good number of hours to complete the install).



                    Hemant K Chitale
                    • 7. Re: Clusterware with ASM on NFS / NAS
                      Kramer
                      Hi..

                      I understand that this thread is done already. But I met exactly the same problem and I did a few search on the internet. Mostly answered by you. So I want to ask for help from you.
                      Ok this is my situation.

                      I`m trying to install 11.0.2.3 grid on OEL6 (two nodes)
                      I use a nfs as the shared storage. But I planted to create ASM disk groups on the nfs (use the dd command ) to store ocr, voting disk and database files.
                      I`m using oracle account only, the group for oracle is oinstall and dba. No asmadmin.

                      The problem is that OUI can not list the disks I created on nfs with dd command. Any idea why please?
                      This is the step after OUI ask where you want to put the ocr and voting disk. I choose the asm. So it prompt to ask create asm disk group. Just it can not list the disks


                      I also find this thread which you answerd
                      disks not showup for ASM  while insalling  11gr2 on solaris 10 using gui
                      You said from 11.1, the asm user group should not be dba but asmadmin. But I checked the 11.2 doc, it said using oracle to install everything is ok. And i can use oinstall and dba group without asmadmin group
                      Can explain please?

                      Regards
                      Kramer

                      Edited by: Kramer on Jun 6, 2013 1:36 AM
                      • 8. Re: Clusterware with ASM on NFS / NAS
                        Levi Pereira
                        Hi,

                        Please, open a new thread about this issue.

                        And post output of "kfod" using option "_asm_allow_only_raw_disks=FALSE".


                        Run kfod on all nodes and put output here:

                        $ kfod asm_diskstring='<path_location>*' disks=all asmallow_only_raw_disks=FALSE status=TRUE



                        Note:
                        Keep in mind that the Grid Infrastructure software does not need to be installed to use kfod, since kfod is located as well on the staging area where you uncompressed the Grid Infrastructure software, as the following example:

                        /stage/grid/stage/ext/bin/kfod
                        Where: "/stage" is the directory where I uncompressed the source software for the Grid Infrastructure software.

                        Regards,
                        Levi Pereira