Using Oracle RMAN with SPARC S7-2L NFS Servers on Oracle MiniCluster

Version 3

    by Dean Halbeisen

     

    This article provides a step-by-step guide on how to configure Oracle MiniCluster to perform Oracle Recovery Manager (Oracle RMAN) database backups on external storage on Oracle's SPARC S7-2L servers.

     

    Table of Contents

     

    Introduction

     

    This article starts at the point where the Oracle MiniCluster initial installation and application installation have been completed or are nearly complete. Oracle MiniCluster must be installed and configured before the backup configuration process can begin.

     

    The SPARC S7-2L server from Oracle is ideally suited for use as a backup and recovery server for Oracle MiniCluster. When configured with a twelve-disk chassis, it provides up to 61 TB of usable capacity for NFS and backup services. For more information on the SPARC S7-2L server, go to oracle.com/servers/sparc/s7-2l.

     

    Be sure to read through the entire article in order, so you don't miss any key steps.

     

    Configuration Walk-Through

     

    The following sections provide an overview of the steps that need to be performed on Oracle MiniCluster to configure Oracle RMAN database backups to SPARC S7-2L servers.

     

    Create and Configure the ZFS Storage Pools

     

    ZFS storage pools must be created and configured before setting up backups to the SPARC S7-2L servers.

     

    A mirrored RAID configuration is preferred to provide the highest performance for all workloads and the best performance for mixed workloads. RAIDZ1 can be used and does provide more usable capacity, with the trade-off of lower mixed workload performance and reliability. It is best to configure at least one storage pool per controller in cluster configurations. Table 1 compares RAID configurations for ZFS storage pools on SPARC S7-2L servers.

     

    Table 1. ZFS Storage Pool Configurations on SPARC S7-2L Servers

                                                  

    RAID ConfigurationNumber of Spare DisksUsable CapacityBackup PerformanceRestore Performance
    6 x 2 disk mirrors041 TB3.8 TB/hour~3.8 TB/hour
    4 x 3 disk mirrors028 TB2.5 TB/hour~2.5 TB/hour
    5 x 2 disk mirrors235 TB3.3 TB/hour~3.3 TB/hour
    3 x 3 disk mirrors321 TB1.8 TB/hour~1.8 TB/hour
    4 x 3 disk RAIDZ1056 TB5.1 TB/hour~5.1 TB/hour
    3 x 4 disk RAIDZ1061 TB3.9 TB/hour~3.9 TB/hour
    3 x 3 disk RAIDZ1 +
      1 x 2 disk RAIDZ1
    149 TB4.4 TB/hour~4.4 TB/hour
    2 x 4 disk RAIDZ1 +
    1 x 3 disk RAIDZ1
    154 TB4.6 TB/hour~4.6 TB/hour

     

    The following example creates a ZFS storage pool and shares it over NFS.

     

    # zpool create -f backups raidz c0t5000CCA2543D58D0d0 \
    c0t5000CCA2543D464Cd0 c0t5000CCA2543F28FCd0 c0t5000CCA2543FCADCd0 raidz \
    c0t5000CCA2543FEC38d0 c0t5000CCA25410CBB8d0 c0t5000CCA25410D278d0 \
    c0t5000CCA25410DC10d0 raidz c0t5000CCA25410DED8d0 c0t5000CCA25439BD1Cd0 \
    c0t5000CCA25442F2C0d0 c0t5000CCA25442F27Cd0
     
    # zfs set share=name=backups,path=/backups,prot=nfs,sec=sys,rw=*,public, \
    root=oos-dbg1-vm1-mc14-n1.us.oracle.com:oos-dbg1-vm1-mc14-n2.us.oracle.com backups

     

    You can choose to have ZFS make the RAID configurations, or you can configure the storage pool manually. When setting up the NFS shares for the new storage pool, allow only the Oracle MiniCluster virtual machines (VMs) to have NFS root access.

     

    For more information on ZFS and sharing ZFS file systems, refer to the following resources:

     

     

    Configure the Network for Performance and Reliability

     

    Configure as many network ports as possible, using either Link Aggregation Control Protocol (LACP) or IP network multipathing (IPMP).

     

    • LACP provides performance, reliability, and automated load balancing.
    • IPMP provides performance and reliability; it must be combined with the Direct NFS Client (dNFS) feature of Oracle Database to provide automated load balancing.

     

    Add up to four virtual network interface cards (VNICs) per controller in the configuration to help increase dNFS performance and provide better load balancing.

     

    Configure Routing for Multihomed Servers

     

    If you are using multiple ports and multiple addresses on SPARC S7-2L servers, you need to configure adaptive routing. The following guidelines apply:

     

    • Multiple interfaces require adaptive routing.
    • The default routing policy type setting in Oracle Solaris is Loose; this must be changed to Adaptive.
    • Adaptive routing can be set without disruption of service.

     

    Tune Virtual Machines

     

    Check the TCP/IP and NFS tuning on each VM that is using the external storage for Oracle RMAN backups. Depending on the code level running on the Oracle MiniCluster systems, you might not need to add any of these settings, because the installation process configures these settings automatically.

     

    Make sure the following entries are in the /etc/system file on each VM that is using the external storage for Oracle RMAN backups. Any changes made to the /etc/system file will require a reboot of the VM to take effect.

     

    set rpcmode:clnt_max_conns = 8
    set nfs:nfsv4_bsize=131072

     

    To tune the TCP/IP stack for higher performance, the following ipadm commands can be added to boot scripts, be run on the command line, or be added to backup scripts.

     

    ipadm set-prop -p max_buf=2097152 tcp
    ipadm set-prop -p send_buf=1048576 tcp
    ipadm set-prop -p recv_buf=1048576 tcp

     

    Tune the Oracle Database Instance for Oracle RMAN

     

    Tuning Oracle RMAN for high performance can be done with SQL*Plus or Oracle RMAN. Table 2 lists the Oracle Database parameters that need tuning.

     

    Table 2. Oracle Database Parameters

     

        

    ParameterDescription
    _backup_disk_bufcntNumber of buffers used to process backup sets
    _backup_disk_bufszSize of the buffers used to process backup sets
    _backup_file_bufcntNumber of buffers used to process image copies
    _backup_file_bufszSize of the buffers used to process image copies

     

    The Oracle RMAN tuning can be set with SQL*Plus to apply the settings to all Oracle RMAN sessions, as follows:

     

    SQL> alter system set "_backup_disk_bufcnt"=64; 
    SQL> alter system set "_backup_file_bufcnt"=64; 
    SQL> alter system set "_backup_disk_bufsz"=1048576; 
    SQL> alter system set "_backup_file_bufsz"=1048576;

     

    The Oracle RMAN tuning can also be set in each Oracle RMAN session by adding the following parameters to the run block of Oracle RMAN scripts:

     

    sql 'alter system set "_backup_disk_bufcnt"=64';
    sql 'alter system set "_backup_disk_bufsz"=1048576'; 

     

    Create Directories on the Shared Storage

     

    You must create a directory on the /sharedstore mount point, because the system configures the default mount point owner to be root and this does not meet Oracle RMAN requirements. This directory must be owned by the oracle user and either the oinstall or dba group on the database VMs to meet Oracle RMAN requirements. The directory for scripts is not required, but it is best practice to store Oracle RMAN scripts on the shared storage to enable both nodes to access the same scripts. The DatabaseName directory is also not required, but this is a good way to keep the Oracle RMAN output for each database together.

     

    # mkdir /sharedstore/rman
    # mkdir /sharedstore/rman/scripts
    # mkdir /sharedstore/rman/DatabaseName
    # chmod -R 775 /sharedstore/rman
    # chown -R oracle:dba /sharedstore/rman

     

    Configuring the ORANFSTAB File for dNFS

     

    Using dNFS is required for all database backups to NFS shares on Oracle MiniCluster, because NFSv3 is not supported and using dNFS is the only way to use NFSv4 with Oracle RMAN. Databases created with the Oracle MiniCluster management utility should already be configured for dNFS by default, if shared storage was enabled when you created the database. You can also create the file manually, if necessary.

     

    If you are using an Oracle Real Application Clusters (Oracle RAC) database, you must set up ORANFSTAB files on both cluster nodes. The default location for the ORANFSTAB files is the $ORACLE_HOME/dbs directory. Note that the command syntax for the export and mount parameters requires that both entries be on the same line, even if the text wraps to a new line on the screen while you are adding the text to the file. If you make sure this file is configured before enabling or using dNFS, this will avoid the need to restart the database an additional time.

     

    The following are key settings not typically found in the ORANFSTAB file:

     

    • nfs_version: NFSv3 is the default for Oracle Database, but NFSv4 is required.
    • local: This setting is required on Oracle MiniCluster because there are always multiple networks on the VMs and you need to direct the NFS traffic over the correct ports. This setting is especially helpful if you are using the extra network ports on the systems for backup and recovery.

     

    server: 10.129.73.170
    nfs_version: NFSv4
    local: 10.129.73.174
    path: 10.129.73.170
    export: /sharedstore/sharedstore mount: /sharedstore

     

    Enable dNFS

     

    Databases created with the Oracle MiniCluster management utility should be already configured for dNFS by default, if shared storage was enabled when you created the database. You can also enable dNFS manually, if necessary, by running the following commands. If you are using an Oracle RAC database, you must set up ORANFSTAB files on both nodes. No matter what type of database you are using, the database must be restarted to enable dNFS.

     

    # cd $ORACLE_HOME/lib
    # make -f ins_rdbms.mk dnfs_on
    # mv $ORACLE_HOME/lib/libodm12.so $ORACLE_HOME/lib/libodm12.so.dnfsenabled;
    # cp $ORACLE_HOME/lib/libnfsodm12.so $ORACLE_HOME/lib/libodm12.so

     

    Create Cluster Services for Oracle RMAN

     

    Create cluster services to allow Oracle RMAN to connect to the database regardless of which node the backup or the database is being run from. Cluster services are also a great way to load balance the Oracle RMAN backups across Oracle RAC nodes or networks.

     

    The following is the general command to create cluster services:

     

    # srvctl add service -d DatabaseName -s ServiceName -r PrimaryNode -a SecondaryNodes

     

    As an example, the following commands create cluster services RMAN1 and RMAN2 for database VIS.

     

    # srvctl add service -d VIS -s RMAN1 -r VIS001 -a VIS002
    # srvctl add service -d VIS -s RMAN2 -r VIS002 -a VIS001

     

    If you have password and login issues when connecting to the services, you might need to copy the local password files to external password files, as seen in the following example.

     

    # cp $ORACLE_HOME/dbs/orapwDBName $ORACLE_HOME/dbs/orapwDBInstanceName

     

    As an example, the following commands copy the local password file to external password files.

     

    # cp $ORACLE_HOME/dbs/orapwVIS $ORACLE_HOME/dbs/orapwVIS001
    # cp $ORACLE_HOME/dbs/orapwVIS $ORACLE_HOME/dbs/orapwVIS002

     

    If the services are ever running on a secondary (and not primary) node, you can stop and start the services to reset them back to their primary nodes, as shown in the following example.

     

    # srvctl stop service RMAN1
    # srvctl stop service RMAN2
    # srvctl start service RMAN1
    # srvctl start service RMAN2

     

    Example Scripts

     

    The following sections contain example Oracle RMAN backup scripts. Examples include the following:

     

     

    Example Oracle RMAN Full Backup Script

     

    This example Oracle RMAN script performs a full backup of an Oracle RAC database.

     

    The following are key points about using the script:

     

    • Make sure to load balance the channels by switching nodes every other channel allocation.
    • Ideally, use the same mount points on each node.
    • Use more Oracle RMAN channels to raise performance and use fewer Oracle RMAN channels to lower performance.
    • Store the database on flash storage to make it easy to tune the number of channels to any backup target.

     

    alter database begin backup;
    run {
    allocate channel ch001 device type disk connect 'sys/SYSPWD@DBscan/RMAN1' format '/backups/rman/DBNAME/%U';
    allocate channel ch002 device type disk connect 'sys/SYSPWD@DBscan/RMAN2' format '/backups/rman/DBNAME/%U';
    allocate channel ch003 device type disk connect 'sys/SYSPWD@DBscan/RMAN1' format '/backups/rman/DBNAME/%U';
    allocate channel ch004 device type disk connect 'sys/SYSPWD@DBscan/RMAN2' format '/backups/rman/DBNAME/%U';
    allocate channel ch005 device type disk connect 'sys/SYSPWD@DBscan/RMAN1' format '/backups/rman/DBNAME/%U';
    allocate channel ch006 device type disk connect 'sys/SYSPWD@DBscan/RMAN2' format '/backups/rman/DBNAME/%U';
    allocate channel ch007 device type disk connect 'sys/SYSPWD@DBscan/RMAN1' format '/backups/rman/DBNAME/%U';
    allocate channel ch008 device type disk connect 'sys/SYSPWD@DBscan/RMAN2' format '/backups/rman/DBNAME/%U';
     
    backup archivelog all not backed up 1 times delete input
    TAG 'logs_bb' FORMAT '/backups/rman/DBNAME/%d_%s_%p_%c_%t.arc.rman';
    BACKUP database
    TAG 'full_bkset' FORMAT '/backups/rman/DBNAME/%d_%s_%p_%c_%t.db.rman';
    backup archivelog all not backed up 1 times delete input
    TAG 'logs_bb' FORMAT '/backups/rman/DBNAME/%d_%s_%p_%c_%t.arc.rman';
    backup current controlfile
    TAG 'ctl_bk' FORMAT '/backups/rman/DBNAME/%d_%s_%p_%c_%t.ctl.rman';
    backup spfile
    TAG 'spfile_bk' FORMAT '/backups/rman/DBNAME/%d_%s_%p_%c_%t.spfile.rman';
    DELETE FORCE NOPROMPT obsolete;
     
    release channel ch001;
    release channel ch002;
    release channel ch003;
    release channel ch004;
    release channel ch005;
    release channel ch006;
    release channel ch007;
    release channel ch008;
    }
    alter database end backup;
    exit

     

    Example Oracle RMAN Incremental Level 1 Backup Script

     

    This example Oracle RMAN script performs an incremental level 1 backup of an Oracle RAC database.

     

    The following are key points about using the script:

     

    • Make sure to load balance the channels by switching nodes every other channel allocation.
    • Ideally, use the same mount points on each node.
    • Use more Oracle RMAN channels to raise performance and use fewer Oracle RMAN channels to lower performance.
    • Use block change tracking to reduce the runtime of incremental backups.

     

    alter database begin backup;
    run {
    allocate channel ch001 device type disk connect 'sys/SYSPWD@DBscan/RMAN1' format '/backups/rman/DBNAME/%U';
    allocate channel ch002 device type disk connect 'sys/SYSPWD@DBscan/RMAN2' format '/backups/rman/DBNAME/%U';
    allocate channel ch003 device type disk connect 'sys/SYSPWD@DBscan/RMAN1' format '/backups/rman/DBNAME/%U';
    allocate channel ch004 device type disk connect 'sys/SYSPWD@DBscan/RMAN2' format '/backups/rman/DBNAME/%U';
    allocate channel ch005 device type disk connect 'sys/SYSPWD@DBscan/RMAN1' format '/backups/rman/DBNAME/%U';
    allocate channel ch006 device type disk connect 'sys/SYSPWD@DBscan/RMAN2' format '/backups/rman/DBNAME/%U';
    allocate channel ch007 device type disk connect 'sys/SYSPWD@DBscan/RMAN1' format '/backups/rman/DBNAME/%U';
    allocate channel ch008 device type disk connect 'sys/SYSPWD@DBscan/RMAN2' format '/backups/rman/DBNAME/%U';
     
    backup archivelog all not backed up 1 times delete input
    TAG 'logs_bb' FORMAT '/backups/rman/DBNAME/%d_%s_%p_%c_%t.arc.rman';
    backup INCREMENTAL LEVEL 1 database
    TAG 'full_bkset' FORMAT '/backups/rman/DBNAME/%d_%s_%p_%c_%t.db.rman';
    backup archivelog all not backed up 1 times delete input
    TAG 'logs_bb' FORMAT '/backups/rman/DBNAME/%d_%s_%p_%c_%t.arc.rman';
    backup current controlfile
    TAG 'ctl_bk' FORMAT '/backups/rman/DBNAME/%d_%s_%p_%c_%t.ctl.rman';
    backup spfile
    TAG 'spfile_bk' FORMAT '/backups/rman/DBNAME/%d_%s_%p_%c_%t.spfile.rman';
    DELETE FORCE NOPROMPT obsolete;
     
    release channel ch001;
    release channel ch002;
    release channel ch003;
    release channel ch004;
    release channel ch005;
    release channel ch006;
    release channel ch007;
    release channel ch008;
    }
    alter database end backup;
    exit

     

    Example Oracle RMAN Image Copy Backup Script

     

    This example Oracle RMAN script performs an image copy backup of an Oracle RAC database.

     

    The following are key points about the script:

     

    • It creates copies of database files.
    • It enables mounting the database from the copies made by Oracle RMAN.
    • It provides the highest performing backup.

     

    alter database begin backup;
    run {
    allocate channel ch001 device type disk connect 'sys/SYSPWD@DBscan/RMAN1' format '/backups/rman/DBNAME/%U';
    allocate channel ch002 device type disk connect 'sys/SYSPWD@DBscan/RMAN2' format '/backups/rman/DBNAME/%U';
    allocate channel ch003 device type disk connect 'sys/SYSPWD@DBscan/RMAN1' format '/backups/rman/DBNAME/%U';
    allocate channel ch004 device type disk connect 'sys/SYSPWD@DBscan/RMAN2' format '/backups/rman/DBNAME/%U';
    allocate channel ch005 device type disk connect 'sys/SYSPWD@DBscan/RMAN1' format '/backups/rman/DBNAME/%U';
    allocate channel ch006 device type disk connect 'sys/SYSPWD@DBscan/RMAN2' format '/backups/rman/DBNAME/%U';
    allocate channel ch007 device type disk connect 'sys/SYSPWD@DBscan/RMAN1' format '/backups/rman/DBNAME/%U';
    allocate channel ch008 device type disk connect 'sys/SYSPWD@DBscan/RMAN2' format '/backups/rman/DBNAME/%U';
     
    backup archivelog all not backed up 1 times delete input
    TAG 'logs_bb' FORMAT '/backups/rman/DBNAME/%d_%s_%p_%c_%t.arc.rman';
    backup incremental level 1 for recover of copy with 
    tag 'IMAGECOPY' database;
    backup archivelog all not backed up 1 times delete input
    TAG 'logs_bb' FORMAT '/backups/rman/DBNAME/%d_%s_%p_%c_%t.arc.rman';
    backup current controlfile
    TAG 'ctl_bk' FORMAT '/backups/rman/DBNAME/%d_%s_%p_%c_%t.ctl.rman';
    backup spfile
    TAG 'spfile_bk' FORMAT '/backups/rman/DBNAME/%d_%s_%p_%c_%t.spfile.rman';
    DELETE FORCE NOPROMPT obsolete;
     
    release channel ch001;
    release channel ch002;
    release channel ch003;
    release channel ch004;
    release channel ch005;
    release channel ch006;
    release channel ch007;
    release channel ch008;
    }
    alter database end backup;
    exit

     

    Example Oracle RMAN Image Merge Script

     

    This example Oracle RMAN script performs an image merge backup of an Oracle RAC database.

     

    The following are key points about the script:

     

    • It is a very I/O-intensive process that must be run on database nodes.
    • It enables faster restores with less log replaying.
    • It must be run after image copy backups.
    • It works only on images copies.

     

    run {
    allocate channel ch001 device type disk connect 'sys/SYSPWD@DBscan/RMAN1' format '/backups/rman/DBNAME/%U';
    allocate channel ch002 device type disk connect 'sys/SYSPWD@DBscan/RMAN2' format '/backups/rman/DBNAME/%U';
    allocate channel ch003 device type disk connect 'sys/SYSPWD@DBscan/RMAN1' format '/backups/rman/DBNAME/%U';
    allocate channel ch004 device type disk connect 'sys/SYSPWD@DBscan/RMAN2' format '/backups/rman/DBNAME/%U';
    allocate channel ch005 device type disk connect 'sys/SYSPWD@DBscan/RMAN1' format '/backups/rman/DBNAME/%U';
    allocate channel ch006 device type disk connect 'sys/SYSPWD@DBscan/RMAN2' format '/backups/rman/DBNAME/%U';
    allocate channel ch007 device type disk connect 'sys/SYSPWD@DBscan/RMAN1' format '/backups/rman/DBNAME/%U';
    allocate channel ch008 device type disk connect 'sys/SYSPWD@DBscan/RMAN2' format '/backups/rman/DBNAME/%U';
     
    recover copy of database with 
    tag 'IMAGECOPY';
     
    release channel ch001;
    release channel ch002;
    release channel ch003;
    release channel ch004;
    release channel ch005;
    release channel ch006;
    release channel ch007;
    release channel ch008;
    }
    exit

     

    Example Oracle RMAN Restore Validate Script

     

    This example Oracle RMAN script validates the restore operation of an Oracle RAC database without actually restoring anything. You can remove the word validate from the example script to make it a restore database script.

     

    The following are key points about the script:

     

    • It verifies that a restore could be done.
    • It shows how the restore would perform if the script were run.
    • It should be run after any changes are made to backup scripts.

     

    run {
    allocate channel ch001 device type disk connect 'sys/SYSPWD@DBscan/RMAN1' format '/backups/rman/DBNAME/%U';
    allocate channel ch002 device type disk connect 'sys/SYSPWD@DBscan/RMAN2' format '/backups/rman/DBNAME/%U';
    allocate channel ch003 device type disk connect 'sys/SYSPWD@DBscan/RMAN1' format '/backups/rman/DBNAME/%U';
    allocate channel ch004 device type disk connect 'sys/SYSPWD@DBscan/RMAN2' format '/backups/rman/DBNAME/%U';
    allocate channel ch005 device type disk connect 'sys/SYSPWD@DBscan/RMAN1' format '/backups/rman/DBNAME/%U';
    allocate channel ch006 device type disk connect 'sys/SYSPWD@DBscan/RMAN2' format '/backups/rman/DBNAME/%U';
    allocate channel ch007 device type disk connect 'sys/SYSPWD@DBscan/RMAN1' format '/backups/rman/DBNAME/%U';
    allocate channel ch008 device type disk connect 'sys/SYSPWD@DBscan/RMAN2' format '/backups/rman/DBNAME/%U';
     
    restore validate database;
     
    release channel ch001;
    release channel ch002;
    release channel ch003;
    release channel ch004;
    release channel ch005;
    release channel ch006;
    release channel ch007;
    release channel ch008;
    }
    exit

     

    See Also

     

    For more information, please see the following resources:

     

     

    About the Author

     

    Dean Halbeisen is a solutions manager at Oracle. He has over 20 years of IT experience and is an expert in enterprise computing solutions, most recently applying these practices to next-generation data center solutions, integrated systems, and Oracle engineered systems. In his current role, he is responsible for solution architecture and development around Oracle Optimized Solutions, including communicating about Oracle's systems, solutions, technology strategies, and roadmaps to customers, partners, and internal stakeholders.