Using Oracle Optimized Solution for Secure Backup and Recovery with Oracle SuperCluster

steph-choyer-Oracle

by Dean Halbeisen

Oracle Optimized Solution for Secure Backup and Recovery provides a comprehensive, secure, and high-performance solution to protect Oracle SuperCluster systems. This solution includes pretested and recommended best practices, and the flexible and scalable architecture supports tape, disk, and cloud-based backups.

Table of Contents
Solution Overview
Architecture Overview
Protecting Oracle SuperCluster Infrastructure
Protecting Unstructured Data
Protecting Oracle Database 11g Release 2 or Higher
Providing Efficiency Savings and High Availability with Remote DR Sites
Best Practices for Secure Backup and Recovery Implementation
Backup Schedules
Monitoring and Troubleshooting
Key Performance Observations and Backup and Restore Rates
Connecting Backup and Recovery Devices to Oracle SuperCluster
Conclusion
See Also
About the Author

Solution Overview Oracle Optimized Solution for Secure Backup and Recovery can be used as a complete solution for performing backups of Oracle SuperCluster systems. Leveraging built-in integration of Oracle hardware and software, it provides pretested, recommended solutions for the backup and recovery of Oracle SuperCluster and has the following characteristics: Comprehensive. This solution provides complete end-to-end data protection for Oracle SuperCluster, and it supports a wide range of backup options including disk, tape, and cloud-based backups. Secure. Oracle Optimized Solutions leverage Oracle's SPARC-based systems, which offer high-performance security using cryptographic instruction accelerators that are directly integrated into the processor cores.

Oracle Optimized Solutions provide tested and proven best practices for how to run software products on Oracle systems. Learn more.

• High performance. Native high-performance networking built into Oracle SuperCluster engineered systems provides faster backup and recovery of applications and databases. Inherent scalability and no single point of failure help ensure service-level agreements (SLAs) are met as data capacity grows.
  
  
• Lower cost. This solution supports disk-only backup and recovery options that do not rely on tape backups. These configurations eliminate the need to license additional tape management software, thereby reducing costs.
  
  
• Simplified. This solution uses tested configurations that have been optimized for Oracle hardware and software, and Oracle Enterprise Manager provides end-to-end management and monitoring. Consolidation of disparate legacy backup solutions onto a single multipurpose backup and recovery platform can eliminate the need for specialized backup and recovery hardware.

Architecture Overview

Oracle Optimized Solution for Secure Backup and Recovery features a flexible and scalable architecture for performing backups of Oracle SuperCluster (see Figure 1). This solution provides options for traditional disk and tape backups as well as cloud-based backups, and includes Oracle ZFS Storage Appliance and Oracle's StorageTek tape libraries. In addition, Oracle's Zero Data Loss Recovery Appliance can be deployed to revolutionize database protection with its incremental-forever backup strategy, which eliminates potential loss and dramatically decreases backup overhead.

Figure 1. Oracle Optimized Solution for Secure Backup and Recovery supports different backup target devices and cloud database backups for Oracle SuperCluster systems.

Designed to be software agnostic, the solution can be implemented with the Oracle Recovery Manager (Oracle RMAN) feature of Oracle Database, Oracle Secure Backup, Symantec NetBackup, or other third-party backup software. For illustration purposes, this paper refers to the use of Oracle Secure Backup software throughout for the tape management component of the solution. Note that in disk-only environments—such as those using Oracle Exadata storage, Oracle's Exadata Storage Expansion Racks, or Oracle ZFS Storage Appliance—no additional backup software is required. Disk backups can be completed using Oracle SuperCluster tools, operating system utilities, and Oracle Database tools (such as Oracle RMAN) alone.

For more information on the architecture and components in Oracle Optimized Solution for Secure Backup and Recovery, refer to the Oracle website oracle.com/solutions/optimized-solutions/backup-and-recovery.

Disk and Tape Backup and Recovery Options

Oracle Optimized Solution for Secure Backup and Recovery, when used with Oracle SuperCluster, provides a choice of backup and restore methods:

• Tape-based backup (using Oracle's StorageTek modular library systems)
• Disk-based backup (using Oracle ZFS Storage Appliance or Exadata Storage Expansion Racks)
• Zero Data Loss Recovery Appliance
• Disk-to-disk-to-tape backups

Disk and tape backups each have their advantages. Disk backups can provide backed-up data immediately in the event of a disaster, while a restore operation from tape might take longer than desired for a business recovery solution. Disk-only backups provide faster recovery times for data and logical corruptions and some tablespace point-in-time recovery (TSPITR) scenarios. Backups to disk also provide the ability to use backups directly—with no restore needed—by switching to a copy of the database, tablespace, or data file after performing image backups.

Tape backups can provide cost-effective, long-term storage for valuable enterprise data. Economical for archiving, tape media are also portable, enabling off-site storage for purposes such as disaster recovery (DR) protection and regulatory compliance.

Disk-to-disk-to-tape backups combine both disk and tape backups: Data is initially backed up to disk and then copied again to tape. Performing disk-to-disk-to-tape backups enables administrators to leverage the advantages of both disk and tape backups and provide for various contingencies.

When used with Oracle SuperCluster, Oracle Optimized Solution for Secure Backup and Recovery includes two disk-based options: Oracle ZFS Storage Appliance or Exadata Storage Expansion Racks. The different configurations are designed to meet a variety of backup solution requirements. In general, Exadata Storage Expansion Racks provide the highest performance and serviceability, while Oracle ZFS Storage Appliances provide flexibility and a lower-cost option.

Optionally, Zero Data Loss Recovery Appliance can be used for database protection. This engineered system can be used to provide a centralized backup strategy for hundreds to thousands of databases in the enterprise, using fully fault-tolerant hardware and storage.

Cloud Deployment

Oracle Optimized Solution for Secure Backup and Recovery supports cloud-based backups. Oracle Database instances can be backed up to the public cloud using Oracle Database Backup Cloud Service, or Oracle Secure Backup Cloud Module can be used to back up databases to cloud-based storage services offered by Amazon Simple Storage Service (Amazon S3). Third-party backup and recovery cloud offerings are also available to back up applications and other unstructured data.

Enterprises can also choose to deploy backup and recovery solutions in private cloud environments. Isolation technologies inherent to this solution's cloud infrastructure design provide comprehensive security and data protection. For example, the following hardware and software isolation features help provide end-to-end data security:

• Zero Data Loss Recovery Appliance: User ID ownership of all resources isolates databases and helps protect against unauthorized access from other databases on the cloud-based backup.
• Oracle ZFS Storage Appliance: Virtualized networking and ZFS storage pools provide isolation and protection of resources. Encryption provides an additional layer of protection for data resources.
• StorageTek tape libraries: Hardware and software virtualization options can be used to logically or physically isolate both tapes and media. Encryption provides an additional layer of protection for data resources.

Oracle Enterprise Manager 13c provides a single interface for administrators to manage both their on-premises deployments and those deployed in Oracle Cloud.

Requirements for Complete Coverage for Oracle SuperCluster Recovery

To provide complete protection against loss of data and operations in Oracle SuperCluster, the backup and recovery plan must include the cluster infrastructure, unstructured data—including the operating system and applications—and Oracle Database (see Table 1).

Table 1. Oracle SuperCluster Backup and Recovery Frequency

           

Category	Components	Backup Frequency
Infrastructure	- Control domain configurations - Service Processor (SP) and Oracle Integrated Lights Out Manager (Oracle ILOM) configuration - Switch configurations - Internal Oracle ZFS Storage Appliance configuration - Operating systems of domains and zones - Oracle SuperCluster–specific data and configuration - Oracle SuperCluster tools	Backed up less frequently, based upon upgrades and system changes
Unstructured data	- Infrastructure backups - Operating system -  Applications - NFS shares	Backed up daily and after upgrades and system changes
Database	Oracle Database	Backed up daily and after upgrades and system changes

The following list provides more details about the backup requirements for the three categories:

• Oracle SuperCluster infrastructure. Oracle SuperCluster infrastructure configuration information, including the networking and switch configuration, must be backed up to preserve the settings and provide for a quick recovery process in the event of a disaster. The Oracle SuperCluster configuration backup tool (osc-config-backup) is used to back up configuration information. The operating systems of all domains and zones in the cluster are also backed up when this tool is run.
• Unstructured data. Applications and other data stored on the internal Oracle ZFS Storage Appliance, including the infrastructure backups, must also be backed up regularly. In addition, daily backups of the operating system are recommended (in addition to the less frequent backups provided by the osc-config-backup tool). These backups are similar to standard backups for any server in an enterprise environment.
• Oracle Database. Oracle Database is a vital element of Oracle SuperCluster and must be backed up to ensure against loss of business- or mission-critical data.

Infrastructure data requires less-frequent backups; these backups are based primarily on upgrades and changes to the Oracle SuperCluster system. Unstructured data and databases are typically backed up daily and upon upgrades and system changes. The following sections discuss backup and recovery of Oracle SuperCluster and include best practices designed for high availability, optimal performance, and maximum data protection.

Protecting Oracle SuperCluster Infrastructure

Oracle SuperCluster configuration information, including switch configurations for the InfiniBand (IB) fabric, is created when Oracle SuperCluster is first deployed. This configuration information must be backed up to preserve the settings and enable quick recovery in case of human error or a disaster. Backups of the infrastructure configuration are recommended after installation and upon major system updates or configuration changes.

Backing Up Oracle SuperCluster Infrastructure with osc-config-backup

The Oracle SuperCluster configuration backup tool, osc-config-backup, is used to create a snapshot of the entire Oracle SuperCluster infrastructure. (At the time of publication, the osc-config-backup tool supports only Oracle SuperCluster M7 and T5-8 systems.) The built-in osc-config-backup tool is included with the Oracle SuperCluster software and provided at no additional cost. It is intended for backing up the system configuration rather than for performing daily backups. This tool should be used to perform configuration backups for the following events:

• After initial deployment, to provide an initial backup snapshot of the Oracle SuperCluster infrastructure
• Before and after reconfiguration of significant operating parameters
• Before and after installation of each quarterly full stack download patch (QFSDP) for Oracle SuperCluster
• As part of a regular maintenance schedule, at a frequency that is practical and feasible (monthly, for example)

The osc-config-backup tool automates the process of backing up system configuration information for all components in Oracle SuperCluster—including the networking configuration, and Oracle Solaris domains and Oracle Solaris Zones on the compute nodes—in a standard way. This tool performs multiple steps sequentially to prepare the environment and capture the configuration information. Specifically, the following Oracle SuperCluster components are backed up by the osc-config-backup tool:

• Configuration information for Logical Domains (LDoms) and Oracle Solaris Zones on the Oracle SuperCluster compute nodes
• rpool—and u01-pool, if it exists—on domains and zones
• Configuration information for InfiniBand and management switches
• Oracle Explorer Data Collector information from each LDom running either a database or application
• Oracle SuperCluster deployment configuration information
• Oracle ZFS Storage Appliance configuration information
• Configuration information for service processors
• Oracle ILOM configuration information for hardware components

It is equally important to note that the following are not backed up by the osc-config-backup tool: Oracle Database instances and other information stored on NFS shares on the internal Oracle ZFS Storage Appliance, including applications and other unstructured data. For complete protection of Oracle SuperCluster, backups of this information are also recommended.

The backups created by the osc-config-backup tool are stored on the Oracle ZFS Storage Appliance that is internal to Oracle SuperCluster. It is critical that this backup data is, in turn, backed up outside Oracle SuperCluster as part of an enterprise backup and recovery strategy to safeguard against catastrophic failures and disasters. Replication, tape backups, and disk-to-disk-to-tape backups are strategies that can be used to back up the information on the internal Oracle ZFS Storage Appliance.

For the latest information on using the osc-config-backup tool, refer to My Oracle Support Document 1934129.1, "SuperCluster—How to back up a SuperCluster using the osc-config-backup tool."

Restoring Oracle SuperCluster Infrastructure

Information on how to restore Oracle SuperCluster from backups performed by the osc-config-backup tool is contained in My Oracle Support Document 1934130.1, "How to Recover SuperCluster Components That Were Backed Up with osc-config-backup." The backup data, stored on the internal Oracle ZFS Storage Appliance, supports a large scope of recovery scenarios.

Each component that is backed up can be restored independently. As an example, a single Oracle Solaris Zone can be restored directly, without having to restore the entire Oracle SuperCluster. Such a single-component recovery is straightforward and can typically be performed by administrators without additional support from Oracle.

However, some recovery scenarios induce more complexity. For example, restoring a database domain that is hosting multiple zones or restoring an entire server is a more complex procedure. A significant part of the complexity results from the dependency between the different components that need to be restored, and restoration details vary among components. For these more-intricate restoration scenarios, Oracle SuperCluster support services are available to help execute the recovery procedure.

Protecting Local Backups

System configuration backups of the Oracle SuperCluster infrastructure created using the osc-config-backup tool are stored on the internal Oracle ZFS Storage Appliance. The internal Oracle ZFS Storage Appliance can also be used for daily backups of the Oracle SuperCluster operating system and applications. For greater protection in the event of local failures or disasters, these backups should be protected with remote replication to an off-site copy, be backed up to a tape device, or both.

• Remote replication. Data that is stored on the internal Oracle ZFS Storage Appliance can be protected using Oracle ZFS Storage Appliance replication (see Figure 2). With Oracle ZFS Storage Appliance replication, a copy of the backup data is copied from the internal Oracle ZFS Storage Appliance (the source) to an external Oracle ZFS Storage Appliance (the target) through an interconnecting TCP/IP network. This target storage appliance can be a standalone appliance or an internal Oracle ZFS Storage Appliance in a separate Oracle SuperCluster. The target appliance can be located virtually any distance from the source, as long as the interconnecting network has sufficient bandwidth to carry the data. Data on the source system is periodically replicated to the target at user-defined intervals. Data can be transmitted securely using SSL.
  
  

  

  Figure 2. Backup using replication to another Oracle ZFS Storage Appliance.

  For more information on replication implementation details, see the Oracle white paper "Architecture Principles and Implementation Practices for Remote Replication Using Oracle ZFS Storage Appliance."

• Tape backups. In configurations that use tape-based backups, backup and recovery software such as Oracle Secure Backup can be used to perform Network Data Management Protocol (NDMP) backups of the internal Oracle ZFS Storage Appliance to tape (see Figure 3).
  
  

  

  Figure 3. Tape-based backups using Oracle's StorageTek SL150 modular tape library.

• Disk-to-disk-to-tape backups. Data can also be protected using disk-to-disk-to-tape backups (see Figure 4), with the data replicated to an Oracle ZFS Storage Appliance and also backed up to tape. More information on disk-to-disk-to-tape backups is included later in this paper (see "Disk-to-Disk-to-Tape Backups").
  
  

  

  Figure 4. Disk-to-disk-to-tape backup and recovery of Oracle SuperCluster.

Protecting Unstructured Data

Day-to-day backups are required to provide ongoing protection of applications and other unstructured data installed on NFS shares on the internal Oracle ZFS Storage Appliance. Daily operating system backups of the Oracle Solaris domains and Oracle Solaris Zones within Oracle SuperCluster are also necessary. Although the Oracle SuperCluster configuration backup tool (osc-config-backup) does back up the operating systems of domains and zones, these backups are not performed daily. Additional daily backups provide the usual protection from unintentional file deletion, unexpected problems, and administrator error; and they are needed to provide point-in-time recovery past the latest full system backup.

Daily Oracle SuperCluster backups of unstructured data (that is, non-database backups) are similar to backups for other Oracle Solaris servers. Systems are backed up and restored in normal operations using backup and recovery software clients on the operating systems (Oracle Solaris domains and Oracle Solaris Zones) in the system.

Note: The operating system on the internal Oracle Exadata Storage Servers does not have to be backed up. Recovery of those servers is accomplished in a similar manner to restoring the firmware on a switch.

Backing Up the Operating System on Compute Nodes

The conventional method for backing up an operating system is to install backup software and back up the operating system to disk or tape. Operating system backups are critical and should be made regularly as well as before and after every significant change to Oracle SuperCluster software. In Oracle Solaris 10 and Oracle Solaris 11, ZFS file system snapshots can also be used for fast backup of the operating system.

Operating system backups should be performed regularly (typically daily) and before and after the following procedures:

• Installation or reconfiguration of significant software or applications
• Reconfiguration of significant operating parameters
• Before and after each quarterly full stack download patch (QFSDP) for Oracle SuperCluster

Backing Up Applications and iSCSI LUNs

Backup and recovery of applications is supported in NFS, and backup and recovery procedures remain the same for snapshots and replication of NFS shares, as well as for backup to tape using NDMP. To back up NFS shares, use the internal Oracle ZFS Storage Appliance with NDMP using the dump backup type. The dump backup type provides high-performance, block-level backups. For more information on using NDMP with Oracle ZFS Storage Appliance, please refer to the Oracle white paper "NDMP Implementation Guide for the Oracle ZFS Storage Appliance."

Within Oracle SuperCluster, Oracle Solaris Zones typically contain applications or databases for different use cases. Oracle Solaris Zones can be installed on Oracle SuperCluster internal drives or on iSCSI LUNs.

To back up the iSCSI LUNs, use the internal Oracle ZFS Storage Appliance with NDMP using the zfs backup type. The zfs backup type does not support file history or direct access recovery (DAR), but it might be faster for some data sets. The iSCSI LUNs can also be backed up with ZFS snapshots and replication on the internal Oracle ZFS Storage Appliance, providing a high-performance method that works well for restoring an entire zone.

Protecting Oracle Database 11g Release 2 or Higher

A complete strategy for high availability and disaster recovery requires proven and dependable data backup, restore, and recovery procedures. Oracle RMAN provides a comprehensive foundation for efficiently backing up and recovering Oracle Database. Designed to work intimately with the database server, Oracle RMAN provides block-level corruption detection during backup and restore. It also optimizes performance and space consumption during backups through file multiplexing and backup set compression.

Oracle RMAN takes care of all underlying database procedures before and after backup or restore operations, eliminating a dependency on operating system and SQL*Plus scripts. Oracle RMAN provides a common interface, by means of the command line and Oracle Enterprise Manager, for backup tasks across different host operating systems. Oracle RMAN offers features—such as parallelization of backup and restore data streams, retention policies for backup files, and detailed backup histories—that are not available through user-managed methods. Oracle RMAN integrates with Oracle Secure Backup as well as third-party media management products for tape backup.

Best Practice: Always Use a Fast Recovery Area

The following sections include best practices for Oracle Database backup and recovery in Oracle SuperCluster when using Exadata Storage Expansion Racks, Oracle ZFS Storage Appliance, tape-based backups, or Zero Data Loss Recovery Appliance. One best practice recommendation applies to all use cases: Always deploy the database with a fast recovery area (FRA).

The FRA is a storage location where the database stores all its information for Oracle Flashback technologies, snapshots, database backups, rollbacks, and more. The FRA enables the database to protect itself, rolling the database back to a certain point in time and restoring it within a matter of seconds instead of minutes or hours when conventional backup and recovery methods are used. Consequently, database recovery time is limited only by the speed of the device on which the FRA resides, and the FRA should be located on the fastest performing storage device possible. Oracle engineered systems such as Oracle SuperCluster are designed with the FRA built in, because placing the FRA on anything but Oracle Exadata internal storage or an external Exadata Storage Expansion Rack will lengthen the amount of time it takes for recovery. Exadata Storage Expansion Racks can be used for extra space for the FRA, as needed.

Database backups can, if desired, be excluded from the FRA and stored instead on Oracle ZFS Storage Appliance. In this case, the FRA on Oracle Exadata can be configured to be much smaller, because space for the full and incremental backups is not required in the FRA. This configuration can provide cost savings, because lower-cost storage can be used for database backups, without sacrificing system performance.

Using Exadata Storage Expansion Racks to Protect Oracle Database

Using Exadata Storage Expansion Racks for Oracle Database backups provides the highest performance and service levels, and this configuration is the general recommendation for performing Oracle SuperCluster database backups to disk. Oracle Database backup technologies—Oracle RMAN and Oracle Secure Backup—perform especially well on Oracle Exadata with its high-bandwidth InfiniBand network and Oracle Exadata Storage Server grid.

The Oracle Maximum Availability Architecture operational and configuration practices for Oracle Exadata provide the most-comprehensive high availability solution for Oracle Database. The following references provide additional information on Oracle Exadata backup and recovery of Oracle Database:

• "Deploying Oracle Maximum Availability Architecture with Exadata Database Machine"
• "Backup and Recovery Performance and Best Practices for Exadata Cell and Oracle Exadata Database Machine"
• My Oracle Support Document 1558851.1, "Oracle Optimized Solution for Secure Backup and Recovery"
• My Oracle Support Document 13549801.1, "Oracle ZFS Storage: FAQ: Exadata RMAN Backup with the Oracle ZFS Storage Appliance"

Best Practices Using Exadata Storage Expansion Racks

General best practices for Oracle SuperCluster configurations using Exadata Storage Expansion Racks include the following:

• Scale backup rates for disk:
  
  - Use all instances and start with two Oracle RMAN channels per instance.
  - Continue to add an additional two Oracle RMAN channels for performance.
  
  
• Utilize the automatic configuration of the Oracle Exadata Storage Server grid disk group layout during deployment for better performance. This approach assigns the faster (outer) 40 percent of the disk to the DATA area and the slower (inner) 60 percent of the disk to the FRA (RECO) area.
• An alternative strategy is to purchase additional serial ATA (SATA) Oracle Exadata storage specifically for storing the FRA. This configuration allows the application to leverage the full Oracle SuperCluster storage grid, allows the use of lower-cost storage for backups, and provides better failure isolation by using separate backup hardware. To reserve more space and bandwidth for the DATA disk group, Oracle recommends using a tape-based backup solution or, at the very least, a hybrid approach where full database backups are written to tape and incremental disk backups are written to the FRA. Note, however, that the FRA should never be placed on Oracle ZFS Storage Appliances. For optimal performance, the FRA should remain on the Oracle Exadata Storage Servers internal to Oracle SuperCluster or on Exadata Storage Expansion Racks.
• Configure a high-redundancy DATA disk group to contain the Oracle Cluster Registry file, Oracle SuperCluster voting disk, spfiles, data files, redo log groups, and control files for any disk-based backup solution.

Best Practices for Performing Local Disk-Based Backups with Oracle RAC

Scale backup rates for data written to local disk in the FRA on an Oracle Database Appliance by using an Oracle Real Applications Clusters (Oracle RAC) configuration, as follows:

• Use multiple instances and start with one Oracle RMAN channel per instance.
• Continue to add Oracle RMAN channels for performance per instance. Optimal backup rates were observed with all Oracle RAC instances and one to four Oracle RMAN channels.

Best Practices for Performing Local Disk-Based Backups with Oracle RAC One Node

Scale backup rates for data written to local disk in the FRA on Oracle Database Appliance by using a single-instance and Oracle RAC One Node configuration, as follows:

• Start with one Oracle RMAN channel.
• Continue to add Oracle RMAN channels to the single database instance to increase performance. Optimal backup rates were observed with two to four Oracle RMAN channels.

Using an External Oracle ZFS Storage Appliance to Protect Oracle Database

An external Oracle ZFS Storage Appliance can be connected with a 10 GbE connection or directly to the InfiniBand network of Oracle SuperCluster to provide a high-performance backup solution for Oracle SuperCluster. Coengineered with Oracle Database, Oracle ZFS Storage Appliance benefits from features such as Hybrid Columnar Compression and Oracle Intelligent Storage Protocol, which are not available to third-party storage systems. Oracle ZFS Storage Appliances can be specifically tuned for Oracle engineered systems to provide optimum performance.

This solution uses Oracle RMAN with the storage appliance, eliminating the need for additional backup software or a media server and associated software licenses. License-free data services, including snapshots and compression, reduce costs; and with built-in snapshot and cloning capabilities, Oracle ZFS Storage Appliance can be used for value-added work such as test, development, and quality assurance (QA). Oracle Snap Management Utility for Oracle Database is supported with Oracle SuperCluster, automating the creation and management of snapshots and clones on Oracle ZFS Storage Appliances.

The QDR InfiniBand fabric provides a direct high-bandwidth connection between Oracle ZFS Storage Appliance and the Oracle SuperCluster InfiniBand backplane. Backup and restore operations can be parallelized, significantly reducing backup and recovery times compared to a traditional NAS storage system. Example testing has shown that Oracle ZFS Storage Appliance can perform a backup of an Oracle SuperCluster T5-8 half-rack configuration at a data rate of up to 14 TB/hour, and it can perform a restore at a data rate of up to 7 TB/hour. Configurations that use 10 GbE network connectivity can provide nearly equal performance to those using InfiniBand connections.

The following references provide additional information on using Oracle ZFS Storage Appliance for backup and recovery of Oracle Database:

• Oracle technical white paper "Configuring an Oracle ZFS Storage ZS3-BA with an Oracle SuperCluster for Oracle Database Backup and Recovery"
• My Oracle Support Document 1517107.1, "RMAN Backup from SPARC SuperCluster to Sun ZFS Backup Appliance"
• My Oracle Support Document 1354980.1, "Oracle ZFS Storage: FAQ: Exadata RMAN Backup with the Oracle ZFS Storage Appliance"
• Oracle white paper "NDMP Implementation Guide for the Oracle ZFS Storage Appliance"

Configuring an External Oracle ZFS Storage Appliance with Oracle SuperCluster

Correct configuration of an external Oracle ZFS Storage Appliance is required for the best performance of Oracle SuperCluster backups. This section provides an overview of the configuration process. For complete details, see the Oracle technical white paper "Configuring an Oracle ZFS Storage ZS3-BA with an Oracle SuperCluster for Oracle Database Backup and Recovery."

• Physically connect the Oracle ZFS Storage Appliance. If InfiniBand connectivity is used, the Oracle ZFS Storage Appliance must be physically connected to each Oracle SuperCluster system's InfiniBand infrastructure. Both primary and failover paths for Oracle ZFS Storage Appliance must be configured to allow for data availability. Oracle ZFS Storage Appliance can be connected directly to the Oracle SuperCluster InfiniBand switches (if it is the only device connected to the infrastructure), or it can be connected to external InfiniBand leaf switches (if additional appliances or devices will be connected). Once the connections are made, the ports on the Oracle ZFS Storage Appliance must be activated and then configured on the InfiniBand switches.
• Configure the Oracle ZFS Storage Appliance. After the storage appliance is physically connected, it must be properly configured. This configuration includes the following tasks:
  
  

  - Set up InfiniBand/10 GbE networking. Key steps for InfiniBand connectivity include configuring the Oracle ZFS Storage Appliance InfiniBand data links; reconfiguring the Oracle SuperCluster InfiniBand switches to include the GUIDs of the Oracle ZFS Storage Appliance InfiniBand HBA ports; and configuring Oracle ZFS Storage Appliance networking for either a single IP connection or active-active IPMP connection (for configurations with external leaf switches).

  - Configure Oracle ZFS Storage Appliance storage pools. A pool configuration assigns physical disk drive resources to logical storage pools for backup data storage. To maximize system throughput, configure two equally sized storage pools by assigning half of the physical drives in each drive tray to each storage pool.

• Use Oracle Engineered Systems Backup Utility for Oracle ZFS Storage Appliance. Lastly, run the Oracle Engineered Systems Backup Utility for Oracle ZFS Storage Appliance to complete the configuration on the Oracle Solaris 11–based Oracle Database 11g Release 2 database server nodes. This tool, which can be used to configure the Oracle ZFS Storage Appliance that is internal to the Oracle SuperCluster and any additional external Oracle ZFS Storage Appliance for backup, verifies the system configuration and automates the necessary configuration steps, including the following:
  
  

  - Configuring Oracle ZFS Storage Appliance. This includes creating a project and shares for Oracle RMAN.

  - Configuring the Oracle SuperCluster database nodes. This includes enabling direct NFS (dNFS), creating mount points, and adding backup services.

  - Creating the final Oracle RMAN scripts. (Note: The Oracle RMAN run block scripts are intended for samples only, and should be reviewed and modified as necessary to meet site-specific requirements.)

Although these configuration steps can be performed manually, use of the Oracle Engineered Systems Backup Utility is recommended to reduce the risk of accidental misconfiguration.

The Oracle Engineered Systems Backup Utility for Oracle ZFS Storage Appliance can be downloaded from the Oracle ZFS Storage Appliance Plug-in Downloads web page.

Recommended Best Practices for Database Backup and Restore Using Oracle ZFS Storage Appliance

Oracle recommends using a combination of level-0 and level-1 backup sets when backing up Oracle Database to the Oracle ZFS Storage Appliance. The frequency of backups is dictated by recovery time objectives (RTOs) and recovery point objectives (RPOs). Note that when using Oracle ZFS Storage Appliance, Oracle recommends testing an incrementally updated backup strategy using a combination of data file copies and incremental backup sets that are subsequently merged into the data file copies before deployment into production. When an incrementally updated backup strategy is selected, the backup solution becomes I/O bound during the merge process, and alleviating this bottleneck requires a significant number of available disk spindles in order to achieve the required IOPS rate.

For most IT departments, the following recommendations should be sufficient:

• The Oracle Database FRA should remain on the Oracle Exadata Storage Servers that are part of Oracle SuperCluster and should not be put on Oracle ZFS Storage Appliance.
• The weekly full level-0 backup should be written to Oracle ZFS Storage Appliance.
• A daily incremental level-1 backup should be written to Oracle ZFS Storage Appliance. A maximum of 16 Oracle RMAN channels should be allocated and configured for both weekly level-0 and daily level-1 backups to write to one of the 16 Oracle ZFS Storage Appliance shares created using the Oracle Engineered Systems Backup Utility (or created manually).
• Two separate backup jobs should be submitted if Oracle RMAN compression is used to preserve space on Oracle ZFS Storage Appliance but the database consists of a mixture of uncompressed data and compressed data or data that cannot be compressed further. One of the backup jobs will create an uncompressed backup, and the second will create a compressed backup. Otherwise, compressing data that does not typically yield good compression ratios lengthens the backup window significantly, consumes unnecessary CPU resources, and results in very little space savings.
• Oracle RMAN offers an option to amortize the backup over a given time period and minimize the performance impact to critical applications: the Oracle RMAN BACKUP DURATION command with the MINIMIZE LOAD option. For example, if there is an eight-hour backup window available for the weekly level-0 backup of a 20 TB database, but critical database functions are still running during that backup window, Oracle RMAN can spread the work over the entire backup window.
• The use of Oracle RMAN options such as sectionsize is recommended. These options are included in the backup scripts generated by the Oracle Engineered Systems Backup Utility. The sectionsize parameter breaks BIGFILE tablespaces into more-manageable amounts. Oracle recommends that the largest tablespaces be broken into equal sized sections for each Oracle RMAN channel. For instance, if there are two tablespaces of 10 TB and 14 TB, and there are 16 Oracle RMAN channels, the sectionsize parameter should be set to approximately 320 GB.
• The Oracle Engineered Systems Backup Utility does not enable Oracle RMAN or Oracle ZFS Storage Appliance compression, because this capability is dependent on the data to be backed up and the database licensing options purchased. Enabling Oracle RMAN compression increases CPU utilization.
• Although use of the Oracle ZFS Storage Appliance that is internal to the Oracle SuperCluster is supported, using an external Oracle ZFS Storage Appliance is a recommended best practice. Backups inherently consume bandwidth and latency. If the internal storage appliance is used, response time of other executables reading and writing to this storage will be impacted during backup and recovery. Regardless of the storage appliance that is used, the Oracle Engineered Systems Backup Utility should be used to do the final configuration.

Using Tape to Protect Oracle Database

Tape-based backups provide cost-effective data protection for Oracle SuperCluster. Tape backups employ the lowest-cost tape storage media, which is ideal for older backup copies and long-term data retention. Oracle's StorageTek tape libraries and Oracle's StorageTek tape drives work in combination with InfiniBand fabric to provide a high-performance, cost-effective tape-based backup solution for Oracle SuperCluster (see Figure 5). Oracle Optimized Solution for Secure Backup and Recovery is software agnostic, and it supports the use of Oracle Secure Backup, Symantec NetBackup, and other third-party backup software.

Figure 5. Oracle SuperCluster uses backup and recovery software such as Oracle Secure Backup to perform tape-based backups.

Backup and Recovery Software

Organizations have a choice of backup and recovery software when using Oracle Optimized Solution for Secure Backup and Recovery with Oracle SuperCluster. However, Oracle recommends using Oracle Secure Backup for low-cost, fast tape backups. This tape management software component from Oracle carries a low-cost, one-time software licensing fee per tape drive, resulting in significant savings compared to the cost of most competing products. Oracle Secure Backup provides the fastest database backup to tape due to its tight integration with Oracle products, including Oracle RMAN and Oracle Database. Using Oracle Secure Backup enables the unused-block optimization capability. However, if the backup is made directly to tape using a third-party media management product, this capability does not have any effect, because unused-block optimization is available only with Oracle Secure Backup.

Media management software such as Oracle Secure Backup uses a client/server architecture with one administrative server, one or more media servers, and the backup clients. Any server in the domain can act as the administrative server. Note, however, that the media server cannot be configured on Oracle SuperCluster itself; a separate server must be configured. Although installing backup software on Oracle SuperCluster is permitted and necessary, installing additional hardware (such as Fibre Channel cards on an Oracle SuperCluster database node) is not supported. All Oracle SuperCluster backups to tape must use the InfiniBand or 10 GbE network connections through a separate media server to the tape library.

Although Figure 5 shows a single media server, multiple media servers can be used for increased performance. A single server can be used for both the administrative server and media server roles; however, for optimal performance and availability, these roles should not be provided by a single server. Oracle's x86-based systems and Oracle's SPARC servers are recommended as administrative and media servers because of their I/O bandwidth, performance, and scalability.

Database Best Practices for Tape-Based Backups

In addition to the general best practices for Oracle Database backups, best practices for tape-based backups include the following:

• Keep the number of archive logs to a minimum to avoid adversely impacting backup rates.
• Remember that heavy loads on an active database and fully consumed CPUs will affect backup rates.
• Configure persistent bindings for tape devices. It is very important that the environment maintains consistent device addresses.
• Perform daily backups of the Oracle Secure Backup or third-party backup and recovery software catalog. This catalog maintains backup metadata, scheduling, and configuration details for the backup domain, and should be backed up on a regular basis. For more information, see My Oracle Support Document 1558851.1.
• Configure one Oracle RMAN channel per tape drive and add tape drives to scale backup rates. Backup performance scales when more tape drives and Oracle RMAN channels are added, assuming there is available throughput on the media server.
• Configure dedicated 10 gigabit Ethernet (GbE) or InfiniBand. Using a dedicated interface for the transport eliminates the impact on the client access network. Use InfiniBand for the best backup rates, especially for larger databases that require fast backup rates and low CPU overhead. When not using InfiniBand, use 10 GbE.
• Configure an Oracle RAC service for backups running on all database instances. This step reduces CPU utilization on the database and media server nodes and load balances the backups.
• Use Oracle SQL*Net Connect service load balancing to distribute Oracle RMAN channels evenly among the allocated instances.

More Information

For more information on tape backups, see the Oracle white paper "Protecting SPARC SuperCluster—Tape Backup with Symantec NetBackup."

Disk-to-Disk-to-Tape Backups

The disk-to-disk-to-tape approach first backs up data to disk and then copies the data to tape, providing the combined advantages of both disk and tape backups. As with tape-only backups, disk-to-disk-to-tape backups in Oracle SuperCluster are software agnostic: Oracle Secure Backup, Symantec NetBackup, and other third-party backup software are supported. When not using Oracle Secure Backup, additional licensing might be required for third-party backup software.

There are two general methods for performing disk-to-disk-to-tape backups. The first method uses Oracle RMAN to first perform a database backup to external Oracle ZFS Storage Appliance or Exadata Storage Expansion Rack (see Figure 6). The administrator then uses Oracle RMAN to perform a backup BACKUPSET operation, which copies the backup set from disk to tape. Oracle Secure Backup or third-party backup and recovery software such as Symantec NetBackup handles the tape management portion of the backup. Oracle RMAN can restore the database from either disk or tape in a single restore process. (Note: the Oracle Engineered Systems Backup Utility tool does not create example disk-to-disk-to-tape scripts. Details on making copies of the backup sets from disk to tape can be found in the Oracle RMAN documentation.)

With this approach, Oracle RMAN is aware of all backups and manages the retention periods for all backups. For example, Oracle RMAN can set tape retention periods for all tape backups, and then each day delete obsolete backups from disk based on the number of days of backups that should be retained. This method does push the data through the database server twice. However, the process of database servers performing a backup BACKUPSET operation is not resource intensive; therefore, this method has a very low impact on CPU utilization and I/O bus traffic, and it does not significantly affect database performance.

Figure 6. Disk-to-disk-to-tape backups with Oracle RMAN.

The second method for disk-to-disk-to-tape backup first uses Oracle RMAN to perform a database backup to an external Oracle ZFS Storage Appliance, and then it uses Network Data Management Protocol (NDMP) to back up the Oracle ZFS Storage Appliance to tape (see Figure 7). With this method, it is not necessary for the data to be pushed back through the database server during the NDMP process. The downside of this method, however, is that Oracle RMAN is not aware of all the copies, because Oracle RMAN is not used to back up the Oracle ZFS Storage Appliance to tape.

When using this second method, restoring backup data requires an intermediate step. The administrator must first restore the backup from tape to the Oracle ZFS Storage Appliance, and then use Oracle RMAN to restore the data from there back to the database. This procedure can take a long time if the database is very large, and it requires sufficient available space on the Oracle ZFS Storage Appliance. The frequency and time sensitivity of restores from tape should be considered when choosing this backup method.

Figure 7. Disk-to-disk-to-tape backups using NDMP.

Both methods of disk-to-disk-to-tape backup are supported when Oracle Optimized Solution for Secure Backup and Recovery is used with Oracle SuperCluster. Each approach has its advantages and might be preferred for certain scenarios (see Table 2). The choice of method should take the database characteristics and recovery requirements into account. For example, if tape backups are performed primarily for archive and compliance reasons and restoration from tape is expected to occur rarely, the second method using NDMP to perform tape backups might be preferred for its simplicity and reduced load on the database servers during backup. Conversely, if a fast recovery option from tape is required, the first method using Oracle RMAN to copy the backup set to tape might be the better choice.

Table 2. Disk-to-Disk-to-Tape Backup Options

        

Method	Advantages	Disadvantages
1. Perform an Oracle RMAN database backup to Oracle ZFS Storage Appliance or Exadata Storage. 2. Use Oracle RMAN to copy backup set from disk to tape.	- Faster and easier to restore from tape (can restore directly from tape)   - Oracle RMAN aware of all copies of the backup set	Additional load on database servers when performing tape backup
1. Perform an Oracle RMAN database backup to Oracle ZFS Storage Appliance. 2. Use NDMP to back up Oracle ZFS Storage Appliance to tape.	No additional load on database servers when performing tape backup	- Requires two-step procedure to restore from tape   - Longer recovery time from tape - Oracle RMAN catalog would not be updated by the NDMP backup of the backup set files on Oracle ZFS Storage Appliance

Offload Database Backups with Data Guard

Data Guard—a feature included in Oracle Database, Enterprise Edition—is the recommended disaster recovery solution to protect mission-critical databases residing on Oracle SuperCluster. Data Guard physical standby databases support all Oracle data types and features, including Exadata Hybrid Columnar Compression, and they are able to support the very high transaction volume driven by Oracle SuperCluster.

Data Guard standby databases are also used to offload backups from a primary production database. Both disk-based and tape-based backups can be performed using a physical standby database. Oracle Active Data Guard, a priced option that extends Data Guard functionality, can be used to offload fast incremental backups (Oracle RMAN block change tracking), further reducing backup times and the impact on the primary database. Additional benefits of Oracle Active Data Guard include offloading read-only queries and reports from the primary database to a synchronized physical standby database and performing automatic block repair if the software detects a block corruption. All Data Guard standby databases can also be used to detect lost-write corruptions and for database rolling upgrades and other maintenance while also providing disaster protection.

For more information, please see the Oracle white paper "Oracle Data Guard: Disaster Recovery for Oracle Exadata Database Machine."

Oracle's Zero Data Loss Recovery Appliance

Zero Data Loss Recovery Appliance—an engineered system for database backup that eliminates data loss exposure without impacting the performance of production environments—is an option that should be considered when traditional backup and recovery approaches are not sufficient to meet enterprise requirements. Compute, network, and storage are integrated into a massively scalable appliance with a cloud-scale architecture that provides fully automated database backup and recovery for multiple databases.

Featuring an incremental-forever backup strategy, the recovery appliance provides minimal-impact backups. The databases send only changes, and all backup and tape processing is offloaded from the production servers to the recovery appliance for improved system performance. Real-time database redo block information is transmitted, eliminating potential data loss and providing instant protection of new transactions. Database recoverability is improved with end-to-end reliability, visibility, and control of the database as a whole, rather than as a disjoint set of files.

Zero Data Loss Recovery Appliance is a complementary technology to other backup and recovery options such as Oracle ZFS Storage Appliance and Oracle Active Data Guard. For example, an enterprise backup and recovery solution could use Zero Data Loss Recovery Appliance to provide a centralized backup service for all databases, use the snapshot and cloning capabilities of Oracle ZFS Storage Appliance for development and test environments, and use Oracle Active Data Guard to provide fast failover capabilities for critical databases.

Oracle Secure Backup Cloud Module

Oracle Secure Backup Cloud Module provides the flexibility to back up databases to cloud-based storage services offered by Amazon Simple Storage Service (Amazon S3). It is compatible with Oracle Database version 9i Release 2 and later. Oracle Secure Backup Cloud Module is implemented using the Oracle RMAN System Backup to Tape (SBT) interface, which enables external backup libraries to be seamlessly integrated with Oracle RMAN. With this cloud offering, local disk backups are sent directly to Amazon S3 for off-site storage and are fully integrated with Oracle RMAN.

Oracle Secure Backup Cloud Module provides the following advantages over traditional off-site tape-based backups:

• Continuous accessibility. Backups stored in the cloud are always accessible—in much the same way local disk backups are. As such, there is no need to call anyone and no need to ship or load tapes before a restore can be performed. Administrators can initiate restore operations using their standard tools (Oracle Enterprise Manager, scripts, and so on) exactly as if the off-site backup were stored locally. This process can restore data faster and reduce downtime from days to hours or even minutes in many cases.
• Better reliability. Storage clouds are disk-based and, thus, inherently more reliable than tape media. Additionally, cloud vendors typically keep multiple redundant copies of data for availability and scalability purposes.
• Cost savings. Cloud storage backups can reduce or eliminate upfront capital expenditures, as well as tape backup licensing and off-site storage costs.

Securing data, especially when stored off premises, is critical. To ensure data is properly secured, Oracle RMAN backup encryption is recommended as a standard part of the cloud-based backup process.

Providing Efficiency Savings and High Availability with Remote DR Sites

Figure 8 shows the ultimate method for providing the ability to restore IT operations while spending less for backup and recovery. Traditional methods require expensive backup and recovery equipment to restore the production system as quickly as possible. An alternative method is to have one or more remote sites with replicated applications and data. Using a primary site and one or more remote secondary sites eliminates the need for high-performance systems to recover production operations. Instead of deploying Oracle SuperCluster and a high-performance disk array to back it up, a secondary Oracle SuperCluster with lower cost Oracle ZFS Storage Appliance or tape can be deployed. In the event of a failure, production can be switched over immediately to the remote site. The primary site's recovery time is irrelevant because production continues on the secondary systems. This approach provides better business continuance and results in significant savings because backup and recovery system costs are lower.

When not running production operations as part of a business continuance effort, the remote replicated location can be used for testing, development, and other functions. The ability to generate reports from a copy of the secondary site's production database—a capability for which Oracle continues to add support—enables IT staff to offload work from the production system and keep the secondary database current. This method provides savings for backup and recovery and maximizes system availability. Many organizations distribute their recovery systems in this manner, with applications that can fail over from city to city. In fact, this design can be used almost universally, with the exception of firms that are required to have large and elaborate backup and recovery systems for compliance reasons.

Figure 8. Oracle technologies enable maximum protection plus cost savings.

Best Practices for Secure Backup and Recovery Implementation

Enterprise-level backup environments typically use a centralized backup and recovery model, which simplifies administration as the size of the data continues to grow. However, this centralization also presents a potential security threat: If an intruder gains access to a centralized backup or media server, data for many systems across the enterprise can be compromised.

Physical security is, of course, fundamental, and access to servers and media libraries must be protected and audited. However, backup and recovery systems cannot rely solely on perimeter security. A combination of system-wide security measures and best practices—including the rule of least privilege, strong authentication, access control, encryption, auditing, disabling of unnecessary services, and antimalware protections—should also be implemented for secure operations. Backup servers, clients, networks, storage, and tape libraries should all be configured following recommended security guidelines to provide end-to-end security. Data encryption is also recommended to protect data at rest and while in transit across networks as well as to and from storage media.

Creating More Secure Backup and Recovery Environments

The following steps can help create a more secure backup and recovery environment:

• Simplify the infrastructure. Most backup and recovery environments are based on a complex infrastructure, making implementation and management complicated. This complexity increases the risk of security vulnerabilities. A backup and recovery implementation as a whole is only as secure as its most vulnerable component, and it can be challenging to securely configure the myriad interacting components and products in a heterogeneous system. Oracle Optimized So