14 Replies Latest reply: Feb 29, 2012 9:07 AM by John-MK RSS

    RMAN backup on RAC

    John-MK
      Hi,

      DB=10.2.0.4 OS=RHL (RAC with 2 nodes db01,db02). Following are the types of automated RMAN backups taken on db01.

      - OCR
      - Voting Disk
      - Talespaces Snapshot
      - Archivelog (every hour)
      - Backup Incremental level 0 (SUNDAY)
      -- backup incrementa level 1 (Mon-SAT)
      - Autobackup of Control file and SPFILE (CONTROLFILE AUTOBACKUP ON)

      Two main questions please. First, are these backups/backup types are enough FOR FULL RESTORE when talking about RMAN backup on RAC environment with 2 nodes? If not please suggest what else I need to backup?

      Secondly, backup home location is /rman which I can see on db01 ONLY. This /rman directory I cannot see on db02. My question is that In case of db02 failure can I restore from this /rman directory? or do I need to automate backup schedules also on db02 separately?
      [root@db01~]# df -h
      Filesystem            Size  Used Avail Use% Mounted on
      /dev/mapper/VolGroup00-LogVol00
                            3.9G  2.9G  814M  79% /
      /dev/sda3             190M   34M  147M  19% /boot
      tmpfs                  16G     0   16G   0% /dev/shm
      /dev/mapper/VolGroup00-LogVol05
                            3.9G  137M  3.6G   4% /home
      /dev/mapper/VolGroup00-LogVol03
                            3.9G  225M  3.5G   6% /tmp
      /dev/mapper/VolGroup00-LogVol02
                            7.8G  3.7G  3.8G  50% /usr
      /dev/mapper/VolGroup00-LogVol04
                            3.9G  379M  3.4G  11% /var
      /dev/mapper/VolGroup00-ORACLE_BASE
                             12G  8.7G  2.6G  78% /oracle
      10.10.28.140:/vol/px4cinconso
                             40G  1.4G   39G   4% /inconso_px
      /dev/mapper/36001ec9000d5ea810000061349015d98
                            1.3T  551G  665G  46% /rman
      [root@db02~]# df -h
      Filesystem            Size  Used Avail Use% Mounted on
      /dev/mapper/VolGroup00-LogVol00
                            3.9G  887M  2.9G  24% /
      /dev/sda3             190M   34M  147M  19% /boot
      tmpfs                  16G     0   16G   0% /dev/shm
      /dev/mapper/VolGroup00-LogVol05
                            3.9G  137M  3.6G   4% /home
      /dev/mapper/VolGroup00-LogVol03
                            3.9G  261M  3.5G   7% /tmp
      /dev/mapper/VolGroup00-LogVol02
                            7.8G  3.7G  3.8G  50% /usr
      /dev/mapper/VolGroup00-LogVol04
                            3.9G  355M  3.4G  10% /var
      /dev/mapper/VolGroup00-ORACLE_BASE
                             12G  6.5G  4.8G  58% /oracle
      10.10.28.140:/vol/px4cinconso
                             40G  1.4G   39G   4% /inconso_px
      [oracle@db01~]$ rman target /
      
      Recovery Manager: Release 10.2.0.5.0 - Production on Fri Feb 24 11:49:59 2012
      
      Copyright (c) 1982, 2007, Oracle.  All rights reserved.
      
      connected to target database: F1 (DBID=1245145263)
      
      RMAN> show all;
      
      using target database control file instead of recovery catalog
      RMAN configuration parameters are:
      CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 28 DAYS;
      CONFIGURE BACKUP OPTIMIZATION ON;
      CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
      CONFIGURE CONTROLFILE AUTOBACKUP ON;
      CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '/rman/f1/fullbackup/%F';
      CONFIGURE DEVICE TYPE DISK BACKUP TYPE TO COMPRESSED BACKUPSET PARALLELISM 1;
      CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
      CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
      CONFIGURE MAXSETSIZE TO UNLIMITED; # default
      CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
      CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
      CONFIGURE ARCHIVELOG DELETION POLICY TO NONE; # default
      CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/oracle/product/10.2.0/db/dbs/snapcf_f11.f'; # default
      # Script for backup
      
      level0 ()
      run {
        allocate channel c1 type disk  maxpiecesize 2000M format '/rman/$ORACLE_SID/fullbackup/$ORACLE_SID-data-%U-%t';
        allocate channel c2 type disk  maxpiecesize 2000M format '/rman/$ORACLE_SID/archive/$ORACLE_SID-arch-%U-%t';
        backup incremental level 0 tag = weeklyfull database channel c1 
        plus archivelog tag = ARCH_$TODAY delete all input channel c2;
        release channel c1;
        release channel c2;
      }
      
      run {
        allocate channel c1 type disk;
        delete noprompt obsolete;
        release channel c1;
      }
      Bundle of thanks.

      Regards,

      Edited by: 910385 on Feb 24, 2012 3:00 AM
        • 1. Re: RMAN backup on RAC
          19426
          The backup strategy is ok.
          Secondly, backup home location is /rman which I can see on db01 ONLY
          You should send the backups to destination accessible to all nodes in a RAC environment (shared filesystem,NFS,...). Don't backup a second time on node 2.

          Werner
          • 2. Re: RMAN backup on RAC
            John-MK
            Hi Werner,

            Thanks alot for the straight forward reply. Very helpful !

            I asked one of our Administrator, and he said that this /rman directory cannot be mounted simultaneously on both nodes. It can only be mounted on one of the node at a time (ICSC system something strategy). Any further suggestion please to get the things done or suggest admins to do/create another shared directory which will be accessible to both the nodes (don't have much knowledge of filesystem,etc).

            Thank you.

            Regards,
            • 3. Re: RMAN backup on RAC
              912595
              910385 wrote:
              Hi,

              DB=10.2.0.4 OS=RHL (RAC with 2 nodes db01,db02). Following are the types of automated RMAN backups taken on db01.

              - OCR
              - Voting Disk
              - Talespaces Snapshot
              - Archivelog (every hour)
              - Backup Incremental level 0 (SUNDAY)
              -- backup incrementa level 1 (Mon-SAT)
              - Autobackup of Control file and SPFILE (CONTROLFILE AUTOBACKUP ON)

              Two main questions please. First, are these backups/backup types are enough FOR FULL RESTORE when talking about RMAN backup on RAC environment with 2 nodes? If not please suggest what else I need to backup?
              Backup is very much ok
              Secondly, backup home location is /rman which I can see on db01 ONLY. This /rman directory I cannot see on db02. My question is that In case of db02 failure can I restore from this /rman directory? or do I need to automate backup schedules also on db02 separately?
              What do you mean by db02 failure? I hope you are talking about machine failure. So in db02 machine goes down or something happens to it, still oracle database will be working. Since it have 2 instance running from node 1 i.e. db01 and from node 2 i.e. db02
              If node 2 comes up, there is no need for rman backups.

              Typically backups are sent to shared location. However if shared location is not available for backup then this can done on non-shared directory as well.
              • 4. Re: RMAN backup on RAC
                19426
                Is the mountpoint /rman dedicated to node 1 or can you mount it to node 2 if needed? There could be only a problem when

                1) node 1 goes down
                2) node 2 is still alive and you need something to restore.

                In this case you need access to /rman. As said the easiest way is a shared location, but if /rman is not dead when node 1 is dead (that would be true for an internal drive), you have simply to mount the destination to node 2.


                Werner
                • 5. Re: RMAN backup on RAC
                  Kavanagh
                  I think , for RAC set ups, you need to allocate one channel for each Instance
                  So, your persistent setting should have
                  CONFIGURE CHANNEL 1 DEVICE TYPE DISK connect 'sys/secret@rac1'  FORMAT   '/rmanbkp_directory/%U.rmbk';
                  CONFIGURE CHANNEL 2 DEVICE TYPE DISK connect 'sys/secret@rac2' FORMAT   '/rmanbkp_directory/%U.rmbk';
                  
                  rac1 -----> TNS entry for instance1
                  rac2 -----> TNS entry for instance2
                  • 6. Re: RMAN backup on RAC
                    19426
                    That doesn't make sense for OP, because he told us the backup destination can only be seen from ONE node.

                    Werner
                    • 7. Re: RMAN backup on RAC
                      Steve_74
                      I think , for RAC set ups, you need to allocate one channel for each Instance
                      So, your persistent setting should have

                      CONFIGURE CHANNEL 1 DEVICE TYPE DISK connect 'sys/secret@rac1'  FORMAT   '/rmanbkp_directory/%U.rmbk';
                      CONFIGURE CHANNEL 2 DEVICE TYPE DISK connect 'sys/secret@rac2' FORMAT   '/rmanbkp_directory/%U.rmbk';

                      rac1 -----> TNS entry for instance1
                      rac2 -----> TNS entry for instance2


                      Yes. This is a pre-requisite if you have RAC configurations where archived logs are stored locally (and cross mounted to either of the nodes). With archived logs stored locally, if you fail to configure one channel for Node2 (assuming that you initiated the backup from Node1) , then you'll get
                      Starting backup at 12-MAR-11
                      current log archived
                      using channel ORA_DISK_1
                      RMAN-00571: ===========================================================
                      RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
                      RMAN-00571: ===========================================================
                      RMAN-03002: failure of backup plus archivelog command at 03/12/2011 02:32:27
                      RMAN-06059: expected archived log not found, lost of archived log compromises recoverability
                      ORA-19625: error identifying file /u04/grish199/arch_382083_2_47455.arc
                      ORA-27037: unable to obtain file status
                      Here RMAN is trying to backup the archived log stored in Node2 and is failing because it does not have a channel to Node2.
                      • 8. Re: RMAN backup on RAC
                        John-MK
                        Hello,

                        I have asked our Admin and he said, yes we can mount /rman on node 2 if node 1 goes down.

                        Here, I am little confused regarding this channel allocation thing. I've read that when you want to distribute backup over multiple locations then you use channels - restore requires then to restore from multiple locations. But as It is being said in the above post that It is necessary to allocate channels to both the nodes incase archived logs are stored on local nodes.

                        I'm not able to satisfy this situation that in our case, whether the Archive logs are saved in the shared location or what. Because, If I am not reading it wrong the backup log suggests that archive logs are backed up from both the nodes. Following are the initial few lines of the backup logs. Please suggest that all the archived logs are backed up or not?
                        SYS@fp11 AS SYSDBA> archive log list
                        Database log mode            Archive Mode
                        Automatic archival            Enabled
                        Archive destination            USE_DB_RECOVERY_FILE_DEST
                        Oldest online log sequence     62850
                        Next log sequence to archive   62854
                        Current log sequence            62854
                        SYS@fp12 AS SYSDBA> archive log list
                        Database log mode            Archive Mode
                        Automatic archival            Enabled
                        Archive destination            USE_DB_RECOVERY_FILE_DEST
                        Oldest online log sequence     65838
                        Next log sequence to archive   65842
                        Current log sequence            65842
                        Starting backup at 27-FEB-12
                        current log archived
                        channel c2: starting compressed archive log backupset
                        channel c2: specifying archive log(s) in backup set
                        input archive log thread=1 sequence=62842 recid=124701 stamp=776316006
                        input archive log thread=2 sequence=65832 recid=124700 stamp=776316006
                        channel c2: starting piece 1 at 27-FEB-12
                        channel c2: finished piece 1 at 27-FEB-12
                        piece handle=/rman/fp11/archive/fcp11-arch-jbn4b93i_1_1-776316018 tag=ARCH_MON comment=NONE
                        channel c2: backup set complete, elapsed time: 00:00:03
                        channel c2: deleting archive log(s)
                        archive log filename=+REDO1/fp1/archivelog/2012_02_27/thread_1_seq_62842.481.776316007 recid=124701 stamp=776316006
                        archive log filename=+REDO1/fp1/archivelog/2012_02_27/thread_2_seq_65832.543.776316005 recid=124700 stamp=776316006
                        Finished backup at 27-FEB-12
                        The main thing confusing me is that whether I should allocate 2 channels in RMAN configuration parameter like 1 channel for node 1 and 1 for node 2? (keeping in mind the all above, that looks OK?)
                        As I have not configured C2 for example for Node 2, but I have not got any error as described in above post.

                        THank you.

                        Regards

                        Edited by: 910385 on Feb 27, 2012 4:27 AM
                        • 9. Re: RMAN backup on RAC
                          19426
                          archive log filename=+REDO1/fp1/archivelog/2012_02_27/thread_1_seq_62842.481.776316007 recid=124701 stamp=776316006
                          Your archivelogs are stored in an ASM diskgroup accessible from both nodes. Both nodes send their archivelogs to this common location. So the RMAN job running on node 1 has no problems to access to archivelogs from node.
                          In other words you don't have the situation where archivelogs are stored locally on one node.

                          As long as you your backup destination at a certain point in time is accessible from one node only, it doesn't make sense to allocate channels to the node, which cannot access this backup location.

                          Werner
                          • 10. Re: RMAN backup on RAC
                            John-MK
                            Hi,

                            THanks alot for your feedback. Makes my life easier :) . But actually I started studying this all subject because of the following error message in the log file. I found out about this error that doesn't seem to harm. Just wait for the job to finish or it will sort out automatically. Here are the lines from the log file.
                            Fehler in Backuplogs gefunden:/oracle/log/fcp11-backup-2300.log: RMAN-00571:
                            ===========================================================brRMAN-00569: ===============
                            ERROR MESSAGE STACK FOLLOWS ===============brRMAN-00571:
                            ===========================================================brRMAN-03009: failure of
                            Control File and SPFILE Autobackup command on c1 channel at 02/08/2012
                            06:10:41brORA-00230: operation disallowed: snapshot control file enqueue unavailable
                            I studied few important lines from one of my book, Oracle 11g RMAN backup and recovery (Roberg G Freeman) that I am going to write after the following MAIN ISSUE:
                            Node 1
                            RMAN> show all;
                            - - - 
                            CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/oracle/product/10.2.0/db/dbs/snapcf_fp11.f'; # default
                            
                            Node 2
                            RMAN> show all;
                            - - - 
                            CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/oracle/product/10.2.0/db/dbs/snapcf_fp12.f'; # default
                            
                            Thing which I just noticed regarding the above RMAN error is that snapcf_fp11.f can be seen on the Node 1 on todays date but snapcf_fp12.f was only on 10th jan.
                            
                            [oracle@db01dbs]$ ls -lrth
                            -rw-r----- 1 oracle oinstall  20M Feb 27 15:10 snapcf_fcp11.f
                            
                            [oracle@db02dbs]$ ls -lrth
                            -rw-r----- 1 oracle oinstall  20M Jan 10 13:28 snapcf_fcp12.f
                            THe few lines from the book which I mentioned above are: (Though you might not need to read those lines):

                            "You need to move the snapshot control file to a shared location if you plan to run backups from more than one node. If you do not move the snapshot control file to a shared file system such as OCFS or an ASM disk group, then you must make sure that the local destination is identical on all nodes."

                            My question is of course, how to sort out the above log file error. Becuse in our case, backups are run only from one node, and snaphost control file is not being moved to ASM disk groups. Local destination is also same on both the Nodes, isn't it (with different names). Why on Node 2 last snapshot was taken on 10th jan while on Node 1 it is todays date which seems fine.

                            Control file and SPFILE which I noticed are stored on ASM disk groups also, so being shared between the two Nodes.
                             (Sanme names/same location for control file and SPFILE on Node 1 and Node 2)
                            Control file:
                            +REDO1/fp1/controlfile/current.256.652803827
                            +DATA1/fp1/controlfile/current.441.652869697
                            
                            SPFILE:
                            +DATA1/fp1/spfilefcp1.ora
                            Please suggest.

                            Thank you.

                            Regards,

                            Edited by: 910385 on Feb 27, 2012 7:43 AM
                            • 11. Re: RMAN backup on RAC
                              19426
                              My question is of course, how to sort out the above log file error
                              I think it's not a configuration issue,it's simply a bug, from MOS:

                              Bug 7450366 - RMAN Autobackup fails with unable to obtain snapshot enqueue (ORA-230) [ID 7450366.8]

                              The error does not happen,when datafiles and archivelogs are backed up,it's a specific autobackup problem. I'm not sure whether a patch for 10.2.0.4 is available,at least for 10.2.0.5 there is one.

                              Werner
                              • 12. Re: RMAN backup on RAC
                                John-MK
                                Hello,

                                Folling is the message of Alert.log of Node 2. Till now, error came 72 times in the Alert.log file (No erros in Node 1 Alert.log file). I searched alot about ORA-27040, but the thing is, sometimes these kind of errors varry. I mean if I just follow its solution/action, then it doesn't look that tricky or complicated.

                                Now you know all the scnerio how backup is being taken from Node 1 and everything is being backed up on /rman (on ASM's disk groups), why this disk write/space/backup is coming in Alert.log file of Node 2 ? Do I need to set some RMAN paramter or ..? There is no logic that this error comes in the Alert.log of Node 2. If I checked all the errors detail, it is always related to the backup piece of Full database (/rman/fp1/fullbackup). It means it has no problem with Archive log backup.

                                Please suggest.

                                THanks alot.

                                Regards,

                                [root@db02~]# grep ORA-27040 /oracle/admin/fp1/bdump/alert_fp12.log -c
                                72
                                ********************  WARNING ***************************
                                The errors during Server autobackup are not fatal, as it
                                is attempted after sucessful completion of the command.
                                However, it is recomended to take an RMAN control file
                                backup as soon as possible because the Autobackup failed
                                with the following error:
                                ORA-19624: operation failed, retry possible
                                ORA-19504: failed to create file "/rman/fp1/fullbackup/c-1245145263-20120210-11"
                                ORA-27040: file create error, unable to create file
                                Linux-x86_64 Error: 2: No such file or directory
                                ********************  END OF WARNING *******************
                                [root@db02~]# grep ORA-27040 /oracle/admin/fp1/bdump/alert_fp12.log -c
                                72
                                Edited by: 910385 on Feb 28, 2012 5:05 AM
                                • 13. Re: RMAN backup on RAC
                                  tychos
                                  Hi 910385,
                                  The error on node db02
                                  ORA-19504: failed to create file "/rman/fp1/fullbackup/c-1245145263-20120210-11"
                                  is caused by your setup.
                                  You have a mount point /rman only visible on node db01.
                                  The controlfile autobackup setting will trigger the creation of a backup of the controlfile:
                                  - during rman backup (this seems to be running only on db01)
                                  - during a structural change at the database level (eg. adding a datafile).
                                  Sessions making a structural change can be connected to any of your two instances but the instance on db02 cannot create the controlfile autobackup in /rman.
                                  You should change the setup and put the backup to a shared disk seen by both systems.
                                  Strangely you have a shared filesystem:
                                  10.10.28.140:/vol/px4cinconso 40G 1.4G 39G 4% /inconso_px
                                  Best would be to make a second one for rman....
                                  Regards,
                                  Tycho
                                  • 14. Re: RMAN backup on RAC
                                    John-MK
                                    Hi,

                                    Thanks alot for the reply. It means alot to me. Really helpful.

                                    I have observed as according to your explanation that during a structural change to the db level, error came on the node db02 logfile, and only when a structural change is connected to Node 2. THough data files are added often but not every time error come. Therefore, I can ignore this error. THat shared file system you mentioned, I enquired about it from one of the Admin guy. It seems it's not possible in our setup to create some other shared location (space might be one factor), and he said there are different something (didn't understand the logic clearly).

                                    But, from this thread at least I came to know that backup is proper ,there are no worries for node backup shares, and there is no need to take backup on the Node 2 again. we can mount /rman to Node 2 incase something happens.

                                    Regards,