1 2 Previous Next 23 Replies Latest reply: Jul 28, 2011 5:41 AM by Catch-22 RSS

    Integration of RMAN with virtual tape library

    Catch-22
      OS: Oracle Linux 5.5 UEK (Virtualbox)
      Oracle: 11gR2 11.2.0.1 Enterprise + ASM (Virtualbox)

      Hello,

      I would like to test something about RMAN and writing directly to tape. For this I installed mhvtl, a virtual tape library. The installation was pretty easy. http://sites.google.com/site/linuxvtl2/

      RMAN does not recognize any tape in the system.
      ORA-19554: error allocating device, device type: SBT_TAPE, device name:
      ORA-27211: Failed to load Media Management Library

      What I found out is that in order to make backups to a media manager, RMAN must communicate with a media management library, which is a 3rd party library that is not supplied by Oracle. So I guess unless there is a Oracle MML library available for mtx I' out of luck here.

      Any ideas? Thanks!

      rpm -ivh mhvtl-0.18-13.x86_64.rpm
      rpm -ivh mhvtl-0.18-13.src.rpm
      cd /usr/src/redhat/SOURCES/
      tar zxvf ../SOURCES/mhvtl-2011-01-11.tgz
      cd mhvtl-0.18/kernel
      make
      make install
      /etc/init.d/mhvtl start

      yum install mtx

      The tape library and using it for backup from system seems to work fine:
      # mtx status
      Storage Changer /dev/changer:4 Drives, 43 Slots ( 4 Import/Export )
      Data Transfer Element 0:Full (Storage Element 1 Loaded):VolumeTag = E03001L4
      etc.
      # mtx load 1
      Drive 0 Full (Storage Element 1 loaded)
      # tar cf /dev/ntape ./
      # tar tvf /dev/ntape

      Edited by: Dude on Jan 20, 2011 5:22 PM
        • 1. Re: Integration of RMAN with virtual tape library
          19426
          To be honest I'm not familiar with this stuff. But you may check whether Oracle Secure Backup is a possible solution here. This is the installation and configuration guide and there's also a dedicated forum:

          http://download.oracle.com/docs/cd/E14812_01/doc/doc.103/e12835/toc.htm

          Secure Backup

          Werner
          • 2. Re: Integration of RMAN with virtual tape library
            Catch-22
            Thanks for the tip!

            I installed Oracle Secure Backup Express and spend all afternoon with this strange product, mainly to troubleshoot the installation and setup. I'm not really sure everything works - it's quite a piece of software, as usual.

            I ran into a couple of issues that were not mentioned in the installation documentation:
            - install sg3_utils
            - use obtool to register the server as mediaserver

            # sg_map -i -x
            /dev/sg9 3 0 1 0 1 /dev/nst0 IBM ULT3580-TD5 550V
            /dev/sg10 3 0 2 0 1 /dev/nst1 IBM ULT3580-TD5 550V
            /dev/sg11 3 0 3 0 1 /dev/nst2 IBM ULT3580-TD4 550V
            /dev/sg12 3 0 4 0 1 /dev/nst3 IBM ULT3580-TD4 550V
            /dev/sg13 3 0 9 0 1 /dev/nst4 IBM ULT3580-TD4 550V
            /dev/sg14 3 0 10 0 1 /dev/nst5 IBM ULT3580-TD4 550V
            /dev/sg15 3 0 11 0 1 /dev/nst6 IBM ULT3580-TD4 550V
            /dev/sg16 3 0 12 0 1 /dev/nst7 IBM ULT3580-TD4 550V
            /dev/sg17 3 0 0 0 8 STK L700 550V
            /dev/sg18 3 0 8 0 8 SPECTRA PYTHON 550V


            # obtool
            ob> chhost --addrole mediaserver ol55
            ob> mkdev -t library -o -a ol55:/dev/sg18 library1
            ob> mkdev -t tape -o -a ol55:/dev/sg9 -l library1 -d 1 tape1

            RMAN was not happy with it:

            ORA-19554: error allocating device, device type: SBT_TAPE, device name:
            ORA-27023: skgfqsbi: media manager protocol error
            ORA-19511: Error received from media manager layer, error text:
            sbt__rpc_connect: Internal error - could not connect to obproxyd (Oracle Secure Backup error: 'bad port name specified (OB SBT proxy manager)').

            # cat /usr/tmp/obproxyd.log
            2011/01/23.02:11:10 [5431] can't set root privileges; continuing anyway - program isn't stamped setuid-root (OB utilities)
            2011/01/23.02:11:10 [5431] can't listen for connections - Operation not permitted

            Document ID 396046.1 solution outlines to correct the hostname or mediaserver name. However, this was not the problem. I finally figured that /etc/obproxyd is a symlink to a .wrapper script which points to the actual proxy application. The fix was:

            #chmod +s /usr/local/oracle/backup/.etc.linux86_64obproxyd

            After this the error changed:

            RMAN-03009: failure of backup command on ORA_SBT_TAPE_1 channel at 01/23/2011 02:43:36
            ORA-19506: failed to create sequential file, name="3cm2q2an_1_1", parms=""
            ORA-27028: skgfqcre: sbtbackup returned error
            ORA-19511: Error received from media manager layer, error text:
            sbt__rpc_cat_query: Query for piece 3cm2q2an_1_1 failed.
            (Oracle Secure Backup error: 'no preauth config found for OS user (OB tools) oracle').

            The fix for this error:

            https://10.0.0.6, login as 'admin'
            Configure->Users->Select Admin->Edit->Preauthorized Access->Add user 'Oracle' (select host and rman)


            RMAN> backup database;

            Starting backup at 23-JAN-11
            using channel ORA_SBT_TAPE_1
            channel ORA_SBT_TAPE_1: starting full datafile backup set
            channel ORA_SBT_TAPE_1: specifying datafile(s) in backup set
            input datafile file number=00001 name=/scratch/oradata/test/system01.dbf
            input datafile file number=00002 name=/scratch/oradata/test/sysaux01.dbf
            input datafile file number=00003 name=/scratch/oradata/test/undotbs01.dbf
            input datafile file number=00005 name=/scratch/oradata/test/users02.dbf
            input datafile file number=00004 name=/scratch/oradata/test/users01.dbf
            channel ORA_SBT_TAPE_1: starting piece 1 at 23-JAN-11

            Now it just sits there forever and the log just keeps adding the following lines:

            # cat obproxyd.log
            2011/01/23.03:00:31 [5662] handle_qsp_msg: received get job state request
            2011/01/23.03:00:31 [5662] job id 'admin/5.1'

            Strange, backup to tape with 'mtx' and 'tar' works, so the problem might be OSB.

            Edited by: Dude on Jan 22, 2011 8:32 PM

            I finally got it running. I deleted the current library and drive configuration and used the webtool to configure the other library at /dev/sg17. it was also necessary to use the webtool to load a volume into the drive. Unfortunately I did not note down the steps in details. Maybe next time.

            channel ORA_SBT_TAPE_1: starting piece 1 at 23-JAN-11
            channel ORA_SBT_TAPE_1: finished piece 1 at 23-JAN-11
            piece handle=3im2qc42_1_1 tag=TAG20110123T053042 comment=API Version 2.0,MMS Version 10.2.0.3
            channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:03:26
            Finished backup at 23-JAN-11

            Starting Control File and SPFILE Autobackup at 23-JAN-11
            piece handle=c-2036659790-20110123-00 comment=API Version 2.0,MMS Version 10.2.0.3
            Finished Control File and SPFILE Autobackup at 23-JAN-11
            • 3. Re: Integration of RMAN with virtual tape library
              775317
              I managed to get OSB integrated with MHVTL such that RMAN appears to do sbt backups. However, obtool doesn't see the pieces in the VTL and restores are not possible.

              Did you get as far as being able to restore pieces and query them from either obtool or RMAN list backup?
              • 4. Re: Integration of RMAN with virtual tape library
                Catch-22
                I just did a backup and can list the backup in RMAN alright, but restore fails, error:

                ORA-19507: failed to retrieve sequential file, handle="45m3bs9o_1_1", parms=""
                ORA-27029: skgfrtrv: sbtrestore returned error
                ORA-19511: Error received from media manager layer, error text:
                sbtrestore: piece '45m3bs9o_1_1' does not exist in the backup catalog.

                Sorry, no idea what is wrong. I stopped playing with it, only needed to test some SBT channel configuration options.
                • 5. Re: Integration of RMAN with virtual tape library
                  775317
                  Dude wrote:
                  I just did a backup and can list the backup in RMAN alright, but restore fails, error:

                  ORA-19507: failed to retrieve sequential file, handle="45m3bs9o_1_1", parms=""
                  ORA-27029: skgfrtrv: sbtrestore returned error
                  ORA-19511: Error received from media manager layer, error text:
                  sbtrestore: piece '45m3bs9o_1_1' does not exist in the backup catalog.

                  Sorry, no idea what is wrong. I stopped playing with it, only needed to test some SBT channel configuration options.
                  Thanks for the response.

                  It seems that OSB and MHVTL don't get along very well. You can do backups but the pieces are latent insofar as they remain accessible for a very short time. Not sure what is happening. I have managed to retrieve a control file backup. Archivelogs and datafile backups just don't seem to hang around.
                  • 6. Re: Integration of RMAN with virtual tape library
                    Catch-22
                    Restore tests from the shell using tar work fine, which makes me think that it might be some OSB configuration issue. The software is complex and takes a bit to get familiar with it. I was planning to look into it again later, and try the full version of OSB.
                    • 7. Re: Integration of RMAN with virtual tape library
                      775317
                      I'm beginning to think that the issue lies with the device drivers to which OSB attaches. Could it be possible that OSB is using a rewinding device instead of non-rewinding when it sets up the obt devices?

                      Only last backup seems to remain and all previous pieces on a given tape are wiped; the size of the VT (found under /opt/mhvtl/<VOLUME> - there is a file data and a file index) drops back down again. I see that there are various devices set up from mhvtl:

                      [root@node4 ~]# ls -l /dev/nst5* /dev/st5* /dev/obt5

                      crw-rw---- 1 root disk 9, 133 Jan 29 19:33 /dev/nst5
                      crw-rw---- 1 root disk 9, 229 Jan 29 19:33 /dev/nst5a
                      crw-rw---- 1 root disk 9, 165 Jan 29 19:33 /dev/nst5l
                      crw-rw---- 1 root disk 9, 197 Jan 29 19:33 /dev/nst5m

                      crw-rw-rw- 1 root sys 16, 144 Jan 29 19:56 /dev/obt5

                      crw-rw---- 1 root disk 9, 5 Jan 29 19:33 /dev/st5
                      crw-rw---- 1 root disk 9, 101 Jan 29 19:33 /dev/st5a
                      crw-rw---- 1 root disk 9, 37 Jan 29 19:33 /dev/st5l
                      crw-rw---- 1 root disk 9, 69 Jan 29 19:33 /dev/st5m

                      If I write something using dd to a VT using the nst device, then everything written is preserved; as soon as I use the st device, then all previous dumps are erased.

                      This gives me the impression that the obt device is using a rewind mechanism.

                      This OSB installation guide merely asks me to supply device information which I can obtain from lsscsi:

                      [root@node4 ~]# lsscsi -g
                      [0:0:6:0] tape SEAGATE DAT DAT72-052 A16E /dev/st0 /dev/sg0
                      [1:0:8:0] mediumx SPECTRA PYTHON 550V - /dev/sg10
                      [1:0:9:0] tape IBM ULT3580-TD4 550V /dev/st5 /dev/sg5
                      [1:0:10:0] tape IBM ULT3580-TD4 550V /dev/st6 /dev/sg6
                      [1:0:11:0] tape IBM ULT3580-TD4 550V /dev/st7 /dev/sg7
                      [1:0:12:0] tape IBM ULT3580-TD4 550V /dev/st8 /dev/sg8


                      Clearly, the installation of mhvtl has set up rewinding devices; I am not sure how to modify this but I think the issue clearly lies with mhvtl - I'll take it up with the forum there.
                      • 8. Re: Integration of RMAN with virtual tape library
                        775317
                        I've discovered a bit more about the missing pieces.

                        The problem seems to be that the volume gets re-labelled. OSB seems to pick the mounted drive with the required media family and then re-label the tape:

                        [root@node4 ~]# obtool lsvol --all
                        VOID OOID Seq Volume ID Barcode Family Created Attributes
                        283 283 1 RMAN-DEFAULT-000077 E03006L4 RMAN-DEFAULT 01/30.20:52 never closes; content manages reuse


                        RMAN-DEFAULT-000077 got re-labelled from RMAN-DEFAULT-000076 when the backup started.

                        It's as though OSB thinks the tape in the drive is unlabelled when it starts the backup but prior to a new backup, it can display the new name for it.

                        I will see if I can inventory the tape without doing any backups, i.e., read its label.
                        • 9. Re: Integration of RMAN with virtual tape library
                          775317
                          I have run

                          obtool identifyvol --import

                          on the drive of concern; it correctly identifies the volume that it has in its catalog and this corresponds to the barcode identification of the volume too.

                          But when I do another backup, the volume is re-labelled!

                          It's as though the volume should be immediately re-cycled or is expired.

                          Wierd.

                          From identifyvol I get the following:

                          ob> identifyvol drive  ULT3580-TD8   import 6

                          Seq Volume Volume Archive Client Backup Archive Create
                          # ID Tag File Sect Host Level Date & Time
                          1 RMAN-DEFAULT-000079 E03006L4 1 1 D node5 0 2011/01/30 21:27:21

                          End of volume set.



                          I wonder why "End of volume set." is issued along with the volume details. And why would this mean that the volume should be re-cycled.
                          • 10. Re: Integration of RMAN with virtual tape library
                            775317
                            >
                            Seq Volume Volume Archive Client Backup Archive Create
                            # ID Tag File Sect Host Level Date & Time
                            1 RMAN-DEFAULT-000079 E03006L4 1 1 D node5 0 2011/01/30 21:27:21

                            End of volume set.



                            I wonder why "End of volume set." is issued along with the volume details. And why would this mean that the volume should be re-cycled.
                            End of volume set means "the last volume of a set of volumes" and not that some marker in the volume has been set.
                            • 11. Re: Integration of RMAN with virtual tape library
                              775317
                              Having upgraded to 10.3, I can report that the volumes are not removed anymore.

                              OSB with MHVTL seems to work reasonably well at a basic level; the robotics are working OK.

                              I haven't done much testing though.
                              • 12. Re: Integration of RMAN with virtual tape library
                                Catch-22
                                Unfortunately I still haven't had a chance to get back to it yet. Thanks for the feedback!
                                • 13. Re: Integration of RMAN with virtual tape library
                                  Jurajl-Oracle
                                  Test setup - osb 10.3.0.3.0 on OEL 5U2; mhvtl 0.18-13 on OEL 5U5

                                  Filesystem and rman backups + restores work fine, until another backup reuses their tape.

                                  1.st backup works fine (is backed up and can be restored)- i.e. obtool lspiece for rman backups prints backup pieces fine - even after unloading volume from tape drive back to library, identify volume, change tape's library positions, loading volume back to tape drive, restoring from it.
                                  After 2.nd backup is issued, initial volume loads into tape drive, but gets relabeled and all previous backups are lost (for rman and filesystem backups same behavior).

                                  With real hardware (same type as being emulated in test setup of mhvtl - library HP MSL G3 Series, tape HP Ultrium 4-SCSI), same operations work with no problem.

                                  Any ideas?
                                  • 14. Re: Integration of RMAN with virtual tape library
                                    Catch-22
                                    Sorry I have lost track of this and will have to rebuild the test environment. Did you check the feedback of Graeme Lawton above?
                                    1 2 Previous Next