This discussion is archived
1 2 Previous Next 26 Replies Latest reply: Oct 25, 2012 5:57 AM by Klaas-Jan Jongsma Go to original post RSS
  • 15. Re: Exadata Backup job is taking very long time
    Daryl E. Explorer
    Currently Being Moderated
    Already some success .. the unix team entered a static route (hope thats the correct terminology) such that backup traffic actually routes thru the backup (eth3) interface. We went from 1MB on the interface to over 30MB/s
  • 16. Re: Exadata Backup job is taking very long time
    Daryl E. Explorer
    Currently Being Moderated
    Using iperf, we have confirmed things are normal for a 1 GigE interface.
    [ ID] Interval Transfer Bandwidth
    [  3] 0.0-10.0 sec 1.08 GBytes 927 Mbits/sec
  • 17. Re: Exadata Backup job is taking very long time
    Daryl E. Explorer
    Currently Being Moderated
    Just wanted to add something to this thread that I just found out about .. "SECTION SIZE" parameter to the rman backup command. This helps to keep the multi-channel backups somewhat balanced when dealing with the large datafiles often found in the Exadata environment. In my case, I have 1 channel chewing away for 4 days now on the last datafile, an 8TB datafile. Would have been nice to have the whole power of the exadata pushing that datafile to tape.
  • 18. Re: Exadata Backup job is taking very long time
    Spyros Kaparelis Newbie
    Currently Being Moderated
    Daryl, from what i read i assume that you have a very big database with very big datafiles (datafiles with more than 1 TB size)
    I had the opposite problem with my DWH database. It had 3500 datafiles (many of them very small size). I reorganized the datafiles
    to have only 8 datafiles for every 1 TB (128GB each datafile). Only from that change now the hole database has only about 100 datafiles.
    Without changing anything else the backup went from 7.30h to 5.30h
  • 19. Re: Exadata Backup job is taking very long time
    ababb Newbie
    Currently Being Moderated
    Please accept my apologies if I cover something’s again.
    h1. First – Verify the bus and device speeds from the DB Nodes through to the Tape Drives.
    1. DB Nodes send out on eth3 at a rate of 1Gbit/sec – there are 8 DB Nodes – so maximum transfer rate across all 8 DB Nodes should be 8Gbit/sec
    2. The Media Servers are receiving data in on a 10GigE card that is installed into a PCI Express 1.0 (X4) slot so should be able to receive at 8Gbit/sec. If the 10GigE card is installed into a PCI Express 1.0 (x8) or PCI Express 2.0 (x4) slot then this increases to 16Gbit/sec
    3. The Media Server sends data to the FC SAN using a single 4Gbaud Fiber Channel card so the maximum transfer rate to the tape library is 3400Mbit/sec. The 4Gbaud Fiber Channel is installed into a PCI Express 1.0 (x4) slot which is 8Gbit/sec. If the Fiber Channel card is a dual ported 4GBaud FC card – then the maximum transfer rate to the SAN is 6800Mbit/sec.
    4. Tape drives are typically specified in order of MB/sec (not Mbit/sec) so our 6800Mbit/sec becomes 850MB/sec.

    If we are writing to LTO4 Tape Drives – the native backup rate is 120MB/sec – so worse case – you need 8 tape drives (960MB/sec) to write the 850MB/sec – 7 tape drives offer 840MB/sec native.

    If the data being backed up is EHCC compressed – then you are unlikely to be able to get any more compression on the tape drives, but if the data is either uncompressed or compressed using either BASIC compression or OLTP compression, then the data will be hardware compressible – and therefore you might need less tape drives to backup the 850MB/sec that the 2 x 4GFC cards can send.

    h2. Second – verify the transfer rate into the Media Server.
    Unfortunately – I do not have a 1GigE (eth3) on DB Nodes – to 10GigE on my media servers in my lab – so I cannot show you real numbers – but the same steps below can be used to verify transfer rate between the DB Nodes and the Media Servers. I am using the IB link between the DB Nodes and a single Media Server on an X2-2 DB Machine. The best practice paper has two Media Servers.

    NOTE – If you are using InfiniBand – ensure Connected Mode is setup and the MTU size is increased to 65520 as per the best practice paper.
    NOTE – If you are able to utilize Jumbo Frames, especially on the 10GigE links – ensure that is setup correctly.

    This procedure does utilize the dcli utility running across the root accounts of the DB Nodes

    On the Media Server – obtain the IP Address of the network that you are expecting to send the network traffic across – for me this is
    # ifconfig bondib0
    bondib0 Link encap:InfiniBand HWaddr 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00
    inet addr:192.168.75.129 Bcast:192.168.75.255 Mask:255.255.252.0
    inet6 addr: fe80::221:2800:13e:54c7/64 Scope:Link
    UP BROADCAST RUNNING MASTER MULTICAST MTU:65520 Metric:1
    RX packets:37753974 errors:0 dropped:0 overruns:0 frame:0
    TX packets:15310099 errors:0 dropped:30 overruns:0 carrier:0
    collisions:0 txqueuelen:0
    RX bytes:1916620197564 (1.7 TiB) TX bytes:857611086 (817.8 MiB)

    Now using qperf – setup the server side sessions one for each DB Node that will be sending data. I use ports 4001 through 4008
    # qperf -lp 4001 &
    # qperf -lp 4002 &
    # qperf -lp 4003 &
    # qperf -lp 4004 &
    # qperf -lp 4005 &
    # qperf -lp 4006 &
    # qperf -lp 4007 &
    # qperf -lp 4008 &

    Now on the first node – where SSH is established – we can run the following command
    # dcli -g dbs_group -l root "node=\`hostname | cut -c 8-9\`; qperf 192.168.75.129 -lp 40\${node} -uu -m 65520 -t 30 tcp_bw quit"

    Note that we are assuming the hostname is 9 characters long – with the last two characters being a numerical number – 01 through 08. If your hostname is shorter – then adjust appropriately. We then are running a 30 second (-t) qperf session to the IP Address 192.168.75.129 – with each DB Nodes sending to a different listener port (-lp) db01 to 4001, db02 to 4002 etc… The –m 65520 is setting the MTU Size of 65520 (you can get this from the ifconfig output). The final two options is tcp_bw which is the test you want to run – “TCP streaming one way bandwidth” and quit – which simply tells the server side qperf sessions to terminate when the 30 second run is over.

    The output I got is as follows – I simply changed the hostnames to protect the innocent.
    db01: tcp_bw:
    db01: bw = 394661904 bytes/sec
    db01: quit:
    db02: tcp_bw:
    db02: bw = 393475992 bytes/sec
    db02: quit:
    db03: tcp_bw:
    db03: bw = 393591744 bytes/sec
    db03: quit:
    db04: tcp_bw:
    db04: bw = 393711864 bytes/sec
    db04: quit:
    db05: tcp_bw:
    db05: bw = 393537283 bytes/sec
    db05: quit:
    db06: tcp_bw:
    db06: bw = 393482665 bytes/sec
    db06: quit:
    db07: tcp_bw:
    db07: bw = 393521990 bytes/sec
    db07: quit:
    db08: tcp_bw:
    db08: bw = 393546022 bytes/sec
    db08: quit:
    If we sum up the bytes/sec each DB Node was able to achieve – I get approx 3,000MB/sec

    On a single 10GigE Link – this is probably going to be in the order of 1100MB/sec I believe – please let me know what you find. If you aggregate multiple 10GigE Links with 802.3ad (or LACP / etherchannel) then depending upon the hashing algorithm used by LACP – you might get a higher rate.
    Obviously – if you only have 2 DB Nodes configured with the Media Server software – then run the dcli command using just the 2 DB Nodes.

    h1. Third – reading the data off the Exadata Storage Cells

    People commented about the backupksfq_bufcnt and backupksfq_bufsz parameters. During the Exadata 11.2.1.2.x time line – we made a change to the DBCA template used by the ACS engineers during the DB Installation to set these parameters away from their defaults.

    Before going any further though – these parameters do consume memory on the DB Nodes – and are not recommended for an Oracle/HP Database Machine (V1) unless you know you have the memory available on the system. On an Oracle Database Machine (V2, X2-2 and X2-8) the ability to use these parameters are a little easier – although we recommend that the parameters are set and the resulting runs verified.

    So – what about these parameters –
    backupksfq_bufsz should be set to 4194304
    backupksfq_bufcnt should be set to 64

    What does this do to the memory used by RMAN? Each RMAN Channel that is run – so one RMAN Channel per tape drive – will obtain 64 Input and 64 Output Buffers (total 128Buffers) of 4MB each – for a total memory consumption of 512MB. This will result in 1GB of memory being consumed, if you have 2 channels running on each DB Node. This is the reason why these parameters are not recommended on an Oracle/HP Database Machine (V1) system – where memory is relatively small. Similarly – if the Database Nodes in a V2 or X2-2 or X2-8 system fully utilize the memory in the system – be careful about setting these parameters.

    Also – In Oracle 11gR2 Patchset 1 (11.2.0.2) being installed by default on X2-2 and X2-8 systems – these parameters do NOT need to be set explicitly as they are automatically set to the correct values on a Database Machine.

    During the running of an RMAN Backup – you can check the values of these parameters by querying
    SQL> select buffer_count,buffer_size from v$backup_async_io where type = 'INPUT';

    BUFFER_COUNT BUFFER_SIZE
    64 4194304

    h1. Fourth – Verify the correct network is being used for the backup
    This might sound obvious – but it is always worth validating. The default route out of the DB Nodes in the Database Machine is the Client Access Network – which is typically running at 1GigE – and is also supporting other traffic.

    Using sar -n DEV 1 100 – verify that the packets are going out of the DB Nodes on the correct interface and being received by the media server on the correct interface. The following shows a backup running into a Media Server via the default 1GigE interface, at a rate of approx 120MB/sec (wire speed).

    07:26:14 AM IFACE rxpck/s txpck/s rxbyt/s txbyt/s rxcmp/s txcmp/s rxmcst/s
    07:26:15 AM eth0 81157.00 2675.00 122843758.00 178326.00 0.00 0.00 0.00

    Oracle Secure Backup controls this via the concept of a Preferred Network Interface. I know Symantec NetBackup recommends that this is specified during installation time. I am not sure about other products.

    These are the primary setup verification checks that I go through whenever I get involved in a backup exercise.

    h1. Fifth – Handling BigFile Tablespaces
    If you have a number of BIGFILE Tablespaces you will want the tablespace to be broken up into little parts and run in parallel. The SECTION SIZE parameter can be used in RMAN for a Level 0 backup and will do this for you, but the SECTION SIZE parameter is not applicable for a Level 1 backup. However, if you have enabled the Block Change Tracking file – the use of the Block Change Tracking file will generally alleviate the need to break the backup into multiple sections depending upon how much data is changed in a given day.

    You probably want to set the section size such that your largest tablespace is evenly distributed between the available tape drives (within reason). For instance – if you have a 2TB BIGFILE Tablespace – and you have access to 8 tape drives – then set the SECTION SIZE to 256GB. If you have other BIGFILE Tablespace that are only 1TB in size – then you might want to set SECTION SIZE to 128GB. The goal is to have all tape drives spinning until the last moments of the backup. RMAN will backup the BIGFILES first – and then work towards the smaller files such as SYSTEM as SYSAUX.
  • 20. Re: Exadata Backup job is taking very long time
    Daryl E. Explorer
    Currently Being Moderated
    Sorry to resurect and old thread but I have come back to revisit this.. Seems we cant push our 1Gb backup interface hard enough to get the 100MB/s capacity. If I run 3-4 channels thru there it can. Is that "normal" should 1 channel be able to fill the backup interface connection?
    From my sar outputs I see only about 15-20Mb/s with 1 channel running (of course I have the other 7 db nodes also pushing that same 15-20mb/s). Can I some how boost that one connection?

    By running 3 channels per host I got 95Mb/s in sar
    10:32:34 PM      eth3  33086.00  66209.00 1985100.00 100188034.00      0.00      0.00      0.00
    I looked at the
    _backup_ksfq_bufsz should be set to 4194304
    _backup_ksfq_bufcnt should be set to 64
    but I see that being on 11gr2, its already 4194304 & *84*.

    Thoughts? Any harm in running 3 or 4 channels per node? I would think that would need a similar number of tape drives on the other end?
  • 21. Re: Exadata Backup job is taking very long time
    ababb Newbie
    Currently Being Moderated
    You should be able to get close to 120MB/sec across the link, assuming you are not employing RMAN compression or similar.

    The difference between 64 and 84 bufcnt will not play this heavily on the issue.

    Would you be able to dump out the RMAN> 'show all' output, as well as the backup script you are using.

    I recently had a problem which turned out to be a tape drive FC issue, and this was only tracked down by looking at the Brocade Switch statistics. Do you see any errors when you run porterrshow (or equivalent command)
    FID128:admin> porterrshow
    frames enc crc crc too too bad enc disc link loss loss frjt fbsy
    tx rx in err g_eof shrt long eof out c3 fail sync sig
    =========================================================================================================
    0: 3.2k 5.6k 0 0 0 0 0 0 0 0 0 0 0 0 0
    1: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    2: 153.0m 44.2m 0 0 0 0 0 0 0 0 0 0 0 0 0
    3: 153.3m 44.1m 0 0 0 0 0 0 0 0 0 0 0 0 0
    4: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    5: 1.9m 4.7m 26.0k 25.6k 25.6k 0 0 1.0k 1.0m 0 0 0 1 0 0
    6: 153.1m 44.2m 0 0 0 0 0 0 0 0 0 0 0 0 0
    7: 153.2m 44.2m 0 0 0 0 0 0 0 0 0 0 0 0 0
    8: 153.3m 44.2m 0 0 0 0 0 0 0 0 0 0 0 0 0
    9: 1.9m 2.5m 2.7k 2.7k 2.6k 0 0 138 196.5k 0 0 0 1 0 0
    10: 2.0m 7.4m 82.1k 79.8k 79.7k 0 0 1.0k 466.9k 0 0 0 1 0 0

    My bad ports are 5,9 and 10 in the output below.

    I am assuming the previous suggestions did not help
  • 22. Re: Exadata Backup job is taking very long time
    Daryl E. Explorer
    Currently Being Moderated
    Here is the outputs:
    RMAN> show all;
    
    RMAN configuration parameters for database with db_unique_name PCINF are:
    CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 30 DAYS;
    CONFIGURE BACKUP OPTIMIZATION OFF; # default
    CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
    CONFIGURE CONTROLFILE AUTOBACKUP OFF;
    CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F'; # default
    CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE SBT_TAPE TO '%F'; # default
    CONFIGURE DEVICE TYPE DISK BACKUP TYPE TO BACKUPSET PARALLELISM 1;
    CONFIGURE DEVICE TYPE 'SBT_TAPE' BACKUP TYPE TO BACKUPSET PARALLELISM 8;
    CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
    CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE SBT_TAPE TO 1; # default
    CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
    CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE SBT_TAPE TO 1; # default
    CONFIGURE MAXSETSIZE TO UNLIMITED; # default
    CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
    CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
    CONFIGURE COMPRESSION ALGORITHM 'BASIC' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE ; # default
    CONFIGURE ARCHIVELOG DELETION POLICY TO NONE; # default
    CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/u01/app/oracle/product/11.2.0/dbhome_1/dbs/snapcf_pcinf1.f'; # default
  • 23. Re: Exadata Backup job is taking very long time
    Daryl E. Explorer
    Currently Being Moderated
    Your previous answer was very informative and helpful - but the situation is still not ideal. Full backups taking days and prone to failure with the media mgmt layer having issues with tapes and such for such long running jobs.

    I dont think there is anything to unusual here - but if you see any improvements, let me know. Thanks!
    ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1;
    export ORACLE_HOME;
    ORACLE_SID=pcinf1;
    export ORACLE_SID;
    TNS_ADMIN=/u01/app/oracle/product/11.2.0/dbhome_1/network/admin;
    export TNS_ADMIN;
    /u01/app/oracle/product/11.2.0/dbhome_1/bin/rman target / catalog rman11g/rman@dbe12ykf:1524/PDBTOOLS msglog /usr/openv/netbackup/logs/user_ops/dbext/rmanlogs/pcinf1_0402_Full.log append << EOF
    ALLOCATE CHANNEL FOR MAINTENANCE DEVICE TYPE DISK;
    ALLOCATE CHANNEL FOR MAINTENANCE DEVICE TYPE 'SBT_TAPE';
    RUN {
    CONFIGURE RETENTION POLICY TO RECOVERY WINDOW of 30 DAYS;
    SET COMMAND ID TO 'Full';
    CONFIGURE BACKUP OPTIMIZATION CLEAR;
    CONFIGURE CONTROLFILE AUTOBACKUP OFF;
    CONFIGURE DEVICE TYPE DISK BACKUP TYPE TO BACKUPSET PARALLELISM 1;
    #
    ALLOCATE CHANNEL ch00 TYPE 'SBT_TAPE' MAXOPENFILES 32 connect sys/xxx@exdb01:1521/pcinf;
    ALLOCATE CHANNEL ch01 TYPE 'SBT_TAPE' MAXOPENFILES 32 connect sys/xxx@dbexdb02:1521/pcinf;
    ALLOCATE CHANNEL ch02 TYPE 'SBT_TAPE' MAXOPENFILES 32 connect sys/xxx@dbexdb03:1521/pcinf;
    ALLOCATE CHANNEL ch03 TYPE 'SBT_TAPE' MAXOPENFILES 32 connect sys/xxx@dbexdb04:1521/pcinf;
    ALLOCATE CHANNEL ch04 TYPE 'SBT_TAPE' MAXOPENFILES 32 connect sys/xxx@dbexdb05:1521/pcinf;
    ALLOCATE CHANNEL ch05 TYPE 'SBT_TAPE' MAXOPENFILES 32 connect sys/xxx@dbexdb06:1521/pcinf;
    ALLOCATE CHANNEL ch06 TYPE 'SBT_TAPE' MAXOPENFILES 32 connect sys/xxx@dbexdb07:1521/pcinf;
    ALLOCATE CHANNEL ch07 TYPE 'SBT_TAPE' MAXOPENFILES 32 connect sys/xxx@dbexdb08:1521/pcinf;
    send 'NB_ORA_POLICY=cnc_RMAN_PCINF_Full';
    send 'NB_ORA_PC_SCHED=Full';
    CONFIGURE EXCLUDE FOR TABLESPACE 'USERS';
    CONFIGURE EXCLUDE FOR TABLESPACE 'L_SRC';
    BACKUP INCREMENTAL LEVEL=0
    SKIP INACCESSIBLE
    FORMAT 'bk_%d_%s_%p_%t'
    FILESPERSET=3
    DATABASE;
    CONFIGURE EXCLUDE FOR TABLESPACE 'USERS' CLEAR;
    CONFIGURE EXCLUDE FOR TABLESPACE 'L_SRC' CLEAR;
    BACKUP SECTION SIZE 200G TABLESPACE 'USERS';
    BACKUP SECTION SIZE 200G TABLESPACE 'L_SRC';
    }
    SHOW ALL;
    RELEASE CHANNEL;
    EXIT;
    EOF
    If I double up the number of channels I seem to push the backup interface eth3 a bit closer to the theoretical max ~50Mb/s (would be nice to get ~100Mb/s)
    Is there a better way then to hard code all those ALLOCATE CHANNEL commands? Using a service doesnt seem to load balance very well at all.

    Thanks
    Daryl.
  • 24. Re: Exadata Backup job is taking very long time
    ababb Newbie
    Currently Being Moderated
    Symantec have a backup paper called "Protecting an Exadata Database Machine with NetBackup for Oracle" which talks about how to test backup rates to a disk pool (to verify that the network portion of RMAN & NBU are correct) and then goes into the tape portion of the job. It also covers certain NetBackup tuning parameters. You should be able to get a copy from Symantec and this might help.

    I do not see any issues with the "RMAN" portions of the script below, but I am not sure what the different SEND commands are doing in Symantec.
  • 25. Re: Exadata Backup job is taking very long time
    969598 Newbie
    Currently Being Moderated
    Hello

    Could it be possible to get a link to the "Protecting an Exadata Database Machine with NetBackup for Oracle" document?

    Or could you tell me how to get it?

    Best Regards
  • 26. Re: Exadata Backup job is taking very long time
    Klaas-Jan Jongsma Newbie
    Currently Being Moderated
    Hi,

    I am wondering why you are limiting your backup pieces with FILESPERSET=3? You have parallelism set to 8, maxsetsize set to unlimited, the only reason i can think why you would do this is to minimize the amount time during a restore when looking for the correct piece? Otherwise i would just remove that parameter and make way bigger pieces.

    Klaas-jan
1 2 Previous Next

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points