This discussion is archived
1 2 Previous Next 21 Replies Latest reply: Apr 9, 2013 4:38 PM by user5749629 RSS

btrfs slows down with expdp in high parallel on large tables

user5749629 Newbie
Currently Being Moderated
We are trying out btrfs. The main reason for doing so over ext4 is that we need to create a file system > 16TB.

This file system will be mounted on Oracle DBs to serve as an export directory for expdp exports of the databases.

Btrfs is very simple and friendly, and for small tables works very well.

However - the bigger the table and higher the parallel - it seems to slow down to a virtual standstill after a while. "-m raid0" lasts a little longer than the defaults but it, too, experiences this.

We understand it's new and experimental.

Any words of wisdom of what settings might be good for high parallel expdp?
Any way to make ext4 go above 16TB and avoid btrfs altogether for now - because performance-wise ext4 on UEK2 is fantastic in managing parallel access over a mount - it's fast - it scales - it's lean - substantially better than RHEL5. We just need the same experience for an 80TB filesystem.

Thanks !
  • 1. Re: btrfs slows down with expdp in high parallel on large tables
    Dude! Guru
    Currently Being Moderated
    From what I understand, the main focus of btrfs are in areas of data redundancy, snapshot ability, and continued efforts to improve performance.
    The main reason for doing so over ext4 is that we need to create a file system > 16TB.
    According to wikipedia, the limit is 1 EiB or 1024 TB. The issue that limits the file system to 16 TB are the available tools to create and maintain the file system. Version 1.42 of e2fsprogs (e4fsprogs) supports the creation of ext4 file systems larger than 16TB.

    You can download http://e2fsprogs.sourceforge.net/
    E2fsprogs 1.42 (November 29, 2011): This release of e2fsprogs has support for file systems > 16TB

    Version 1.42 has apparently still not made into the current RHEL distributions, currently 1.41.12. Makes me wonder why RHEL even bothered to release ext4 yet.

    However, neither btrfs nor ext4 are recommended by Oracle to store oracle database files. Oracle ASM supports file sizes up to 128 TB according to http://docs.oracle.com/cd/E11882_01/server.112/e17110/limits002.htm#REFRN0042
  • 2. Re: btrfs slows down with expdp in high parallel on large tables
    user5749629 Newbie
    Currently Being Moderated
    Thanks. Clearly if you have $$$ - the recommended solution is to get a NAS. Otherwise you are left to improvise.

    We will check e2fsprogs.

    Yes, ASM supports datafiles for Bigfile Tablespaces of up to 128TB. We have those - and performance is outstanding. This is exactly the issue - we'd like to export them into a single filesystem.

    I guess the next thing to try is ADVM volumes within a large ASM diskgroup.

    Thank you for the tip !
  • 3. Re: btrfs slows down with expdp in high parallel on large tables
    Dude! Guru
    Currently Being Moderated
    Support for ADVM is kernel version and database version depending and requires access to Oracle support. Perhaps it would have been a better idea, certainly more practical and compatible, to design tablespaces that comprise of smaller datafiles on separate partitions in your case.
  • 4. Re: btrfs slows down with expdp in high parallel on large tables
    user5749629 Newbie
    Currently Being Moderated
    Very true. We are on 11.2.0.3 - so ADVM works - we just need to see how it scales.

    The problem is that the SCHEMAs are 40-60TB and typicaly a SCHEMA resides in a single tablespace - hence the large tablespaces.

    Ideally we would like to export an entire tablespace (2-3 schemas) but that's even more ambitious. For now we are content to export a single SCHEMA at a time - but that still requires a landing zone filesystem > 16TB.

    Thanks !
  • 5. Re: btrfs slows down with expdp in high parallel on large tables
    Dude! Guru
    Currently Being Moderated
    You can add datafiles to a tablespace as you require or desire and to my knowledge there is no constrain between schema and datafile. It is a common practice, for instance, to simply add another datafile to a tablespace if you need more space.

    The Oracle datapump or export utility can split the dumpfile file into several smaller chunks of data and even define that the output goes to different file systems. Such questions are however better discussed in the database forum.

    I found an old related post that demonstrates the use of pipes and how to split and automatically compress the export output: What to do when not all pipes was consumed?

    You can also define the output using the FILE parameter and thereby create dumpfiles on different volumes, for instance: FILE=/u01/prodex/exp01.dmp,/u02/prodex/exp02.dmp,/u03/prodex/exp03.dmp FILESIZE=2G
  • 6. Re: btrfs slows down with expdp in high parallel on large tables
    bigdelboy Pro
    Currently Being Moderated
    I may be making a stupid suggestion here, but would XFS suit your purposes for that filesystem?

    http://xfs.org/docs/xfsdocs-xml-dev/XFS_User_Guide//tmp/en-US/html/index.html

    Having said that I would probably be going into splitting the export into chunks myself as Dude suggested.
  • 7. Re: btrfs slows down with expdp in high parallel on large tables
    alvaromiranda Explorer
    Currently Being Moderated
    Have a look at OCFS2.

    You can test the block and cluster size for the size you want.

    You can go for single server, no cluster, and it's fully supported/certified for oracle rdbms files.

    Alvaro.
  • 8. Re: btrfs slows down with expdp in high parallel on large tables
    user5749629 Newbie
    Currently Being Moderated
    Dude - that's a clever workaround ! You are right - you can run expdp with DUMPFILE=DIR1:FILE1_%U,DIR2:FILE2_%U and get 2 sets of dump files, so DIR1 and DIR2 can be 2 different <16TB ext4 file systems !!!

    Compression also helps - whether through the OS or the export utility.

    And you're right - it's not about the tablespace size - it's the SCHEMAs that are large and need to be exported as a single chunk for ease of administration.

    Our hope, of course, was to have one big room where you can just throw everything - instead of multiple smaller cabinets - but if >16TB (btrfs or ext4) is a little bleeding edge - multiple DUMP destinations is a great way to get around it.

    Thanks !
  • 9. Re: btrfs slows down with expdp in high parallel on large tables
    user5749629 Newbie
    Currently Being Moderated
    bigdelboy - thanks - it's actually a very good suggestion to use XFS. They are supposed to be the "big scalable" filesystem. They were our "plan B" for the following reasons -

    (1) there seems to be a licensing surcharge for the >16TB filesystem feature
    (2) we are all UEK2 - and on large volume, large performance - uniformity of OS is a virtue - if only for support purposes - who knows - maybe there's some tiny incompatibility between UEK2 and XFS ?
    (3) if >16TB is bleeding edge on ext4 and btrfs - is it truly mature on XFS?
    (4) we don't have much experience with XFS - so a new OS to get used to - little things no doubt - but still a learning curve

    So we were going to try and exhaust the OEL offerings first - but XFS is definitely high on the fallback list.

    Thanks !
  • 10. Re: btrfs slows down with expdp in high parallel on large tables
    user5749629 Newbie
    Currently Being Moderated
    Alvaro - thanks - that's another great idea.

    We don't use OCFS2 today - so we didn't know if it would scale beyond 16TB, and we kind of assumed it's for clusters - but you're right - no reason why it couldn't run standalone, plus it's built into the UEK2 kernel already.

    We'll give it a try.

    Thanks everyone - a lot of good suggestions !
  • 11. Re: btrfs slows down with expdp in high parallel on large tables
    alvaromiranda Explorer
    Currently Being Moderated
    For local, you should be able to use something like this:

    #mkfs.ocfs2 -F -T datafiles -M local --fs-features=refcount /dev/<device>

    And when you do the mount entry on fstab, just skip the _netdev, as won't require network to work.

    /dev/<device> /<mount_point> ocfs2 datavolume,nointr,noatime 0 0

    Test the default mkfs.ocfs2 first, then, start playing with the custom values of block and cluster device.

    be nice, and tell us back how it went. :D
  • 12. Re: btrfs slows down with expdp in high parallel on large tables
    alvaromiranda Explorer
    Currently Being Moderated
    As you said you are on UEK, if you want to test ACFS also, take note, as today, you can run ACFS with UEK2 (2.6.39-300) on 11.2.0.3.4 oct 2012 or 11.2.0.3.5 jan 2013

    On Grid Home, install

    #112034 Oct12
    patch 14275572

    then, this patch does the trick.

    #ACFS UEK - required 112034
    patch 12983005

    The rdbms db oh, can be Jan2013 11.2.0.3.5 and works fine.

    #112035 Jan13
    patch 14727347

    Update:

    As today, 12983005 is out for 11.2.0.3.5 Jan 2013 also, so you can be on Oct2012 or Jan 2013.. Bleeding edge in patches. :D

    Alvaro.

    Edited by: Alvaro Miranda on Feb 26, 2013 5:50 PM
  • 13. Re: btrfs slows down with expdp in high parallel on large tables
    user5749629 Newbie
    Currently Being Moderated
    Alvaro - thanks. The thing is that we are on this version of UEK2 -

    Linux (servername) 2.6.39-200.34.1.el6uek.x86_64 #1 SMP Thu Oct 18 17:00:17 PDT 2012 x86_64 x86_64 x86_64 GNU/Linux

    and on 11.2.0.3.0

    so making a case for patches / upgrades would be another challenge.

    Will ACFS2 with the desired functionality not work on the above?
  • 14. Re: btrfs slows down with expdp in high parallel on large tables
    alvaromiranda Explorer
    Currently Being Moderated
    well,

    don't panic!


    #ACFS UEK - required 112034
    patch 12983005

    enables the acfs to work with UEK2 2.6.39-300 but it requires the OCT 2012 cpu patch.

    not big deal.

    I will give you the steps:

    http://kikitux.blogspot.co.nz/2013/02/acfs-with-uek2-kernel.html
1 2 Previous Next

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points