2 Replies Latest reply on Aug 28, 2013 6:40 PM by Carlos Azevedo

    Poor performance on SSD-backed ZFS pools in mirror configuration only

    Tim Creswick

      We are currently debugging a system with 12x Toshiba 512GB SSDs on Solaris 11.1 (Oracle Solaris


      The system has each SSD directly attached to a dedicated port on an LSI 9206-16e card, so there should be full bandwidth to each device all the way through the SAS and PCIe bus.


      We're running dd write-tests from /dev/zero to a test filesystem. We've tried most tunables with no success; at the moment a pool of 6 mirrors always tops out at a meagre 750 MB/s. CPU utilisation is actually quite low whilst this is happening.


      By contrast, if we create a pool of 3 SSDs in a stripe, we get write speeds of around 1.5 GB/s.


      A pool with a single SSD in it gives write speeds of around 480 MB/s, which is what we expect.


      As soon as we put the SSDs into a mirror, the performance drops off, approximately in half.


      Interestingly, we have also tried offlining half the drives in the pool using 'zpool offline testpool <dev>'. When we do this, the performance picks up to the expected levels.


      Is there a bug in the mirror vdev code and/or a tunable for vdevs that we've missed?