This discussion is archived
2 Replies Latest reply: Aug 28, 2013 11:40 AM by Carlos Azevedo RSS

Poor performance on SSD-backed ZFS pools in mirror configuration only

TimCreswick Newbie
Currently Being Moderated

We are currently debugging a system with 12x Toshiba 512GB SSDs on Solaris 11.1 (Oracle Solaris 11.1.10.5.0).

 

The system has each SSD directly attached to a dedicated port on an LSI 9206-16e card, so there should be full bandwidth to each device all the way through the SAS and PCIe bus.

 

We're running dd write-tests from /dev/zero to a test filesystem. We've tried most tunables with no success; at the moment a pool of 6 mirrors always tops out at a meagre 750 MB/s. CPU utilisation is actually quite low whilst this is happening.

 

By contrast, if we create a pool of 3 SSDs in a stripe, we get write speeds of around 1.5 GB/s.

 

A pool with a single SSD in it gives write speeds of around 480 MB/s, which is what we expect.

 

As soon as we put the SSDs into a mirror, the performance drops off, approximately in half.

 

Interestingly, we have also tried offlining half the drives in the pool using 'zpool offline testpool <dev>'. When we do this, the performance picks up to the expected levels.

 

Is there a bug in the mirror vdev code and/or a tunable for vdevs that we've missed?

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points