We are currently debugging a system with 12x Toshiba 512GB SSDs on Solaris 11.1 (Oracle Solaris 22.214.171.124.0).
The system has each SSD directly attached to a dedicated port on an LSI 9206-16e card, so there should be full bandwidth to each device all the way through the SAS and PCIe bus.
We're running dd write-tests from /dev/zero to a test filesystem. We've tried most tunables with no success; at the moment a pool of 6 mirrors always tops out at a meagre 750 MB/s. CPU utilisation is actually quite low whilst this is happening.
By contrast, if we create a pool of 3 SSDs in a stripe, we get write speeds of around 1.5 GB/s.
A pool with a single SSD in it gives write speeds of around 480 MB/s, which is what we expect.
As soon as we put the SSDs into a mirror, the performance drops off, approximately in half.
Interestingly, we have also tried offlining half the drives in the pool using 'zpool offline testpool <dev>'. When we do this, the performance picks up to the expected levels.
Is there a bug in the mirror vdev code and/or a tunable for vdevs that we've missed?
Yes, very curious indeed. I would expect the mirrored configuration to perform better not slower.
Most of my resources who could look at this are tied up in meetings all week but I have a few
1. What kind of write workload is this? Can you provide your test dd syntax?
2. Do you see any SSD related errors in /var/adm/messages?
3. Any unusual SSD temp readings?