Hi everyone. Hoping someone can provide some guidance here.
We are trying to develop some low-end NAS devices using Solaris 11.1 to house virtual machines. Between the RAID configuration and the SSDs we have been able to obtain very solid performance under max load. The problem is when a drive fails.
Disk latency, more precisely read latency, goes extremely high. An example is we have one recently built and not housing many machines. start a resilver, read latency jumps from 25ms to 235ms. A bit of a problem for VMs.
1.37TB used on a 14TB pool, almost 200GB to resilver.
Hardware controllers in my experience are bale to rebuild a drive and the performance impact is not noticable. Different situation I understand.
I am trying to locate a tunable or some other suggestion to help resolve this issue. I suppose the same can be said for scrubs. We currently do not perform scrubs because it brings the system to its knees.
Is this a mirrored ZFS pool or a ZFS pool that is sitting on top of a mirrored H/W config? What kind of hardware? Is the pool built directly on whole disks and not virtual disks?
We have very large mirrored ZFS pools built on hardware arrays in JBOD mode for large build servers, lots of compiling and stuff and I don't think anyone notices when a disk resilvers. At least, they don't mention it.
My nearby coworkers who know resilvering and scrubs best are in constant meetings this week and last week. I'm hesitant to hand-out tuning info on this discussion list without knowing this stuff really well. However, you might start by tuning the zfs_resilver_delay parameter, which is the number of clock ticks to delay resilvering when other work is present. What concerns me is if you slow down resilvering, do you risk another disk failure while the first disk is still resilvering? The default is 2. O means ignore all other work and increase resilvering speed. You might try setting this to 4:
Really, your best approach is to file a MOS SR to get this investigated.
I understand but if I am not mistaken compiling is more CPU intensive on the client than disk I/O on the storage side.
This is a ZFS pool consisting of 8 mirrors. This is all physical, no virtual hardware. The pool built directly on whole disks being presented by an LSI HBA. No H/W RAID.
Thank you for the delay parameter this is exactly what I was looking for. Something to slow it down just a bit. I am hesitant to change the default values but I may need to test to see if going to 4 would help us. I am not so concerned about losing another disk during resilver as long as it is not the disk in the same vdev which I believe should be unlikely. Thank you for you help. Depending on how this goes I may open that MOS SR.
Another quick question. I've seen documentation stating resilver can be stopped but haven't found the command to do so. I know how to stop a scrub "zpool scrub -s".
Ha...the joke's on me. I'll have that text fixed because it implies that you can stop a resilver but what it means is that if the system reboots or the power fails, resilvering will continue where it left of when the OS is running again.
Are any of the iostat -En errors seen by FMA or ZFS?
I know that updating the HBA/disk firmware can sometimes reduce the transport errors. I'm not sure about the hard errors. Which LSI HBA model is this?