Most of the changes were visible in iostat, but still caused the timeouts. I thought the timeouts only started yesterday, but I see from /var/adm/messages that they have been going back at least a week, showing up here and there. When this happens, the machine effectively freezes for up to a minute then the timeout clears and things start back up again.
The following dtrace program (which I grabbed off the interweb; not sure from where) shows a retry when the machine stops, which corresponds to 100% busy of one or more disks for about 60 seconds.
#pragma D option quiet
printf("Tracing... output every 10 seconds.\n");
The machine is a HP DL585 G2 with 24GB of RAM. The i/o is fibre channel from 2x Qlogic 2342 dual port 2gb HBAs (4x 2gb ports). The HBAs feed a Silkworm 3200 and 3250, which in turn connect to two A5200 arrays each in split loop mode (8x 1gb loops). The arrays house 44 total disks roughly split between 15k and 10k spindles.
I do not recall seeing this on a previous v40z server which was replaced three weeks ago with the DL585. It almost seems like the server is overwhelming the arrays but I cannot confirm why or how to stop it.
Does anyone have any suggestions on how to fix this or how to troubleshoot it further?
I know this is an old thread, but there is precious little information out there, so maybe this will help the next guy. After studying mode page settings, it seems that ARRE and AWRE set to one have reduced (but not eliminated) the timeouts. I have played with a LOT of settings. After globally setting the values as described below, the timeouts have been reduced. Prior to setting the values, the drives were a mixed bag of values.
bash-4.1$ for d in /dev/rdsk/c0t2*d0; do sudo sdparm -S -s AWRE=1 $d; done
bash-4.1$ for d in /dev/rdsk/c0t2*d0; do sudo sdparm -S -s ARRE=1 $d; done
If anyone else knows of good ways to reduce these timeouts (on drives that otherwise appear OK), please speak up.
I found a pattern with this. The more disks in the vdev, the higher the chances of getting this timeout. Previously, a pool with 6x vdevs each RADIZ1 with 4 disks had no timeouts. It was recently changed to 3x vdevs RADZ2 each with 8 disks and has some sporadic timeouts. Another pool with the same model of disks is configured as a single vdev RAIDZ3 with 20 disks and times out regularly under load.
It is either the spindles / vdev ratio or the RAIDZ level, or some combination.