3 Replies Latest reply on May 12, 2013 4:08 AM by martyscholes-JavaNet

    Strange fc scsi_vhci timeouts during heavy disk activity

      After a recent hardware upgrade, I noticed strange timeouts while replicating a zfs pool under Solaris 11. They all look similar to:

      Mar 23 04:33:20 dl585 scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0):
      Mar 23 04:33:20 dl585 /scsi_vhci/disk@g2000001862b56107 (sd33): Command Timeout on path fp4/disk@w2200001862b56107,0

      I have played with setting various kernel parameters such as the following with no improvement.

      * sd_io_time
      * zfs_vdev_max_pending
      * sd_max_throttle

      Most of the changes were visible in iostat, but still caused the timeouts. I thought the timeouts only started yesterday, but I see from /var/adm/messages that they have been going back at least a week, showing up here and there. When this happens, the machine effectively freezes for up to a minute then the timeout clears and things start back up again.

      The following dtrace program (which I grabbed off the interweb; not sure from where) shows a retry when the machine stops, which corresponds to 100% busy of one or more disks for about 60 seconds.

      #!/usr/sbin/dtrace -s

      #pragma D option quiet

           printf("Tracing... output every 10 seconds.\n");

           @[xlate <devinfo_t *>(args[1])->dev_statname,
           xlate <devinfo_t *>(args[1])->dev_major,
           xlate <devinfo_t *>(args[1])->dev_minor] = count();

           printf("\n%Y:\n", walltimestamp);
           printf("%28s %-3s,%-4s %s\n", "DEVICE", "MAJ", "MIN", "RETRIES");
           printa("%28s %-03d,%-4d %@d\n", @);

      The machine is a HP DL585 G2 with 24GB of RAM. The i/o is fibre channel from 2x Qlogic 2342 dual port 2gb HBAs (4x 2gb ports). The HBAs feed a Silkworm 3200 and 3250, which in turn connect to two A5200 arrays each in split loop mode (8x 1gb loops). The arrays house 44 total disks roughly split between 15k and 10k spindles.

      I do not recall seeing this on a previous v40z server which was replaced three weeks ago with the DL585. It almost seems like the server is overwhelming the arrays but I cannot confirm why or how to stop it.

      Does anyone have any suggestions on how to fix this or how to troubleshoot it further?

        • 1. Re: Strange fc scsi_vhci timeouts during heavy disk activity
          Hi Marty

          We have a very similar issue, the re are maybe 10 timeouts a day, the mode for the timeout is 55 seconds.

          There is a different hardware confguration, but the sympton is uncannely similar.

          It would be very interesting to know what the cause of the issue was.
          • 2. Re: Strange fc scsi_vhci timeouts during heavy disk activity
            I know this is an old thread, but there is precious little information out there, so maybe this will help the next guy. After studying mode page settings, it seems that ARRE and AWRE set to one have reduced (but not eliminated) the timeouts. I have played with a LOT of settings. After globally setting the values as described below, the timeouts have been reduced. Prior to setting the values, the drives were a mixed bag of values.

            bash-4.1$ for d in /dev/rdsk/c0t2*d0; do sudo sdparm -S -s AWRE=1 $d; done
            bash-4.1$ for d in /dev/rdsk/c0t2*d0; do sudo sdparm -S -s ARRE=1 $d; done

            If anyone else knows of good ways to reduce these timeouts (on drives that otherwise appear OK), please speak up.

            • 3. Re: Strange fc scsi_vhci timeouts during heavy disk activity
              I found a pattern with this. The more disks in the vdev, the higher the chances of getting this timeout. Previously, a pool with 6x vdevs each RADIZ1 with 4 disks had no timeouts. It was recently changed to 3x vdevs RADZ2 each with 8 disks and has some sporadic timeouts. Another pool with the same model of disks is configured as a single vdev RAIDZ3 with 20 disks and times out regularly under load.

              It is either the spindles / vdev ratio or the RAIDZ level, or some combination.