IO latency of 60s
Dear Community,
I am facing an IO hang issue on a T4 server with Solaris 11.3. Sporadically, we have IO hangs which last approx. 60 seconds. During this time, iostat shows one device with 100% busy but with no r/s or w/s.
Tuesday, April 25, 2017 04:18:20 PM CEST
extended device statistics
device r/s w/s kr/s kw/s wait actv svc_t %w %b
sd3 0.0 18.6 0.0 266.5 0.0 0.0 1.4 0 1
sd4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
sd5 0.0 18.4 0.0 266.5 0.0 0.0 1.8 0 1
sd6 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
ssd0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
ssd1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
ssd9 0.0 17.0 0.0 574.5 0.0 0.1 3.1 0 1
ssd10 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
ssd11 0.0 17.2 0.0 574.6 0.0 0.1 3.1 0 2
ssd12 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
ssd13 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
ssd14 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
ssd15 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
ssd19 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
ssd20 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
ssd21 0.0 17.4 0.0 574.3 0.0 0.1 4.4 0 2
ssd22 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
ssd23 0.2 17.2 0.1 574.6 0.0 0.1 3.1 0 2
ssd24 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
ssd25 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
ssd26 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
ssd27 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
ssd28 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
ssd29 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
ssd30 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
ssd31 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
ssd32 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0 100. <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
ssd33 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
ssd34 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
ssd35 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
zpool iostat shows zero read/write operations during this 60s period. The dtrace script zfsslower.d reports latencies around 60s during the hang.