Did you try to match the ZFS Max Pending? It needs to match the queue depth.
I have 8 T4-4s with about a 400 LUNs each and I have had to play around with the settings until I found the best I/O performance was with a queue depth of 4:
The problem that I have now is we are expecting to have over 1000 LUNs on each server and by following Hitachi's best practice my queue depth should now go to 2.
Hitachi's Formula to calculate queue depth:
(Number of LUs) × (queue_depth) ≤ 2048 and queue_depth ≤ 32
1000 * 4 = 4000 (which is greater than 2048)
1000 * 2 = 2000 (This works, but am I limited to only a 1000 luns before I/O problems occur?)
Hope the zfs_vdev_max_pending helps.