6 Replies Latest reply on Mar 2, 2013 10:10 PM by diamaunt

    irregular i/o on zpool

      Hi Guys,

      I just migrated a pool to another system with an onboard raid controller. My thought was that having all my discs on the same controller may help performance, but as I'm posting here, I'm clearly having issues. I'm wondering if there's any suggestions to improve this, but here's what I see

      solaris@srv-data:~$ zpool iostat tank -v 3
      cannot open '-v': name must begin with a letter
      capacity operations bandwidth
      pool alloc free read write read write
      ---------- ----- ----- ----- ----- ----- -----
      tank 4.73T 6.14T 0 0 51.8K 14.3K
      tank 4.73T 6.14T 0 0 0 0
      tank 4.73T 6.14T 0 0 0 0
      tank 4.73T 6.14T 0 277 170 31.4M ---- this where i started writing a large file with DD
      tank 4.73T 6.14T 0 0 0 0
      tank 4.74T 6.14T 0 745 0 91.0M
      tank 4.74T 6.14T 0 41 0 4.88M
      tank 4.74T 6.14T 0 734 0 90.2M
      tank 4.74T 6.14T 0 808 0 99.9M
      tank 4.74T 6.14T 0 24 0 173K
      tank 4.74T 6.14T 0 885 0 107M
      tank 4.74T 6.14T 0 104 0 10.3M

      the numbers are up an down. I took a screen shot as well on a test pool I had done with a pair of old mismatched 80gb drives before i migrated the pool


      it seems to do these regular drops in i/o rate, seems like a buffer issue of sorts. I did some google searching and found 1-2 similar issues, but the solution they had was going to a larger LSI 9260-16i controller which is not practical here. I'm using an onboard LSI 1068E controller

      the system is on a supermicro X8DT3 board with two Xeon 5570's and 32gb ram running solaris 11.1

      the pool is consists of 6x 2tb WD RE4's

      any input would be greatly appreciated
        • 1. Re: irregular i/o on zpool
          Bjoern Rost
          are you sure this is not dd blocking? were are you reading from? Also, the '-v' in your command is in the wrong place, the correct syntax is
          zpool iostat -v tank 3
          • 2. Re: irregular i/o on zpool
            If i transfer files over the network, it seems to transfer, then pause, and continue transferring the same way so it doesn't seem to be just dd. my write speeds are significantly lower too, 100~mb is less than half of what I had before

            I should also note the controller is in pass-through mode in case that question comes up

            I've also updated to the latest bios on the motherboard as well, though I'm not sure if that touched any LSI bios/firmware or not, or even if i can update the lsi firmware seperately
            • 3. Re: irregular i/o on zpool
              Well I've been at it today

              I tried updating the firmware for the onboard controller, that has had no change

              I also tried putting in an add-in lsi 9240 controller and I'm seeing the same kind of performance with it.

              Is it just the LSI driver in solaris that's terrible or something? I'm at a loss here
              • 4. Re: irregular i/o on zpool
                take a look at http://www.cuddletech.com/blog/pivot/entry.php?id=1015

                but, yes, zfs caches writes, then flushes them to disk, it doesn't write constantly, unless you''re writing data faster than it can flush to disk.
                • 5. Re: irregular i/o on zpool
                  this is where I have an issue. these numbers were taken on the previous core2quad q6600, with 8gb ram on an asus p5q premium. the disks attached to 4 of the onboard intel sata ports and two on a secondary controller that's on the board

                  he lower numbers are files synching from my linux machine over gigabit lan, the last few intervals are the result of me running DD to write a large file to the drive locally. not bad

                  solaris@SRV-DATA:~$ zpool iostat tank -v 3
                  cannot open '-v': name must begin with a letter
                  capacity operations bandwidth
                  pool alloc free read write read write
                  ---------- ----- ----- ----- ----- ----- -----
                  tank 892G 10.0T 39 211 37.8K 22.1M
                  tank 894G 10.0T 11 264 11.5K 31.0M
                  tank 895G 10.0T 29 219 27.4K 23.6M
                  tank 896G 10.0T 29 215 30.5K 22.9M
                  tank 903G 9.99T 26 1.33K 24.0K 165M
                  tank 912G 9.98T 8 1.57K 8.80K 197M
                  tank 937G 9.96T 6 1.50K 6.47K 189M
                  tank 946G 9.95T 11 1.58K 11.0K 198M
                  tank 954G 9.94T 8 1.63K 7.52K 205M
                  tank 972G 9.93T 11 1.60K 10.8K 202M
                  tank 991G 9.91T 7 1.67K 7.62K 211M

                  so previously i could get a solid 200~mb/ps. between 190/200 at peak and had no issue maxing out transfers over gigabit lan

                  now i've moved to a dual xeon 5570 with 32gb ram on a supermicro X8DT3-F using the onboard lsi controller thinking I would get better performance with all the drives on a dedicated controller, and i get a fluctuating 0-115 which is horrible in comparison

                  I did go through that article and i've pulled some additional information. I saved a lot of output, so I'll link the two files, one for iostat, one for fsstat


                  looking a the write statistics in the fsstat.txt, I really feel I should be able to write at more than 60~mb/ps with this system though. I understand the cache -> write -> flush concept, but what I don't understand is why the io performance seems worse on what should be a far greater system.

                  Edited by: Stlouis1 on Mar 2, 2013 8:05 AM
                  • 6. Re: irregular i/o on zpool
                    you might try putting the -v where it belongs, as mentioned earlier.

                    try running a zpool iostat -v tank 1 and see what's happening second by second.