8 Replies Latest reply: Jul 8, 2010 12:33 AM by 807567 RSS

    Poor iSCSI Performance over 10GbE connection.

    807567
      Hi Solaris Folks,

      I'v a setup of two SunFire X4540 connected over a 10GbE link (a Cisco Catalyst 4507 in the middle) using Sun's Dual 10 GbE NIC (X1027A-Z) on both machines.
      The plan is to provide the 2nd machine a ZFS volume via iSCSI from the 1st machine. The 1st machine has a ZPool wiht 22x 2-way mirror vdevs and a locale
      FileBench test shows good results.

      I've now configured the 1st machine to share a ZFS volume via iSCSI and on the 2nd machine I used that target as a raw-device for a new ZPool. If I now run
      FileBench againt the "iSCSI-ZPool" I get results which are far away from the local benchmarks and far away from really using the 10GbE link (most times the
      bandwidth is around 1,5 GBit/s).

      Did a couple of performance tuning which I found in the Sun Blogs and Wikis but without any luck to increase the performance significantly.
      Anybody out there who tried the same (iSCSI over 10GbE)? Any experience with iSCSI performance? Suggestions?

      Here is an overview about the changes I made:

      iSCSI Initiator only:
      tcp-nodelay=1; in /kernel/drv/iscsi.conf

      iSCSI Targer only:
      Configured a TPGT so that only the 10GbE connection can be used.

      On both machines:
      tcp_recv_hiwat 400000, tcp_xmit_hiwat 400000, tcp_max_buf 2097152, tcp_cwnd_max 2097152 on /dev/tcp via ndd

      soft-lso-enable = 1; and accept_jumbo=1; within nxge.conf

      MTU on both NICs is set to 9194. Jumbo Frames are also configured on the Cisco switch ports.

      Regards,
      Lemmy
        • 1. Re: Poor iSCSI Performance over 10GbE connection.
          Robert Cohen
          What version of solaris are you using? If its not opensolaris, that would be worth a try so you can use the comstar iscsi framework..
          I believe thats supposed to have much better performance..
          • 2. Re: Poor iSCSI Performance over 10GbE connection.
            807567
            Both machines running Solaris 10 5/09 (Update 7). I found a couple of sites which recommend using OpenSolaris with the new COMSTAR framework.
            It's just I would prefer to get some comments from other users who tried Solaris 10 with iSCSI before installing OpenSolaris, so I can make sure it is not worth to do further testing&tuning.

            Bye
            Lemmy
            • 3. Re: Poor iSCSI Performance over 10GbE connection.
              807567
              I increased the number of sessions for the target on the initiator to 4-sessions which almost doubled the performance.
              But still far away from 10G. Does anybody know about a limitation regarding the number of sessions for a target?
              Four sessions was the highest value iscsiadm accepted.
              • 4. Re: Poor iSCSI Performance over 10GbE connection.
                Robert Cohen
                Well, your luck your at least at Solaris 5/09. Prior to that release, Solaris 10 iSCSI performance had real problems..

                Still its probably worth patching up to date or running recommended patch set through.
                Just in case theres something that helps..
                • 5. Re: Poor iSCSI Performance over 10GbE connection.
                  807567
                  I'm working with a USS7410C with 10Gb Sun Ethernet Cards and I'm not
                  able to take more than 100MB/sec (write).

                  I think with 10Gb card unmask iSCSI throughput problems, otherwise covered with
                  1Gb onboard interfaces (take into account 100MB/sec are on the edge of 1Gb interface).

                  We have an scalation open with Sun support but we are still working ...

                  I'm sorry

                  Cesar
                  • 6. Re: Poor iSCSI Performance over 10GbE connection.
                    807567
                    were you able to get higher than 100MB/s ?
                    • 7. Re: Poor iSCSI Performance over 10GbE connection.
                      807567
                      The answer could be .. don't use ZFS volumes

                      I have an SSD and a simple Seagate disk. Both on SATA. Same capacity of 74.53GB

                      If I use those disks to create a pool and make volumes on the pool and then
                      use the shareiscsi property to create a target (and there is plenty of documentation
                      describing that as the easy way to do it ) the performance
                      is dreadful.

                      If on the other hand I use iscsitadm to make a target out of the devices of two disks using commands like this
                      for example

                      iscsitadm create target --type raw -b /dev/dsk/c4t0d0
                      iscsitadm create target --type raw -b /dev/dsk/c4t1d0

                      and then on the initiator create the pool then the performance is entirely different!

                      A bit more detail:


                      For example I built a target out of a zfs volume then on the initiator I wrote to the pool using zfs send/recv with about 3.2GB. It took 26 minutes 24 sec - pathetic.
                      I almost gave up waiting!
                      So i exported the these useless zfs pools, removed the static-configs etc and cleared the zfs property etc and started again.

                      If the target is instead built "manually" using iscsitadm e.g. "iscsitadm create target --type raw -b /dev/dsk/c4t0d0" (dsk or rdsk it doesn't seem to make much difference)
                      the same write test takes just 3 min 12sec.
                      So in this example Solaris 10 10/09 (x86) the performance degrades by a factor of 8 if you use zfs volumes. Nice idea ... easy to use but ... terrible overhead.
                      At least for SATA disks.
                      Incidentally for a SATA SSD disk the factor is not so bad (5 min vs 3 min) . However an ordinary disk like ST3808110AS .. 8 times slower with zfs volumes.
                      No idea why .. but easily reproducible.
                      So .. what do you see if you don't use zfs volumes but instead use the disk device name itself (whole disk remember - not a slice)

                      For target and initiator I was using Solaris 10 10/09 (x86) on a v20z with 3rd party eSATA card. SunOS kernel patched recently to 5.10 Generic_142901-10 on the target.
                      • 8. Re: Poor iSCSI Performance over 10GbE connection.
                        807567
                        [The post of Gfmitchell is detailed , and i know more about iSCSI from these.!|http://www.highwaytowatches.com]