5 Replies Latest reply: Sep 25, 2013 4:32 PM by 842251 RSS

    Solaris 11.1 Comstar FC target

    0bb5646e-768f-405e-8a84-472884245952

      Hello,

       

      I have a problem with the comstar as a FC target.

       

      New install of Solaris 11.1

      HBA is an Emulex LPe11002

      Brocade 5100B switches

      2x 10x 3TB NL-SAS disks in raidz2 in pool

       

      It all works, but the speed is unusable slow to the LUN.

      iSCSI work and I am able to hit the max of the network so there is no problems with access to access the disks.

       

      HBA info

      HBA Port WWN: 10000000c98e9712

              Port Mode: Target

              Port ID: 12000

              OS Device Name: Not Applicable

              Manufacturer: Emulex

              Model: LPe11002-E

              Firmware Version: 2.80a4 (Z3F2.80A4)

              FCode/BIOS Version: none

              Serial Number: VM92923844

              Driver Name: emlxs

              Driver Version: 2.70i (2012.02.10.12.00)

              Type: F-port

              State: online

              Supported Speeds: 1Gb 2Gb 4Gb

              Current Speed: 4Gb

              Node WWN: 20000000c98e9712

      HBA Port WWN: 10000000c98e9713

              Port Mode: Target

              Port ID: 22000

              OS Device Name: Not Applicable

              Manufacturer: Emulex

              Model: LPe11002-E

              Firmware Version: 2.80a4 (Z3F2.80A4)

              FCode/BIOS Version: none

              Serial Number: VM92923844

              Driver Name: emlxs

              Driver Version: 2.70i (2012.02.10.12.00)

              Type: F-port

              State: online

              Supported Speeds: 1Gb 2Gb 4Gb

              Current Speed: 4Gb

              Node WWN: 20000000c98e9713

       

      iostat 2 sec apart:

       

      pool        alloc   free   read  write   read  write

      ----------  -----  -----  -----  -----  -----  -----

      dipool    44.1M  54.5T      0     19  1.01K   134K

      dipool    44.1M  54.5T      0      2      0   196K

      dipool    45.0M  54.5T      0     50      0   210K

      dipool    45.0M  54.5T      0      0      0  64.0K

      dipool    45.8M  54.5T      0     50      0   274K

      dipool    45.8M  54.5T      0      0      0  64.0K

      dipool    45.8M  54.5T      0      0      0      0

      dipool    45.0M  54.5T      0     35      0   125K

      dipool    45.0M  54.5T      0      0      0  64.0K

      dipool    44.5M  54.5T      0     34      0  61.0K

      dipool    44.5M  54.5T      0      0      0  64.0K

      dipool    44.5M  54.5T      0      0      0  64.0K

      dipool    44.6M  54.5T      0     34      0  61.0K

      dipool    44.6M  54.5T      0      0      0  64.0K

       

       

      I also tried openindiana, the speed was good, but link will die and then capturing stmf debug shows the following when using the Emulex.

      FROM STMF:210406652: abort_task_offline called for LPORT: lport abort timed out, 1000's of them

      Jun  7 14:02:18 emlxs: [ID 349649 kern.info] [ 5.0608]emlxs1: NOTICE: 730: Link reset. (Disabling link...)

      Jun  7 14:02:18 emlxs: [ID 349649 kern.info] [ 5.0333]emlxs1: NOTICE: 710: Link down.

      Jun  7 14:04:41 emlxs: [ID 349649 kern.info] [ 5.055D]emlxs1: NOTICE: 720: Link up. (4Gb, fabric, target)

      Jun  7 14:04:41 fct: [ID 132490 kern.notice] NOTICE: emlxs1 LINK UP, portid 22000, topology Fabric Pt-to-Pt,speed 4G

      Jun  7 14:10:19 emlxs: [ID 349649 kern.info] [ 5.0608]emlxs1: NOTICE: 730: Link reset. (Disabling link...)

      Jun  7 14:10:19 emlxs: [ID 349649 kern.info] [ 5.0333]emlxs1: NOTICE: 710: Link down.

      Jun  7 14:12:40 emlxs: [ID 349649 kern.info] [ 5.055D]emlxs1: NOTICE: 720: Link up. (4Gb, fabric, target)

      Jun  7 14:12:40 fct: [ID 132490 kern.notice] NOTICE: emlxs1 LINK UP, portid 22000, topology Fabric Pt-to-Pt,speed 4G

       

      I also tried a Qlogic QLE2460-SUN and that has the same problem in both OI and Solaris, ultra slow

      HBA Port WWN: 2100001b3280b

              Port Mode: Target

       

              Port ID: 12000

              OS Device Name: Not Applicable

              Manufacturer: QLogic Corp.

              Model: QLE2460

              Firmware Version: 5.2.1

              FCode/BIOS Version: N/A

              Serial Number: not available

              Driver Name: COMSTAR QLT

              Driver Version: 20100505-1.05

              Type: F-port

       

              State: online

              Supported Speeds: 1Gb 2Gb 4Gb

              Current Speed: 4Gb

              Node WWN: 2000001b3280b

       

       

      It seems no one is using Solaris as a FC target anymore and since we do not have 10Gbe in our lab and some systems cannot communicate via IP to others, FC is the only form of backup.

       

      Can someone please let me know if they are using Solaris as an FC target and perhaps some pointers. On the example above I am trying to clone using VMware from a LUN on an EMC array to the Solaris node. As I mentions the speed is good in OI, but then it seems there is a driver issue.

       

      Cloning in OI from the EMC LUN to the back server:


      1 sec apart.

           alloc free read write read write

      >>>>>>> ----- ----- ----- ----- ----- -----

      >>>>>>> 309G 54.2T 81 48 452K 1.34M

      >>>>>>> 309G 54.2T 0 8.17K 0 258M

      >>>>>>> 310G 54.2T 0 16.3K 0 510M

      >>>>>>> 310G 54.2T 0 0 0 0

      >>>>>>> 310G 54.2T 0 0 0 0

      >>>>>>> 310G 54.2T 0 0 0 0

      >>>>>>> 310G 54.2T 0 10.1K 0 320M

      >>>>>>> 311G 54.2T 0 26.1K 0 820M

      >>>>>>> 311G 54.2T 0 0 0 0

      >>>>>>> 311G 54.2T 0 0 0 0

      >>>>>>> 311G 54.2T 0 0 0 0

      >>>>>>> 311G 54.2T 0 10.6K 0 333M

      >>>>>>> 313G 54.2T 0 27.4K 0 860M

      >>>>>>> 313G 54.2T 0 0 0 0

      >>>>>>> 313G 54.2T 0 0 0 0

      >>>>>>> 313G 54.2T 0 0 0 0

      >>>>>>> 313G 54.2T 0 9.69K 0 305M

      >>>>>>> 314G 54.2T 0 10.8K 0 337M

       

       

      We have tons of other devices connected to the Brocade 5100B switches. I tried connecting the system to two different switches individually with the same result. We are basically 100% Emulex shop and I only have the one qlt card


      I have now tried a brand new Emulex LPe11002 card in a different PCI-E slot, new cable and different FC switch.

      I have similar problems with Openindiana and no problems with any of the emc vnx/cx/data domain connected to the same switches or any of the hosts connected to them as the targets using the same LPe10000/LPe11002/LPe12002 cards.


      Any help/pointers would be greatly appreciated.


      Thanks,



        • 1. Re: Solaris 11.1 Comstar FC target
          0bb5646e-768f-405e-8a84-472884245952

          I tried the qlt in the solaris 11.1 node also and it works for a few min. I get ok speed and then after some time it slows down to again only a few k/s.

          Useless.... We use to use this heavily in build 99 and 128 of opensiolaris and worked fine.  I give up Slowlaris

          • 2. Re: Solaris 11.1 Comstar FC target
            Cindys-Oracle

            Sounds frustrating. We discussed some target/initiator issues in this thread:

             

            /thread/2460513?tstart=0

             

            But your issues are somewhat different.

             

            The only other issue I think of that might be impacting overall system performance is if you are using SSDs and you are impacted by the unmap performance issue described in bug 15804599. This feature is disabled in SRU 3.4.

             

            Thanks, Cindy

            • 3. Re: Solaris 11.1 Comstar FC target
              0bb5646e-768f-405e-8a84-472884245952

              Hi Cindy,

               

              Thank you for the response.

              We have no SSD drives in the system yet. Once thing I would like to note is that as a last resort I tried an OS called OmniOS since I got a response of someone using it as an FC target. Lucky I created the pool with ver 28 and zfs ver 5. Imported the pool and the problem is not present in this OS when I use the Qlogic card. It has the same problem with the Emulex card.

               

              I am using the same hardware and same HBA's as with Solaris 11.1. I have done extensive testing to the fc target and it is working very well.

              I am convinced there is a problem here with the Comstar and it being a FC target in newer releases. I cant seem to find anyone really using it as a FC target, or they just do not have problems. We have tons of devices connected to the same fabric there are no problems. I tried different ports and a different switch all resulting in the same problem.

               

              Would you be able to tell me if we buy the license ( i am on a 30 day trail version of Solaris since I ran into the problem on OI)  if we would able to to open a support case? It is not running on Sun hardware, the Qlogic card is a Sun part.

               

              Thank you,

              • 4. Re: Solaris 11.1 Comstar FC target
                Cindys-Oracle

                Hi--

                 

                I'm not really good on the support side but I believe you can buy license/support for non-Sun/Oracle hardware.

                 

                My other minor suggestion is to make sure you check the Solaris 11 HCL to review whether your gear has been tested/verified.

                 

                Thanks, Cindy

                • 5. Re: Solaris 11.1 Comstar FC target
                  842251

                  Accidentally found this. Some comments.  Unfortuinately, we run Solaris FC as initiators/clients to san target LUNs on NetApp and HP.

                  1. There were patches in sol 10 for fc cards for bugs, driver and firmware upgrades, fcode/bios which I believe is for san boot only.

                  I read somewhere these patches for sun/oracle branded FC cards would not be released under Sol 11, have not looked.

                  Sample patches for Qlogic for Sol 10 are 114874-07 for fcode/bios and 149175-03 for everything else.  Unfortunately we're most Qlogic and only have a couple Emulex in Linux systems.  So is 11.1 really supporting these FC cards now or is the user responsible for downloading drivers and firmware from vendors and installing?.

                  2.  Have heard when zfs gets to around 80% capacity i/o performance can suffer.  This may have been fixed, been avoiding with quotas.

                  Then of course if looking for continuous speed don't turn on compression.

                  3.  Do you have/need the sol 11.1 multi-path package when Solaris has the targets/LUNs?  Are you configured for MPxIO?

                  pkg info system/storage/multipath-utilities

                  4. Do you need any kernel changes to /etc/system for performance? Some below are x86.

                  set sd:sd_max_throttle=64   sparc

                  set ssd:ssd_max_throttle=64   x86

                  set maxphys=1048576

                  set ssd:ssd_io_time=60        x86

                  5.   Do you need to worry about 4K alignment from client side?

                  These are all things I worry about but Solaris is an initiator in our environment along with every other platform.

                  This is old and hopefully resolved by this time!

                   

                   

                   

                   

                   

                   

                   

                  where (s)sd_max_throttle= 256
                  / # of LUNs