12 Replies Latest reply on Aug 1, 2018 11:17 AM by Adam Boliński

    Solaris 11.4 Beta: ZFS sudden performance drop during iozone (sequential)

    Tuxwielder

      Dear List,

       

      I'm running some performance test on Solaris 11.4 Beta/ZFS using a 36-disk server (2x5118 CPU, 512GB, 2x NVME).

       

      I'm running iozone sequential write/read tests on RAIDZ2 (5x7) and get adequate performance in the first few runs, e.g.:

       

              File size set to 545259520 kB

              Command line used: /opt/feiog/bin/iozone -a -r 256k -r 1M -r 4M -s 520g -i 0 -i 1

              Output is in kBytes/sec

              Time Resolution = 0.000001 seconds.

              Processor cache size set to 1024 kBytes.

              Processor cache line size set to 32 bytes.

              File stride size set to 17 * record size.

                                                                    random    random     bkwd    record    stride

                    kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  freread

             545259520     256  3233358  3311229  1454387  1485596

             545259520    1024  3270404  3304932  1449425  1461943

             545259520    4096  2914228  2862887  1462135  1414928

       

      But then there is a sudden drop (starts at recordsize 1024):

       

                                                                    random    random     bkwd    record    stride

                    kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  freread

             545259520     256  2990823  3097740  1443773  1493013

             545259520    1024  2457356  1642110   322637  1408447

             545259520    4096  2636473  1553800   327611  1458264

       

      Which remains, until reboot (after which the same happens).

       

      Zpool iostat shows regular output when everything seems in order, and frequent stalls (seconds of 0-byte written/read) when performance drops. This is reflected by 'iostat -xn', which shows the disks idling for a few seconds at the same time.

       

      Has anyone else encountered this?

       

      With kind regards,

       

      Jeroen Roodhart

        • 1. Re: Solaris 11.4 Beta: ZFS sudden performance drop during iozone (sequential)
          Adam Boliński

          Hello what is machine where you run Solaris 11.4 ? Is it Oracle machine or Commodity hardware ?

          • 2. Re: Solaris 11.4 Beta: ZFS sudden performance drop during iozone (sequential)
            Tuxwielder

            Hi Adam,

             

            You may want to call it commodity From the Solaris dashboard (nice!):

             

             

             

             

             

             

            Configuration


             

             

             

             

             

             

             

             

            This system has 36 10TB disks and is equipped with two Intel P4800X's (slog), although initial benchmarks seem to indicate we only need one.

            Zpool layout: 5x7 RAIDZ2, 2 log devices and 1 spare. We use an AVAGO 3108 MegaRAID HBA to connect it up to the SES-chassis (LSI-SAS3x28, SMC-SC846P).

            The OS is installed on 2 SSD's (connected to  MoBo SATA).

             

             

             

            • 3. Re: Solaris 11.4 Beta: ZFS sudden performance drop during iozone (sequential)
              Adam Boliński

              Thanks for quick reply , I just need to ask you about it because I have a little problem with nvmeadm list, I get the output Platform Not Supported so I just would like to know if this problem is not related to server manufacture. I just have also Supermicro server. So maybe you know why I get this output ?

              • 4. Re: Solaris 11.4 Beta: ZFS sudden performance drop during iozone (sequential)
                Tuxwielder

                Adam Boliński wrote:

                 

                I have a little problem with nvmeadm list, I get the output Platform Not Supported so I just would like to know if this problem is not related to server manufacture. I just have also Supermicro server. So maybe you know why I get this output ?

                 

                I get the same. The devices seem to be working fine though.

                 

                Looks like (from looking at a truss-trace of the command) nvmeadm uses the "Oracle hardware management framework" and the manpage for the cli states:

                 

                "hwmgmtcli is a command line utility to display high-level hardware configuration/topology and status of supported Oracle servers."

                • 5. Re: Solaris 11.4 Beta: ZFS sudden performance drop during iozone (sequential)
                  Adam Boliński

                  Ok so how you list and monitor NVMe device without this command ?

                  • 6. Re: Solaris 11.4 Beta: ZFS sudden performance drop during iozone (sequential)
                    Tuxwielder

                    Using diskinfo you can list the devices:

                     

                    [root@ivi-s0 ~]$ diskinfo

                    D:devchassis-path                                      c:occupant-compdev

                    -----------------------------------------------------  ---------------------

                    /dev/chassis/SYS/CPU1-SLOT-6-PCI-E-3.0-X8/NVME/disk    c3t1d0

                    /dev/chassis/SYS/CPU2-SLOT-5-PCI-E-3.0-X16/NVME/disk   c4t1d0

                    ...

                     

                    Monitoring I expect monitoring to work through ZFS-tooling (i.e. zpool consistency reports).

                    • 7. Re: Solaris 11.4 Beta: ZFS sudden performance drop during iozone (sequential)
                      3123398

                      Hello I just resolve this issue and yes diskinfo is working but I saw that for example did not recognise this small 32GB Optane M.2 sticks , but I have other issue and maybe you can help me with this , do you try to install gcc or storage server  ?

                      When I try to do it I get following errors (related I think with packages dependency).

                       

                       

                      pkg install group/feature/storage-server

                      Creating Plan (Solver setup): -

                      pkg install: No matching version of group/feature/storage-server can be installed:

                        Reject:  pkg://solaris/group/feature/storage-server@0.5.11-0.175.0.0.0.2.1

                                   to

                                 pkg://solaris/group/feature/storage-server@0.5.11-0.175.3.0.0.30.0

                        Reason:  This version is excluded by installed incorporation consolidation/osnet/osnet-incorporation@11.4-11.4.0.0.0.12.2

                       

                      BR

                      Adam

                      • 8. Re: Solaris 11.4 Beta: ZFS sudden performance drop during iozone (sequential)
                        Alanc-Oracle

                        Make sure you've correctly set up the beta package repo, including configuring the certificates you receive when signing up online if you're using Oracle's server and not a local repo:

                        https://blogs.oracle.com/solaris/installing-packages-%e2%80%94-oracle-solaris-114-beta

                        • 9. Re: Solaris 11.4 Beta: ZFS sudden performance drop during iozone (sequential)
                          Adam Boliński

                          Many thanks for help this is working perfect.

                          • 10. Re: Solaris 11.4 Beta: ZFS sudden performance drop during iozone (sequential)
                            Tuxwielder

                            Just making sure that Adam's affirmation is not considered an answer to my original question. We still see this performance drop 

                            • 11. Re: Solaris 11.4 Beta: ZFS sudden performance drop during iozone (sequential)
                              3123398

                              I just wondering if it is normal I have disk :

                               

                              D:devchassis-path              c:occupant-compdev

                              -----------------------------  ------------------

                              /dev/chassis/SYS/MB/NVME/disk  c14t1d0

                               

                              But when I list /dev/rdsk/*  I have a lot symlinks to this devices but there is no c14t1d0 ? I would like to share via iSER this disk "completely" and there is no partition on it ?

                              Can someone smarter then me tell me what is going on ?

                               

                              lrwxrwxrwx   1 root     root          61 Apr 19 15:27 /dev/rdsk/c14t1d0p4 -> ../../devices/pci@0,0/pci8086,2f04@2/pci1c58,3@0/disk@1:u,raw

                              lrwxrwxrwx   1 root     root          61 Apr 19 15:27 /dev/rdsk/c14t1d0p3 -> ../../devices/pci@0,0/pci8086,2f04@2/pci1c58,3@0/disk@1:t,raw

                              lrwxrwxrwx   1 root     root          61 Apr 19 15:27 /dev/rdsk/c14t1d0p2 -> ../../devices/pci@0,0/pci8086,2f04@2/pci1c58,3@0/disk@1:s,raw

                              lrwxrwxrwx   1 root     root          61 Apr 19 15:27 /dev/rdsk/c14t1d0p1 -> ../../devices/pci@0,0/pci8086,2f04@2/pci1c58,3@0/disk@1:r,raw

                              lrwxrwxrwx   1 root     root          61 Apr 19 15:27 /dev/rdsk/c14t1d0p0 -> ../../devices/pci@0,0/pci8086,2f04@2/pci1c58,3@0/disk@1:q,raw

                              lrwxrwxrwx   1 root     root          61 Apr 19 15:27 /dev/rdsk/c14t1d0s15 -> ../../devices/pci@0,0/pci8086,2f04@2/pci1c58,3@0/disk@1:p,raw

                              lrwxrwxrwx   1 root     root          61 Apr 19 15:27 /dev/rdsk/c14t1d0s14 -> ../../devices/pci@0,0/pci8086,2f04@2/pci1c58,3@0/disk@1:o,raw

                              lrwxrwxrwx   1 root     root          61 Apr 19 15:27 /dev/rdsk/c14t1d0s13 -> ../../devices/pci@0,0/pci8086,2f04@2/pci1c58,3@0/disk@1:n,raw

                              lrwxrwxrwx   1 root     root          61 Apr 19 15:27 /dev/rdsk/c14t1d0s12 -> ../../devices/pci@0,0/pci8086,2f04@2/pci1c58,3@0/disk@1:m,raw

                              lrwxrwxrwx   1 root     root          61 Apr 19 15:27 /dev/rdsk/c14t1d0s11 -> ../../devices/pci@0,0/pci8086,2f04@2/pci1c58,3@0/disk@1:l,raw

                              lrwxrwxrwx   1 root     root          61 Apr 19 15:27 /dev/rdsk/c14t1d0s10 -> ../../devices/pci@0,0/pci8086,2f04@2/pci1c58,3@0/disk@1:k,raw

                              lrwxrwxrwx   1 root     root          61 Apr 19 15:27 /dev/rdsk/c14t1d0s9 -> ../../devices/pci@0,0/pci8086,2f04@2/pci1c58,3@0/disk@1:j,raw

                              lrwxrwxrwx   1 root     root          61 Apr 19 15:27 /dev/rdsk/c14t1d0s8 -> ../../devices/pci@0,0/pci8086,2f04@2/pci1c58,3@0/disk@1:i,raw

                              lrwxrwxrwx   1 root     root          61 Apr 19 15:27 /dev/rdsk/c14t1d0s7 -> ../../devices/pci@0,0/pci8086,2f04@2/pci1c58,3@0/disk@1:h,raw

                              lrwxrwxrwx   1 root     root          61 Apr 19 15:27 /dev/rdsk/c14t1d0s6 -> ../../devices/pci@0,0/pci8086,2f04@2/pci1c58,3@0/disk@1:g,raw

                              lrwxrwxrwx   1 root     root          61 Apr 19 15:27 /dev/rdsk/c14t1d0s5 -> ../../devices/pci@0,0/pci8086,2f04@2/pci1c58,3@0/disk@1:f,raw

                              lrwxrwxrwx   1 root     root          61 Apr 19 15:27 /dev/rdsk/c14t1d0s4 -> ../../devices/pci@0,0/pci8086,2f04@2/pci1c58,3@0/disk@1:e,raw

                              lrwxrwxrwx   1 root     root          61 Apr 19 15:27 /dev/rdsk/c14t1d0s3 -> ../../devices/pci@0,0/pci8086,2f04@2/pci1c58,3@0/disk@1:d,raw

                              lrwxrwxrwx   1 root     root          61 Apr 19 15:27 /dev/rdsk/c14t1d0s2 -> ../../devices/pci@0,0/pci8086,2f04@2/pci1c58,3@0/disk@1:c,raw

                              lrwxrwxrwx   1 root     root          61 Apr 19 15:27 /dev/rdsk/c14t1d0s1 -> ../../devices/pci@0,0/pci8086,2f04@2/pci1c58,3@0/disk@1:b,raw

                              lrwxrwxrwx   1 root     root          61 Apr 19 15:27 /dev/rdsk/c14t1d0s0 -> ../../devices/pci@0,0/pci8086,2f04@2/pci1c58,3@0/disk@1:a,raw

                              • 12. Re: Solaris 11.4 Beta: ZFS sudden performance drop during iozone (sequential)
                                Adam Boliński

                                Somebody knows how to resolve this issue ?

                                 

                                # diskinfo

                                diskinfo: di_cro_init() failed: does '/var/fm/fmd/topo/latest/cro_db' exist? is 'svc:/system/fmd' running?

                                 

                                # svcs /system/fmd

                                STATE          STIME    FMRI

                                online         13:15:25 svc:/system/fmd:default

                                 

                                BR

                                A