Solaris 11.3.34.4.0 ram speed is too slow on DDR4 rdim

Ergi

Hey all,

We spect our a Supermicro box  with the latest  greatest  hardware. Skylake CPU gold, (678G DDR4 rdim ecc registered) ram and some nvme drives. Solaris 11.3 is certified to run on the box. We install the O/S and start doing some benchmark test. Disk IO is awesome (can get 5GB/sec on write to zpool), no hardware errors in the /var/adm/messages or fmadm faulty. One things that we stumble upon is the  read speed, to be more specific  MEMORY READ SPEED. Usually when you run a DD command will run it from the memory,  if the file size is smaller than the amount of ram will load the whole thing in the arc and you get the awesome speed of RAM, thank you ZFS for that. Supermicro  ran the speed test on Redhat 7.3 using "STREAM Memory benchmark" and they get about 200GB/sec sustainable memory bandwidth. Yes that is 200GB/sec is not a typo. We on the other hand on dd can not get more than 2.7GB/sec not matter what variations of file size file count, zpool vdevs, etc....

On other similar boxes on older generation cpu and ram we easily get 7GB/sec. I have an open ticket with Solaris kernel dev team and they are looking into data as well. I been looking around to find a MEMORY test utility on solaris like memtest or STREAM but i can not. I have already use the vmstat  pstat... iostat utilties.

The hardware specs to be more specific are

CPU Intel(R) Xeon(R) Gold 5122 CPU @ 3.60GHz

RAM SK Hynix  2666 MHz  HMA84GR7AFR4N-VK.

Mobo X11DSC+

Any help is appreciated 

Steffen Moser

Hi Ergi,

unfortunately, I don't have a solution or idea about what's causing it. You could try to compile the STREAM memory benchmark under Solaris 11.3 using Developer Studio. According to https://blogs.oracle.com/bestperf/memory-and-bisection-bandwidth%3a-sparc-m8-performance, it seems to be possible for the SPARC architecture, so I am quite sure that it is also possible to build it for x86-64.

Kind regards,

Steffen