Hardware

You can find it here: Vdbench Downloads

Initially I had removed 50405 from the download page, but it turns out that SNIA is still using that, so I added it back in.

Below are the release notes for 50407:

Have fun.

Henk

50407rc29

• Message “Warning: total amount of i/o per second per slave (123,456) greater than 100,000.” The 100,000 limit can now be controlled using the ‘ios_per_jvm=nnnnnn’ General parameter with the default set to 100,000. This basically gives you control over the i/o limit per slave that vdbench will report on. This means that YOU will be responsible if you are overloading your slaves.
• A ‘debug=128’ general parameter: use at your own discretion. Normally all file system workloads defined using the fwd= parameter are all controlled workloads. Vdbench does its best to let them all run at their requested skew, or by default, 100% divided by the number of workloads. This debug=128 parameter (or –d128 execution parameter) will let each workload run as fast as it can.
• A ‘debug=129’ general parameter: use at your own discretion. This option adds an extra column to the file system reporting, reporting the average percentage of time that all requested FWD threads are busy. This is useful if, as mentioned above, some of your workloads are throttled to accommodate other workloads that cannot keep up. Note that this is an approximation; some busy values due to idle time falling across reporting intervals may be reported in adjacent intervals.
• As usual, ‘debug=’ options are merely debugging aids or problem workarounds and will never be documented in the official Vdbench doc.
• The data_errors=remove option has been enhanced to data_errors=(remove,nn) (For raw workloads only). The failing SD will not be sent any new i/o operation for ‘nn’ seconds after which normal work will continue.
• The check to make sure you are running java 1.7 or higher has been removed.
• A new ‘fwd_thread_adjust=no’ parameter. Example: you are running with the forthreads=1 RD parameter, but you have TWO FWD workloads. Vdbench automatically adjusts this thread count from one to two to make sure each workload gets something to do. This can confuse people if they really, really want to run with only one thread. This parameter will cause the code to NOT make this adjustment and instead abort.
• No code changes, just a fix to the C compile for Linux. The old Linux system used for C compile died of old age. The new (virtual) system I received was oel7, which requires a newer version of GLIBC which older oel6 systems don’t have. This caused incompatibility issues. Symptom: /lib64/libc.so.6: version `GLIBC_2.14' not found.  Code in rc27 was recompiled with oel6.
• Though when running without Data Validation Vdbench should not care about data integrity, there is one place where it does matter: for File System functionality (FSD/FWD) Vdbench writes a small Control File into the target FSD. Twice over the years we have seen corruptions in this file which means that when Vdbench restarts using this corrupted Control File all bets are off. For diagnostics purposes this file is now re-read immediately after it is created and its internal checksum is compared. I also now write the FSD anchor name inside of the file plus the process-id of the Vdbench slave doing the writing. This will hopefully help us identifying future corruption issues, or, cases where possibly two or more users are writing to the same FSD anchor at the same time.
• Small fix: the Histogram for each Workload Definition (WD) was missing.
• Data Validation: There will be a new 'concept' in Vdbench: the 'Owner ID'.
  That will usually be the process ID of the Vdbench MASTER (no longer the slave) that is running.
  When using journaling however the Owner ID will be the process ID of the very first Vdbench MASTER creating the journal file. Even if you run journal recovery a dozen times, this Owner ID will never change.
  This Owner ID will be as usual in bytes 12 - 15 in each sector (used to be process ID), for DUPLICATE blocks however it will be in bytes 8-11, so if you see a corrupted duplicate block you can immediately see if it was not written by you, though of course, if YOUR block gets lost any other old garbage may still show....
• The new timeout= parameter will from now on not look at SDs or FSDs that completed their work early for instance due to a format.
• A new ‘-r rdname’ execution parameter allows for a restart of a parameter file containing multiple Run Definitions, e.g. ‘-r rd3’ If you have rd1 through rd5 in a parameter file, Vdbench will rerun rd3, rd4, and rd5.
• Added a ‘search’ option to Vdbench Compare allowing you to be more selective comparing specific Vdbench runs.
• Fixed a bug in the code where using dedup without Data Validation caused a java memory exception.
• Added ‘validate=reportdedupsets’, where Vdbench Data Validation, when it detects a corruption to a DUPLICATE block will report the lba and ‘last time used’ for all the duplicates of that corrupted block. Yep, when you have one million copies of the same block, which one is the bad one? J

50407rc27

50407rc26

50407rc25:

50407rc24:

50407rc23:

50407rc22:

• Startcmd=, endcmd=, config.sh and my_config.sh. The user may optionally create a new script directory under the PARENT directory of where Vdbench resides. E.g. /var/vdbench50407 contains Vdbench, user may create /var/vdbscript/. Any script or command will be first searched for in /vdbscript/ and then in /var/vdbench50407/solaris(or)linux. If found that script will be used. This new option allows you to now have a permanent place for scripts called by vdbench without worrying that a re-install of Vdbench will overlay whatever you have there.
• Vdbench sometimes needed 10 minutes or more of cpu cycles to figure out where what workload would run if you used loads of SDs and/or slaves. That bug has been fixed.
• The skew report now does a numeric sort of SD names instead of an alpha sort, allowing sd1 and sd2 to be in the proper order instead of having sd1,sd10 and then sd2.
• The big ‘skew’ fix in 50407rc19 did not work. I did a major rewrite in that area, so hopefully things are all better now.
• Introduced seekpct=seqnz, fileselect=seqnz and fileio=seqnz.   Sequential i/o always started at block zero, with as result that if the run did not last long enough the next Vdbench execution would just find everything in cache. Adding ‘nz’ causes all i/o to start at a random point in the lun or file so there now is a better chance that we will not find all data in cache. Again, if your lun/file size is relatively small you are STILL going to find it all in cache. ‘fileselect=seq’ basically does the same: it does not start at file0, but somewhere with a random file number.
• The ‘$host’ logic now also works inside of a Run definition (RD).f

50407rc21:

50407rc19:

• Symptom: Intermittent incomplete workload when doing a controlled workload AND multiple 100% sequential workloads in one Run Definition (RD) AND multiple slaves AND not enough threads for each slave to have at least one thread. Yep, a pretty small window.

  style="margin-left:.25in"

• (It took only 15 years for this 'problem' to show up J) 

50407rc18:

• A fix to the way /etc/mnttab was parsed: for the first time a 64bit value showed up in the 'dev=' mount output.
• A nasty bug in Data Validation. In 50407 a new key value would no longer always start with key=01, it would start with a random value. On rare occasions this random value would result in key=0xffffff, and that of course caused havoc since it may not be larger than 7 bits.
• Vdbench compare: new syntax:
  • ./vdbench compare old_output_dir new_output_dir [-o out.csv ]    Using '-o' will bypass the GUI and directly create a CSV.
  • ./vdbench compare old olddir-1 olddir-2 olddir-n new newdir-1 newdir-2 newdir-n  [-o out.csv]  (wildcarding is OK)

  style="margin-left:1.0in"

• Also added '$error', which will contain either 'r' or 'w' for a normal read or write i/o error, or a to-be-determined hex value in case of a Data Validation error.

• Just before the above 'data_errors=scriptxxx.sh' is called, no new I/O is started, thus giving the script some extra control over what's happening while the script is running.

50407rc13:

• When using the report=no_sd_detail parameter the histogram files were still generated, again with 2000 luns this option better work.

• Now of course we also have the report=no_fsd_detail parameter, though no_sd_detail will imply this also.

• Workload reports no longer will be generated when a Workload Definition (WD) is NOT used.

• timeout=(60,600) Will print message every 60 seconds and then will abort at 600 seconds if problem has not been resolved.

• timeout=(60,abort)  will abort Vdbench if it does not see any i/o completions for 60 seconds

• timeout=(60,/xxx/script)    will call '/xxx/script', and if the script returns 'abort' Vdbench will abort. Any other value returned by the script will cause vdbench to continue. The script will be called again 60 seconds later if the timeout continues.

• A fix to ./vdbench rsh which aborted when Vdbench was killed with ctrl-c.

• A fix to ./vdbench rsh where the java port number of its rsh daemon was accidentally incremented.

• A fix to a 45-second reporting anomaly on Linux, caused by not setting setTcpNodelay(true).

• A fix to forseekpct=0 where Vdbench did not force the sequential work to be run on a single JVM.

• A fix where an I/O error to an FSD allowed the block to be used again.

• A fix for sparc linux where the openflags= values used by did not match the OS.

• The skew.html file now will include individual SD or FSD status, even if report=no_sd_detail  was used.

• ./vdbench print by default ran using directio, that is now optional by adding the '-f directio' flag. The problem with using directio by default was that you would NOT be able to print the in-cache copy of the block.

• validate=xfersize         This remembers and stores the xfersize used for each key block of the LAST write done to this key block. It requires an extra four bytes per key block of java heap space.  This option has been very useful chasing a data corruption. This xfersize then will be reported when a corruption is found. Not available with journaling.

• The 'max response time' column has been replaced by 'max read response' and 'max write response'.

• At then end of a file system functionality test (FSD/FWD) two new lines are reported, with 'max' and 'standard deviation' for appropriate columns.

• The flatfile parser now allows tab separated files, not only command separated files. It also allows for '-c all' where ALL available columns will be reported.

• fileselect= [empty full notfull]: causes file selection to only look for empty or full, or partially full files.

• The FSD= 'count=' parameter now also allows use of a printf mask, e.g. fsd=fsd%02d,anchor=/tmp/dir%03d,…,count=(1,10)