Just a small hint (you guys ask hard questions) but I noticed in an internal discussion of using Solaris 11 with NFSv3, using the forcedirectio option is only recommended when using large block sizes.
If you are setting a 32K NFS size and using an 8 KB mkfile to test, what is the actual workload size? The caution with using forcedirectio with large write workloads like 1 MB, for example, is that the NFSv3 I/O size needs to match.
the reason we're using this is to stay supported with the oracle rac cluster. It has to be set to sensure data consistency. wsize and rsize have to be set to 32k. We also tried different io sizes up to 64k, but the result does not change much. I'm really wondering what could limit the number of requests pers sec for a single job.