2 Replies Latest reply: Jun 5, 2014 5:01 AM by 1305030 RSS

    Solaris 10 nfs and forcedirectio

    1305030

      Hi,

       

      we're running a Solaris 10 U11 sparc Generic_150400-11 system and want to use a nfs share mounted using the following options:

       

      remote/read/write/setuid/nodevices/rstchown/bg/hard/nointr/rsize=32768/wsize=32768/proto=tcp/forcedirectio/vers=3/xattr

       

       

      The underlying network is an aggr of 2 10gbit interfaces. With buffering enabled (no forcedirectio) a mkfile with 8k blksize runs with around 400mbytes/sec.

      When forcedirectio is used, the number of ios and throughput decrease dramatically.

      Although it's clear to me, that we're facing syncronous writes now, I'm wondering about the limiting factor: when a 2nd job is started the number of io/sec and troughput doubles, latency does ot change.

      The same is true with every now job startet in parallel. io/sec and troughput/sec scale, despite the fact, they are still very low. Every process of these jobs is at 99%slp state.

       

       

      #pragma D option quiet

       

      sched:::off-cpu

      /curlwpsinfo->pr_state == SSLEEP && curpsinfo->pr_pid == $1/

      {

              self->ts = timestamp;

      }

       

      sched:::on-cpu

      /self->ts/

      {

              @[execname,stack()] = sum(timestamp - self->ts);

              self->ts = 0;

      }

       

      tick-15s

      {

              normalize(@, 1000000);

              printa("%20s %k %@u\\n", @);

              exit(0);

      }

       

       

       

      unix`_resume_from_idle+0x1e0

      genunix`cv_timedwait_hires+0xb0

      rpcmod`clnt_cots_kcallit+0x5f4

      nfs`rfscall+0x5f4

      nfs`rfs3call+0x60

      nfs`nfs3write+0x12c

      nfs`nfs3_write+0x7f0

      genunix`fop_write+0x20

      genunix`write+0x268

      unix`syscall_trap32+0xcc

       

       

       

      Any hints?

       

      thank you very much,

       

      cheers, Frank

        • 1. Re: Solaris 10 nfs and forcedirectio
          Cindys-Oracle

          Hi Frank,

           

          Just a small hint (you guys ask hard questions) but I noticed in an internal discussion of using Solaris 11 with NFSv3, using the forcedirectio option is only recommended when using large block sizes.

          If you are setting a 32K NFS size and using an 8 KB mkfile to test, what is the actual workload size? The caution with using forcedirectio with large write workloads like 1 MB, for example, is that the NFSv3 I/O size needs to match.

           

          Thanks, Cindy

          • 2. Re: Solaris 10 nfs and forcedirectio
            1305030

            Hi Cindy,

             

             

            the reason we're using this is to stay supported with the oracle rac cluster. It has to be set to sensure data consistency. wsize and rsize have to be set to 32k. We also tried different io sizes up to 64k, but the result does not change much. I'm really wondering what could limit the number of requests pers sec for a single job.

             

            cheers, Frank