6 Replies Latest reply: Mar 10, 2011 6:24 PM by EJP RSS

    FileChannel.write(ByteBuffer[]) only writes out the first 16 ByteBuffer

    846113
      Hi,
      This does NOT happen on Windows. I've only had this problem on Solaris (10 u8).
      I've tried this in jdk1.4, 1.5 and 1.6. All showed the same behavior on Solaris.
      I tried to write bytes to a file using FileChannel.write(ByteBuffer[]) method.
      I noticed that, if the ByteBuffer array contains more than 16 ByteBuffer, the FileChannel.write(ByteBuffer[] src) method only writes out contents of the first 16 ByteBuffer (src[0] thru src[15]).
      It worked fine if I combined all bytes into a single (big) ByteBuffer. (FileChannel.write(ByteBuffer) works fine)
      Has anyone experienced similar problem? Is it a bug in the JRE?

      Thanks.
        • 1. Re: FileChannel.write(ByteBuffer[]) only writes out the first 16 ByteBuffer
          EJP
          That's probably the value of IOV_MAX in <limits.h> in Solaris, and if so you are stuck with it. See man writev(1), which is the system call underlying this Java method. As it's specified to fail if the count is too large I speculate that Java caps it for you. Whether it should really do that, or loop, or fail itself, is another question.

          Never seen anybody use 16 buffers to write to a file before!
          • 2. Re: FileChannel.write(ByteBuffer[]) only writes out the first 16 ByteBuffer
            846113
            Thank you very much for the response. That might explain the behavior. I was just trying to optimize some existing code and found that write(ByteBuffer[]) is available in the API.
            The code currently calls FileChannel.write(ByteBuffer) for each ByteBuffer. I was hoping I could batch it up by calling the write(ByteBuffer[]) method (to reduce the number of writes).
            • 3. Re: FileChannel.write(ByteBuffer[]) only writes out the first 16 ByteBuffer
              EJP
              It does indeed reduce the number of writes, but if they are above about 1k each there is no significant saving. You would get the same effect with much simpler coding via e.g. new BufferedOutputStream(new FileOutputStream(..));
              • 4. Re: FileChannel.write(ByteBuffer[]) only writes out the first 16 ByteBuffer
                800381
                shawnstc wrote:
                Hi,
                This does NOT happen on Windows. I've only had this problem on Solaris (10 u8).
                I've tried this in jdk1.4, 1.5 and 1.6. All showed the same behavior on Solaris.
                I tried to write bytes to a file using FileChannel.write(ByteBuffer[]) method.
                I noticed that, if the ByteBuffer array contains more than 16 ByteBuffer, the FileChannel.write(ByteBuffer[] src) method only writes out contents of the first 16 ByteBuffer (src[0] thru src[15]).
                It worked fine if I combined all bytes into a single (big) ByteBuffer. (FileChannel.write(ByteBuffer) works fine)
                Has anyone experienced similar problem? Is it a bug in the JRE?

                Thanks.
                You do check and properly handle partial success for every single write you do, correct?
                • 5. Re: FileChannel.write(ByteBuffer[]) only writes out the first 16 ByteBuffer
                  846113
                  Yes. Based on the returned value (i.e, the number of bytes actually written), "partial success" can be handled. Thanks.
                  • 6. Re: FileChannel.write(ByteBuffer[]) only writes out the first 16 ByteBuffer
                    EJP
                    The correct way to do that is something like this:
                    long count = 0;
                    ByteBuffer[] buffers; // buffers to write, assumed to be unflipped at this point
                    long max; // maximum data to transfer, i.e. sum of available() over buffers
                    do
                    {
                      for (ByteBuffer buffer : buffers)
                      {
                        if (offset+1 < length && buffer.available() == 0)
                          offset++;
                        buffer.flip();
                      }
                      try
                      {
                        count += fch.write(buffers, offset, length);
                      }
                      finally
                      {
                        for (ByteBuffer buffer : buffers)
                          buffer.compact();
                      }
                    } while (count < max);
                    Warning: totally untested. But you get the idea. It caters for partial writes and also for the case where the number of iovecs is capped as appears to be happening here.