7 Replies Latest reply: May 16, 2011 12:15 AM by EJP RSS

    how to serialize a stream

    862053
      Hi. How to serialize a stream? My code is like this:
      public class BinaryFile implements Serializable
      {
         ...
      
         private void writeObject(ObjectOutputStream os) throws IOException
         {
            os.defaultWriteObject();
            int total_length = inputStream.available();
            os.writeInt(total_length);
            byte[] data = new byte[inputStream.available()];
            inputStream.read(data);
            os.write(data);
         }
      
         private void readObject(ObjectInputStream in) throws IOException, ClassNotFoundException
         {
            in.defaultReadObject();
            int total_length = in.readInt();
            remoteData = new byte[total_length];
            in.read(remoteData);
            remoteStream = new ByteArrayInputStream(remoteData);
         }
      My test code writes a string, file input stream, string to an ObjectOutputStream using writeObject. It then reads the data back. The first object is the string and is correct. The second object is the BinaryFile, which is correct, and the length of the file is correct too. However the contents of the file are not correct. The file is a PDF. Opening the file with PDF says the file is corrupted. In fact, using vi to look at the file shows that both files are different.

      Thanks.
        • 1. Re: how to serialize a stream
          796440
          I don't know if this is this source of your problem, but you'll probably never have a reason to use available(). It does not do what you think it does.

          If you're reading a byte stream in the general case, you loop on read(byte[] buf) until it returns -1.
          • 2. Re: how to serialize a stream
            jtahlborn
            if you are doing this so that you can send a stream over rmi, you should check out the rmiio project ( http://openhms.sourceforge.net/rmiio/ ). even if you are not using rmi, however, there is a class in that project which does effectively what you are looking for, check out DirectRemoteInputStream ( http://openhms.sourceforge.net/rmiio/xref/com/healthmarketscience/rmiio/DirectRemoteInputStream.html ).
            • 3. Re: how to serialize a stream
              EJP
              In addition to what jverd said, you should use a loop in the ReadFully() method, and check the result of read() every time. At the moment you're just assuming you filled the buffer. As this is a network, you almost certainly didn't.
              • 4. Re: how to serialize a stream
                862053
                Just to be clear I did some simplification in my previous post. The real writeObject does use a loop to copy the file.
                   private void writeObject(ObjectOutputStream os) throws IOException
                   {
                      os.defaultWriteObject();
                      int total_length = inputStream.available();
                      os.writeInt(total_length);
                      StreamUtils.copy(inputStream, os);
                   }
                where
                   public static void copy(InputStream source, OutputStream dest) throws IOException
                   {
                      byte[] chunk = new byte[65536];
                      try
                      {
                         while (true)
                         {
                            int actual = source.read(chunk);
                            if (actual <= 0) break;
                            dest.write(chunk, 0, actual);
                         }
                      }
                Regarding the use of InputStream.available, it is true that the Javadoc says "Note that while some implementations of InputStream will return the total number of bytes in the stream, many will not. It is never correct to use the return value of this method to allocate a buffer intended to hold all data in this stream.". However in my case the InputStream is either a FileInputStream or ByteArrayInputStream and available() does return the length of the file. Maybe I'll make the code more robust by not using available(), but that's another issue.
                • 5. Re: how to serialize a stream
                  EJP
                  That code still doesn't work. You're still using available, and in readObject() you aren't limiting the reading to the length you passed. You need to track how much you've read, and limit the read via
                  int total = in.readInt();
                  int count = 0;
                  int actual;
                  while ((actual = in.read(buffer, 0, (int)Math.min(buffer.length, total-count)) > 0)
                  {
                      count += actual;
                      out.write(buffer, 0, count);
                  }
                  Also you should arguably be using a long for the file length, although I guess if you're transferring files that large you already have much bigger problems. At least check that the file isn't too large.
                  Regarding the use of InputStream.available() ... "It is never correct to use the return value of this method to allocate a buffer intended to hold all data in this stream".
                  QED. No 'maybe'about it. It isn't specified to do what you're using it for, so don't use it for that. It does something else. Sometimes.
                  • 6. Re: how to serialize a stream
                    862053
                    Yes, the problem is that in.read(buffer) where buffer is an array of 400 thousands or so bytes does not read the entire array into the buffer. I thought it does this, internally reading chunks and appending it to buffer. But it turns out it only reads first 1020 bytes. So had to keep reading in a loop. It does work.
                    • 7. Re: how to serialize a stream
                      EJP
                      That's one of the problems. There are others. I've described them.