4 Replies Latest reply: Apr 13, 2012 9:20 AM by greybird RSS

    Compression/Decompression of byte-array in TupleOutput/TupleInput

    921522
      Hello,

      how do I compress the whole data before I flush everything out? I simply want to write to the internal buffer using TupleOutput methods and after that compress the buffer. The decompressing once using the TupleInput reference.

      For instance Snappy.compress(byte[]) and Snappy.decompress(byte[]).

      Thanks.

      kind regards,
      Johannes
        • 1. Re: Compression/Decompression of byte-array in TupleOutput/TupleInput
          greybird
          Hi Johannes,

          It is possible to convert between a byte array and a TupleInput or TupleOutput, in both directions. Is that what you're asking how to do?

          Or are you asking how to do this "in place" within the context of a TupleBinding method?

          --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
          • 2. Re: Compression/Decompression of byte-array in TupleOutput/TupleInput
            921522
            Well, I've found a method to do it based on a SnappyOutputStream.

            /**
            * Constructor.
            *
            * @param pOutput
            * to be wrapped
            * @throws NullPointerException
            * if {@code pOutput} is {@code null}
            */
            public TupleOutputSink(@Nonnull final TupleOutput pOutput) {
            mOutput = checkNotNull(pOutput);
            try {
            mOut = new SnappyOutputStream(new ByteArrayOutputStream());
            mOutput.writeTo(mOut);
            } catch (final IOException e) {
            throw new RuntimeException(e);
            }
            }

            But I don't know about the other side:

            public class NodePageContainerBinding extends TupleBinding<NodePageContainer> {

            @Override
            public NodePageContainer entryToObject(@Nonnull final TupleInput pInput) {
            final ITTSource source = new TupleInputSource(checkNotNull(pInput));
            final NodePage current = (NodePage)PagePersistenter.createPage(source);
            final NodePage modified = (NodePage)PagePersistenter.createPage(source);
            return new NodePageContainer(current, modified);
            }

            @Override
            public void objectToEntry(@Nonnull final NodePageContainer pPageContainer,
            @Nonnull final TupleOutput pOutput) {
            pPageContainer.serialize(checkNotNull(pOutput));
            }
            }

            I can't find a method to specify an InputStream to read from for a TupleInput reference. TupleInputSource is just a wrapper which implements an interface for the BerkeleyDB backend. Somehow I need to add SnappyInputStream somwhere, such that the data first is decompressed and then normally read via TupleInput.

            Edit: Sorry obviously writing to an OutputStream doesn't has any value if the TupleOutput is directly used...

            Edited by: 918519 on Apr 10, 2012 2:30 PM
            • 3. Re: Compression/Decompression of byte-array in TupleOutput/TupleInput
              921522
              So I don't know how to do it in TupleOutput/TupleInput directly :(

              kind regards,
              Johannes
              • 4. Re: Compression/Decompression of byte-array in TupleOutput/TupleInput
                greybird
                Perhaps you're expecting to be able to use TupleInputStream as a FilterInputStream, which is is not.

                Have you tried implementing the EntryBinding interface? The entryToObject(DatabaseEntry) method gives you the byte array. You can decompress the byte array and then use the byte array to call your tuple binding's entryToObject(DatabaseEntry) method.

                In other words, compose two bindings. One binding does the compression/decompression, the other does the tuple serialize/deserialize using the uncompressed byte array.

                --mark