This discussion is archived
4 Replies Latest reply: Apr 13, 2012 7:20 AM by greybird RSS

Compression/Decompression of byte-array in TupleOutput/TupleInput

921522 Newbie
Currently Being Moderated
Hello,

how do I compress the whole data before I flush everything out? I simply want to write to the internal buffer using TupleOutput methods and after that compress the buffer. The decompressing once using the TupleInput reference.

For instance Snappy.compress(byte[]) and Snappy.decompress(byte[]).

Thanks.

kind regards,
Johannes
  • 1. Re: Compression/Decompression of byte-array in TupleOutput/TupleInput
    greybird Expert
    Currently Being Moderated
    Hi Johannes,

    It is possible to convert between a byte array and a TupleInput or TupleOutput, in both directions. Is that what you're asking how to do?

    Or are you asking how to do this "in place" within the context of a TupleBinding method?

    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
  • 2. Re: Compression/Decompression of byte-array in TupleOutput/TupleInput
    921522 Newbie
    Currently Being Moderated
    Well, I've found a method to do it based on a SnappyOutputStream.

    /**
    * Constructor.
    *
    * @param pOutput
    * to be wrapped
    * @throws NullPointerException
    * if {@code pOutput} is {@code null}
    */
    public TupleOutputSink(@Nonnull final TupleOutput pOutput) {
    mOutput = checkNotNull(pOutput);
    try {
    mOut = new SnappyOutputStream(new ByteArrayOutputStream());
    mOutput.writeTo(mOut);
    } catch (final IOException e) {
    throw new RuntimeException(e);
    }
    }

    But I don't know about the other side:

    public class NodePageContainerBinding extends TupleBinding<NodePageContainer> {

    @Override
    public NodePageContainer entryToObject(@Nonnull final TupleInput pInput) {
    final ITTSource source = new TupleInputSource(checkNotNull(pInput));
    final NodePage current = (NodePage)PagePersistenter.createPage(source);
    final NodePage modified = (NodePage)PagePersistenter.createPage(source);
    return new NodePageContainer(current, modified);
    }

    @Override
    public void objectToEntry(@Nonnull final NodePageContainer pPageContainer,
    @Nonnull final TupleOutput pOutput) {
    pPageContainer.serialize(checkNotNull(pOutput));
    }
    }

    I can't find a method to specify an InputStream to read from for a TupleInput reference. TupleInputSource is just a wrapper which implements an interface for the BerkeleyDB backend. Somehow I need to add SnappyInputStream somwhere, such that the data first is decompressed and then normally read via TupleInput.

    Edit: Sorry obviously writing to an OutputStream doesn't has any value if the TupleOutput is directly used...

    Edited by: 918519 on Apr 10, 2012 2:30 PM
  • 3. Re: Compression/Decompression of byte-array in TupleOutput/TupleInput
    921522 Newbie
    Currently Being Moderated
    So I don't know how to do it in TupleOutput/TupleInput directly :(

    kind regards,
    Johannes
  • 4. Re: Compression/Decompression of byte-array in TupleOutput/TupleInput
    greybird Expert
    Currently Being Moderated
    Perhaps you're expecting to be able to use TupleInputStream as a FilterInputStream, which is is not.

    Have you tried implementing the EntryBinding interface? The entryToObject(DatabaseEntry) method gives you the byte array. You can decompress the byte array and then use the byte array to call your tuple binding's entryToObject(DatabaseEntry) method.

    In other words, compose two bindings. One binding does the compression/decompression, the other does the tuple serialize/deserialize using the uncompressed byte array.

    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points