2 Replies Latest reply on Nov 27, 2012 12:18 PM by 976184

    Data compression


      In our company we are using ODSI 10g R3 to produce something like an account statement of a customer.
      The service is exposed through HTTP/SOAP.
      I'm wondering if there are any compression options we could use. Currently the xml message may be 100s of MB in size, and because it has to travel through our middleware (IBM Message Broker, MQ-based), I'm quite worried about network/disk/queue-size bottlenecks.
      Best solution in my opinion would be to compress the response on the ODSI/weblogic side, send the binary data over middleware, and decompress on the requesting application side.
      Quick search didn't produce any promising results.

      Thank you for your help!
        • 1. Re: Data compression
          Measuring the performance to see if you really have a problem would be a good place to start. Today's networks are gigabit and disks are terabyte. The first place I would expect to see a problem is memory usage of the ODSI server - since exposing a data service as SOAP requires the complete payload to be materialized at once (using the ODSI Java Mediator does not), and materialization of xml in java can take on the order of ten times the size of the xml (your case could require gigabytes of memory for a single result).
          Compression in the ODSI stack would not reduce the volume of data outside of the ODSI stack (IBM Message Broker and the consuming application).
          Results returned from ODSI server to the ODSI java mediator are in binary xml, which is considerably more compact than raw xml. Results returned from ODSI server as SOAP are SOAP - not compressed (your case).

          You certainly could write a data service that returns compressed data, and have the client application decompress it. You will find compression utilities in java.util.zip.
          1 person found this helpful
          • 2. Re: Data compression
            Thanks for the tips.

            I'll admit that we haven't tried the Java mediator approach and don't know much about it, however from what I understand this approach requires us to code some stuff in MB, essentially serializing objects returned from ODSI into xml. This is extra work compared to simply calling a web service.
            What I'd like to have in the "ideal" scenario is some interface through which we could get the entire response from ODSI in binary/compressed XML format. Doing any compression on MB side is one step too late and we can't afford such CPU intensive operations in it.
            Perhaps I'm misunderstanding something, please correct me then.