3 Replies Latest reply: May 17, 2013 3:10 AM by Mark BT52 RSS

    Performance across network

    Mark BT52
      Don't understand the performance we are seeing.

      Have a setup with the database in Luxemburg. The Apex listener or Weblogic Server in UK. Client PCs in UK.

      Uploading a 2Mb file to the database using embedded Gateway = 2s
      Uploading a 2Mb file to the database using Listener (standalone or through Weblogic) = 49s (even when listener is on client PC, so it's not a local network thing)
      Uploading a 2Mb file to the database using Application Server = 8 mins

      Set up the Apex listener on a server in Luxemburg instead
      Uploading a 2MB file to the database using Luxemburg Listener (standalone) = 2s (File being uploaded still from UK PC, so still going across thje same distance

      So why is the upload taking so long using the Listener compared to the embedded Gateway, and why is it only slow when the connection between the Listener and the Database is remote, rather than when the connection between the PC and the Listener is remote?

      Is it our network config? Is the file going across the network in a different way when the Listener is local to the PC than when it is local to the database?

      What mechanism does the listener use to actually communicate with the database across the network that would be different to using the embedded gateway?

      (NOTE: response time between the server in UK and Luxembourg is 22ms. Is this a factor affecting the behaiviour of the Listener?)

      Cheers
      Mark

      Edited by: Mark BT52 on May 16, 2013 8:53 AM
        • 1. Re: Performance across network
          Udo
          Hi Mark,
          Have a setup with the database in Luxemburg. The Apex listener or Weblogic Server in UK. Client PCs in UK.
          [...]
          Uploading a 2Mb file to the database using Listener (standalone or through Weblogic) = 49s (even when listener is on client PC, so it's not a local network thing)
          [...]
          Uploading a 2MB file to the database using Luxemburg Listener (standalone) = 2s

          So why is the upload taking so long using the Listener compared to the embedded Gateway.
          It's not really poor performance with APEX Listener compared to EPG, as you've found out yourself by locating APEX Listener on your database host.
          It seems there is some delay on your link between the database in Luxemburg and the app server in the UK. If it's the same network connection that is also used for the client PCs, it's probably something in your app servers configuration. In many cases there are delays due to issues with your DNS configuration, because DNS timeouts can be quite long and simple things like log entries including host names may result poor response times.
          Is the file going across the network in a different way when the Listener is local to the PC than when it is local to the database?
          As mentioned before, this depends on how your clients and the app server are connected to the database. I'll try to point out the difference in request-response paths:

          1. Client to EPG/APEX Listener on DB-Host
          Client ->link1-> DB-Host ->link1-> Client
          2. Client to APEX Listener on App-Server
          Client ->link2-> App-Server ->link3-> DB-Host ->link3-> App-Server ->link2-> Client

          If link1 and link3 appear to be the same, this doesn't necessarily mean that the configuration (network in general, dns as one possible service in particular) of your clients and your app server are similar.
          In addition, there is link2. Though it seems to be a local connection in your setup, it could be something like the client-related name resolution that your database host can handle properly but the app server cannot.

          For troubleshooting, you should check every step of your slow request path. You should find at least one of them to be much slower than expected.
          Afterwards, you should be able to answer questions like the follwing ones:

          - Are requests from your client to your app server slow, e. g. when requesting static files like the apex_version.txt out of the APEX images from your app server?
          - Are requests from your app server to your database slow, e. g. when pinging of the hostname you've configured for APEX Listener's database connection or by using wget or something similar to request some static file from your EPG?
          - Are requests from your database server to your app server slow, e. g. when pinging the hostname your app server uses when requesting your database or using wget or something similar to request some static file from your app server?
          [...]

          The answers will hint where to seek for configuration issues...

          -Udo
          • 2. Re: Performance across network
            Colm Divilly
            If you have an app server in one country and the database in another, then you are going to have latency issues, this is not specific to Listener. Best practice is to co-locate the app server and database.

            The closer your app server is to your database, the better the performance. In my experience Net8 (the on-the-wire protocol of Oracle) does not work well over typical wide area networks.

            You say you are trying to upload a 2mb file, the Oracle JDBC driver is not going to send this all in one network request, it will most likely be splitting the file into smaller chunks and sending multiple network requests between the app server and the database, all these extra round-trips increase latency (more round-trips == more chances for one or more round-trisp getting held up) and reduce the overall transfer speed.

            You may be able to tweak the JDBC settings (consult the JDBC documentation) to affect how many round-trips are required by increasing buffer sizes, but it would just be better to put the app server in Luxemburg.
            • 3. Re: Performance across network
              Mark BT52
              Thanks for that.

              I also suspected it was splitting into smaller chunks.
              From the logs with the appliction server I got the impression that it was splitting into 700byte chunks and then logging in and out again for each chunk, (this one took 8 mins)

              I have been unable to find any kind of config setting in the application server to control this, and neither for the Listener though.

              Will look into the JDBC settings. Also going to use Wireshark to see exactly what is going on. Maybe we can improve the 22ms latency.


              Would be a shame to have to have the Application Server on the Luxembourg side as that would require an additional licence.