4 Replies Latest reply on Nov 27, 2012 1:28 PM by Jackie_C

    Ingestion hangs/Bulk load limitation?


      I've been running a simple graph that reads in a large amount of data from a DB, does a simple reformat to create a recspec before calling the Bulk Add/Replace Records.
      However, my graph has been consistently hanging when it hits a certain number of records, in my case just under 500K.
      I don't believe it's the DB connection, but I can't be sure either, but I'm wondering if there's a limitation to the number of records the bulk add/replace can handle? When checking the processes, the CPU is sitting at 99-100% on the dgraph process but the number of records in the logs isn't increasing at all.


        • 1. Re: Ingestion hangs/Bulk load limitation?
          I should also add that if I connect the reformatter to Trash and run it, the graph runs perfect which is why it leads me to believe the issue is at the Endeca Server or the Bulk load.
          • 2. Re: Ingestion hangs/Bulk load limitation?
            Dan at Branchbird

            Have you tried to enable debug on the edge leading into the bulk writer? I wonder if there is any odd characteristics to your data. It might help to look at the data to be sure you don't have duplicate record specs, etc. Also, assuming you're doing this locally, how much available RAM do you have on your laptop? I wonder if you haven't saturated your physical memory and started paging to virtual memory. From the task manager, how much memory is the dgraph taking up? How much available memory remains on your machine?

            • 3. Re: Ingestion hangs/Bulk load limitation?
              Hey Dan,

              Thanks for the quick response.
              I've checked the data beforehand to ensure there's no duplicates or any funny characteristics.
              I'm running the graphs on separate servers (don't have the specs handy) but from what I can tell, the dgraph process is using 6% of the RAM while it's maxing out the CPU during that time.

              I just tried running the same graph in a new test data store on their staging server which and got this error:
              Exception in thread "main" com.sun.xml.ws.client.ClientTransportException: HTTP transport error: java.net.ConnectException: Connection timed out: connect
                   at com.sun.xml.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:131)
                   at com.sun.xml.ws.transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:151)
                   at com.sun.xml.ws.transport.http.client.HttpTransportPipe.processRequest(HttpTransportPipe.java:93)
                   at com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:598)
                   at com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:557)
                   at com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:542)
                   at com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:439)
                   at com.sun.xml.ws.client.Stub.process(Stub.java:222)
                   at com.sun.xml.ws.client.sei.SEIStub.doProcess(SEIStub.java:135)
                   at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:109)
                   at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:89)
                   at com.sun.xml.ws.client.sei.SEIStub.invoke(SEIStub.java:118)
                   at $Proxy31.getGraphExecutionStatus(Unknown Source)
                   at com.cloveretl.gui.server.runtime.CloverServerProxy.A(Unknown Source)
                   at com.cloveretl.gui.server.runtime.CloverServerProxy.main(Unknown Source)
              Caused by: java.net.ConnectException: Connection timed out: connect
                   at java.net.PlainSocketImpl.socketConnect(Native Method)
                   at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
                   at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
                   at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
                   at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
                   at java.net.Socket.connect(Socket.java:529)
                   at java.net.Socket.connect(Socket.java:478)
                   at sun.net.NetworkClient.doConnect(NetworkClient.java:163)
                   at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
                   at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
                   at sun.net.www.http.HttpClient.<init>(HttpClient.java:233)
                   at sun.net.www.http.HttpClient.New(HttpClient.java:306)
                   at sun.net.www.http.HttpClient.New(HttpClient.java:323)
                   at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:860)
                   at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:839)
                   at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:726)
                   at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:904)
                   at com.sun.xml.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:119)
                   ... 14 more

              The clover server has now hung, but when I check using endeca-cmd, there are no jobs listed. However when I try and add that datastore as a data source in Studio, it can't validate it though it can validate other datastores on that Endeca server.

              Thoughts? :)
              • 4. Re: Ingestion hangs/Bulk load limitation?
                Looks like the issue had to deal with running out of disk space on the server due to the massive log files being generated by Tomcat.
                Once we changed the logging.properties and freed up the old logs for space, everything seemed to work well.

                Thanks for the help!