This discussion is archived
4 Replies Latest reply: Nov 27, 2012 5:28 AM by Jackie_C RSS

Ingestion hangs/Bulk load limitation?

Jackie_C Newbie
Currently Being Moderated
Hi,

I've been running a simple graph that reads in a large amount of data from a DB, does a simple reformat to create a recspec before calling the Bulk Add/Replace Records.
However, my graph has been consistently hanging when it hits a certain number of records, in my case just under 500K.
I don't believe it's the DB connection, but I can't be sure either, but I'm wondering if there's a limitation to the number of records the bulk add/replace can handle? When checking the processes, the CPU is sitting at 99-100% on the dgraph process but the number of records in the logs isn't increasing at all.

Thoughts?

Thanks,
Chris
  • 1. Re: Ingestion hangs/Bulk load limitation?
    Jackie_C Newbie
    Currently Being Moderated
    I should also add that if I connect the reformatter to Trash and run it, the graph runs perfect which is why it leads me to believe the issue is at the Endeca Server or the Bulk load.
  • 2. Re: Ingestion hangs/Bulk load limitation?
    Dan at Branchbird Journeyer
    Currently Being Moderated
    Chris,

    Have you tried to enable debug on the edge leading into the bulk writer? I wonder if there is any odd characteristics to your data. It might help to look at the data to be sure you don't have duplicate record specs, etc. Also, assuming you're doing this locally, how much available RAM do you have on your laptop? I wonder if you haven't saturated your physical memory and started paging to virtual memory. From the task manager, how much memory is the dgraph taking up? How much available memory remains on your machine?

    HTH,
    Dan
    http://branchbird.com
  • 3. Re: Ingestion hangs/Bulk load limitation?
    Jackie_C Newbie
    Currently Being Moderated
    Hey Dan,

    Thanks for the quick response.
    I've checked the data beforehand to ensure there's no duplicates or any funny characteristics.
    I'm running the graphs on separate servers (don't have the specs handy) but from what I can tell, the dgraph process is using 6% of the RAM while it's maxing out the CPU during that time.

    I just tried running the same graph in a new test data store on their staging server which and got this error:
    >
    Exception in thread "main" com.sun.xml.ws.client.ClientTransportException: HTTP transport error: java.net.ConnectException: Connection timed out: connect
         at com.sun.xml.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:131)
         at com.sun.xml.ws.transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:151)
         at com.sun.xml.ws.transport.http.client.HttpTransportPipe.processRequest(HttpTransportPipe.java:93)
         at com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:598)
         at com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:557)
         at com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:542)
         at com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:439)
         at com.sun.xml.ws.client.Stub.process(Stub.java:222)
         at com.sun.xml.ws.client.sei.SEIStub.doProcess(SEIStub.java:135)
         at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:109)
         at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:89)
         at com.sun.xml.ws.client.sei.SEIStub.invoke(SEIStub.java:118)
         at $Proxy31.getGraphExecutionStatus(Unknown Source)
         at com.cloveretl.gui.server.runtime.CloverServerProxy.A(Unknown Source)
         at com.cloveretl.gui.server.runtime.CloverServerProxy.main(Unknown Source)
    Caused by: java.net.ConnectException: Connection timed out: connect
         at java.net.PlainSocketImpl.socketConnect(Native Method)
         at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
         at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
         at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
         at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
         at java.net.Socket.connect(Socket.java:529)
         at java.net.Socket.connect(Socket.java:478)
         at sun.net.NetworkClient.doConnect(NetworkClient.java:163)
         at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
         at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
         at sun.net.www.http.HttpClient.<init>(HttpClient.java:233)
         at sun.net.www.http.HttpClient.New(HttpClient.java:306)
         at sun.net.www.http.HttpClient.New(HttpClient.java:323)
         at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:860)
         at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:839)
         at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:726)
         at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:904)
         at com.sun.xml.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:119)
         ... 14 more
    >

    The clover server has now hung, but when I check using endeca-cmd, there are no jobs listed. However when I try and add that datastore as a data source in Studio, it can't validate it though it can validate other datastores on that Endeca server.

    Thoughts? :)
  • 4. Re: Ingestion hangs/Bulk load limitation?
    Jackie_C Newbie
    Currently Being Moderated
    Looks like the issue had to deal with running out of disk space on the server due to the massive log files being generated by Tomcat.
    Once we changed the logging.properties and freed up the old logs for space, everything seemed to work well.

    Thanks for the help!

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points