Our sync script for connector has gone from running in minutes to taking over 40 minutes per run. Normally I would think that was caused by amount to process but when I look at the logs on the MR script, 41 minutes was spent in "Get Input Data" phase while only a minute and a half is spent in Mad-Shuffle-Reduce-Summarize all together.
To help see the problem (source is locked) I set the script logging to debug. The number of items to sync is at 37,899 which is strange since we only have 36,000 item records and of those only 8000 need to sync 8000 with NS Connector. The number to sync keeps growing and it keeps slowing. despite is saying it synced 50 records each run. (runtime parameter it set to its max value of 50)
What is the criteria to add something to sync list? Trying to figure out if we are inadvertently adding thing to queue that shouldn't be. Any known problems of things not being removed from queue after they sync?
Running version 1.6.8.
Any insight appreciated.
Rob