Discussions
NS Connector sync script "FA | MR Sync - Update FA" running out of control
Our sync script for connector has gone from running in minutes to taking over 40 minutes per run. Normally I would think that was caused by amount to process but when I look at the logs on the MR script, 41 minutes was spent in "Get Input Data" phase while only a minute and a half is spent in Mad-Shuffle-Reduce-Summarize all together.
To help see the problem (source is locked) I set the script logging to debug. The number of items to sync is at 37,899 which is strange since we only have 36,000 item records and of those only 8000 need to sync 8000 with NS Connector. The number to sync keeps growing and it keeps slowing. despite is saying it synced 50 records each run. (runtime parameter it set to its max value of 50)
Rob Speer
Principal Consultant
SpeerIT Consulting
312-480-0538
rob@speerit.net