We are getting ready to migrate lots of people and events. The dry run that we just completed had 3.6 million events that were processed in 6-8 hours (not bad). We had a scheduling queue on both backend Calendar Stores that peaked at 130,000 items and took over 24 hours to process, pretty maxed out at about 5,000/hour. Is there any way to tweak the scheduling, even if only temporary?
The trouble with using regular PUT for migration is that, in the case of meetings with attendees, depending on the order of the insertion, events may be written twice (e.g. organizer calendar is migrated, this triggers scheduling and generation of attendees copy, then attendees calendars are migrated, causing an update or vice versa).
Did you try temporarily setting notification.dav.enablejmsnotif to false ? The perf improvement may be marginal but at least, you will avoid generation of jms messages and potentially loads of SMTP notification. But you may have figured that out by now.
You can try to temporarily set davcore.scheduling.synchronousdelivery to true. This will slow down your insertion rate as each PUT will internally do the whole scheduling processing before returning but maybe this will create less contention.
Are you establishing multiple HTTP connections or working with a single one ?
Do the 3.6 million event correspond to your whole legacy database or did you extract only recent events? Assuming that it is acceptable for your end users, you might want to consider filtering out old events (e.g. migrate only if year > 2011) if you have not done so already.
To answer questions:
Yes, we have used notification.dav.enablejmsnotif=false.
Yes, we are using multiple HTTP connections.
Yes, we have filtered out some old events, but there was some pushback so we could not do as much as we wanted.
We are just not sure about using: davcore.scheduling.synchronousdelivery=true.
We can live with it taking 24 hours for the scheduling queue to flush out, I was just looking for an easy change to make it happen faster.