9:40 p.m. EDT
Eloqua deployed release 2012-R3 during a planned maintenance window on September 9 from 5:00AM to 11:00AM EDT. After the release, some customers advised us that some of their contacts were receiving again emails that were previously sent, with some emails being re-sent several times over the course of a few hours.
Duration of incident: Sept 9 11:00AM EDT to Sept 10 11:00AM EDT
Incident Root Cause
Based on our investigation, we have determined the following:
- This software defect was introduced during the R3 release as part of a project to improve the scalability and performance of segment and recipient creation.
- The defect caused segment members (contacts) to be added to an active Campaign, when in fact the contacts were either already in the Campaign, or had previously run through it and should not have re-entered.
- As a result of re-entering the Campaign, if the Campaign contained an email-send step that specifically allowed resending of an email, then the email would have been re-sent, resulting in duplicated emails. In addition, other Campaign steps would have also executed, such as “Cloud Connectors”, “Add To Program” and “Add to Campaign”.
- The root cause of the bug was addressed via a code change.
- Customer impact was minimized by removing duplicate Campaign members from affected campaigns.
- Campaign automation QA test coverage will be corrected to ensure that this specific scenario cannot re-occur.
- Our development team will improve unit and integration tests in our workflow engine to catch these types of scenarios.
11:50 a.m. EDT
Campaign Services were resumed at approximately 11:30 AM EDT.
11:05 a.m. EDT
We are now in the process of deploying the resolution for this issue to the Eloqua platform. Before restarting Campaign Services, we need to take some additional time to ensure that there are no contacts queued up to receive emails that they shouldn't. Once the services are restarted, processing will begin again, and normal operation is expected to resume by 11:30 AM EDT.
10:20 a.m. EDT
A configuration issue with Eloqua's campaign automation services is causing unexpected behavior for campaigns that have their emails set to be allowed to be resent. There is the potential for contacts that should not be resent an email to have it sent to them again should the automation flow allow them to be re-entered into the campaign. In order to properly address this issue we are briefly pausing Campaign Services until approximately 10:30 ET. While these services are paused any campaign actions such as email sends will be queued. We will update this post as soon as services are re-enabled. Thank you for your patience.