Forum Stats

  • 3,678,175 Users
  • 7,778 Discussions
  • 13,052 Comments

Discussions

GREEN / Temporary Pause in Campaign Services / Some Customers Affected -- Resolved

Ken Carneiro-Oracle
Ken Carneiro-Oracle Posts: 175 Silver Trophy
edited Sep 10, 2012 9:47PM in Eloqua System Status

**UPDATE**

9:40 p.m. EDT

Incident Description

Eloqua deployed release 2012-R3 during a planned maintenance window on September 9 from 5:00AM to 11:00AM EDT.  After the release, some customers advised us that some of their contacts were receiving again emails that were previously sent, with some emails being re-sent several times over the course of a few hours.


Duration of incident: Sept 9 11:00AM EDT to Sept 10 11:00AM EDT

Incident Root Cause

Based on our investigation, we have determined the following:

    1. This software defect was introduced during the R3 release as part of a project to improve the scalability and performance of segment and recipient creation.
    2. The defect caused segment members (contacts) to be added to an active Campaign, when in fact the contacts were either already in the Campaign, or had previously run through it and should not have re-entered.
    3. As a result of re-entering the Campaign, if the Campaign contained an email-send step that specifically allowed resending of an email, then the email would have been re-sent, resulting in duplicated emails.  In addition, other Campaign steps would have also executed, such as “Cloud Connectors”, “Add To Program” and “Add to Campaign”.

Corrective Action

      1. The root cause of the bug was addressed via a code change.
      2. Customer impact was minimized by removing duplicate Campaign members from affected campaigns.
      3. Campaign automation QA test coverage will be corrected to ensure that this specific scenario cannot re-occur.
      4. Our development team will improve unit and integration tests in our workflow engine to catch these types of scenarios.

**UPDATE**

11:50 a.m. EDT

Campaign Services were resumed at approximately 11:30 AM EDT.

**UPDATE**

11:05 a.m. EDT

We are now in the process of deploying the resolution for this issue to the Eloqua platform.  Before restarting Campaign Services, we need to take some additional time to ensure that there are no contacts queued up to receive emails that they shouldn't.  Once the services are restarted, processing will begin again, and normal operation is expected to resume by 11:30 AM EDT.

**UPDATE**

10:20 a.m. EDT

A configuration issue with Eloqua's campaign automation services is causing unexpected behavior for campaigns that have their emails set to be allowed to be resent.  There is the potential for contacts that should not be resent an email to have it sent to them again should the automation flow allow them to be re-entered into the campaign.  In order to properly address this issue we are briefly pausing Campaign Services until approximately 10:30 ET.  While these services are paused any campaign actions such as email sends will be queued.  We will update this post as soon as services are re-enabled.  Thank you for your patience.

Post edited by Unknown User on
Tagged:

Comments

  • ChrisJones-Oracle
    ChrisJones-Oracle Posts: 2 Silver Badge

    Hi Ken, is this going to affect Program Builder in E9 in any way?  Or is this restricted only to E10's campaigns?

  • Ken Carneiro-Oracle
    Ken Carneiro-Oracle Posts: 175 Silver Trophy

    Hi Chris,

    Thanks for the question. No, E9 Program Builder emails are not affected in any way.

    Thanks,

    Ken

  • krista.seidemann
    krista.seidemann Posts: 30 Gold Trophy

    Is there any way to tell if one of our campaigns has been affected?

  • is there any way to get an alert on something like this in the future? vs. having to figure it out by simply not receiving an email that was set to go out?

    or perhaps a message upon login that there's an issue?

    this is critical information but it's buried in a Topliners message.

  • 2862153
    2862153 Posts: 2 Bronze Badge

    is there any way to get an alert on something like this in the future? vs. having to figure it out by simply not receiving an email that was set to go out?

    or perhaps a message upon login that there's an issue?

    this is critical information but it's buried in a Topliners message.

    Hi Celine, you can follow the Eloqua System Status group on Topliners and you will receive email notification of all issues as they are posted.  It only takes 5 seconds to setup and you'll be notified of everything in real time.

    Cheers,

    Joe

  • jmugo
    jmugo Posts: 22 Silver Trophy

    is there any way to get an alert on something like this in the future? vs. having to figure it out by simply not receiving an email that was set to go out?

    or perhaps a message upon login that there's an issue?

    this is critical information but it's buried in a Topliners message.

    I agree with Celine. This is not something that should be in Topliners. Not every Eloqua User is a Topliners Community member. This is critical information that at the very least, needs to be a huge banner when we log into the system and only a secondary FYI in Topliners.

  • I just got off the phone with support about a campaign that is stuck and was told it would be turned back on at 8:30am PST, but would have to deal with the backlog this pause is creating.

  • jmugo
    jmugo Posts: 22 Silver Trophy

    I just got off the phone with support about a campaign that is stuck and was told it would be turned back on at 8:30am PST, but would have to deal with the backlog this pause is creating.

    It's 11:35 ET and nothing is happening. No word on how long this "delay" is going to be.

  • jmugo
    jmugo Posts: 22 Silver Trophy
    edited Sep 10, 2012 12:22PM

    Campaign services *not resumed* correctly. Still experiencing issues. On the phone with support

  • schwartzrw
    schwartzrw Posts: 25 Silver Trophy

    Can we get an update one this?

  • jmugo
    jmugo Posts: 22 Silver Trophy

    Can we get an update one this?

    Just fyi -

    Although Eloqua resolved the issue with the system, it still took (and is still taking) a really long time to get emails out the door. Apparently, the issue was with Campaign Canvas.

    What a morning!

  • A temporary pause would have been inconvenient but when Eloqua sent out an old campaign (from April) to our Enterprise-level customers, which is what happened to us with the recent update, it was absolutely unnerving.  I've made sure to disable all older campaigns now, but am very disappointed that none of the safeguards in Eloqua actually took effect.  We sent the same email that we sent in April (now outdated) to the EXACT same customers who received that email already.  I've just spent 2 hours in a very uncomfortable meeting with my boss and his boss.  I am not a happy camper. 

  • schwartzrw
    schwartzrw Posts: 25 Silver Trophy

    A temporary pause would have been inconvenient but when Eloqua sent out an old campaign (from April) to our Enterprise-level customers, which is what happened to us with the recent update, it was absolutely unnerving.  I've made sure to disable all older campaigns now, but am very disappointed that none of the safeguards in Eloqua actually took effect.  We sent the same email that we sent in April (now outdated) to the EXACT same customers who received that email already.  I've just spent 2 hours in a very uncomfortable meeting with my boss and his boss.  I am not a happy camper. 

    Hi Pamela - I agree - it's surprising these kind of things make it through QA.

  • krista.seidemann
    krista.seidemann Posts: 30 Gold Trophy

    Is there any way to tell if one of our campaigns has been affected?

  • To those affected by this issue, we understand this has affected your business.  The head of development and QA and myself just met, and discussed how we ensure we are providing more comprehensive testing in this area.

    Although we want to continually innovate with quarterly releases, we understand it is critical to ensure quality of each of those releases.

    Thank you for your ongoing support.

    Paul

  • craighuff
    craighuff Posts: 10 Silver Badge

    Is there any way to tell if one of our campaigns has been affected?

    I would like to know the same thing. They aren't responding and support doesn't know............

  • jmugo
    jmugo Posts: 22 Silver Trophy

    Is there any way to tell if one of our campaigns has been affected?

    Krista,

    I'd start with Campaigns that have "Allow members to enter more than once" checkbox checked. Pull a Campaign Analysis report and look at your total sends versus total contacts in the segment.

    Additionally you should probably spot check individual contacts in each campaign to look at activity history. That will give you a good idea about whether the email was deployed again for the entire segment.

    Luckily, almost all our Campaigns do not allow members to enter twice and I would recommend this as a default for future campaigns wherever and whenever possible.

  • vallen
    vallen Posts: 1 Red Ribbon

    "If the Campaign contained an email-send step that specifically allowed resending of an email, then the email would have been re-sent"

    Can anyone confirm whether this statement is the fool-proof way of determining whether a camapign's email was resent?