Discussions

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

AB Testing Functionality in Multi Step Campaign

rebecca.vilcsek
rebecca.vilcsek Posts: 6 Green Ribbon
edited January 2017 in Dream It

Right now the AB testing functionality is only available in Simple Campaigns that allow you to test on one single email blast.  We would love to be able to leverage this functionality within a multi-step campaign in canvas so we can test with our canvas programs.

Post edited by Unknown User on
Tagged:
218
228 votes

Under review · Last Updated

«1

Comments

  • Jonas B. G. Wolffsen
    Jonas B. G. Wolffsen Posts: 3 Blue Ribbon

    Increase A/S is currently having an AppCloud Decision running in Beta with this functionality. You can contact me at [email protected] for more information.

  • claudiazoglmann
    claudiazoglmann Posts: 79 Silver Badge

    "Currently the teams focus is on developing the next generation of program builder"

    Which explains the "Program Canvas" glitches.

    How much of the current functionality should we expect to be deprecated with this, @Taha Sheikh-Oracle ?

  • Dominick Gaglio
    Dominick Gaglio Posts: 5 Silver Badge

    There is a way to do this manually that isn't too time consuming,  however I agree it would be nice if Eloqua provided a simple way on the canvas.

  • Taha Sheikh-Oracle
    Taha Sheikh-Oracle Posts: 11 Gold Badge

    "Currently the teams focus is on developing the next generation of program builder"

    Which explains the "Program Canvas" glitches.

    How much of the current functionality should we expect to be deprecated with this, @Taha Sheikh-Oracle ?

    Hi Claudia,

    All use cases from Program Builder will be supported on the Program Canvas (next gen PB), but not necessarily in the same manner.  As an example, rather than cloud connectors, the program canvas will leverage our new AppCloud Developer Framework and associated audience, action and decision apps for more use case based integrations e.g. data cleansing or even CRM integration.  This is another key difference, where instead of autosynchs, external calls, etc. CRM integrations will be app based allowing us to iterate on functionality outside of the core Eloqua platform and also take advantage of new versions of a CRM providers APIs.  We'll have lots more details to share as we get closer to releasing this capability, but you can expect to start hearing some information sooner as we start beta testing. 

    Hope this helps.

    Taha

  • li*2702796*ey
    li*2702796*ey Posts: 4 Green Ribbon

    We currently have to export out the contacts then run a random number in excel and re-import. It's a basic function in almost every other ESP so it doesn't reflect well on Eloqua that we are still waiting on this to be available within campaigns.

    Understand that focus is on the new Program Canvas but it gets frustrating when the updates to Eloqua are so small at the moment. Would save a huge amount of time and effort so interested to see where this goes.

  • James Solo
    James Solo Posts: 16 Bronze Badge

    I would love to see Eloqua have the same degree of A/B testing for multi-step Campaign Canvas that Responsys does.  It also supports multivariate testing.  This is a feature that Oracle/Eloqua is very far behind its peers in providing.  Most email vendors provide A/B or Multivariate testing today (especially SF/ExactTarget).

    @kurtstoll Perhaps we'll start to see some of these capabilities in Eloqua emails/LPs soon from the recent Maxymiser acquisition?

    https://www.maxymiser.com/

  • kurtstoll
    kurtstoll Posts: 20 Bronze Badge

    @kurtstoll Perhaps we'll start to see some of these capabilities in Eloqua emails/LPs soon from the recent Maxymiser acquisition?

    https://www.maxymiser.com/

    Possibly on Eloqua landing pages, but doubtful on emails, because Maxymiser only works with websites today (not emails).

  • Although not ideal, if you create the shared filter (on eloqua id for 50/50 groups) as explained here ;

    You can 'a/b' test in a multistep campaign. It would require a lot of manual work and deciding yourself when to select the winner...time consuming to say the least.

  • David Gutelius
    David Gutelius Posts: 34 Bronze Badge

    For those following this thread, we're in process of building an app to do exactly this as well as using machine learning to segment targets and suggest campaign steps -- at large scale.  Think of it as massively parallel multivariate experimentation that automatically learns what works for whom, and optimizes over time. We're gathering feedback from users now on exactly how you'd like to see this work.  Ping me if you're interested in participating in a design interview, or just want to know more.  Would love to chat.

    cc @rebecca.vilcsek

  • Dominick Gaglio
    Dominick Gaglio Posts: 5 Silver Badge

    You can easily run an A/B test from the Campaign Canvas by moving the segment immediately out to Program Builder, do the random split there using the standard A/B test segment splitting function, and then sending your randomly-split groups back to your Campaign. Send the test emails, pick a winner, and send the winning email to the majority group. Sounds cumbersome but it's actually very simple and the only manual part is to run a Campaign report to pick your ultimate winner. Not perfect but better than using filters to build (not-so) random splits and definitely better than using spreadsheets.

    Hi @kkandler - I think your solution is a great one until a new release that includes this functionality on the canvas.  Would you please elaborate a bit on how you set up the program builder to run A/B tests?  I will probably start using this method.

  • louise.risoli
    louise.risoli Posts: 6 Blue Ribbon

    Great idea! Would be amazing to have AB functionality available in multi-step campaigns. As I workaround for a recent nurture campaign touchpoint, I did the AB simple campaign outside of my nurture flow (using a segment comprising a filter to pull in all members of my nurture campaign stage), but then back in my nurture flow I added two "Sent Email?" decision steps to cover each of the AB emails to filter out those included in the test run and then I routed the remainder to the winning email, I had the box in the AB test to "automatically send the winning email" deselected. This meant I was able to keep all but the test group send within my nurture flow canvas.  A fairly simple work around but a compromise all the same, would love to see this on the multi-step campaign. 

  • Great idea! Would be amazing to have AB functionality available in multi-step campaigns. As I workaround for a recent nurture campaign touchpoint, I did the AB simple campaign outside of my nurture flow (using a segment comprising a filter to pull in all members of my nurture campaign stage), but then back in my nurture flow I added two "Sent Email?" decision steps to cover each of the AB emails to filter out those included in the test run and then I routed the remainder to the winning email, I had the box in the AB test to "automatically send the winning email" deselected. This meant I was able to keep all but the test group send within my nurture flow canvas.  A fairly simple work around but a compromise all the same, would love to see this on the multi-step campaign. 

    Hi Louise.

    Do you have a picture from your canvas to show how you did that?

    It would be nice to test it.

  • kurtstoll
    kurtstoll Posts: 20 Bronze Badge

    For those following this thread, we're in process of building an app to do exactly this as well as using machine learning to segment targets and suggest campaign steps -- at large scale.  Think of it as massively parallel multivariate experimentation that automatically learns what works for whom, and optimizes over time. We're gathering feedback from users now on exactly how you'd like to see this work.  Ping me if you're interested in participating in a design interview, or just want to know more.  Would love to chat.

    cc @rebecca.vilcsek

    Hi David,

    We might be interested in participating in a design interview and would love to know more.  What is the best way to connect?

  • David Gutelius
    David Gutelius Posts: 34 Bronze Badge

    Hi David,

    We might be interested in participating in a design interview and would love to know more.  What is the best way to connect?

    Hey Kurt - Just sent you a PM.  You can also just ping me at [email protected].

  • David Gutelius
    David Gutelius Posts: 34 Bronze Badge

    Great idea! Would be amazing to have AB functionality available in multi-step campaigns. As I workaround for a recent nurture campaign touchpoint, I did the AB simple campaign outside of my nurture flow (using a segment comprising a filter to pull in all members of my nurture campaign stage), but then back in my nurture flow I added two "Sent Email?" decision steps to cover each of the AB emails to filter out those included in the test run and then I routed the remainder to the winning email, I had the box in the AB test to "automatically send the winning email" deselected. This meant I was able to keep all but the test group send within my nurture flow canvas.  A fairly simple work around but a compromise all the same, would love to see this on the multi-step campaign. 

    Cool technique, Louise.  Can I get in touch about something related?  @louise.risoli

  • andThyHolmes
    andThyHolmes Posts: 2 Green Ribbon

    Another way to do A/B testing in a multi-step campaign is to set up a shared filter and add it to the canvas.

    the shared filter should look at the contact id and any contact ending in 0, 2, 4, 6, 8, (Even) pass the filter and go through pipeline. Any contact with an ID ending in 1,3,5,7 (odd) go through the pipeline.

    It is not a perfect 50/50 split but if you have a large enough segment it is close to a perfect 50/50. On top of that once you set up shared filter you can add it to any multi step campaign in a matter of seconds

  • 3036016
    3036016 Posts: 2 Green Ribbon

    For those following this thread, we're in process of building an app to do exactly this as well as using machine learning to segment targets and suggest campaign steps -- at large scale.  Think of it as massively parallel multivariate experimentation that automatically learns what works for whom, and optimizes over time. We're gathering feedback from users now on exactly how you'd like to see this work.  Ping me if you're interested in participating in a design interview, or just want to know more.  Would love to chat.

    cc @rebecca.vilcsek

    Hi David - checking in on the app progress. Anything in market yet?

  • David Gutelius
    David Gutelius Posts: 34 Bronze Badge

    Hi David - checking in on the app progress. Anything in market yet?

    Hi there - Yes, we're making great progress. Ping me at the above address and I can tell you more. We're about to open up our Beta program to a few customers but it's working phenomenally well so far...

  • 3036016
    3036016 Posts: 2 Green Ribbon

    Hi there - Yes, we're making great progress. Ping me at the above address and I can tell you more. We're about to open up our Beta program to a few customers but it's working phenomenally well so far...

    Great! Danielle Brooks from my team will be following up...

  • David Gutelius
    David Gutelius Posts: 34 Bronze Badge

    Great! Danielle Brooks from my team will be following up...

    Got it - thanks. Eager to learn more about your use case there and how you're thinking about campaign experimentation and optimization.

  • DeanQ.
    DeanQ. Posts: 1

    Great idea! Would be amazing to have AB functionality available in multi-step campaigns. As I workaround for a recent nurture campaign touchpoint, I did the AB simple campaign outside of my nurture flow (using a segment comprising a filter to pull in all members of my nurture campaign stage), but then back in my nurture flow I added two "Sent Email?" decision steps to cover each of the AB emails to filter out those included in the test run and then I routed the remainder to the winning email, I had the box in the AB test to "automatically send the winning email" deselected. This meant I was able to keep all but the test group send within my nurture flow canvas.  A fairly simple work around but a compromise all the same, would love to see this on the multi-step campaign. 

    Hi Louise,

    Would love to see how you routed the remainder of your segment to the winning email. Is there a screenshot available of how that was achieved?

  • louise.risoli
    louise.risoli Posts: 6 Blue Ribbon

    Hi Louise,

    Would love to see how you routed the remainder of your segment to the winning email. Is there a screenshot available of how that was achieved?

    Hi just revisited this conversation, someone else asked me to share a screenshot too (sorry about that). See below... this is the part of my flow that I built out to incorporate the AB test.  Whilst the AB Test itself was performed outside of the canvas I sent the winning version through my mutli-step flow.  To replicate this, within your AB test (in section 4) make sure the box to "automatically send the winning version" is NOT selected. By adding these steps and taking the send of the most successful version back into the multi-step flow, my overall campaign reporting was only marginally impacted by the AB Test which happens outside the flow.

    Hope this is helpful.  Any questions please let me know.

    pastedImage_1.png

  • louise.risoli
    louise.risoli Posts: 6 Blue Ribbon

    I really like this, but in this case I think you are sending to all contacts but with a 50/50 split so I guess to test different styles to implement in later emails?
    Would you use this filter approach if you want to do AB test using say 20% of your audience and split the 20% 50/50 to determine the which is the best performing email to send to the other 80%?

    Maybe build a smaller list and have "Shared List Member?" and then go through the filter, but then the split on a smaller list might be a bit skewed...

  • andThyHolmes
    andThyHolmes Posts: 2 Green Ribbon

    I really like this, but in this case I think you are sending to all contacts but with a 50/50 split so I guess to test different styles to implement in later emails?
    Would you use this filter approach if you want to do AB test using say 20% of your audience and split the 20% 50/50 to determine the which is the best performing email to send to the other 80%?

    Maybe build a smaller list and have "Shared List Member?" and then go through the filter, but then the split on a smaller list might be a bit skewed...

    In order to test 20% you can add a filter above the 50/50 filter. Since we will be using the last digit in the contacts ID for the 50/50 split we cannot use the same logic for the 20% split. Instead you can use the 2nd to last digit in the contacts id. To get 20% you would set the rule so contact id ending in 1? or 2?. The ? is a wildcard which is equal to any one digit. This allows us to look at the tens digit without having to specify what the singles digit should be.

    I am not sure how or if you can set up the campaign to automatically send the remaining 80% to the winning email. Typically I would just review the results for the A/B test and adjust the campaign accordingly to ensure the remaining contacts to going to the proper version.

  • HenrikD
    HenrikD Posts: 1

    @Taha any news or best practices you can share on performing automated A/B testing in a multi-step campaign?

  • claudiazoglmann
    claudiazoglmann Posts: 79 Silver Badge
    edited August 2016

    Hi Claudia,

    All use cases from Program Builder will be supported on the Program Canvas (next gen PB), but not necessarily in the same manner.  As an example, rather than cloud connectors, the program canvas will leverage our new AppCloud Developer Framework and associated audience, action and decision apps for more use case based integrations e.g. data cleansing or even CRM integration.  This is another key difference, where instead of autosynchs, external calls, etc. CRM integrations will be app based allowing us to iterate on functionality outside of the core Eloqua platform and also take advantage of new versions of a CRM providers APIs.  We'll have lots more details to share as we get closer to releasing this capability, but you can expect to start hearing some information sooner as we start beta testing. 

    Hope this helps.

    Taha

    Hi @Taha Sheikh-Oracle,

    With the upcoming deprecation of Cloud Connectors in Program Builder, I suppose, a lot of customers will look further into Program Canvas. Are you able to share Oracle's roadmap for the Canvas? The current capabilities seem very limited still.
    Thanks.

    Post edited by Unknown User on
Sign In or Register to comment.