Exporting All Eloqua Activities Using the Bulk API: Flowchart, Best Practices, and Resources

Lou Patrick

Original Blog Post Date: Apr 29, 2016 6:12:16 PM

Before diving into the flowchart for exporting all Eloqua activities using the the Bulk API, let's cover some best practices using the Bulk API focused on exporting activities:

• Use filters to ensure fewer than 5,000,000 records are exported per Bulk API sync
  • A common method of filtering is to use the Activity Date to export a finite time period of activity, such as one month (use larger or smaller time periods based on volume of activity generation)
• Export one Activity Type at a time
  • Do not run syncs for all Activity Types simultaneously
• Make multiple export requests sequentially
  • Allow the sync to finish before executing the next sync

Here are the main resources related to exporting activities using the Bulk API on the Developer Help Center:

• Getting Started
• Exporting Activities
• Activities Export Definitions
• Activity Fields by Activity Type
• Activities Endpoints
• Syncs Endpoints

Exporting All Eloqua Activities Using the Bulk API Flowchart

(click to enlarge)

Notes and Resources by Flowchart Step

Determine Bulk API Base URL and Select Authentication

• Determining base URLs
• Authentication:
  • Basic API authentication
  • Secure API authentication using OAuth2

Select Activity Type to Export

• Activity Types:
  • EmailOpen
  • EmailClickthrough
  • EmailSend
  • Subscribe
  • Unsubscribe
  • Bounceback
  • WebVisit
  • PageView
  • FormSubmit

Create an Activity Export Definition for the selected Activity Type

• (Tutorial) Exporting Activities - Define the Export
• Example export definition requests for all Activity Types - Activities export definitions
• List of Activity Fields per Activity Type - Activity Fields

Create the Sync with the Activity Export Definition URI

• (Tutorial) Exporting Activities - Create the Sync
• Check the Sync status using the GET /syncs/{id} endpoint

Retrieve Sync Logs with the Sync URI's Logs Endpoint

• Syncs Logs Endpoint
• If exporting to CSV:
  • The total number of records returned can only be retrieved using the Syncs Logs Endpoint
• If exporting to JSON:
  • When retrieving data with the Sync URI's Data Endpoint in JSON, the following parameters are returned along with the data, either of which could be used to determine how many calls are necessary to retrieve the remaining data:
    • totalResults
    • hasMore
• When retrieving the count of total records exported from the Syncs Logs endpoint, it's important to note the "message" within that item will be "Successfully exported members to csv file."
• Example Syncs Logs Response with items "count" and "message" highlighted:

             

Retrieve the Data with the Sync URI's Data Endpoint

• (Tutorial) Exporting Activities - Retrieve the Data
• Syncs Data Endpoint
• Example with URL parameters: GET /syncs/{id}/data?limit=50000&offset=0
• Data returned in JSON unless Accept header specifics CSV (API Call Format)
  • Example: Accept: text/csv
• A validation check could be added here. In example, a custom Insight report, looking for volume comparison within 10%.

Retrieve the Remaining Data with the Sync URI's Data Endpoint Using Offset

• Example with URL parameters: GET /syncs/{id}/data?limit=50000&offset=50000
• Example calls to retrieve all data if there were exactly 150,000 activity records:
  • GET /syncs/{id}/data?limit=50000&offset=0 (This is part of "Retrieve the Data with the Sync URI's Data Endpoint" step)
  • GET /syncs/{id}/data?limit=50000&offset=50000
  • GET /syncs/{id}/data?limit=50000&offset=100000
• Data returned in JSON unless Accept header specifics CSV (API Call Format)
  • Example: Accept: text/csv
• A validation check could also be added here.

Create an Activity Export Definition Filtering on Activity Created Date

• Activity volume can be checked using Insight reports
• (Tutorial) Exporting Activities - Define the Export (Includes example time based filter)
• (Tutorial) Filtering