Partitioning of Configurable Account Analysis

Sreedhar Mohan

Problem Description:

Currently Lowe's Configurable Account Analysis (CAA) table contain over 900 million rows just 8 months after go-live. Our Daily Incremental and Frequent Data Refreshes are running longer than the run-times when compared during go-live. As per CEAL team recommendation, we reduced the initial extract date to be 4 months old, but still we see over 360 million rows in CAA.

Proposal:

This Idea request is to implement partition logic as per period-year and period-month.

Or create individual table for each period and create a view unioning all the sub-tables for each period.

Benefits:

1. This way, CAA would become slick and we can even set the initial extract date as just one period start-date, so that we can restrict to just one-period.
2. This might help contain the longer running data-pipeline issues also.