Enhancement Request: Full Historical Data Extraction for Payroll Balance/Costing Pipeline

Anurag Umathe

The current implementation of the Payroll Balance Pipeline's full data load functionality limits data extraction to the three most recent years, regardless of an earlier date set in the "Initial Extract Date" parameter.

This behavior presents a significant operational challenge. Following mandatory Data Warehouse resets in our lower environments, we require the ability to reload the complete history of payroll and costing data from the HCM source to ensure data integrity. The current three-year limitation makes this impossible.

This functionality is also critical for addressing business needs that require older data, such as:

• Investigating and resolving historical payroll discrepancies that fall outside the three-year window.
• Fulfilling legal and compliance obligations that mandate access to a more extensive data history.

To resolve this, we propose that the pipeline be modified to honor the "Initial Extract Date" for all full data extractions. As an alternative, we suggest the addition of a new parameter that allows users to define the number of historical years to be imported.

Expect this functionality in both Non-PROD and PROD FDI environments.