Adding Execution Details for Data Pipeline Runs

Vamsy Mohan

Adding Execution Details for Data Pipeline Runs :

I definitely feel capturing execution timing details like start time, end time , execution duration for Data Pipelines is very crucial for monitoring as well as in decision making on the incremental schedules.

Now with the latest feature to have Frequent Refresh for Module-Area/tables, We can only implement/recommend this feature when we have an understanding on time taken for various Data Pipelines Processes like WAREHOUSE_REFRESH_SCHEDULED, WAREHOUSE_REFRESH_ONBOARDING, WAREHOUSE_REFRESH_ADHOC, OTHERS.

The pipeline run timings will allow us to take decisions if more frequent runs can be scheduled for more real time data.

could you please review and add this feature.

I think we can leverage the existing DW_WH_REFRESH_SUMMARY and DW_WH_REFRESH_DETAIL tables for the same.