We are currently using BICC to extract FSCM, CRM, etc. data to build a data warehouse for analytics. Noticed that BICC Application Data Extracts only captures updated and new records, but doesn't capture deletes. Full table loads is not a sustainable solution as the data continues to grow.
We figured that Active Primary key extracts, will provide the list of all primary keys in a PVO. Using this, we are performing soft deletes in our warehouse.
- However we wanted to understand if there are any other approaches to identify deletes? (was trying to check if there's any column that can indicate if a row is deleted, but couldn't identify one)
- Also, what is the recommended frequency to extract the active primary keys? We see that BICC offers a minimum of 1 hours. But, we were planning to use SOAP APIs to schedule the jobs more frequently. Is it okay to extract Active primary keys every 15 mins/every hour? What's the advisable frequency here?
