Sequences should support Data Replication taks

Gianni Ceresa

(OAC November 2023)

In "Data" you can now add to a sequence data flows, datasets and other sequences.

It is not possible to add Data Replication tasks.

It is fairly annoying because often there are multiple data replication tasks that should run one after the other (to not run them in parallel and not overload the system for that).

Currently it's easy to schedule a sequence to execute, serially, multiple data tasks. While Data Replication tasks require a schedule for each and every single one.

It would just make sense to make it consistent and extend the sequences to support data replication.

Bret Grinslade - Oracle Analytics-Oracle

In general, it is better to not have multiple sets of jobs for data replication but to have them run as one set. What is the scenario for sequencing them?

Gianni Ceresa

Hi Bret,

The situation I have in mind (that I'm facing right now) is for example a customer who has been "moved" to OAC by Oracle Lift: they needed to replicate 8 PVOs and they created 8 replication jobs scheduled at 1 min of interval from others with daily execution.

Now the customer asked if they could run the replication manually during the day for special cases (months closing etc.): they can, but have to execute 8 different replications. Because they take what Oracle Lift did setup as "the right way to handle it", I was looking if at least these 8 could be added in a sequence so they only have a single thing to run.

And another use case for this is when you don't have an ETL but desperately need one: data replication load data in the database, but the format and shape means that workbooks will take 2-3 minutes to load and display something, and up to 15 minutes for more advanced tasks because queries are just extremely bad looking (all the business logic is added in queries). To make it "faster" a view has been created in the database applying all the business logic and turning data in a dimensional-like format. A data flow in OAC is used to read the view (a dataset in OAC) and save it to another dataset materialized in the database.

All these flows doing "ETL-like" activities are in a sequence, but they can't be chained with the data replication because this one can't be added in the sequence as the first task to execute. It's all again scheduled hoping that replication will not run longer than X minutes, or the "ETL" sequence will miss the data and materialize old data only.

Definitely not really what you designed the tool for, but in some cases when all you have is OAC and a database and don't want to start creating a PL/SQL ETL process with database scheduled tasks, OAC become an ETL as well.

Being able to have a sequence running all the tasks needed in sequence mode (with the right order to handle dependencies), it's a key requirement. Being able to add any "data" related operation in sequence will allow to make simple ETL like that, without losing a global vision of what is going on with the data (because everything will be driven by OAC, with logs in there as well).