Behaviour of Incremental Dataflow with "save data to" option set to "db connection"

Question

Content

Hello,

I've created an incremental Dataflow in OAC to over come the problem of "huge record set" in my datasource. My datasource has more than 10 lakhs of records and dataflow processing this dataset fails. To overcome this issue I am using incremental dataflow for which I've set, "save data to" option to "db connection".

Purpose of using this dataflow is to "incrementally add data to existing data" in the db table. I am seeing inconsistent behaviour with this dataflow. Please let me know, if some one has used "incremental dataflow"?. If yes, I would like to understand, the reliability of using it.

I am facing below challenges with incremental dataflow:

1.The physical table, I have created in the database (used with "save data to" option in the dataflow) is not persistent.  First time I run the dataflow, all records in the source table are copied to the physical table (in "save data to" option).

2.Second time, when I run the dataflow, I get the error "table or view does not exist".

Please help!

Thanks in advance,

Chethana

Chethana Kumari-Oracle · Answer

Hey Philipp,

Yes, I found a solution for this.

Let me tell you, how I addressed my challenges

1. The physical table, I have created in the database (used with "save data to" option in the dataflow) is not persistent.  First time I run the dataflow, all records in the source table are copied to the physical table (in "save data to" option) - I created this table manually in DB (earlier this was created by OAC by simply mentioning the name of the table in "Save Data" step in the dataflow)

2.Second time, when I run the dataflow, I get the error "table or view does not exist" - There is no error and the table gets updated with the incremental data

Regards,

Chethana

Philipp Kaufmann-Oracle · Answer

Hi Chethana,

Did you get any feedback on this or found a solution? I would be very interested to hear how you work with the incremental feature of data flows as I've been struggling with it as well in the past.

Kind regards,

Philipp