Data Flow - Add Columns

John Ely

Summary

Data Flow - Add Columns

Content

I'm sourcing a CSV file  (3Gb in size) with around 22million records in it (about 50 columns) and simply creating a new column called "1617".  I'm populating this new column by creating a calculation based on another column in the file "TOTAL COST" and multiplying the values by 1.1. 

 

I've validated the calculation and hit apply.  I execute the query and after about 25mins it just comes back with an error:

 

"Data Flow has produced an error" and that's it - no other details provided.

 

I'm running ODV desktop 12.2.4.2.0-20180220134543  using a windows laptop with 32Gb of RAM and 30Gb of free disk space.

 

Any ideas? Is there a limitation on the number of records that I can process?

John Ely

Many thanks for the response, its really useful to know. In effect we're saying that it has the same limitations as far as rows are concerned as Excel. 

However, if I connect to a live Oracle DB table (e.g. 12C db) - are you saying there is no restriction on rows ?

Luke Sharma

Hi there,

I believe that there are some limits on data:

I think there is also a limit on the number of columns that one can use in data visualizer.

Luke Sharma

No problem.

I haven't really tested this rigorously, but from the looks of documentation then there should be no limit on rows on a live DB connection. Obviously cached "non-live" data DB connections will have some limitation - but live DB connection doesn't appear to!