Maximum data set size / overall data storage in DV

jhall_uk_redux

Summary

What is the maximum data set size you can load into DV?

Content

Hello experts,

Is there an upper limit on the size of an individual data set in DV? - either in terms of MB size or number rows size? What happens if the source connection attempts to load a file or run a query from relational database that is too large? Will the query/load fail, timeout or just restrict the number of records returned?

The follow on to this is if there is a maximum number of Connections or an overall upper limit of all data stored in DV across all data sets loaded?

Is any of this configurable?

Does this differ between DVD and DV within OAC?

Is any of this documented anywhere? If so, please can you send a link.

cheers,

John

Philippe Lions-Oracle

Hi John

the documentation here https://docs.oracle.com/en/cloud/paas/analytics-cloud/acubi/adding-spreadsheets-data-sets-acubi.html#GUID-3F17CDCE-B80A-44C7-98EA-F1CEF3220A9F indicates a limit for size of uploaded files (100MB), but this only means file uploads are limited, not datasets. The size of the actual dataset, irrespective of it's source (file, DB,...), only influences if the system can 'cache' it or not. Datasets that point to database tables can be very larger , OAC will recognize this based on size/# of rows and simply not cache them in memory if they are too large. Instead it will send live queries for every requests. Note that results of queries can be cached subsequently, but that's a different level of caching.

In OAC, the cache max size can be configured (I believe config.ini file), as it depends on the HW ressources. However this configuration is managed by Oracle if using Autonomous OAC, I am not sure exactly what its value is right now, it depends on the selected shape of the service (# of OCPUs). There is also another limit which is the total size of datasets per users. I believe this is 20GB all together. You can monitor this by clicking the Dataset-storage option in the top right menu, see the picture below.

This is not configurable in DVD.

hope that helps

Philippe

jhall_uk_redux

Thanks Philippe - this is very helpful.

If the data set storage quota is defined per user I'm guessing this only affects local data sets generated/loaded from files in DV - live connections to a database source has no impact on data store quota size, even if the source tables loaded are larger than a user's quota. Is that correct?

How does this memory caching work alongside the caching options within a Project? If you do not select "refresh data" in the project does it affect anything relating to what data can be queried from a database? It sounds like live database connections will be unaffected by this Project level cache. Have I understood this correctly?

Finally, does this apply to connections to OTBI in Fusion Cloud (Oracle Applications connection) too? Will the queries to OTBI get reissued each time a Project  page is opened or refreshed - or will the data be loaded /cached once per user until you choose to click on Refresh Data within a Project?

Perhaps the behaviour is different between DVD and DV in OAC.

Cheers,

John

 

Philippe Lions-Oracle

Hi John,

...live connections to a database source has no impact on data store quota size, even if the source tables loaded are larger than a user's quota. Is that correct?

When the datasource from the DB table is set to Automatic Caching (ie not explicitly Live), then if the DB table size is below the limit of single XSA-cache (DSS or XSA = dataset caching) entries size defined in config.ini (either size or # of rows), the system will cache it. I am not sure if this will show-up consuming the 20G quota but it may. If the DB table size is above these limits, it will automatically consider it as live and will not create an XSA cache entry. So no impact in this case.

...How does this memory caching work alongside the caching options within a Project ?

When the dataset is XSA cached, the dataset needs to be actively re-loaded when data changes (Reload Data or Refresh Datasets). Once the XSA data is refreshed, you may sometimes be in a position to need to refresh a project canvas with the latest data, that's the Refresh Data menu option, this does not reload the XSAs. If your DV query is hitting an RPD subject area then other BI Server Cache may kick in, but that's a different topic. I believe live database connections should only be affected by the Refresh Data option (re-sending the SQL to DB)

I am not sure about the OTBI connections (native or via SA connection) as it depends on the version you may be using. Most Likely the data will be cached on OAC via these connections and may need a refresh. The replication feature in DV will allow you to deepen this functional process.

This should be the same behavior in OAC (IE DV), DVD, and On Prem, except that in some cases you can influence the configuration of the limits, on some cases you may not.

Philippe