I came across a rather annoying problem while trying to load data into OEDQ today and wondered if anyone can offer any insight.
2 separate installations (OEDQ 9.0.5) trying to load 2 different data loads (120k records and 35k records) both produced java heap errors while trying to create the datastore connection.
Following the various support articles regarding the java memory settings and systematically "playing" with the various values for min/max and reserved memory setting from dnconfig made no difference, the same java heap error every time.
I then noticed that the 2 data sources were both in the newer .xlsx format having been extracted to excel and then saved, I opened them and re-saved as first .csv and then .xls -- in both cases the data load now worked.
Given that more people will be moving to the newer xlsx format as "default" I wondered if anyone has any ideas on configuration to deal with what appears to be an added overhead when dealing with this format of files? we can of course just tell people not to use it but it would be nice to know why they cant.
The current Excel driver works by pulling the whole of the spreadsheet into memory before capturing it. This can mean that with large spreadsheets it is possible not to have enough heap memory allocated to perform this function. This does not normally happen if using server-side data store connections as the server normally has considerably more memory allocated to it than the client application. If you do encounter this and cannot overcome it by adjusting the relevant heap memory settings, the workaround is as as you say to save as a CSV or other text files which can be streamed in (and out).
Probably the reason that xlsx files fail where xls files succeed is that the xlsx save creates a larger file.
Edited by: MikeMatthews on Nov 6, 2012 3:20 AM