How to design a Java desktop application for processing a file with a size of up to 100GB?

User_I6DXR

A requirement of a standalone Java application is that it shall be able to process a data file with a size of up to 100 GB. The file contains single one line of data. The data type is the IBM dataset. And the same data can be processed multiple times with various parameters.

The challenge is that I can't load all of the data into memory. That occurs at different stages. Reading the data from a file is one. Presenting the processed data on the GUI (JavaFX) is another. And there are one or two in between.

I am thinking of using an embedded database, such as MongoDB to hold the data. How shall I build such an app to meet the requirement?

justsomeone

Hello

not sure if that what you looking for or not

https://stackoverflow.com/questions/49227476/sequential-read-of-a-mainframe-dataset-using-java-program

https://www.ibm.com/docs/en/cloud-paks/z-modernization-stack/2022.3?topic=jzos-recordreader

hope that help and have a nice day :)

User_I6DXR

Thanks very much for the info. I hadn't seen it yet. The question is about how to process the type of data file.

One of the significant issues I run into is large file size

justsomeone

you very welcome

i did not used ibm dataset before so not sure if there away to convert it to database file or not but the read method it will read single record per time

https://www.ibm.com/docs/api/v1/content/SSV97FN_2022.1.1/java/com.ibm.java.zsecurity.api.80.doc/com.ibm.jzos/com/ibm/jzos/RecordReader.html#read-byte:A-

so if you did not have better way then you would read each record then connect to the database you like then insert it there

things to not forget if that dateset file updated by something else which will make you rebuild the database again unless it just append new record

and have a nice day :)

justsomeone

another way to import it into database

first if the dataset file is plain text then you can use

mysqlimport which work for both mysql database and mariadatabase

https://dev.mysql.com/doc/refman/8.0/en/mysqlimport.html

https://mariadb.com/kb/en/mysqlimport/

i guess MongoDB would have something like that but try to which type it support

and have a nice day :)