Forum Stats

  • 3,874,333 Users
  • 2,266,724 Discussions


How to design a Java desktop application for processing a file with a size of up to 100GB?

User_I6DXR Member Posts: 3 Green Ribbon

A requirement of a standalone Java application is that it shall be able to process a data file with a size of up to 100 GB. The file contains single one line of data. The data type is the IBM dataset. And the same data can be processed multiple times with various parameters.

The challenge is that I can't load all of the data into memory. That occurs at different stages. Reading the data from a file is one. Presenting the processed data on the GUI (JavaFX) is another. And there are one or two in between.

I am thinking of using an embedded database, such as MongoDB to hold the data. How shall I build such an app to meet the requirement?