You can give more memory to Data Modeler - you need to change in datamodeler64.conf
Change 768 to 4000 or to value that works for you.
And there is DM v4.0 production.
Are there any errors logged in your log file? This is normally file datamodeler.log in folder datamodeler/datamodeler/log.