6 Replies Latest reply: May 28, 2010 1:44 PM by DrClap RSS

    HasMap & ArrayList memory issue

    843793
      Hey Forum,

      I have a somewhat tricky problem. The program I'm writing has several storage classes (they only store data like strings and such for the user to view) that I keep in memory. I use an custom class with an ArrayList that implements a TableModel interface to make them available. On top of this I have a HashMap that stores <long, storage class> combinations. The long is the unique ID that is generated for each instance of a storage class. This setup gives me an O(1) lookup in the table using the ArrayList and an O(1) lookup for each storage class instance by UID using the HashMap. So far so good.

      Here's my problem; all this stuff is stored in memory. So if the user has a large project these heavy storage classes fill up the memory. What I would like to do is store the storage instances in a separate file and read them out when required. Its relatively easy to accomplish if I only had an ArrayList or only a HashMap but since I have both I'm having some difficulty figuring out how to implement this. I want to maintain O(1) lookup in the table and O(1) lookup in the HashMap and store the objects in separate files and only store each instance once (the ArrayList and HashMap need to point to the same file).

      Ideas?
      Jurgen

      PS. I was thinking storing a list of UIDs in the ArrayList and writing a custom HashMap that overrides the get and put methods. The ArrayList would then retrieve the objects from the map and the map would read and write to file. Is this the best option?
        • 1. Re: HasMap & ArrayList memory issue
          843793
          Oh yeah, small addition.

          I also want to be able to allow the user to filter the data that is shown in the JTable (I'm assuming a RowFilter will do this) as well as put a limit on the amount of records that are shown at any given time. So if the user selects a filter I will show the first X records that match that filter.

          Does this still work with RowFilter? Do I need 2 row filters or...?
          • 2. Re: HasMap & ArrayList memory issue
            843793
            Make a facade to the storage class that transparently handles reading the data from the file, if it is not already loaded (via a Proxy, for example). Then use that proxy as the value in the Map and the List.
            • 3. Re: HasMap & ArrayList memory issue
              843793
              I'm not sure that would work well. All the storage instances are created during a single run so they are already loaded in memory. Lazy loading would then not work. I guess I could somehow trigger this proxy to write to disk and clean its data but I don't know when this is supposed to happen. Is there a way I can trigger an event when the VM tries to garbage collect?
              • 4. Re: HasMap & ArrayList memory issue
                DrClap
                Rival wrote:
                Is there a way I can trigger an event when the VM tries to garbage collect?
                Sort of; have a look at the java.lang.ref package and see if anything there helps you out.

                There's also a WeakHashMap class, but it has weak keys and you want weak values (I think).
                • 5. Re: HasMap & ArrayList memory issue
                  843793
                  In the mean time I have more or less implemented my original idea. Initially I thought I could use Object.finalize() to write out the data. I would simply store everything in a WeakHashMap and let the garbage collector get rid of everything, then when the objects are about to be destroyed I would write them to disk. Since the GC would only finalize objects that aren't being used anymore I wouldn't have to worry about writing objects to disk that were still being modified. And since the GC triggers itself when I have too much memory use it would only run when needed and not on small projects with only a few objects.. It sounded great in my mind. It just didn't work so well.
                  The GC would trigger the objects but only after the memory was really filled up and then it would try to write everything to disk at once. That basically froze the program completely and then after a while it would magically continue as if nothing had happened. And reading files from disk while they are being garbage collected created some weird results, which necessitated synchronizing everything. Which slowed down the garbage collection even further.
                  So I scraped the idea. Now I just keep track of the objects and write them myself. I don't know whether this big map class that I've written actually works. The code compiles and files are written and read from disk and the program still functions and is pretty fast with small projects. But I haven't tested it with large projects yet, which was kinda the point of all this. Lets hope. :)
                  I really hope the WeakHashMap works the way I thought it works, if for some reason the values stay alive even though the keys are gone I'm still nowhere.
                  • 6. Re: HasMap & ArrayList memory issue
                    DrClap
                    Rival wrote:
                    I really hope the WeakHashMap works the way I thought it works, if for some reason the values stay alive even though the keys are gone I'm still nowhere.
                    Indeed, you will have code which accesses the WeakHashMap and gets references to its values. You can't avoid that. Just don't let that code hang on to those references, because if it does than (as you suggest) you're SOL.