This might be a useful addition, and should be easily implemented using existing mechanisms.
1) A NotifyReference would be a strong reference to an object
2) It would always have a ReferenceQueue declared
3) At the point where the JVM determines that memory is required, for any objects reachable only through a NotifyReference that reference would be enqueued to the ReferenceQueue (if not already present)
Enqueing a NotifyReference allows the JVM to indicate to the application that free memory is dwindling.
A well-behaved application would monitor the ReferenceQueue, can make application specific decisions on which of multiple entries to sacrifice, has the full context of the application to perform any necessary cleanup on the selected victims (as opposed to the somewhat sketchy context of a finalizer), and can then remove the victim from the ReferenceQueue and clear it -- so that the target object can now be reclaimed by the JVM. Also, by extending a NotifyReference, the application can easily associate additional information in order to make the choice of victim or cleanup easier (e.g. the computational or I/O cost required to re-create the object if needed again).
The mechanism for a ReferenceQueue is already supported within the JVM, and is required for any PhantomReference.
Probably the easiest time to decide to enque a NofiyReference is at the same point that the JVM decides to clear and reclaim one or more SoftReference instances. Note however that the use of a NotifyReference does introduce a delay -- the application must monitor the ReferenceQueue, clear one or more entries, and the memory will not be reclaimed until the next execution of the GC (with the possible exception of Direct buffers, see below). For this reason, the JVM may decide to post a NotifyReference while there is still more free memory available than would trigger the clearing of a SoftReference.
Essentially, this is just a PhantomReference where the get method still returns the target instance, and it is not enqueued until the JVM needs more memory. The main benefit is to provide the application with the information that free memory is running low, but the JVM has not yet taken any action on these instances -- so the application can decide what should be released. Of course, the downside is that a poorly behaved application does not monitor the ReferenceQueue with an OutOfMemoryException eventually being thrown.
Application to Direct Buffers
There have been a fair number of postings (e.g. see StackOverflow) about handling of the Direct Buffers introduced in the java.nio package.
The general issue is that Direct buffers are not freed directly by the Garbage Collector (since they are not allocated in heap memory). Instead, the actual memory is released when the 'finalizer' is called, which may be delayed for quite some time. (See memory leaks - how to garbage collect a direct buffer java - Stack Overflow) The application may suffer an OOM condition before this memory is released.
If references to Direct buffers were encapsulated in a NotifyReference, then when there are no other references this NotifyReference would be enqueued, and the application would have the opportunity to release the underlying memory. This may result in the release of the memory sooner than it would be by waiting for the finalizer to be executed. Of course, the application must be well behaved and monitor this queue closely.
10/18/13 - Renamed AskMeReference as a NotifyReference. Minor text edits. Added the section about using a NotifyReference to address the problem of quickly releasing memory when using Direct buffers.