I originally wrote this as one single blog, but as I found out with my previous BufferedImage articles, I tend to go on much longer than I inteded at first, so I had to split it up. So now there's 2 parts to this blog: last time's VolatileImage usage discussion, and this week's VolatileImage FAQ. This week, I thought I'd write down various reasonable questions about either VolatileImage objects and usage in general, or questions that might arise from last week's example. These are questions that come up all the time, or that might reasonably surface from the discussion so far. If you have questions of your own, feel free to post them in the talk-back section at the end.

First, let's see that sample code again:

import java.awt.Color;
import java.awt.Component;
import java.awt.Frame;
import java.awt.Graphics;
import java.awt.image.VolatileImage;

public class VImageDemo extends Component {
    
    VolatileImage backBuffer = null;

    void createBackBuffer() {
    if (backBuffer != null) {
        backBuffer.flush();
        backBuffer = null;
    }
    backBuffer = createVolatileImage(getWidth(), getHeight());
    }

    public void paint(Graphics g) {
    if (backBuffer == null) {
        createBackBuffer();
    }
    do {
        // First, we validate the back buffer
        int valCode = backBuffer.validate(getGraphicsConfiguration());
        if (valCode == VolatileImage.IMAGE_RESTORED) {
        // This case is just here for illustration
        // purposes.  Since we are
        // recreating the contents of the back buffer
        // every time through this loop, we actually
        // do not need to do anything here to recreate
        // the contents.  If our VImage was an image that
        // we were going to be copying _from_, then we
        // would need to restore the contents at this point
        } else if (valCode == VolatileImage.IMAGE_INCOMPATIBLE) {
        createBackBuffer();
        }
        // Now we've handled validation, get on with the rendering

        //
        // rendering to the back buffer:
        Graphics gBB = backBuffer.getGraphics();
        gBB.setColor(Color.white);
        gBB.fillRect(0, 0, getWidth(), getHeight());
        gBB.setColor(Color.red);
        gBB.drawLine(0, 0, getWidth(), getHeight());

        // copy from the back buffer to the screen
        g.drawImage(backBuffer, 0, 0, this);

        // Now we are done; or are we?  Check contentsLost() and loop as necessary
    } while (backBuffer.contentsLost());
    }

    public static void main(String args[]) {
    Frame f = new Frame();
    f.setSize(500, 500);
    f.add(new VImageDemo());
    f.setVisible(true);
    }
}

And now, on with the questions:

Q: So when do I use VolatileImage objects? Should I use them instead of BufferedImages, or managed images?
A: Ah yes, the plethora of image types and the confusion that quickly sets in. The quick answer is "You probably do not need to ever use a VolatileImage object directly in your application." This answer assumes that most applications:

  • Are Swing apps, and therefore are already using a VolatileImage back buffer because Swing does it for them
  • Do not need or want the extra complexity inherent in using these images that must be managed.
  • Do not need every ounce of performance and runtime details available
So much for the quick answer. The long and much more descriptive and rewarding answer is "You might want to use VolatileImage objects sometimes." (Actually, this long answer is much shorter than the quick answer above, but you'll see that the underlying explanation behind this glib response is not so brief).

 

The main reason for anyone to ever use a VolatileImage object is when you need a dynamic (frequently updated) image that you would really like to be accelerated if possible, including bothrendering to the image and copying from the image. This is the basic usage model of a back buffer; you want it accelerated and you are rendering to the image frequently (at least once per screen update, by definition). This is why Swing uses a VolatileImage for its back buffer.

Using a BufferedImage or a managed image in this kind of scenario will instead give you software-rendering performance. Because of the volatile nature of accelerated memory (particularly on Windows), any image that is not explicitly managed for loss (which means any image that is not a VolatileImage) mustreside primarly in system memory. That way, any rendering operation to that image is captured in that system memory version and cannot be destroyed through the loss of an accelerated version of the image. This implies that all rendering to the image must occur first through software rendering loops (since we cannot use hardware acceleration to render to a system memory image). Then, if it makes sense, we may copy that image into accelerated memory for future copies from that image. There are two main problems with this for a back-buffer scenario:

  • Performance: for simple rendering, hardware acceleration will generally win out over software rendering performance. There are various reasons for this, including the speed of the graphics processors, the inherent parallelism of offloading operations to the graphics chip, etc. But in the end, just remember that's why they call it "accelerated". So if your back buffer resides in system memory (if you are not using a VolatileImage), then you must make do with software rendering performance.
  • Performance: Although the render-to performance is important for the back buffer, sometimes the bigger win comes in the simple copy-to-screen operation that back buffers do on every frame. Ideally, this copy happens between an accelerated back buffer living in VRAM to the screen, also living in VRAM, and can happen at amazing speeds due to the memory bandwidth in the graphics system. But if your back buffer lives in system memory, this copy must happen over the system bus: reading from system memory (hopefully the on-chip cache, at least) and copying to VRAM down the PCI or AGP bus. Imagine trying to render 60 frames per second in an animation application, with a screen size of 1280x1024 and 32bpp screen depth; that's about 5 megs of data per frame * 60 frames = 300 MBytes of data moving across the bus, just to copy the buffer to the screen, not including anything else that needs to happen over the bus or through the CPU/cache/memory. "But what if the back buffer is a managed image?" Unfortunately, this is no better than an un-managed image; because you are rendering to the back buffer every frame, there is no benefit to our copying that buffer down to a VRAM cached version; we'd have to do it every frame (since the contents of the primary version of the image are being updated every frame), which is no better than simply copying it directly to the screen every frame.


 

Q: Couldn't the contenstLost() rendering loop become an infinite loop?
A: No. At least, not in theory. validate() will attempt to allocate an accelerated surface. Failing that, it will allocate the surface memory in system memory. So when you get either an IMAGE_OK or and IMAGE_RESTORED, you can guarantee that there is somethingthere to render to (and from). And if you get an INCOMPATIBLE error, you must recreate the image completely, which again will result in at least something you can render to and from. So if nothing happens to that surface between that validate and the contentsLost(), then contentsLost() will return false and you can continue. The only situation I can foresee where you might get stuck in this loop more than one time around would be some weird system thrashing case where we are able to allocate VRAM, but then it's clobbered by the time we get around to the end of the loop, then we can get the memory again by the next validate, then it gets clobbered, and so on; this is pretty unlikely in reality, and if this is really happening on the system, there are probably more disturbing things to worry about than spinning in this rendering loop too many times.

Q: What's with the extra null-check condition in createBackBuffer()?
A: This is an optimization tweak for those interested. There are a couple of things going on here:

    backBuffer.flush();

This call tells us to release any memory associated with this image. This will happen anyway when we reassign backBuffer to a new value; the old object will eventually be GC'd (Garbage Collected). But the key word here is "eventually"; we do not know exactly how long it will take for that GC operation to kick in. When dealing with the constraint of small and finite video memory, we need to be as proactive as possible and make sure that we do not have anything wasting valuable VRAM space. For example, what if there was space for exactly one back buffer? If we had an old version of the back buffer around and then went to create a new one without first removing the old one, we would not have space in video memory for the new one and would be forced to create it in system memory. Eventually, the old one would go away and we would re-create the new one in accelerated memory, but why not get it right the first time?

 

The second item is this condition is even tweakier:

    backBuffer = null;

This one is more superfluous, given the flush() call above. I put it in just to bring up an interesting point (and one that was a complete mystery to me until I ran into it). In general, the garbage collector will not collect any objects until they are dead; that is, until there is no reference to them. By assigning null to backBuffer, we ensure that the next time the garbage collector runs, the object formerly pointed at by backBuffer (sounds like the monicker of a rock star...) is up for collection. If we took the more obvious and usual approach of simply reassigning backBuffer to the new object we create, the end result is the same, but the timing is not.

 

Let us say that the operation of assigning the new object to backBuffer causes the collector to run to try to get more memory. It can only get memory from objects that are already dead; but since the old backBuffer object is still alive until this reassignment is done, that object is not available for collection.

As I said, this extra step is somewhat useless here; since we already nuked the video memory taken up by this image by using the more definitive and direct approach of calling flush(). The only possible memory savings here is that Java heap memory taken up by the image data structure. This is not enough to worry about. But I brought it up because I have run into problems with this in other image contexts. For example, let's say (for some unknown and perhaps suspect reason) that our rendering loop creates a new BufferedImage every time through the loop for some intermediate calculation. Images can get pretty big, so this one object can be enough to swamp the young generation of the collector; but if we have to have 2 of these objects live simultaneously, then there's a fair chance that we're going to start chucking objects into the tenured generation and thus requiring Full (aka, "expensive") GC operations to get that memory back. Much better to get rid of the old one before asking for another one.

I don't want to get too deeply into GarbageCollection issues and implementations (and I think I've reached the limit of my knowledge on the subject in any case), but I did want to mention the cases above because as graphics and image users, we tend to run into unique garbage and memory situation, so it's worth understanding more about this stuff.



Q: If I have several VolatileImage objects, can I just call the image-loss methods on one of them? Won't I get the same answers from all of them and thus working with just one of them will be just as good.
A: Yes, but NO.
I say NO with Capital Letters because I have, in fact, run into this exact bug in a simple demo app I was writing recently. 
Whoops!
I t is correct that the images will probably all return the same values (if one image gets lost, all will be lost). At least this is probably true now, but it is not guaranteed to always be true. For example, at some point we will implement image priorities and may actually force some images to be punted out of memory, thus causing selective loss instead of the wholesale loss we now see due to operating-system funkiness. However, even if we could always count on all of the image attributes being the same, that does not mean that reinstating any one image will automatically cause all images to be reinstated.

 

Example time: Let's say we have one VImage sprite and a VIimage backBuffer. We could write a buggy rendering loop like this:

        int valCode = backBuffer.validate(comp.getGraphicsConfig());
        if (valCode == VolatileImage.IMAGE_RESTORED) {
            // backBuffer does not need to be restored,
            // but sprite does - initContents() is a method
            // that re-renders the contents into the sprite
            initSpriteContents();
        } else if (valCode == VolatileImage.IMAGE_INCOMPATIBLE) {
            // recreate all volatile images
            createBackBuffer();
            createSprite();
            // also restore contents of sprite since we now have a
            // completely new image
            initSpriteContents();
        }
        // rendering to the back buffer:
        Graphics gBB = backBuffer.getGraphics();
        gBB.drawImage(sprite, 0, 0, this);
        
        // copy from the back buffer to the screen
        g.drawImage(backBuffer, 0, 0, this);
        
        // Now we are done; or are we?  Check contentsLost() and loop as necessary
        } while (!backBuffer.contentsLost());

This code is exactly the same as the code way above except that we also use the return values from validate() on the backBuffer to do whatever is necessary to the sprite.

 

Here's the problem: images only get reinstated (put into a reasonable state to render to/from) when you call validate() on that image. So if you get a IMAGE_RESTORED value from backBuffer.validate(), then chances are pretty good that the sprite is also lost ... and is staying lost. The code above calls initSpriteContents() to re-render the contents of the sprite, but these operations will fail because the sprite is not in a state that will accept rendering.

The single contentsLost() call will probably work for now, but as I said above, if any images are every selectively punted that could change and the above code would be wrong under the mythical future implementation.

A more correct version of the above looks something like this:

       int valCodeBB = backBuffer.validate(comp.getGraphicsConfig());
        int valCodeSprite = sprite.validate(comp.getGraphicsConfig());
        if (valCodeSprite == VolatileImage.IMAGE_RESTORED) {
            // note that we do not care about the back buffer
            // for restoration since the buffer will be 
            // re-rendered here anyway.
            initSpriteContents();
        }
        if (valCodeBuffer == VolatileImage.IMAGE_INCOMPATIBLE) {
            createBackBuffer();
        }
        if (valCodeSprite == VolatileImage.IMAGE_INCOMPATIBLE) {
            createSprite();
            // also restore contents of sprite since we now have a
            // completely new image
            initSpriteContents();
        }
        // rendering to the back buffer:
        Graphics gBB = backBuffer.getGraphics();
        gBB.drawImage(sprite, 0, 0, this);
        
        // copy from the back buffer to the screen
        g.drawImage(backBuffer, 0, 0, this);
        
        // Now we are done; or are we?  Check contentsLost() and loop as necessary
        } while (!backBuffer.contentsLost() && !sprite.contentsLost());