I ran into a similar problem with pdf files and IE. It turned out to be a problem with IE. You can exclude IE with something like this:
<If $browser = "*MSIE*" && $uri =~ ".flv$">
#IE Bug - do nothing
NameTrans fn="assign-name" from="*" name="compress-on-demand"
You may have to exclude the entire page that serves the flash too.
Compression is not supported on flash files or .swf file and dont enable compression for jpg/jpeg as it is already compressed.
Regarding compression support of IE all IE version above 6 support compression. IE version 6 also supports but sometime your browser may get hung.