Java Media Components

orMaking Java a Player 

The question comes up regularly, and on a more frequent and random basis lately, "What is Sun doing about media support in Java?" And sometimes it is not phrased quite so politely.

Video used to be a feature that we lacked:


Now, it seems sometimes that it is the feature that we lack:


In the current world of YouTube, Flash, and iTunes, video has gone from a niche capability to a checkbox item that all platforms must support.

So what are we doing about it, besides pointing out the obvious issue that it isn't there now?

If you attended the Java Technology Desktop overview session at JavaOne this year, you learned, along 500+ of your closest friends in the room, that we're working on a new project that we call Java Media Components(JMC). This new feature, hopefully shipping in Java SE 7, is intended to support basic playback support for Java applications. JMC is also, eventually, intended to address capture and streaming capabilities.

Playing Fair

There are actually two levels of playback that we would like to support:

  • Native players
  • Java player

Going Native

The native players on most platforms have substantial support for multiple video formats. It would not make sense (read: it would cost serious $$$) for Java to implement players for some of these existing formats. So why not leverage the native platform capabilities instead?

The idea is to expose a simple API for playback control (e.g., play, stop, and rewind), and to allow a Java application to thus control playback of specified content through whatever was available on the native system. For example, playback of an AVI file on Windows might launch Windows Media Player. Moreover, the native player would be exposed through a GUI component that you could integrate into the rest of your application's interface. There would also, depending on the underlying native support, be options to either expose the native controls for the player (e.g., the Play button), or to allow the developer to skin their own controls around the player component.

This native-wrapper functionality is the easiest part of the whole JMC project. The main work is in the simple API design and the glue code to the various players on different platforms. In fact, we already have prototypes of this working for several players, including Windows Media Player and Flash.

Curious readers familiar with lightweight/heavyweight mixing issues in Swing applications might wonder at this point how we will deal with a native player component in a Swing GUI. Good question, thanks for asking. This issue is one of the reasons that we also plan to support mixed heavyweight/lightweight components in Java SE 7. In case I don't write a blog on this feature soon, here's the quick scoop: We can enable this through clipping the heavyweight components against the shapes of the overlaying lightweight components. There are some cases that do not yet work with this mechanism, such as shaped and translucent lightweights, but the system works pretty well in general for other cases.

Playing Nice with Java

Supporting common formats through the native player mechanism will solve a key part of the problem. The ability for a Java application to easily play video that resides on a user's local system through a standard local player is an important case that will be handled through this simple mechanism.

But native player support is not a complete solution. For one thing, if a developer wants a specific video to play through the application, they will have to ensure that the video is available in whatever formats will be supported on all of their users' native platforms. For example, a Quicktime movie would not be playable on Windows unless Quicktime happened to be installed on that system. Similarly, an AVI file may not have much luck playing on Linux.

To solve this problem, we need one common format, and an accompanying codec (coder-decoder), so that any Java application can count on being able to play back any content in that format, regardless of the runtime platform.

This cross-platform format support is a much more involved issue, since it immediately gets into harder issues of:

  • Licensing technology from existing player developers, and/or
  • Creating or integrating new technology into the platform

We plan to tackle this part of the JMC project after the native players are well in hand. Meanwhile, we are trying to figure out the right solution (which formats, which technologies, etc.).

Playback versus Pixel Access

There is an important consideration in both the native and Java player solutions above: How much do players need to simply display their pixels on the screen, and how much do they need to actually expose the raw pixel data? To a great extent, simply playing on the screen is probably good enough. If an application just wants to play a video, then blasting it to the user's screen is fine. But there are subtle cases here that make the issue a bit more complicated.

For example, what if you wanted to do something more interesting with the video content, like play it on a lightweight Swing widget? Remember that I said that we plan to allow mixing of heavyweight and lightweight components, which should allow a JMC player to at least cooperate with Swing widgets. But that's not the same as a JMC component actually being a Swing component. For example, what if you want to use the video as the icon in a JButton? Or what if you want the video playing in a translucent component? Or what if you want to get really crazy and display video in a 3D object, via JOGL?

For any of these cases where you want to do anything with the pixels besides simply show them on the screen, you will need access to the pixel data from the video stream. Many of the native players do not offer this capability, but merely allow the controls of playing, rewinding, and stopping the video. Some native systems, such as DirectShow, do allow more flexibility, so we plan to use these approaches when we can. And the Java Player will certainly allow this capability. Since we will provide the decoder for the video through the Java player, we can have those pixels appear on the screen, or in an image, or wherever the application specifies.

What about Capture and Streaming?

Capturing video from an input device and streaming between Java clients are two areas that we would also like to support eventually. Having the Java player is actually a good initial step toward these capabilities, since we will be able to encode and decode into our cross-platform format. But we are taking baby steps here, and feel that the top priority is solid playback support. We would like to get rolling on that playback project first before we take on even more, and harder, issues.

What about Java Media Framework (JMF)?

JMF is an existing media library, provided as an optional package on top of Java SE. JMF is a very large and powerful media development platform that is capable of doing much more than I've discussed here. In particular, while it's possible to write a complete video editing solution in JMF, we are clearly going nowhere near that level of functionality with the JMC project, but rather focusing squarely on playback at first. Again, we are concentrating on attacking the pain points in our existing video support (or lack thereof), and we feel that playback is simply much more in demand than full-on video editing APIs.

One solution here might be to simply revamp our investment in and support of JMF itself, and make it the platform for future video endeavours. However, the level of effort that we would need to invest to bring JMF up to the current level of requirements, including providing codecs for modern formats, would be huge. Also, at about a 2 Megabyte download size now, we're not sure that simply adding in the current JMF to the core platform is the right approach. It seems better to solve the top priority of simple playback by a targeted solution that is possible with a much smaller footprint impact.

We may actually implement parts of the JMC stack on portions of JMF, which would give us the advantage of re-use of some of JMFs capabilities without taking on the entire library. Also, work on codecs for JMC could end up also benefitting the JMF library eventually in any case.

That's the Plan

Plans change, of course, but this writeup represents what we would like to see happen in Java SE 7. Does it sound reasonable to you? Would having simple playback functionality, through both native players and a new Java player, satisfy your craving for media in Java? And if we spin up a new project on SwingLabssoon, would you be interested in helping out in the design and implementation of these pieces?