Skip to Main Content

Java SE (Java Platform, Standard Edition)

Announcement

For appeals, questions and feedback about Oracle Forums, please email oracle-forums-moderators_us@oracle.com. Technical questions should be asked in the appropriate category. Thank you!

Interested in getting your voice heard by members of the Developer Marketing team at Oracle? Check out this post for AppDev or this post for AI focus group information.

Canvas performance

john16384Sep 2 2012 — edited Sep 6 2012
I've been testing Canvas performance on a very fast machine (quad core Xeon) with a one of the latest NVidia graphics boards, but I'm unable to render full screen 1920x1080 video with this setup using the Canvas.

Is this simply too much to hope for or am I missing some obvious performance improvements?

Here's the code I used:
import java.nio.ByteBuffer;

import javafx.application.Application;
import javafx.scene.Scene;
import javafx.scene.canvas.Canvas;
import javafx.scene.image.PixelFormat;
import javafx.scene.image.PixelWriter;
import javafx.scene.image.WritablePixelFormat;
import javafx.scene.layout.BorderPane;
import javafx.stage.Stage;
import uk.co.caprica.vlcj.component.DirectMediaPlayerComponent;

import com.sun.jna.Memory;
import com.sun.jna.NativeLibrary;

public class VLCDirectTest extends Application {

  public static void main(final String[] args) {
    Application.launch(args);
  }

  private DirectMediaPlayerComponent mp;

  @Override
  public void start(Stage primaryStage) throws Exception {
    NativeLibrary.addSearchPath("libvlc", "c:/program files (x86)/videolan/vlc");

    BorderPane borderPane = new BorderPane();
    final Canvas canvas = new Canvas(1920, 1080);
    borderPane.setCenter(canvas);
    System.out.println(">>> " + canvas.getGraphicsContext2D().getPixelWriter().getPixelFormat());
    Scene scene = new Scene(borderPane);
    final PixelWriter pixelWriter = canvas.getGraphicsContext2D().getPixelWriter();
    final WritablePixelFormat<ByteBuffer> byteBgraInstance = PixelFormat.getByteBgraInstance();

    mp = new DirectMediaPlayerComponent("RV32", 1920, 1080, 1920*4) {
      @Override
      public void display(Memory nativeBuffer) {
        ByteBuffer byteBuffer = nativeBuffer.getByteBuffer(0, nativeBuffer.size());
        pixelWriter.setPixels(0, 0, 1920, 1080, byteBgraInstance, byteBuffer, 1920*4);
      }
    };

    mp.getMediaPlayer().playMedia("L:\\test-movies\\2012.mkv");

    primaryStage.setScene(scene);
    primaryStage.show();
  }
}
It works, but it hickups and distorts a lot when it hickups. Lowering the resolution to say 1600x900 makes it almost smooth. Lowering it further gets the expected 24 frames per second.
This post has been answered by 935214 on Sep 2 2012
Jump to Answer

Comments

935214 Sep 2 2012 — edited on Sep 2 2012
Answer
Try using IntBuffer, it's possible byte buffer get's converted per pixel. I think if the pixel format and the scanline width matches,
then System.arraycopy(...) is used instead of looping / converting over each pixel. It should be enough for 25fps full HD upload.

You also don't need to use the Canvas node, you can just use WritableImage.getImageWriter().setPixels() with IntBuffer.
It's possible the Canvas node dispatches all draw / pixel events to the rendering thread and copies the array / buffer again.
I'd be curious what performance you get.

And nice, I didn't know you could integrate VLC so easily =)

-----

PS:
Oh never mind, for me the WritableImage has a "ByteBgraPre" PixelFormat (premultiplied). So
imgWriter.setPixels(0, 0, w, h, PixelFormat.getByteBgraPreInstance(), pixels.asByteBuffer(), w * 4);
was a bit better than:
imgWriter.setPixels(0, 0, w, h, PixelFormat.getByteBgraInstance(), pixels.asByteBuffer(), w * 4);
For opaque images, premultiplied doesn't matter anyway. I managed to get 45 fps with 1920x1200 pixel upload.

Edited by: Dejay on Sep 2, 2012 9:34 AM
Marked as Answer by john16384 · Sep 27 2020
jsmith Sep 4 2012 — edited on Sep 4 2012
I think there are a bunch of performance optimizations going into the JavaFX8 branch (the most relevant of which is allowing the JavaFX Application Thread and JavaFX Rendering Thread to execute in parallel), so you may get even better performance if you make use of a jdk8 preview build.

http://jdk8.java.net/download.html

When I do stuff like this, I use the same PixelFormat and Writeable image technique which Dejay suggests. I haven't tried it with full screen though, just 1024x1024 writable images (which animated smoothly for me on an i7+460gtx). I'm not exactly sure about where the performance bottlenecks in such implementations are to be found, but I'm guessing it is in memory bandwidth somewhere (either system memory, or system memory => gpu texture upload memory) because I found it hard to max out the cpu.
john16384 Sep 5 2012
Thanks for both the answers, I haven't had time to look into this again, but I will soon.

I donot think the CPU is maxed out at all (maybe one core), and I also donot think this should be a bandwidth issue. Memory bandwidth is on the order of multiple GB/sec these days, and a 1920x1080 stream at 25 fps only consumes around 200 MB/sec. Even with multiple copies being made of each frame, it shouldn't get close to saturating the memory bus.

It shouldn't saturate the graphics card bus either -- other programs can display movies like this just fine, and they would have to get the data to the gfx card as well. I also donot think it is my graphics card -- it is a fairly recent model (asus gtx 550) that can handle a lot of punishment.
john16384 Sep 5 2012 — edited on Sep 6 2012
Wow...

Using PixelFormat.getByteBgraPreInstance makes a huge difference...

Before I got something like this (with over 750 frames dropped):

Frames: 799 Avg.time: 60.4 ms Frames>20ms: 799 (Max)FPS: 17 fps

With those 3 extra letters, with pre mulitplied alpha, I get this:

Frames: 1501 Avg.time: 3.1 ms Frames>20ms: 1 (Max)FPS: 321 fps

This means I will be able to integrate VLC much tighter into my program now -- before I just let it render to its own window and I just overlayed a transparent JavaFX window... with this I can just put the Canvas in a StackPane and overlay as usual :-)

There was no option to get an IntBuffer from JNA, but ByteBuffer performs quite adequately. Full source of this test program for those interested:
import java.nio.ByteBuffer;

import javafx.application.Application;
import javafx.scene.Scene;
import javafx.scene.canvas.Canvas;
import javafx.scene.image.PixelFormat;
import javafx.scene.image.PixelWriter;
import javafx.scene.image.WritablePixelFormat;
import javafx.scene.layout.BorderPane;
import javafx.stage.Stage;
import uk.co.caprica.vlcj.component.DirectMediaPlayerComponent;

import com.sun.jna.Memory;
import com.sun.jna.NativeLibrary;

public class VLCDirectTest extends Application {
  private static final int WIDTH = 1920;
  private static final int HEIGHT = 1080;
  public static void main(final String[] args) {
    Application.launch(args);
  }

  private DirectMediaPlayerComponent mp;

  @Override
  public void start(Stage primaryStage) throws Exception {
    NativeLibrary.addSearchPath("libvlc", "c:/program files (x86)/videolan/vlc");

    BorderPane borderPane = new BorderPane();
    final Canvas canvas = new Canvas(WIDTH, HEIGHT);
    borderPane.setCenter(canvas);
    System.out.println(">>> " + canvas.getGraphicsContext2D().getPixelWriter().getPixelFormat());
    Scene scene = new Scene(borderPane);
    final PixelWriter pixelWriter = canvas.getGraphicsContext2D().getPixelWriter();
    final WritablePixelFormat<ByteBuffer> byteBgraInstance = PixelFormat.getByteBgraPreInstance();

    mp = new DirectMediaPlayerComponent("RV32", WIDTH, HEIGHT, WIDTH*4) {
      private long totalTime;
      private long totalFrames;
      private long tooLateFrames;

      @Override
      public void display(Memory nativeBuffer) {
        long startTime = System.currentTimeMillis();
        ByteBuffer byteBuffer = nativeBuffer.getByteBuffer(0, nativeBuffer.size());
        pixelWriter.setPixels(0, 0, WIDTH, HEIGHT, byteBgraInstance, byteBuffer, WIDTH*4);
        long renderTime = System.currentTimeMillis() - startTime;
        totalTime += renderTime;
        totalFrames++;
        if(renderTime > 20) {
          tooLateFrames++;
        }

        System.out.printf("Frames: %4d   Avg.time: %4.1f ms   Frames>20ms: %d   (Max)FPS: %3.0f fps\n", totalFrames, (double)totalTime / totalFrames, tooLateFrames, 1000.0 / ((double)totalTime / totalFrames));
        if(totalFrames > 1500) {
          System.exit(0);
        }
      }
    };

    mp.getMediaPlayer().playMedia("L:\\Movies\\HD\\2012 [2009, Action Adventure Drama SF Thriller, 1080p].mkv");

    primaryStage.setScene(scene);
    primaryStage.show();
  }
}
jsmith Sep 5 2012
That's very cool John.

When I first trying this technique a couple of months back I was using a different pixel format than the native one reported on my platform (PixelFormat.getByteBgraPreInstance). When I noticed the native format was different, I switched to that, thinking it would offer better performance due to skipping a PixelFormat conversion step, but I never benchmarked to see the difference between the two. It's very interesting to see that the choice of pixel format makes such a big difference.

Were your tests run on JavaFX 2.2 or the JavaFX8 preview?
Richard Bair-Oracle Sep 5 2012
Very cool :-)
john16384 Sep 5 2012
The test was done on JavaFX 2.2, the performance is already very good so I'm definitely going to try integrate it. It will be very nice to do all the things that MediaView can do but supporting all the formats that VLC does :)
MiPa Sep 6 2012 — edited on Sep 6 2012
I tried it too but the results on my machine where not so promising.
Frames:    1   Avg.time: 161,0 ms   Frames>20ms: 1   (Max)FPS:   6 fps
Frames:    2   Avg.time: 149,0 ms   Frames>20ms: 2   (Max)FPS:   7 fps
Frames:    3   Avg.time: 139,3 ms   Frames>20ms: 3   (Max)FPS:   7 fps
Frames:    4   Avg.time: 144,5 ms   Frames>20ms: 4   (Max)FPS:   7 fps
Frames:    5   Avg.time: 138,8 ms   Frames>20ms: 5   (Max)FPS:   7 fps
For fullHD videos the display does not seem to be updated anymore after the first frame although more frames seem to be decoded. It almost works for smaller videos (1280x720) but also not fast enough (~20fps). My machine is a quad-core Dell Precision M6500 with an NVidia GPU. I normally can play all these videos easily with VLC. I did the test with JavaFX 2.2 and the current JDK8.0 preview on Windows 7.

Correction: I just realized that your last example code does not use the ByteBgraPreInstance. After changing this in the code I got a good framerate.
Frames:  110   Avg.time:  7,6 ms   Frames>20ms: 8   (Max)FPS: 132 fps
Frames:  111   Avg.time:  7,5 ms   Frames>20ms: 8   (Max)FPS: 133 fps
Frames:  112   Avg.time:  7,5 ms   Frames>20ms: 8   (Max)FPS: 133 fps
Frames:  113   Avg.time:  7,5 ms   Frames>20ms: 8   (Max)FPS: 134 fps
Frames:  114   Avg.time:  7,4 ms   Frames>20ms: 8   (Max)FPS: 135 fps
Frames:  115   Avg.time:  7,4 ms   Frames>20ms: 8   (Max)FPS: 135 fps
Edited by: MiPa on 06.09.2012 01:34
john16384 Sep 6 2012
Sorry MiPa, I apparently copy/pasted the wrong code, I updated it now.

When the display() method is too slow to handle the frames, then behaviour gets a bit wierd -- it's possible VLC just gives up decoding more in that case.
1 - 9
Locked Post
New comments cannot be posted to this locked post.

Post Details

Locked on Oct 4 2012
Added on Sep 2 2012
9 comments
17,193 views