9 Replies Latest reply: Sep 6, 2012 7:57 AM by JohnHendrikx RSS

    Canvas performance

    JohnHendrikx
      I've been testing Canvas performance on a very fast machine (quad core Xeon) with a one of the latest NVidia graphics boards, but I'm unable to render full screen 1920x1080 video with this setup using the Canvas.

      Is this simply too much to hope for or am I missing some obvious performance improvements?

      Here's the code I used:
      import java.nio.ByteBuffer;
      
      import javafx.application.Application;
      import javafx.scene.Scene;
      import javafx.scene.canvas.Canvas;
      import javafx.scene.image.PixelFormat;
      import javafx.scene.image.PixelWriter;
      import javafx.scene.image.WritablePixelFormat;
      import javafx.scene.layout.BorderPane;
      import javafx.stage.Stage;
      import uk.co.caprica.vlcj.component.DirectMediaPlayerComponent;
      
      import com.sun.jna.Memory;
      import com.sun.jna.NativeLibrary;
      
      public class VLCDirectTest extends Application {
      
        public static void main(final String[] args) {
          Application.launch(args);
        }
      
        private DirectMediaPlayerComponent mp;
      
        @Override
        public void start(Stage primaryStage) throws Exception {
          NativeLibrary.addSearchPath("libvlc", "c:/program files (x86)/videolan/vlc");
      
          BorderPane borderPane = new BorderPane();
          final Canvas canvas = new Canvas(1920, 1080);
          borderPane.setCenter(canvas);
          System.out.println(">>> " + canvas.getGraphicsContext2D().getPixelWriter().getPixelFormat());
          Scene scene = new Scene(borderPane);
          final PixelWriter pixelWriter = canvas.getGraphicsContext2D().getPixelWriter();
          final WritablePixelFormat<ByteBuffer> byteBgraInstance = PixelFormat.getByteBgraInstance();
      
          mp = new DirectMediaPlayerComponent("RV32", 1920, 1080, 1920*4) {
            @Override
            public void display(Memory nativeBuffer) {
              ByteBuffer byteBuffer = nativeBuffer.getByteBuffer(0, nativeBuffer.size());
              pixelWriter.setPixels(0, 0, 1920, 1080, byteBgraInstance, byteBuffer, 1920*4);
            }
          };
      
          mp.getMediaPlayer().playMedia("L:\\test-movies\\2012.mkv");
      
          primaryStage.setScene(scene);
          primaryStage.show();
        }
      }
      It works, but it hickups and distorts a lot when it hickups. Lowering the resolution to say 1600x900 makes it almost smooth. Lowering it further gets the expected 24 frames per second.
        • 1. Re: Canvas performance
          935214
          Try using IntBuffer, it's possible byte buffer get's converted per pixel. I think if the pixel format and the scanline width matches,
          then System.arraycopy(...) is used instead of looping / converting over each pixel. It should be enough for 25fps full HD upload.

          You also don't need to use the Canvas node, you can just use WritableImage.getImageWriter().setPixels() with IntBuffer.
          It's possible the Canvas node dispatches all draw / pixel events to the rendering thread and copies the array / buffer again.
          I'd be curious what performance you get.

          And nice, I didn't know you could integrate VLC so easily =)

          -----

          PS:
          Oh never mind, for me the WritableImage has a "ByteBgraPre" PixelFormat (premultiplied). So
          imgWriter.setPixels(0, 0, w, h, PixelFormat.getByteBgraPreInstance(), pixels.asByteBuffer(), w * 4);
          was a bit better than:
          imgWriter.setPixels(0, 0, w, h, PixelFormat.getByteBgraInstance(), pixels.asByteBuffer(), w * 4);
          For opaque images, premultiplied doesn't matter anyway. I managed to get 45 fps with 1920x1200 pixel upload.

          Edited by: Dejay on Sep 2, 2012 9:34 AM
          • 2. Re: Canvas performance
            jsmith
            I think there are a bunch of performance optimizations going into the JavaFX8 branch (the most relevant of which is allowing the JavaFX Application Thread and JavaFX Rendering Thread to execute in parallel), so you may get even better performance if you make use of a jdk8 preview build.

            http://jdk8.java.net/download.html

            When I do stuff like this, I use the same PixelFormat and Writeable image technique which Dejay suggests. I haven't tried it with full screen though, just 1024x1024 writable images (which animated smoothly for me on an i7+460gtx). I'm not exactly sure about where the performance bottlenecks in such implementations are to be found, but I'm guessing it is in memory bandwidth somewhere (either system memory, or system memory => gpu texture upload memory) because I found it hard to max out the cpu.
            • 3. Re: Canvas performance
              JohnHendrikx
              Thanks for both the answers, I haven't had time to look into this again, but I will soon.

              I donot think the CPU is maxed out at all (maybe one core), and I also donot think this should be a bandwidth issue. Memory bandwidth is on the order of multiple GB/sec these days, and a 1920x1080 stream at 25 fps only consumes around 200 MB/sec. Even with multiple copies being made of each frame, it shouldn't get close to saturating the memory bus.

              It shouldn't saturate the graphics card bus either -- other programs can display movies like this just fine, and they would have to get the data to the gfx card as well. I also donot think it is my graphics card -- it is a fairly recent model (asus gtx 550) that can handle a lot of punishment.
              • 4. Re: Canvas performance
                JohnHendrikx
                Wow...

                Using PixelFormat.getByteBgraPreInstance makes a huge difference...

                Before I got something like this (with over 750 frames dropped):

                Frames: 799 Avg.time: 60.4 ms Frames>20ms: 799 (Max)FPS: 17 fps

                With those 3 extra letters, with pre mulitplied alpha, I get this:

                Frames: 1501 Avg.time: 3.1 ms Frames>20ms: 1 (Max)FPS: 321 fps

                This means I will be able to integrate VLC much tighter into my program now -- before I just let it render to its own window and I just overlayed a transparent JavaFX window... with this I can just put the Canvas in a StackPane and overlay as usual :-)

                There was no option to get an IntBuffer from JNA, but ByteBuffer performs quite adequately. Full source of this test program for those interested:
                import java.nio.ByteBuffer;
                
                import javafx.application.Application;
                import javafx.scene.Scene;
                import javafx.scene.canvas.Canvas;
                import javafx.scene.image.PixelFormat;
                import javafx.scene.image.PixelWriter;
                import javafx.scene.image.WritablePixelFormat;
                import javafx.scene.layout.BorderPane;
                import javafx.stage.Stage;
                import uk.co.caprica.vlcj.component.DirectMediaPlayerComponent;
                
                import com.sun.jna.Memory;
                import com.sun.jna.NativeLibrary;
                
                public class VLCDirectTest extends Application {
                  private static final int WIDTH = 1920;
                  private static final int HEIGHT = 1080;
                  public static void main(final String[] args) {
                    Application.launch(args);
                  }
                
                  private DirectMediaPlayerComponent mp;
                
                  @Override
                  public void start(Stage primaryStage) throws Exception {
                    NativeLibrary.addSearchPath("libvlc", "c:/program files (x86)/videolan/vlc");
                
                    BorderPane borderPane = new BorderPane();
                    final Canvas canvas = new Canvas(WIDTH, HEIGHT);
                    borderPane.setCenter(canvas);
                    System.out.println(">>> " + canvas.getGraphicsContext2D().getPixelWriter().getPixelFormat());
                    Scene scene = new Scene(borderPane);
                    final PixelWriter pixelWriter = canvas.getGraphicsContext2D().getPixelWriter();
                    final WritablePixelFormat<ByteBuffer> byteBgraInstance = PixelFormat.getByteBgraPreInstance();
                
                    mp = new DirectMediaPlayerComponent("RV32", WIDTH, HEIGHT, WIDTH*4) {
                      private long totalTime;
                      private long totalFrames;
                      private long tooLateFrames;
                
                      @Override
                      public void display(Memory nativeBuffer) {
                        long startTime = System.currentTimeMillis();
                        ByteBuffer byteBuffer = nativeBuffer.getByteBuffer(0, nativeBuffer.size());
                        pixelWriter.setPixels(0, 0, WIDTH, HEIGHT, byteBgraInstance, byteBuffer, WIDTH*4);
                        long renderTime = System.currentTimeMillis() - startTime;
                        totalTime += renderTime;
                        totalFrames++;
                        if(renderTime > 20) {
                          tooLateFrames++;
                        }
                
                        System.out.printf("Frames: %4d   Avg.time: %4.1f ms   Frames>20ms: %d   (Max)FPS: %3.0f fps\n", totalFrames, (double)totalTime / totalFrames, tooLateFrames, 1000.0 / ((double)totalTime / totalFrames));
                        if(totalFrames > 1500) {
                          System.exit(0);
                        }
                      }
                    };
                
                    mp.getMediaPlayer().playMedia("L:\\Movies\\HD\\2012 [2009, Action Adventure Drama SF Thriller, 1080p].mkv");
                
                    primaryStage.setScene(scene);
                    primaryStage.show();
                  }
                }
                • 5. Re: Canvas performance
                  jsmith
                  That's very cool John.

                  When I first trying this technique a couple of months back I was using a different pixel format than the native one reported on my platform (PixelFormat.getByteBgraPreInstance). When I noticed the native format was different, I switched to that, thinking it would offer better performance due to skipping a PixelFormat conversion step, but I never benchmarked to see the difference between the two. It's very interesting to see that the choice of pixel format makes such a big difference.

                  Were your tests run on JavaFX 2.2 or the JavaFX8 preview?
                  • 6. Re: Canvas performance
                    Richard Bair
                    Very cool :-)
                    • 7. Re: Canvas performance
                      JohnHendrikx
                      The test was done on JavaFX 2.2, the performance is already very good so I'm definitely going to try integrate it. It will be very nice to do all the things that MediaView can do but supporting all the formats that VLC does :)
                      • 8. Re: Canvas performance
                        MiPa
                        I tried it too but the results on my machine where not so promising.
                        Frames:    1   Avg.time: 161,0 ms   Frames>20ms: 1   (Max)FPS:   6 fps
                        Frames:    2   Avg.time: 149,0 ms   Frames>20ms: 2   (Max)FPS:   7 fps
                        Frames:    3   Avg.time: 139,3 ms   Frames>20ms: 3   (Max)FPS:   7 fps
                        Frames:    4   Avg.time: 144,5 ms   Frames>20ms: 4   (Max)FPS:   7 fps
                        Frames:    5   Avg.time: 138,8 ms   Frames>20ms: 5   (Max)FPS:   7 fps
                        For fullHD videos the display does not seem to be updated anymore after the first frame although more frames seem to be decoded. It almost works for smaller videos (1280x720) but also not fast enough (~20fps). My machine is a quad-core Dell Precision M6500 with an NVidia GPU. I normally can play all these videos easily with VLC. I did the test with JavaFX 2.2 and the current JDK8.0 preview on Windows 7.

                        Correction: I just realized that your last example code does not use the ByteBgraPreInstance. After changing this in the code I got a good framerate.
                        Frames:  110   Avg.time:  7,6 ms   Frames>20ms: 8   (Max)FPS: 132 fps
                        Frames:  111   Avg.time:  7,5 ms   Frames>20ms: 8   (Max)FPS: 133 fps
                        Frames:  112   Avg.time:  7,5 ms   Frames>20ms: 8   (Max)FPS: 133 fps
                        Frames:  113   Avg.time:  7,5 ms   Frames>20ms: 8   (Max)FPS: 134 fps
                        Frames:  114   Avg.time:  7,4 ms   Frames>20ms: 8   (Max)FPS: 135 fps
                        Frames:  115   Avg.time:  7,4 ms   Frames>20ms: 8   (Max)FPS: 135 fps
                        Edited by: MiPa on 06.09.2012 01:34
                        • 9. Re: Canvas performance
                          JohnHendrikx
                          Sorry MiPa, I apparently copy/pasted the wrong code, I updated it now.

                          When the display() method is too slow to handle the frames, then behaviour gets a bit wierd -- it's possible VLC just gives up decoding more in that case.