7 Replies Latest reply: Aug 9, 2011 3:47 AM by chummer RSS

    Very low video framerate when sound activated

      Hi Guys,

      hope you can help me with a problem that already cost me 2 days.
      Basically I used the AVTransmit2 and AVReceive2 classes as a draft for my own implementations (see below).
                      DataSource video_ds      = getVideoSource(new Dimension(640, 480), 30.0f);
                DataSource audio_ds      = getAudioSource();
                      // when I use only one datasource here everything is ok (normal audio or 30fps video)
                DataSource[] sources      = new DataSource[] {video_ds, audio_ds};
                try {
                     processor_ds = Manager.createMergingDataSource(sources);
                } catch (IncompatibleSourceException e) {
                     return "IncompatibleSourceException creating data sources";
                processor = javax.media.Manager.createProcessor(processor_ds);
      The problem I have is when using both datasources, video and audio creating a merged datasource.
      Then the framerate drops down to 1-2 fps. I tested this on 2 different pc's here with 3 different cameras and with JMStudio it seems to be the same.
      When capturing audio and video and transmitting via RTP, the resulting frame is showing video with 1 fps (Monitor seems to be ok).

      When using AVTransmit2 and AVReceive2 without any changes I've the same problem.

      Is this a known problem that a webcam stream with audio can't be transferred fluently?
      When needed, I can post the code of my two classes to be run standalone (It's quite big so I leave it out for the moment).
      Thanks in advance for your help.

      This is the output of my capture program:
      Found video capture device vfw:Microsoft WDM Image Capture (Win32):0
      DataSource created for device
      Supported video format: YUV Video Format: Size = java.awt.Dimension[width=640,height=480] MaxDataLength = 460800 DataType = class [B yuvType = 2 StrideY = 640 StrideUV = 320 OffsetY = 0 OffsetU = 307200 OffsetV = 384000
      Supported video format: RGB, 160x120, Length=57600, 24-bit, Masks=3:2:1, PixelStride=3, LineStride=480, Flipped
      Supported video format: RGB, 176x144, Length=76032, 24-bit, Masks=3:2:1, PixelStride=3, LineStride=528, Flipped
      Supported video format: RGB, 320x240, Length=230400, 24-bit, Masks=3:2:1, PixelStride=3, LineStride=960, Flipped
      Supported video format: RGB, 352x288, Length=304128, 24-bit, Masks=3:2:1, PixelStride=3, LineStride=1056, Flipped
      Supported video format: RGB, 640x480, Length=921600, 24-bit, Masks=3:2:1, PixelStride=3, LineStride=1920, Flipped
      Setting capture format to RGB, 640x480, FrameRate=30.0
      Input format for RTP conversion: RGB, 640x480, FrameRate=30.0, Length=921600, 24-bit, Masks=3:2:1, PixelStride=3, LineStride=1920, Flipped
      Track 0 is set to transmit as: JPEG/RTP, 640x480, FrameRate=30.0
      Input format for RTP conversion: LINEAR, 44100.0 Hz, 16-bit, Stereo, LittleEndian, Signed
      Track 1 is set to transmit as: gsm/rtp, 8000.0 Hz, 8-bit, Mono
      - Set quality to 0.5 on com.sun.media.codec.video.jpeg.NativeEncoder$1$QCA@1ef9f1d
      RTP channel opened to: 5030
      RTP channel opened to: 5032
      and here the output of the receiver / playback program:
        - Open RTP session for: addr: port: 5030 ttl: 1
        - Waiting for RTP data to arrive...
        - Received new RTP stream: gsm/rtp, 8000.0 Hz, Mono
            The sender of this stream had yet to be identified.
        - Received new RTP stream: JPEG/RTP
            The sender of this stream had yet to be identified.
        - A new participant had just joined: otto@meier
        - A new participant had just joined: otto@meier
        - The previously unidentified stream 
            gsm/rtp, 8000.0 Hz, Mono
            had now been identified as sent by: otto@meier
        - A new participant had just joined: otto@meier
        - The previously unidentified stream 
            had now been identified as sent by: rlamotte@chumbell
      Also curious and it would be great if somebody could bring some light in there:
      I thought I would need two RTPManager instances for the two tracks, so one socket for audio and one for video.
      But, when I configure my receiver only for listening on port 5030 with one RTPManager instance, I get both, audio and video.
      So, what's the normal approach to implement a video/audio transmission via RTPManager?

      Edited by: user2293420 on 18.07.2011 08:33
        • 1. Re: Very low video framerate when sound activated
          Basically I'm doing the following:

          1. creates a datasource for audio from javasound://
          2. creates a datasource for video from vfw://0 with RGB 640x480 and 30 FPS
          3. merge the datasources and create a processor
          4. configures the processor
          5. transcode video to JPEG_RTP 640x480 and 30 FPS and audio to GSM_RTP
          5. creates one RTPManager for video and one for audio
          SessionAddress videoAddr = new SessionAddress(InetAddress.getLocalHost(), ANY_PORT);
          sendStream = sendingVideoRTPManager.createSendStream(ds, 0);          
          6. add remote targets to the RTPManager

          1. start two RTPManager, one for video and one for audio
          SessionAddress videoAddr = new SessionAddress(InetAddress.getLocalHost(),     5030);
          2. when capture side adds dynamically the target, a ReceiveStreamEvent is fired on the playback side
          3. Depending of the stream format I decide which player to create (audio or video)
          stream = ((NewReceiveStreamEvent)evt).getReceiveStream();
          DataSource ds = stream.getDataSource();
          Format format = null;
          // Find out the formats.
          RTPControl ctl = (RTPControl)ds.getControl("javax.media.rtp.RTPControl");
          if (ctl != null) {
               format = ctl.getFormat();
               System.out.println("  - Received new RTP stream: " + format);
               System.out.println("  - Received new RTP stream");
          if (format != null && format instanceof VideoFormat) {
               Player video = javax.media.Manager.createRealizedPlayer(ds);
               // synch audio and video                              
               if (this.audio_player != null)
                    this.video_player = video;
               else if (format != null && format instanceof AudioFormat) {
                    Player audio = Manager.createRealizedPlayer(ds);
                    // synch audio and video
                    if (this.video_player != null)
                    this.audio_player = audio;
          This works pretty good but my problem is the low framerate of the RTP stream.
          When I use only the video datasource as input for the processor, I receive a perfect 30 fps JPEG stream. Merging the video source with audio, the result is horrible (normal sound but video 1 fps).
          I've tried now with H263_RTP 352x288 with sound and it works at least at 15-18 fps. At least something...

          Does anybody have the problems when using JPEG 640x480 or does nobody use this resolution for a video conference?

          PS: I'm only testing local or in the lan, so no bandwith problems at all and my pc is also quite new with lots of memory and processor power. ;-)

          I really need the highest possible quality of JMF. Please suggest everything you think it could be affect.
          • 2. Re: Very low video framerate when sound activated
            I really need the highest possible quality of JMF. Please suggest everything you think it could be affect.
            "Highest possible quality" and JMF shouldn't really ever be used in the same sentence. JMF is 10 year old streaming technology designed to run on machines and networks that were medocre 12 years ago.

            That said, you can transmit audio and video on the same stream, but you shouldn't ever do that because it'll decrease your bandwidth. Yes, you're going to say that you have sufficient bandwidth on your machine, but if you use a single send stream, you're forcing everything through one socket which does limit the bandwidth to 1 sockets worth.

            So, make your code work like the AVTransmit2 example works. They wrote the example the way they did for a reason.
                 private String createTransmitter() {        
                 PushBufferDataSource pbds = (PushBufferDataSource)dataOutput;
                 PushBufferStream pbss[] = pbds.getStreams();
                 rtpMgrs = new RTPManager[pbss.length];
                 SessionAddress localAddr, destAddr;
                 InetAddress ipAddr;
                 SendStream sendStream;
                 int port;
                 SourceDescription srcDesList[];
                 for (int i = 0; i < pbss.length; i++) {
                      rtpMgrs[i] = RTPManager.newInstance();
                      port = portBase + 2*i;
                      ipAddr = InetAddress.getByName(ipAddress);
                      localAddr = new SessionAddress( InetAddress.getLocalHost(), port);
                      destAddr = new SessionAddress( ipAddr, port);
                      rtpMgrs.initialize( localAddr);
                      rtpMgrs[i].addTarget( destAddr);          
                      sendStream = rtpMgrs[i].createSendStream(dataOutput, i);          
            Note that they create instances of an RTPManager for each audio and video track, and transmit them to different ports using different send streams. You need to be doing that as well.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
            • 3. Re: Very low video framerate when sound activated
              Hi captfoss,
              thanks for your reply.
              "Highest possible quality" and JMF shouldn't really ever be used in the same sentence. JMF is 10 year old streaming technology designed to run on machines and networks that were medocre 12 years ago.
              I know, I was just referring to the 640x480 resolution. This has to be enough for the moment and maybe next year we will see how to improve that.
              So, make your code work like the AVTransmit2 example works. They wrote the example the way they did for a reason.
              That's exactly how I have it implementing now.
              private String createTransmitter() {
                       DataSource ds = dataOutput;
                       PushBufferDataSource pbds = (PushBufferDataSource)dataOutput;
                       PushBufferStream pbss[] = pbds.getStreams();
                        SendStream sendStream;
                        String portNumber;
                        String media = "";
                        int port = 0;
                       try {
                            for (int i=0; i<pbss.length; ++i) {
                                 SessionAddress address = null;
                                 RTPManager mgr = RTPManager.newInstance();
                                 if (pbss.getFormat() instanceof VideoFormat) {
                                  port = localVideoPort;
                                  media = "Video";
                                  videoRtpManager = mgr;
                             else if (pbss[i].getFormat() instanceof AudioFormat) {
                                  port = localAudioPort;
                                  media = "Audio";
                                  audioRtpManager = mgr;

                             address = new SessionAddress(InetAddress.getLocalHost(), port);
                                  sendStream = mgr.createSendStream(ds, i);          
                                  portNumber = (port == SessionAddress.ANY_PORT) ? "ANY_PORT" : Integer.toString(port);
                                  log.info("RTP Manager for " + media + " capture created on port " + portNumber);

                   } catch (Exception e) {
                        return e.getMessage();
                        return null;
              It's working very good and stable now. When I transmit H263_RTP video with DVI_RTP audio, it works great (apart from the resolution, just the FPS which are constantly > 25). The problem is when I change to JPEG_RTP (640x480) with DVI_RTP, then the framerate drops until 1 FPS. This just the encoding part of the processor, the capture format of the datasource is always RGB and it not touched when I change the encoding format.
              Also, I can transmit JPEG_RTP with 640x480 and 30 FPS but only when I disable the audio track in the processor.
              Any clue why this could happen?                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
              • 4. Re: Very low video framerate when sound activated
                Any clue why this could happen?

                I believe that H263 uses keyframes, whereas I think JPEG video encoding doesn't... so that could have something to do with it. It's presumably much harder to encode into JPEG format than H263...

                That said, have you tried all of the possible audio formats to see if any of them work well with JPEG?
                • 5. Re: Very low video framerate when sound activated
                  It's presumably much harder to encode into JPEG format than H263...
                  Hmmm, I can't really believe that nobody uses JPEG_RTP encoding with an audio track.
                  I've tried with every combination of audio RTP format and it's always the same (independently of the JPEG resolution).
                  • 6. Re: Very low video framerate when sound activated
                    chummer wrote:
                    I can't really believe that nobody uses JPEG_RTP encoding with an audio track
                    I can't really believe that anyone uses JPEG_RTP encoding peroid. JPEG is a horrible, horrible choice for video encoding.
                    • 7. Re: Very low video framerate when sound activated
                      Yes, but unfortunately it seems the only RTP format with 640x480 supported by JMF. :-(
                      I'll have a look at how to integrate H264...