8 Replies Latest reply on Jul 30, 2011 2:01 PM by chummer

    JMF sync audio and video from webcam

    775480
      Hello everyone, I've been trying to record video from a webcam with JMF but I'm having some problem with the audio and video sync. Independently of the audio format choosen the audio starts before the video, some formats end up with more delay. I dunno how to synch the audio and video tracks, all I do is create the both datasources, choose the codec and creathe processor, without time or clock configurations. Any help will be much appretiated, thanks !
      Here's the code I'm using:

      import java.util.Vector;
      import java.awt.*;
      import java.awt.event.*;
      import javax.media.*;
      import javax.media.format.*;
      import javax.media.protocol.*;
      import javax.media.datasink.*;
      
      /**
       * 
       * @author TW Burger from code by pawan_sin99
       */
      public class CaptureDisplay {
      
           /**
            * @param args
            *            the command line arguments
            */
           public static void main(String[] args) {
                // run a video capture object
                VidCap vidCap = new VidCap();
           }
      }
      
      class VidCap {
      
           // pega o tipo de dispositivo de video
           CaptureDeviceInfo device = null;
           //
           MediaLocator ml = null;
           Player player = null;
           Component videoScreen = null;
           Processor processor = null;
           DataSink dataSink = null;
           TheDataSinkListenerVideo dataSinkListener = null;
      
           VidCap() {
                try {
                     // Recebe a lista de dispositivos validos
                     Vector deviceList = CaptureDeviceManager
                               .getDeviceList(new YUVFormat());
                     String str1 = "vfw://0";
                     // Obtém o tipo de video - quase sempre o primeiro é a camera
                     // disponivel
                     device = CaptureDeviceManager.getDevice(str1);
                     // guarda a localização da url
                     ml = new MediaLocator(str1);
      
                     // cria a conexao com o dispositivo
                     DataSource ods = null;
                     ods = Manager.createDataSource(ml);
      
                     // Clone the video source so it can be displayed and used to capture
                     // the video at the same time. Trying to use the same source for two
                     // purposes would cause a "source is in use" error
                     // Clona o vídeo para que possa ser visto e usado para capturar o
                     // vídeo. Caso se tentar usar a mesma fonte para as duas finalidades
                     // causaria um erro "source is in use".
                     DataSource cloneableDS = Manager.createCloneableDataSource(ods);
                     DataSource PlayerVidDS = cloneableDS;
      
                     // O código de captura vai usar o player para poder controlar a
                     // midia
                     DataSource CaptureVidDS = ((javax.media.protocol.SourceCloneable) cloneableDS)
                               .createClone();
      
                     // -----------------------------------------------------------------
                     // Exibe o vídeo para a captura
                     // -----------------------------------------------------------------
      
                     player = Manager.createRealizedPlayer(PlayerVidDS);
                     player.start();
      
                     // Obtém o audio
                     deviceList = CaptureDeviceManager
                               .getDeviceList(new javax.media.format.AudioFormat(null));
                     device = (CaptureDeviceInfo) deviceList.firstElement();
                     ml = device.getLocator();
                     DataSource audioDataSource = Manager.createDataSource(ml);
      
                     // junta o audio com o vídeo
                     // --------------------------
                     DataSource mixedDataSource = null;
                     DataSource dsArray[] = new DataSource[2];
                     dsArray[0] = CaptureVidDS; // this is a cloned datasource
                     // and is controlled by the master clonable data source
                     dsArray[1] = audioDataSource;
                     try {
                          mixedDataSource = javax.media.Manager
                                    .createMergingDataSource(dsArray);
                     } catch (Exception e) {
                          // your exception handling here
                     }
      
                     // setup output file format to msvideo
                     FileTypeDescriptor outputType = new FileTypeDescriptor(
                               FileTypeDescriptor.MSVIDEO);
      
                     // configura a saida video and audio data format
                     Format outputFormat[] = new Format[2];
                     // outputFormat[0] = new VideoFormat(VideoFormat.RGB);
                     // outputFormat[0] = new VideoFormat(VideoFormat.YUV);
                     // outputFormat[0] = new VideoFormat(VideoFormat.H263);
                     outputFormat[0] = new VideoFormat(VideoFormat.H263);
                     outputFormat[1] = new AudioFormat(AudioFormat.IMA4_MS);
      
                     // ----------------------
                     // create a new processor
                     // ----------------------
                     ProcessorModel processorModel = new ProcessorModel(mixedDataSource,
                               outputFormat, outputType);
                     try {
                          processor = Manager.createRealizedProcessor(processorModel);
                     } catch (Exception e) {
                          // your exception handling here
                     }
      
                     try {
                          // get the output of the processor to be used as the datasink
                          // input
                          DataSource source = processor.getDataOutput();
      
                          // create a File protocol MediaLocator with the location
                          // of the file to which bits are to be written
                          MediaLocator mediadestination = new MediaLocator(
                                    "file:C:\\Video\\vidcapaudio.avi");
      
                          // create a datasink to create the video file
                          dataSink = Manager.createDataSink(source, mediadestination);
      
                          // create a listener to control the datasink
                          dataSinkListener = new TheDataSinkListenerVideo();
                          dataSink.addDataSinkListener(dataSinkListener);
                          dataSink.open();
      
                          // now start the datasink and processor
                          dataSink.start();
      
                          processor.start();
                     } catch (Exception e) {
                          e.getMessage();
                     }
      
                     videoScreen = player.getVisualComponent();
      
                     Frame frm = new Frame("Display of Webcam");
                     frm.addWindowListener(new WindowAdapter() {
      
                          @Override
                          public void windowClosing(WindowEvent event) {
                               Frame f = (Frame) event.getSource();
      
                               // Stop the processor doing the movie capture first
                               processor.stop();
                               processor.close();
      
                               // Closing the processor will end the data stream to the
                               // data sink.
                               // Wait for the end of stream to occur before closing the
                               // datasink
                               dataSinkListener.waitEndOfStream(1);
                               dataSink.close();
      
                               // stop and close the player which closes the video data
                               // source
                               player.stop();
                               player.close();
      
                               // ------------------------------------------------
                               // dispose of the frame and close the application
                               // ------------------------------------------------
                               f.dispose();
                               System.exit(0);
                               // ------------------------------------------------
                          }
                     });
      
                     frm.setBounds(10, 10, 300, 300);
                     frm.add(videoScreen);
                     frm.setVisible(true);
      
                     Thread.sleep(65000);
      
                     // Stop the processor doing the movie capture first
                     processor.stop();
                     processor.close();
      
                     // Closing the processor will end the data stream to the data sink.
                     // Wait for the end of stream to occur before closing the datasink
                     dataSinkListener.waitEndOfStream(1);
                     dataSink.close();
      
                     // stop and close the player which closes the video data source
                     player.stop();
                     player.close();
      
                     // ------------------------------------------------
                     // dispose of the frame and close the application
                     // ------------------------------------------------
      
                     System.exit(0);
      
                } catch (Exception e) {
                     System.out.println(e);
                }
           }
      }
        • 1. Re: JMF sync audio and video from webcam
          captfoss
          Not sure if it'll fix your problem, but...
          1) When you're using a cloned datasource to record and preview, ALWAYS use the original to record. NEVER use the clone to record.

          Fix that, and report back if you still have sync errors.
          1 person found this helpful
          • 2. Re: JMF sync audio and video from webcam
            775480
            Hi capt. foss, thanks for the reply
            I've tryed disabling the player and capturing using the original data source for capturing but unfortunately the problem remains. When I use the MJPEG video codec there is a clearly bigger delay than when I use the H.263, which is ligthter. I've also tryed switching between the audio codecs but none solved the problem. Maybe some problem with the mixing perhaps ? When I use the H.263 codec the recording starts fine, but a few seconds later (20-30) the desynch begins to be noticable. The Audio kicks in earlier than the video.
            Also I tryed changing the audio device to use JavaSound intead of DirectSound but nothing changed... I did it in this line: device = (CaptureDeviceInfo) deviceList.get(1);

            Here's my code after the changes (I posted all 3 classes in the case you wish to run yourself):

            CaptureDisplay.java (main class)
            package br.jus.tjpe.video.capture;
            
            import java.awt.Component;
            import java.awt.Frame;
            import java.awt.event.ActionEvent;
            import java.awt.event.ActionListener;
            import java.awt.event.WindowAdapter;
            import java.awt.event.WindowEvent;
            import java.io.IOException;
            import java.text.DateFormat;
            import java.text.MessageFormat;
            import java.text.SimpleDateFormat;
            import java.util.Calendar;
            import java.util.Vector;
            
            import javax.media.CaptureDeviceInfo;
            import javax.media.CaptureDeviceManager;
            import javax.media.DataSink;
            import javax.media.Format;
            import javax.media.Manager;
            import javax.media.MediaLocator;
            import javax.media.Player;
            import javax.media.Processor;
            import javax.media.ProcessorModel;
            import javax.media.format.AudioFormat;
            import javax.media.format.VideoFormat;
            import javax.media.format.YUVFormat;
            import javax.media.protocol.DataSource;
            import javax.media.protocol.FileTypeDescriptor;
            import javax.swing.BoxLayout;
            import javax.swing.JLabel;
            import javax.swing.JPanel;
            import javax.swing.Timer;
            
            /**
             * 
             * @author TW Burger from code by pawan_sin99
             */
            public class CaptureDisplay {
            
                 /**
                  * @param args
                  *            the command line arguments
                  * @throws IOException
                  * @throws InterruptedException
                  */
            
                 static JLabel label = new JLabel();
            
                 public static void main(String[] args) throws IOException,
                           InterruptedException {
            
                      // convertToFLV();
            
                      System.out.println("run");
            
                      new VidCap();
            
                 }
            
                 private static void convertToFLV() throws IOException, InterruptedException {
                      String rootDir = System.getProperty("user.dir");
                      String relativePath = "\\..\\videos\\";
                      String fileName = "testeh263.avi";
            
                      String inPut = rootDir + relativePath + fileName;
                      String outPut = "C:\\Video\\testRelativo.flv";
            
                      String commandPattern = "ffmpeg -i {0} -ar 11025 -b 120000 -s 320x240 -y {1}";
            
                      String commandString = MessageFormat.format(commandPattern, inPut,
                                outPut);
                      System.out.println("Executing " + commandString);
                      Process ffmpegProcess = Runtime.getRuntime().exec(commandString);
                      // correcao para erro de travamento do processo do ffmpeg
                      new PrintStream(ffmpegProcess.getInputStream()).start();
                      new PrintStream(ffmpegProcess.getErrorStream()).start();
                      ffmpegProcess.waitFor();
                 }
            }
            
            class VidCap {
            
                 // pega o tipo de dispositivo de video
                 CaptureDeviceInfo device = null;
                 //
                 MediaLocator ml = null;
                 Player player = null;
                 Component videoScreen = null;
                 Processor processor = null;
                 DataSink dataSink = null;
                 TheDataSinkListenerVideo dataSinkListener = null;
            
                 VidCap() {
                      try {
                           // Recebe a lista de dispositivos validos
                           Vector deviceList = CaptureDeviceManager
                                     .getDeviceList(new YUVFormat());
                           String str1 = "vfw://0";
                           // Obtém o tipo de video - quase sempre o primeiro é a camera
                           // disponivel
                           device = CaptureDeviceManager.getDevice(str1);
                           // guarda a localização da url
                           ml = new MediaLocator(str1);
            
                           // cria a conexao com o dispositivo
                           DataSource ods = null;
                           ods = Manager.createDataSource(ml);
            
                           // Clona o vídeo para que possa ser visto e usado para capturar o
                           // vídeo. Caso se tentar usar a mesma fonte para as duas finalidades
                           // causaria um erro "source is in use".
                           DataSource cloneableDS = Manager.createCloneableDataSource(ods);
                           DataSource PlayerVidDS = cloneableDS;
            
                           // O código de captura vai usar o player para poder controlar a
                           // midia
                           DataSource CaptureVidDS = ((javax.media.protocol.SourceCloneable) cloneableDS)
                                     .createClone();
            
                           // -----------------------------------------------------------------
                           // Exibe o vídeo para a captura
                           // -----------------------------------------------------------------
            
            //               player = Manager.createRealizedPlayer(PlayerVidDS);
            //               player.start();
            
                           // Obtém o audio
                           deviceList = CaptureDeviceManager
                                     .getDeviceList(new javax.media.format.AudioFormat(null));
                           device = (CaptureDeviceInfo) deviceList.get(1);
                           ml = device.getLocator();
                           DataSource audioDataSource = Manager.createDataSource(ml);
            
                           // junta o audio com o vídeo
                           // --------------------------
                           DataSource mixedDataSource = null;
                           DataSource dsArray[] = new DataSource[2];
                           // dsArray[0] = CaptureVidDS; // this is a cloned datasource
                           //dsArray[0] = cloneableDS; // this is a cloned datasource               
                           dsArray[0] = ods; // this is a cloned datasource
                           // and is controlled by the master clonable data source
                           dsArray[1] = audioDataSource;
                           try {
                                mixedDataSource = javax.media.Manager
                                          .createMergingDataSource(dsArray);
                           } catch (Exception e) {
                                e.printStackTrace();
                           }
            
                           // setup output file format to msvideo
                           FileTypeDescriptor outputType = new FileTypeDescriptor(
                                     FileTypeDescriptor.MSVIDEO);
            
                           // configura a saida video and audio data format
                           Format outputFormat[] = new Format[2];
                           // outputFormat[0] = new VideoFormat(VideoFormat.YUV);
                           // outputFormat[0] = new VideoFormat(VideoFormat.H263);
                           outputFormat[0] = new VideoFormat(VideoFormat.MJPG);
                           outputFormat[1] = new AudioFormat(AudioFormat.LINEAR);
            
                           // ----------------------
                           // create a new processor
                           // ----------------------
                           ProcessorModel processorModel = new ProcessorModel(mixedDataSource,
                                     outputFormat, outputType);
                           try {
                                processor = Manager.createRealizedProcessor(processorModel);
                           } catch (Exception e) {
                                // your exception handling here
                           }
            
                           try {
                                // get the output of the processor to be used as the datasink
                                // input
                                DataSource source = processor.getDataOutput();
            
                                // create a File protocol MediaLocator with the location
                                // of the file to which bits are to be written
                                MediaLocator mediadestination = new MediaLocator(
                                          "file:C:\\Video\\vidcapteste3.avi");
            
                                // create a datasink to create the video file
                                dataSink = Manager.createDataSink(source, mediadestination);
            
                                // create a listener to control the datasink
                                dataSinkListener = new TheDataSinkListenerVideo();
                                dataSink.addDataSinkListener(dataSinkListener);
                                dataSink.open();
            
                                // now start the datasink and processor
                                dataSink.start();
            
                                processor.start();
                           } catch (Exception e) {
                                e.getMessage();
                           }
            
                           //videoScreen = player.getVisualComponent();
            
                           Frame frm = new Frame("Display of Webcam");
            
                           final DateFormat timeFormat = new SimpleDateFormat("HH:mm:ss");
                           final int increment = 1000;
                           ActionListener timerListener = new ActionListener() {
                                Calendar cal = Calendar.getInstance();
                                {
                                     cal.set(Calendar.HOUR_OF_DAY, 0);
                                     cal.set(Calendar.MINUTE, 0);
                                     cal.set(Calendar.SECOND, 0);
                                }
            
                                public void actionPerformed(ActionEvent e) {
                                     String time = timeFormat.format(cal.getTime());
                                     cal.add(Calendar.SECOND, increment / 1000);
                                     CaptureDisplay.label.setText(time);
                                }
                           };
                           Timer timer = new Timer(increment, timerListener);
                           timer.setInitialDelay(0);
                           timer.start();
            
                           frm.addWindowListener(new WindowAdapter() {
            
                                @Override
                                public void windowClosing(WindowEvent event) {
                                     Frame f = (Frame) event.getSource();
            
                                     // Stop the processor doing the movie capture first
                                     processor.stop();
                                     processor.close();
            
                                     // Closing the processor will end the data stream to the
                                     // data sink.
                                     // Wait for the end of stream to occur before closing the
                                     // datasink
                                     try {
                                          dataSinkListener.waitEndOfStream(1);
                                     } catch (InterruptedException e) {
                                          e.printStackTrace();
                                     }
                                     dataSink.close();
            
                                     // stop and close the player which closes the video data
                                     // source
            //                         player.stop();
            //                         player.close();
            
                                     // ------------------------------------------------
                                     // dispose of the frame and close the application
                                     // ------------------------------------------------
                                     f.dispose();
                                     System.exit(0);
                                     // ------------------------------------------------
                                }
                           });
            
                           JPanel listPane = new JPanel();
                           listPane.setLayout(new BoxLayout(listPane, BoxLayout.Y_AXIS));
                           //listPane.add(videoScreen);
                           listPane.add(CaptureDisplay.label);
                           frm.add(listPane);
                           frm.pack();
                           frm.setVisible(true);
            
                      } catch (Exception e) {
                           System.out.println(e);
                      }
                 }
            }
            PrintStream.java
            package br.jus.tjpe.video.capture;
            
            /**
             * Classe para fixar erro de travamento no waitFor() no Windows
             * @author acnlf 
             */
            
            class PrintStream extends Thread {
                 java.io.InputStream inputStream = null;
            
                 public PrintStream(java.io.InputStream is) {
                      inputStream = is;
                 }
            
                 public void run() {
                      try {
                           while (this != null) {
                                int character = inputStream.read();
                                if (character != -1)
                                     System.out.print((char) character);
                                else
                                     break;
                           }
                      } catch (Exception e) {
                           e.printStackTrace();
                      }
                 }
            }
            TheDataSinkListenerVideo.java
            package br.jus.tjpe.video.capture;
            
            /**
             * Classe para fixar erro de travamento no waitFor() no Windows
             * @author acnlf 
             */
            
            class PrintStream extends Thread {
                 java.io.InputStream inputStream = null;
            
                 public PrintStream(java.io.InputStream is) {
                      inputStream = is;
                 }
            
                 public void run() {
                      try {
                           while (this != null) {
                                int character = inputStream.read();
                                if (character != -1)
                                     System.out.print((char) character);
                                else
                                     break;
                           }
                      } catch (Exception e) {
                           e.printStackTrace();
                      }
                 }
            }
            Thanks again !

            Edited by: Lupan on 14/07/2011 12:30
            • 3. Re: JMF sync audio and video from webcam
              captfoss
              You could try processing the two streams first, and then merge the output of the processors...
              • 4. Re: JMF sync audio and video from webcam
                814245
                Hey, i suggest you manipulate the mediaTime() methods of both the audio and video processor, setting both times to thesame position at each instance...
                • 5. Re: JMF sync audio and video from webcam
                  878403
                  hi
                  this is rama
                  i got same problem when i was recording video from captured web cam and audio device. please, d u find any solution.
                  • 6. Re: JMF sync audio and video from webcam
                    878403
                    hi
                    this is rama
                    i got same problem when i was recording video from captured web cam and audio device((The Audio kicks in earlier than the video)). please, d u find any solution.
                    • 7. Re: JMF sync audio and video from webcam
                      878403
                      hi lupan,
                      i have solved this problem(The Audio kicks in earlier than the video.); using this A Processor can enable user control over the maximum number of bytes that it can write to its destination by implementing the StreamWriterControl. You find out if a Processor provides a StreamWriterControl by calling getControl("javax.media.datasink.StreamWriterControl") on the Processor.if u want, i will forward my code
                      • 8. Re: JMF sync audio and video from webcam
                        chummer
                        hi rama, can you please share your code? thanks.