1 2 3 Previous Next 30 Replies Latest reply: Apr 28, 2011 11:25 AM by 856191 RSS

    JMF - Concurrent video feed and image capture

    856191
      Hello all,

      (For background) I'm writing a monitoring system for a project in which the basic functionality is completed. Using JMF it uses a camera to periodically take a frame, compare it to a frame previously taken, and determine whether any changes have happened that would constitute movement (based on a pixel-by-pixel comparison). This part is completed. It uses a couple of calculations to determine sensitivity based on (1) what exactly is considered a 'different' colored pixel, and (2) a percentage of the total pixels that are different between the two frames.

      However, I would like to implement an extension of this that would allow me to calibrate the motion detection per the room that it is placed in. This is because there is a large variation in lighting conditions and the darkness of objects that the camera is pointed toward, which need to be accounted for because the camera is fairly imprecise. To do this, I would like to show a real-time video feed with a console printing whether motion is currently being detected. The idea is, if it is detecting motion that isn't there, the settings can be changed in a way that makes it simpler to know what exactly is happening instead of having to guess.

      Breath

      Here's the tough part - I have created the classes to show the feed and print output to a JFrame, and it works perfectly. However, when I put it together with the motion detection, I receive a Null Pointer Exception when it attempts to take the frame in order to perform the comparison because the camera is obviously being used to display the live feed. My question is, how difficult would it be to solve this problem? Is it even possible to have a live feed AND take a frame to perform the comparison while that's running? Any insight would be very helpful.

      Thanks

      Edited by: 853188 on Apr 18, 2011 9:40 PM

      Edited by: 853188 on Apr 18, 2011 9:41 PM

      Edited by: 853188 on Apr 18, 2011 9:46 PM

      Edited by: 853188 on Apr 18, 2011 10:19 PM
        • 1. Re: JMF - Concurrent video feed and image capture
          captfoss
          853188 wrote:
          (based on a pixel-by-pixel comparison).
          Horrible way to do motion detection...
          Breath

          Here's the tough part - I have created the classes to show the feed and print output to a JFrame, and it works perfectly. However, when I put it together with the motion detection, I receive a Null Pointer Exception when it attempts to take the frame in order to perform the comparison because the camera is obviously being used to display the live feed. My question is, how difficult would it be to solve this problem? Is it even possible to have a live feed AND take a frame to perform the comparison while that's running? Any insight would be very helpful.
          Yes, it's possible. Most likely, you're experiecing a bug whereby the first couple of video frames coming through a Processor are null while it's starting up. You're probably assuming you're given a video frame everytime process is called, and that's a bad assumption.

          You haven't specified at all how you've implemented anything, so that's all the help I can give you.
          • 2. Re: JMF - Concurrent video feed and image capture
            856191
            captfoss wrote:

            Horrible way to do motion detection...
            What's a better way then?
            captfoss wrote:
            Yes, it's possible. Most likely, you're experiecing a bug whereby the first couple of video frames coming through a Processor are null while it's starting up. You're probably assuming you're given a video frame everytime process is called, and that's a bad assumption.

            You haven't specified at all how you've implemented anything, so that's all the help I can give you.
            I have not assumed that, in fact, here's the code that checks whether they're null:
            prevPic = newPic;
            newPic = puller.grabImage();
                      
            if (prevPic != null && newPic != null) {
                 movement = comparator.detectMovement(prevPic, newPic);
            }
            Regardless, I was able to solve the problem. I realized that I should use the player that I instantiated to create the feed to also detect movement instead of instantiating one for video and one for motion detection.
            • 3. Re: JMF - Concurrent video feed and image capture
              captfoss
              853188 wrote:
              captfoss wrote:

              Horrible way to do motion detection...
              What's a better way then?
              There are any number of better methods for doing change detection between two pictures...

              At minimum, you should be applying some sort of smoothing filter to cancel out pixel changes due to noise (Median if you want to do it well, Gaussian if you want to do it quickly).

              At medium, you should run your image through something like a Canny edge dector to minimize the effects of different lighting conditions...

              At best, you should probably be running a better similarity measure (like a Kolmogorov-Smirnov test) on your (edge) images than doing a pixel-by-pixel similiarity measure.
              I have not assumed that, in fact, here's the code that checks whether they're null:
              Ok...
              Regardless, I was able to solve the problem. I realized that I should use the player that I instantiated to create the feed to also detect movement instead of instantiating one for video and one for motion detection.
              • 4. Re: JMF - Concurrent video feed and image capture
                856191
                captfoss wrote:

                There are any number of better methods for doing change detection between two pictures...

                At minimum, you should be applying some sort of smoothing filter to cancel out pixel changes due to noise (Median if you want to do it well, Gaussian if you want to do it quickly).

                At medium, you should run your image through something like a Canny edge dector to minimize the effects of different lighting conditions...

                At best, you should probably be running a better similarity measure (like a Kolmogorov-Smirnov test) on your (edge) images than doing a pixel-by-pixel similiarity measure.
                If I had more time, and if this wasn't for a class where the prof said he is accepting of some false positives, I would probably try to do that. Given that he is accepting of false positives, it's almost done, and I have close to no time left, I will turn it in as-is.

                I appreciate the info though. Maybe I should have asked this question earlier.
                • 5. Re: JMF - Concurrent video feed and image capture
                  captfoss
                  853188 wrote:
                  if this wasn't for a class where the prof said he is accepting of some false positives
                  Well, that changes things dramatically.

                  Yeah, if it's just for a grade, do the bare minimum to get an A... if you're using this to setup an automated gun turrent to kill burgalors, you'd want it to be more robust.
                  • 6. Re: JMF - Concurrent video feed and image capture
                    856191
                    captfoss wrote:
                    Well, that changes things dramatically.

                    Yeah, if it's just for a grade, do the bare minimum to get an A... if you're using this to setup an automated gun turrent to kill burgalors, you'd want it to be more robust.
                    While I have you here... one last thing I must do is save video to a file on the HDD while motion is being detected. I have looked over these forums and the internet and have not found a solution. Can you point me in the right direction? At least what I need or the general idea of what is required to do it?

                    Thanks again for the help.

                    Edit: clarification.

                    Edited by: 853188 on Apr 19, 2011 4:45 PM
                    • 7. Re: JMF - Concurrent video feed and image capture
                      captfoss
                      DataSinks are what you use to save files, but you can only use a DataSink to save a file as being outputted by a Processor, not a player, and the stream has to be transcoded by the Processor into the appropriate output format...

                      Best examples of saving files I have is:
                      http://www.cs.odu.edu/~cs778/spring04/lectures/jmfsolutions/examplesindex.html#transcode
                      http://www.cs.odu.edu/~cs778/spring04/lectures/jmfsolutions/examplesindex.html#export

                      You'll lose your ability to "see" the video, but you can get that back by either cloning the output of the Processor and using the clone to drive a Player object, or you can just use the MonitorControl on the Processor to see the stream associated with the Processor.

                      I don't know how your motion capture is currently getting access to the video frames, so I can't comment on how that will affect your actual motion detection code.

                      You should have done it using some code similar to this:
                      http://classic-web.archive.org/web/20080316220305/java.sun.com/products/java-media/jmf/2.1.1/solutions/DataSourceReader.html
                      • 8. Re: JMF - Concurrent video feed and image capture
                        856191
                        captfoss wrote:
                        DataSinks are what you use to save files, but you can only use a DataSink to save a file as being outputted by a Processor, not a player, and the stream has to be transcoded by the Processor into the appropriate output format...

                        Best examples of saving files I have is:
                        http://www.cs.odu.edu/~cs778/spring04/lectures/jmfsolutions/examplesindex.html#transcode
                        http://www.cs.odu.edu/~cs778/spring04/lectures/jmfsolutions/examplesindex.html#export

                        You'll lose your ability to "see" the video, but you can get that back by either cloning the output of the Processor and using the clone to drive a Player object, or you can just use the MonitorControl on the Processor to see the stream associated with the Processor.

                        I don't know how your motion capture is currently getting access to the video frames, so I can't comment on how that will affect your actual motion detection code.

                        You should have done it using some code similar to this:
                        http://classic-web.archive.org/web/20080316220305/java.sun.com/products/java-media/jmf/2.1.1/solutions/DataSourceReader.html
                        Thanks for the info. And the concurrent video feed/image capture for the motion detection is only part of the calibration, which is a feature of the UI application. When the actual system runs at times specified in the UI scheduling options, it does so in the background. Though I will say that the image capture implementation for this background detection service is similar to the video feed display, and uses a player with the FrameGrabbingController's getFrame() method.

                        How would this be affected while recording video?
                        • 9. Re: JMF - Concurrent video feed and image capture
                          captfoss
                          853188 wrote:
                          Though I will say that the image capture implementation for this background detection service is similar to the video feed display, and uses a player with the FrameGrabbingController's getFrame() method.
                          FrameGrabbingControl is designed to be used for prerecorded videos for single-frame extraction... so you're not using it as it was intended to be used, so you really shouldn't be using it.

                          Specifically, the way you've currently got it implemented, you'll be missing frames quite frequently... which could cause you miss objects that quickly cross in front of the camera, like a cat.

                          You would be very well served at this point to immediately switch that code over to use a custom pass-through codec to gain access to the frame data directly... without missing frames.
                          http://classic-web.archive.org/web/20080216001555/http://java.sun.com/products/java-media/jmf/2.1.1/solutions/FrameAccess.html
                          How would this be affected while recording video?
                          Ignoring the fact you can't record with a player? You're not guaranteed to always be able to get a FrameGrabbingControl, so that would prevent your entire system from working.

                          If you switch to the above method, you can do all 3 things (run your motion detection, view the video, record the video) reliably.
                          • 10. Re: JMF - Concurrent video feed and image capture
                            856191
                            captfoss wrote:

                            Ignoring the fact you can't record with a player? You're not guaranteed to always be able to get a FrameGrabbingControl, so that would prevent your entire system from working.

                            If you switch to the above method, you can do all 3 things (run your motion detection, view the video, record the video) reliably.
                            That link seems to be having issues.. either loading reeeeeeeeeeeeeeeeeeeeeeeaaaaaaaalllllllllly slow or not allowing me to access it. I realize that this system could have been implemented better. Even so, I'm not sure I have enough time to completely re-do it now. Time is severely of the essence. So unless that other method makes my job considerably easier, I am ok with a hackish solution if I can find one.

                            I don't seem to notice that I have lost any frames... the detection that I have in place actually seems to be working fairly well considering it has to deal with a large amount of image noise. And I will never be running detection AND recording video AND viewing video all at the same time. The possible states the system can be in are detecting motion, detecting motion and recording video (only when motion is detected), or detecting motion and viewing video (only to calibrate the detection sensitivity settings).

                            Most of the time that the system is running, it will simply be in the background, pulling frames and assessing whether motion has happened. When this background detection is running, the only way one would know is if they happened to be looking at the folder where video is to be saved, and motion occurs which triggers recording and the creation of a new video file.
                            • 11. Re: JMF - Concurrent video feed and image capture
                              856191
                              I'm now looking at that FrameAccess class, and I have yet to quite figure out how to use it in terms of what to do with the URL, what the URL is used for, or why I am using a URL at all because I'm not dealing with anything online.
                              • 12. Re: JMF - Concurrent video feed and image capture
                                captfoss
                                URL is used to load the video... if you're using a live video source, you swap out the URL portion for your CaptureDevice's MediaLocator.
                                • 13. Re: JMF - Concurrent video feed and image capture
                                  856191
                                  Ok, I think I mostly understand what's going on here (frame access, codecs, etc), but I have one more question. How exactly does the 'callback' work in the codec.process() method? Will I need to create a loop to continue taking frames or will it do that itself?
                                  • 14. Re: JMF - Concurrent video feed and image capture
                                    856191
                                    853188 wrote:
                                    Ok, I think I mostly understand what's going on here (frame access, codecs, etc), but I have one more question. How exactly does the 'callback' work in the codec.process() method? Will I need to create a loop to continue taking frames or will it do that itself?
                                    Check that, solved that problem. Only problem now is how to record and save video. Any tips?
                                    1 2 3 Previous Next