Saving key event video clips with OpenCV

Last week’s blog post taught us how to write videos to file using OpenCV and Python. This is a great skill to have, but it also raises the question:

How do I write video clips containing interesting events to file rather than the entire video?

In this case, the overall goal is to construct a video synopsis, distilling the most key, salient, and interesting parts of the video stream into a series of short video files.

What actually defines a “key or interesting event” is entirely up to you and your application. Potential examples of key events can include:

  • Motion being detected in a restricted access zone.
  • An intruder entering your house or apartment.
  • A car running a stop sign on a busy street by your home.

In each of these cases, you’re not interested in the entire video capture — instead, you only want the video clip that contains the action!

To see how capturing key event video clips with OpenCV is done (and build your own video synopsis), just keep reading.

Looking for the source code to this post?
Jump right to the downloads section.

Saving key event video clips with OpenCV

The purpose of this blog post is to demonstrate how to write short video clips to file when a particular action takes place. We’ll be using our knowledge gained from last week’s blog post on writing video to file with OpenCV to implement this functionality.

As I mentioned at the top of this post, defining “key” and “interesting” events in a video stream is entirely dependent on your application and the overalls goals of what you’re trying to build.

You might be interesting in detecting motion in a room. Monitoring your house. Or creating a system to observe traffic and store clips of motor vehicle drivers breaking the law.

As a simple example of both:

  1. Defining a key event.
  2. Writing the video clip to file containing the event.

We’ll be processing a video streaming and looking for occurrences of this green ball:

Figure 1: An example of the green ball we are going to detect in video streams.

Figure 1: An example of the green ball we are going to detect in video streams.

If this green ball appears in our video stream, we’ll open up a new file video (based on the timestamp of occurrence), write the clip to file, and then stop the writing process once the ball disappears from our view.

Furthermore, our implementation will have a number of desirable properties, including:

  1. Writing frames to our video file a few seconds before the action takes place.
  2. Writing frames to file a few seconds after the action finishes — in both cases, our goal is to not only capture the entire event, but also the context of the event as well.
  3. Utilizing threads to ensure our main program is not slowed down when performing I/O on both the input stream and the output video clip file.
  4. Leveraging built-in Python data structures such as deque  and Queue  so we need not rely on external libraries (other than OpenCV and imutils, of course).

Project structure

Before we get started implementing our key event video writer, let’s look at the project structure:

Inside the pyimagesearch  module, we’ll define a class named KeyClipWriter  inside the keyclipwriter.py  file. This class will handle accepting frames from an input video stream ad writing them to file in a safe, efficient, and threaded manner.

The driver script, save_key_events.py , will define the criteria of what an “interesting event” is (i.e., the green ball entering the view of the camera), followed by passing these frames on to the KeyClipWriter  which will then create our video synopsis.

A quick note on Python + OpenCV versions

This blog post assumes you are using Python 3+ and OpenCV 3. As I mentioned in last week’s post, I wasn’t able to get the cv2.VideoWriter  function to work on my OpenCV 2.4 installation, so after a few hours of hacking around with no luck, I ended up abandoning OpenCV 2.4 for this project and sticking with OpenCV 3.

The code in this lesson is technically compatible with Python 2.7 (again, provided you are using Python 2.7 with OpenCV 3 bindings), but you’ll need to change a few import  statements (I’ll point these out along the way).

Writing key/interesting video clips to file with OpenCV

Let’s go ahead and get started reviewing our KeyClipWriter  class:

We start off by importing our required Python packages on Lines 2-6. This tutorial assumes you are using Python 3, so if you’re using Python 2.7, you’ll need to change Line 4 from from queue import Queue  to simply import Queue .

Line 9 defines the constructor to our KeyClipWriter , which accepts two optional parameters:

  • bufSize : The maximum number of frames to be keep cached in an in-memory buffer.
  • timeout : An integer representing the number of seconds to sleep for when (1) writing video clips to file and (2) there are no frames ready to be written.

We then initialize four important variables on Lines 18-22:

  • frames : A buffer used to a store a maximum of bufSize  frames that have been most recently read from the video stream.
  • Q : A “first in, first out” (FIFO) Python Queue data structure used to hold frames that are awaiting to be written to video file.
  • writer : An instantiation of the cv2.VideoWriter  class used to actually write frames to the output video file.
  • thread : A Python Thread  instance that we’ll use when writing videos to file (to avoid costly I/O latency delays).
  • recording : Boolean value indicating whether or not we are in “recording mode”.

Next up, let’s review the update  method:

The update  function requires a single parameter, the frame  read from our video stream. We take this frame  and store it in our frames  buffer (Line 26). And if we are already in recording mode, we’ll store the frame  in the Queue  as well so it can be flushed to video file (Lines 29 and 30).

In order to kick-off an actual video clip recording, we need a start  method:

First, we update our recording  boolean to indicate that we are in “recording mode”. Then, we initialize the cv2.VideoWriter  using the supplied outputPath , fourcc , and fps  provided to the start  method, along with the frame spatial dimensions (i.e., width and height). For a complete review of the cv2.VideoWriter  parameters, please refer to this blog post.

Line 39 initializes our Queue  used to store the frames ready to be written to file. We then loop over all frames in our frames  buffer and add them to the queue.

Finally, we spawn a separate thread to handle writing frames to video — this way we don’t slow down our main video processing pipeline by waiting for I/O operations to complete.

As noted above, the start  method creates a new thread, calling the write  method used to write frames inside the Q  to file. Let’s define this write  method:

Line 53 starts an infinite loop that will continue polling for new frames and writing them to file until our video recording has finished.

Lines 55 and 56 make a check to see if the recording should be stopped, and if so, we return from the thread.

Otherwise, if the Q  is not empty, we grab the next frame and write it to the video file (Lines 59-63).

If there are no frames in the Q , we sleep for a bit so we don’t needlessly waste CPU cycles spinning (Lines 67 and 68). This is especially important when using the Queue  data structure which is thread-safe, implying that we must acquire a lock/semaphore prior to updating the internal buffer. If we don’t call time.sleep  when the buffer is empty, then the write  and update  methods will constantly be fighting for the lock. Instead, it’s best to let the writer sleep for a bit until there are a backlog of frames in the queue that need to be written to file.

We’ll also define a flush  method which simply takes all frames left in the Q  and dumps them to file:

A method like this is used when a video recording has finished and we need to immediately flush all frames to file.

Finally, we define the finish  method below:

This method indicates that the recording has been completed, joins the writer thread with the main script, flushes the remaining frames in the Q to file, and finally releases the cv2.VideoWriter  pointer.

Now that we have defined the KeyClipWriter  class, we can move on to the driver script used to implement the “key/interesting event” detection.

Saving key events with OpenCV

In order to keep this blog post simple and hands-on, we’ll define our “key event” to be when this green ball enters our video stream:

Figure 2: An example of a key/interesting event in a video stream.

Figure 2: An example of a key/interesting event in a video stream.

Once we see this green ball, we will call KeyClipWriter  to write all frames that contain the green ball to file. Essentially, this will give us a set of short video clips that neatly summarizes the events of the entire video stream — in short, a video synopsis.

Of course, you can use this code as a boilerplate/starting point to defining your own actions — we’ll simply use the “green ball” event since we have covered it multiple times before on the PyImageSearch blog, including tracking object movement and ball tracking.

Before you proceed with the rest of this tutorial, make sure you have the imutils package installed on your system:

This will ensure that you can use the VideoStream  class which creates a unified access to both builtin/USB webcams and the Raspberry Pi camera module.

Let’s go ahead and get started. Open up the save_key_events.py  file and insert the following code:

Lines 2-8 import our necessary Python packages while Lines 11-22 parse our command line arguments. The set of command line arguments are detailed below:

  • --output : This is the path to the output directory where we will store the output video clips.
  • --picamera : If you want to use your Raspberry Pi camera (rather than a builtin/USB webcam), then supply a value of --picamera 1 . You can read more about accessing both builtin/USB webcams and the Raspberry Pi camera module (without changing a single line of code) in this post.
  • --fps : This switch controls the desired FPS of your output video. This value should be similar to the number of frames per second your image processing pipeline can process.
  • --codec : The FourCC codec of the output video clips. Please see the previous post for more information.
  • --buffer-size : The size of the in-memory buffer used to store the most recently polled frames from the camera sensor. A larger --buffer-size  will allow for more context before and after the “key event” to be included in the output video clip, while a smaller --buffer-size  will store less frames before and after the “key event”.

Let’s perform some initialization:

Lines 26-28 initialize our VideoStream  and allow the camera sensor to warmup.

From there, Lines 32 and 33 define the lower and upper color threshold boundaries for the green ball in the HSV color space. For more information on how we defined these color threshold values, please see this post.

Line 37 instantiates our KeyClipWriter  using our supplied --buffer-size , along with initializing an integer used to count the number of consecutive frames that have not contained any interesting events.

We are now ready to start processing frames from our video stream:

On Line 41 we start to looping over frames from our video stream. Lines 45 and 46 read the next frame  from the video stream and then resizes it to have a width of 600 pixels.

Further pre-processing is done on Lines 50 and 51 by blurring the image slightly and then converting the image from the RGB color space to the HSV color space (so we can apply our color thresholding).

The actual color thresholding is performed on Line 56 using the cv2.inRange  function. This method finds all pixels p that are greenLower <= p <= greenUpper . We then perform a series of erosions and dilations to remove any small blobs left in the mask.

Finally, Lines 61-63 find contours in the thresholded image.

If you are confused about any step of this processing pipeline, I would suggest going back to our previous posts on ball tracking and object movement to further familiarize yourself with the topic.

We are now ready to check and see if the green ball was found in our image:

Line 66 makes a check to ensure that at least one contour was found, and if so, Line 69 and 70 find the largest contour in the mask (according to the area) and use this contour to compute the minimum enclosing circle.

If the radius of the circle meets a minimum suze of 10 pixels (Line 74), then we will assume that we have found the green ball. Lines 78-80 reset the number of consecFrames  that do not contain any interesting events (since an interesting event is “currently happening”) and draw a circle highlighting our ball in the frame.

Finally, we make a check if to see if we are currently recording a video clip (Line 83). If not, we generate an output filename for the video clip based on the current timestamp and call the start  method of the KeyClipWriter .

Otherwise, we’ll assume no key/interesting event has taken place:

If no interesting event has happened, we update consecFrames  and pass the frame  over to our buffer.

Line 101 makes an important check — if we are recording and have reached a sufficient number of consecutive frames with no key event, then we should stop the recording.

Finally, Lines 105-110 display the output frame  to our screen and wait for a keypress.

Our final block of code ensures the video has been successfully closed and then performs a bit of cleanup:

Video synopsis results

To generate video clips for key events (i.e., the green ball appearing on our video stream), just execute the following command:

I’ve included the full 1m 46s video (without extracting salient clips) below:

After running the save_key_events.py  script, I now have 4 output videos, one for each the time green ball was present in my video stream:

Figure 3: Creating a separate video clip for each interesting and key event.

Figure 3: Creating a separate video clip for each interesting and key event.

The key event video clips are displayed below to demonstrate that our script is working properly, accurately extracting our “interesting events”, and essentially building a series of video clips functioning as a video synopsis:

Video clip #1:

Video clip #2:

Video clip #3:

Video clip #4:

Summary

In this blog post, we learned how to save key event video clips to file using OpenCV and Python.

Exactly what defines a “key or interesting event” is entirely dependent on your application and the goals of your overall project. Examples of key events can include:

  • Monitoring your front door for motion detection (i.e., someone entering your house).
  • Recognizing the face of an intruder as they enter your house.
  • Reporting unsafe driving outside your home to the authorities.

Again, exactly what constitutes a “key event” is near endless. However, regardless of how you define an interesting event, you can still use the Python code detailed in this post to help save these interesting events to file as a shortened video clip.

Using this methodology, you can condense hours of video stream footage into seconds of interesting events, effectively yielding a video synopsis — all generated using Python, computer vision, and image processing techniques.

Anyway, I hope you enjoyed this blog post!

If you did, please consider sharing it on your favorite social media outlet such as Facebook, Twitter, LinkedIn, etc. I put a lot of effort into the past two blog posts in this series and I would really appreciate it if you could help spread the word.

And before you, be sure to signup for the PyImageSearch Newsletter using the form below to receive email updates when new posts go live!

Downloads:

If you would like to download the code and images used in this post, please enter your email address in the form below. Not only will you get a .zip of the code, I’ll also send you a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL! Sound good? If so, enter your email address and I’ll send you the code immediately!

, , , , , ,

35 Responses to Saving key event video clips with OpenCV

  1. Alexander Kurilov February 29, 2016 at 12:15 pm #

    Dude I hope you know how cool you are. As an enthusiastic reader of programming blogs, I’m always excited when you post something knew. The format of this blog is something that everybody who writes programming tutorials should strive for. Your books are pretty cool, too.

    I know this is kinda spammy but I really wanted to tell you this.

    • Adrian Rosebrock February 29, 2016 at 3:26 pm #

      It’s by no means spammy at all Alexander. After a rough day, your comment certainly put a smile on my face 🙂

  2. Daniel Fang March 1, 2016 at 4:12 am #

    Thank you for giving me the meat! Every weeks your share always make my eyes light up with delight. : )

    • Adrian Rosebrock March 1, 2016 at 3:38 pm #

      Thanks Daniel, you certainly put a smile on my face 🙂

  3. Kenny March 1, 2016 at 4:43 am #

    Cheers Adrian! As an avid reader of yours, this post is awesome! Thank you for your enthusiasm and hardwork as always! 😀

    • Adrian Rosebrock March 1, 2016 at 3:38 pm #

      Thanks Kenny — I’m glad you enjoyed the blog post! 🙂

  4. Karim March 1, 2016 at 10:11 am #

    Hey there, you’re doing an awesome job and I’ve been reading your blog for some time now. Keep up the good stuff, I can already see your next project combining this and virtual reality!

  5. Johnny March 14, 2016 at 12:11 pm #

    Another great blog Adrian. You continue to bring it sir, thank you.

    Johnny

    • Adrian Rosebrock March 14, 2016 at 3:16 pm #

      Thanks Johnny!

  6. Hilman May 24, 2016 at 10:56 pm #

    Hey Adrian. As usual, nice post.

    I am using Python 2.7 and there is also other thing needed to be changed to make this code works (If I ever missed the fact that you have wrote this in your post, pardon me, because I didn’t really read in details this post yet). Also, this is just a quick corrections I have tried. Maybe it is not perfect or not the best way etc. (still a novice here).

    First way of correcting it:
    In line 39 of the keyclipwriter.py file, the code

    “self.Q = Queue()”

    need to be changed to

    “self.Q = Queue.Queue”

    Second way of correcting it:
    Replace the line

    “from queue import Queue”

    to

    “from Queue import Queue” instead of “import Queue”

    • Hilman May 29, 2016 at 10:46 pm #

      I think I’ve made a mistake. Instead of changing to

      “self.Q = Queue.Queue”,

      change to

      “self.Q = Queue.Queue()”

  7. Hilman May 25, 2016 at 4:32 am #

    Adrian, I realized that in your save_key_events.py file, in line 20 (referring to the above), the key to access the value of the buffer size if it is given by the user would be

    “buffer-size”.

    But in line 101 (again, referring to the above) of the same file, the buffer size value (if it is given by the user) is being accessed with key

    “buffer_size”.

    A mistake perhaps? Or I am not in the same page here?

    • Adrian Rosebrock May 25, 2016 at 3:22 pm #

      The argparse library automatically converts dashes to underscores. Using the key buffer_size is correct.

      • Hilman May 26, 2016 at 3:46 am #

        Thanks for the reply. A new knowledge for me =/

        • Adrian Rosebrock May 26, 2016 at 6:16 am #

          I admit, it’s a little bit of a nuance and something that has to be learned with experience 🙂

  8. Mike July 4, 2016 at 11:04 am #

    I’ve spent the last 3 days embedded in this article and others on your blog. Thanks very much for your work. It is great.

    I’ve implemented your program here and I’ve noticed that my recorded video is like 2-3x faster than the reality or what I see on the original playback. It’s like watching something in fast forward. Is it possible this is a frame dropping issue? It doesn’t look like the library is designed to just save key frames, so I can’t think of what else this could be. Have you ever seen what I’m talking about or have other people reported it? I’m doing this on an rpi 3 having followed your rpi 3 opencv 3 tutorial.

    Thanks,

    Mike

    • Adrian Rosebrock July 5, 2016 at 1:49 pm #

      Hey Mike — the likely issue is that you have set a high FPS in the cv2.VideoWriter, but in reality, your video processing pipeline is not capable of processing that many frames per second. For example, if I want to write 32 FPS to file, but my processing loop can only process 10 frames per second, then my output video will look like it’s in “fast forward” mode.

      • Mike July 16, 2016 at 8:29 am #

        Thanks for the reply Adrian. This was exactly what was happening.

        To improve…

        I was able to increase the FPS a little bit by allocating more memory to the GPU by changing the /boot/config.txt gpu_mem to 512.

        I also got a big performance gain by making the image width 300px vs 600px (although I don’t want to do this).

        The program I’m using is a mixture of this script and your motion capture script. Basically motion capture is the trigger, but it’s using the rolling buffer to save the full event. I’m not sure exactly, but I’d guess I’m getting < 10 FPS. I've removed all of the green ball detection and any unneeded lines in the main processing loop.

        Using your FPS testing script, I get 30 FPS normal and 100+ FPS when threaded.

        I've been trying to optimize and figure out how to get the FPS up on the motion capture script, but I'm thinking maybe it is just not possible on the PI3, using CV2 go get past 10 FPS? Do you have a benchmark for this? Or any suggestions. I'm trying to do a simple motion capture app, but need a high FPS, I'm mainly trying to figure out what the PIs limits are, so if I'm hitting them, I can just accept that.

        Thanks for all the great work you do on this blog! I've been glued to it for a few weeks now.

        • Adrian Rosebrock July 18, 2016 at 5:20 pm #

          The FPS testing script is mainly used to test out how frames per second you can process, which essentially measures the throughput of your pipeline. It is not meant to be a true measure of the physical FPS of the device.

          10 FPS sounds a bit low, but you could certainly be hitting a ceiling depending on how many images are going into the average computation. One suggestion I would have is to re-implement your code in C/C++. This should give you a noticeable performance increase as well.

  9. Kartik December 29, 2016 at 2:37 am #

    Hey man!

    First of all, Thanks for this great tutorial and all others too.

    I am a novice and I am facing a few difficulties. I am getting this error:-

    “from pyimagesearch.keyclipwriter import KeyClipWriter
    ImportError: No module named pyimagesearch.keyclipwriter”

    I understood your KeyClipWriter code but I am not getting how to incorporate into the main code. Also, I am not able to find “pyimagesearch” using pip. Can you tell me what I need to do with the KeyClipWriter code?

    Many thanks in advance!

    • Adrian Rosebrock December 31, 2016 at 1:26 pm #

      Hey Kartik — it sounds like you’re copying and pasting the code as you work through the tutorial. That’s a great way to learn; however, I would wait until you have a bit more experience before doing that. Please use the “Downloads” section of this tutorial to download the code. This will give you the proper directory structure for the project and allow you to execute the code.

  10. Tira Sundara June 24, 2017 at 4:22 am #

    Hi Adrian, great tutorial as always.
    But, can i change the extension of the output video to .mp4 or .webm? Because i need to play it on the web. Since w3shools.com says “AVI (Audio Video Interleave). Developed by Microsoft. Commonly used in video cameras and TV hardware. Plays well on Windows computers, but not in web browsers.” (Source: https://www.w3schools.com/html/html_media.asp)

    • Adrian Rosebrock June 27, 2017 at 6:40 am #

      I cover how to save video files to disk in various formats in this post. However, keep in mind that saving videos to dish can be a bit of a pain and you might need to install different codecs and re-install OpenCV. In many ways, it’s a trial and error process.

  11. Mark Z December 24, 2017 at 3:51 pm #

    Love the blog! I am getting an error that I can’t figure out when I start the call kcw.start(__):

    [DEBUG] Filename: /home/pi/raspsecurity/videos/20171224-204800.avi
    Traceback (most recent call last):
    File “pi_surveillance.py”, line 133, in
    kcw.start(p, cv2.VideoWriter_fourcc(*’MJPG’), 20)
    File “/home/pi/raspsecurity/pyimagesearch/keyclipwriter.py”, line 40, in start
    (self.frames[0].shape[1], self.frames[0].shape[0]), True)
    IndexError: deque index out of range

    I tried changing all off the variables passed to start() to constants and still get the error, any guidance how what is going wrong?

    • Adrian Rosebrock December 26, 2017 at 4:05 pm #

      It looks like there might be action starting from the very first frame when the deque hasn’t had a chance to fill up. This code assumes there are already frames in the deque. I’m pretty sure this is the root cause of the error but make sure you debug this as well.

    • Thakur Rohit May 2, 2018 at 2:10 am #

      Hello, I am also getting the same error. If you solved it can you share your solution?

  12. Kartik Desai May 17, 2018 at 5:35 am #

    Instead of detecting green ball i.e green lower and green upper
    I want to save video locally when there is motion captured in camera.
    So how can I modify contour part.
    I have seen you tutorial of Room status occupied and unoccupied.And also this tutorial of saving .avi videos locally when green ball is detected but I am getting errors when I try to combine can you please help how it can be done or can you please upload a tutorial regarding that. As it will be really helpful for others as well.

    • Kartik Desai May 17, 2018 at 5:36 am #

      I am using raspberry pi camera

    • Adrian Rosebrock May 17, 2018 at 6:33 am #

      It sounds like you’ve already seen my motion detection tutorial using thresholding + contours. As you noted, you will need the combine the two to build the solution. I’m not sure what errors you may be getting, could you be more specific?

  13. Kartik Desai May 17, 2018 at 6:47 am #

    Can you please share a tutorial combining the two?
    It will be really helpful

    instead of detecting green ball it is needed to detect the motion in the frame and capture the video locally on raspberry pi either in avi or mp4 format till the motion in going on in the frame.

    for example:
    Just as like when ever there is room status: occupied save video locally .

    • Adrian Rosebrock May 17, 2018 at 7:22 am #

      Hey Kartik — I don’t have any plans on combining the two but if I do, I will let you know. I’m happy to help but I cannot write the code for you. If you’re getting stuck, no worries! That happens if you’re new to OpenCV or image processing. Instead of getting discouraged I would suggest working through Practical Python and OpenCV to help you learn the fundamentals first. It can be challenging to combine computer vision/OpenCV techniques without a strong understanding of the fundamentals. Take a look, I have faith in you and I know you can do it! 🙂

  14. usuf May 28, 2018 at 3:03 am #

    thank for amazing tuto
    how can i store that motion part as a image file instead of video any idea?

    • Adrian Rosebrock May 28, 2018 at 9:27 am #

      You can use the cv2.imwrite function to write images/frames to disk.

  15. srinivas December 3, 2018 at 6:50 am #

    Can anyone help me how to change the green ball to some vehicle or knife. Whether this code is sufficient to detect any object or i have to write code for detecting the objects i want. If i have to write on my own then how to achieve it?

    Here the key event is detecting green ball. How can i change it to Reporting unsafe driving outside my home to the authorities? Some one help me please.

Leave a Reply

[email]
[email]