Object tracking with dlib

Click here to download the source code to this post.

This tutorial will teach you how to perform object tracking using dlib and Python. After reading today’s blog post you will be able to track objects in real-time video with dlib.

A couple months ago we discussed centroid tracking, a simple, yet effective method to (1) assign unique IDs to each object in an image and then (2) track each of the objects and associated IDs as they move around in a video stream.

The biggest downside to this object tracking algorithm is that a separate object detector has to be run on each and every input frame — in most situations, this behavior is undesirable as object detectors, including HOG + Linear SVM, Faster R-CNNs, and SSDs can be computationally expensive to run.

An alternative approach would be to:

  1. Perform object detection once (or once every N frames)
  2. And then apply a dedicated tracking algorithm that can keep tracking of the object as it moves in subsequent frames without having to perform object detection

Is such a method possible?

The answer is yes, and in particular, we can use dlib’s implementation of the correlation tracking algorithm.

In the remainder of today’s blog post, you will learn how to apply dlib’s correlation tracker to track an object in real-time in a video stream.

To learn more about dlib’s correlation tracker, just keep reading.

Looking for the source code to this post?
Jump right to the downloads section.

Object tracking with dlib

We’ll start off today’s tutorial with a brief discussion of dlib’s implementation of correlation-based object tracking.

From there I will show you how to utilize dlib’s object tracker in your own applications.

Finally, we’ll wrap up today by discussing some of the limitations and drawbacks of dlib’s object tracker.

What are correlation trackers?

The dlib correlation tracker implementation is based on Danelljan et al.’s 2014 paper, Accurate Scale Estimation for Robust Visual Tracking.

Their work, in turn, builds on the popular MOSSE tracker from Bolme et al.’s 2010 work, Visual Object Tracking using Adaptive Correlation Filters. While the MOSSE tracker works well for objects that are translated, it often fails for objects that change in scale.

The work of Danelljan et al. proposed utilizing a scale pyramid to accurately estimate the scale of an object after the optimal translation was found. This breakthrough allows us to track objects that change in both (1) translation and (2) scaling throughout a video stream — and furthermore, we can perform this tracking in real-time.

For a detailed review of the algorithm, please refer to the papers linked above.

Project structure

To see how this project is organized, simply use the tree  command in your terminal:

We have three directories:

  • input/ : Contains input videos for object tracking.
  • output/ : Our processed videos. In the processed video, the tracked object is annotated with a box and label.
  • mobilenet_ssd/ : The Caffe CNN model files are contained within this directory.

Today we’ll be reviewing one Python script: track_object.py .

Implementing our dlib object tracker

Let’s go ahead and get started implementing our object tracker using dlib.

Open up track_object.py  and insert the following code:

Here we import our required packages. Notably, we’re using dlib, imutils, and OpenCV.

From there, let’s parse our command line arguments:

Our script has four required command line arguments:

  • --prototxt : Our path to the Caffe deploy prototxt file.
  • --model : The path to the Caffe pre-trained model.
  • --video : The path to the input video file. Today’s script works with video files rather than your webcam (but you could easily change it to support a webcam stream).
  • --label : A class label that we are interested in detecting and tracking. Review the next code block for the available classes that this model supports.

And two optional ones:

  • --output : An optional path to an output video file if you’d like to save the results of the object tracker.
  • --confidence : With a default=0.2 , this is the minimum probability threshold and it allows us to filter weak detections from our Caffe object detector.

Let’s define the classes that this model supports and load our network from disk:

We’ll be using a pre-trained MobileNet SSD to perform object detection in a single frame. From there the object location will be handed off to dlib’s correlation tracker for tracking throughout the remaining frames of the video.

The model included with the “Downloads” supports 20 object classes (plus 1 for the background class) on Lines 27-30.

Note: If you’re using a different Caffe model, you’ll need to redefine this CLASSES  list. Similarly, don’t modify this list if you’re using the model included with today’s download. If you’re confused about how deep learning object detectors work, be sure to refer to this getting started guide.

Prior to looping over frames, we need to load our model into memory. This is handled on Line 34 where all that is required to load a Caffe model is the path to the prototxt and model files (both available in our command line args  dictionary).

Now let’s perform important initializations, notably our video stream:

Our video stream, tracker , and video writer  objects are initialized on Lines 39-41. We also initialize our textual label  on Line 42.

Our frames-per-second estimator is instantiated on Line 45.

Now we’re ready to begin looping over our video frames:

We begin our while  loop on Line 48 and proceed to grab a frame  on Line 50.

Our frame is resized and the color channels are swapped on Lines 58 and 59. Resizing allows for faster processing — you can experiment with the frame dimensions to achieve higher FPS. Converting to RGB color space is required by dlib (OpenCV stores images in BGR order by default).

Optionally, at runtime, an output video path can be passed via command line arguments. So, if necessary, we’ll initialize our video writer  on Lines 63-66. For more information on writing video to disk with OpenCV, see this previous post.

Next, we’ll need to detect an object for tracking (if we haven’t already):

If our tracker  object is None  (Line 71), we first need to detect objects in the input frame . To do so, we create a blob  (Line 74) and pass it through the network (Lines 78 and 79).

Let’s handle the detections  now:

If our object detector finds any objects (Line 82), we’ll grab the one with the largest probability (Line 88).

We’re only demonstrating how to use dlib to perform single object tracking in this post, so we need to find the detected object with the highest probability. Next week’s blog post will cover multi-object tracking with dlib.

From there, we’ll grab the confidence ( conf ) and label  associated with the object (Lines 92 and 93).

Now it’s time to filter out the detections. Here we’re trying to ensure we have the right type of object which was passed by command line argument:

On Line 97 we check to ensure that conf  exceeds the confidence threshold and that the object is actually the class type we’re looking for. When we run the script later, we’ll use “person” or “cat” as examples so you can see how we can filter results.

We determine bounding box  coordinates of our object on Lines 100 and 101.

Then we establish our dlib object tracker and provide the bounding box coordinates (Lines 106-108). Future tracking updates will be easy from here on.

A bounding box rectangle and object class label  text is drawn on the frame  on Lines 111-114.

Let’s handle the case where we’ve already established a tracker :

This else  block handles the case where we’ve already locked on to an object for tracking.

Think of it like a dogfight in the movie, Top Gun. Once the enemy aircraft has been locked on by the “guidance system”, it can be tracked via updates.

This requires two main actions on our part:

  1. Update our tracker object (Line 121) — the heavy lifting is performed in the backend of this update  method.
  2. Grab the position ( get_position ) of our object from the tracker  (Line 122). This would be where a PID control loop would come in handy if, for example, a robot seeks to follow a tracked object. In our case, we’re just going to annotate the object in the frame with a bounding box and label on Lines 131-134.

Let’s finish out the loop:

If the frame  should be written to video, we do so on Lines 137 and 138.

We’ll show the frame  on the screen (Line 141).

If the quit key (“q”) is pressed at any point during playback + tracking, we’ll break  out of the loop (Lines 142-146).

Our fps  estimator is updated on Line 149.

Finally, let’s perform print out FPS throughput statistics and release pointers prior to the script exiting:

Housekeeping for our script includes:

  • Our fps  counter is stopped and the FPS information is displayed in the terminal (Lines 152-154.
  • Then, if we were writing to an output video, we release the video writer  (Lines 157 and 158).
  • Lastly, we close all OpenCV windows and release the video stream (Lines 161 and 162).

Running dlib’s object tracker in real-time

To see our dlib object tracker in action, make sure you use the “Downloads” section of this blog post to download the source code.

From there, open up a terminal and execute the following command:

Usain Bolt (Olympic World Record holder) was detected initially with highest confidence at the beginning of the video. From there, he is tracked successfully throughout his 100m race.

The full video can be found below:

Below we have a second example of object tracking with dlib:

The cat above was part of a BuzzFeed segment cat owners trying to take their cats for a walk (as if they were dogs). Poor cats!

Drawbacks and potential improvements

If you watched the full output video of the demo above, you would have noticed the object tracker behaving strangely towards the end of the demo, as this GIF demonstrates.

So, what’s going on here?

Why is the tracker losing the object?

Keep in mind there is no such thing as a “perfect” object tracker — and furthermore, this object tracking algorithm is not requiring you to run a more expensive object detector on each and every frame of the input image.

Instead, dlib’s correlation tracker is combining both (1) prior information regarding the location of the object bounding box in the previous frame along with (2) data garnered from the current frame to infer where the new location of the object is.

There will certainly be times when the algorithm loses the object.

To remedy this situation, I recommend occasionally running your more expensive object detector to (1) validate the object is still there and (2) reseed the object tracking with the updated (and ideally correct) bounding box coordinates. August’s blog post on people counting with OpenCV accomplished exactly this, so be sure to check it out.

What about multi-object tracking?

Undoubtedly, I know there will be PyImageSearch readers wishing to apply this method to multi-object tracking rather than single object tracking.

Is it possible to track multiple objects using dlib’s correlation tracker?

The answer is yes, absolutely!

I’ll be covering multi-object tracking next week, so stay tuned.

Video credits

To create the examples for this tutorial I needed to use clips from two different videos. A big thank you and credit to BuzzFeed Video and GERrevolt.

Summary

In today’s blog post we discussed dlib’s object tracking algorithm.

Unlike July’s tutorial on centroid tracking, dlib’s object tracking algorithm can update itself utilizing information garnered from the input RGB image — the algorithm does not require that a set of bounding boxes be computed for each and every frame in the input video stream.

As we found out, dlib’s correlation tracking algorithm is quite robust and capable of running in real-time.

However, the biggest drawback is that the correlation tracker can become “confused” and lose the object we wish to track if viewpoint changes substantially or if the object to be tracked becomes occluded.

In those scenarios we can re-run our (computationally expensive) object detector to re-determine the location of our tracked object — be sure to refer to this blog post on people counting for such an implementation.

In our next blog post we’ll be discussing multi-object tracking with dlib — to be notified when the next blog post goes live (and download the source code to today’s post), just enter your email address in the form below.

Downloads:

If you would like to download the code and images used in this post, please enter your email address in the form below. Not only will you get a .zip of the code, I’ll also send you a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL! Sound good? If so, enter your email address and I’ll send you the code immediately!

, ,

26 Responses to Object tracking with dlib

  1. Ebrahim Rupawala October 22, 2018 at 11:22 am #

    Hello Adrian! As usual, your blogs are so useful and amazing, they make our projects alot easier. I was wondering if you can do a tutorial on the Intel’s new Open VINO toolkit, so that we could make use of open cv’s algorithms as well as use the Optimization, deployment tools provided by Intel in the open VINO toolkit to rapidly prototype our computer vision applications. Thank you so much!

    • Adrian Rosebrock October 22, 2018 at 12:54 pm #

      Thanks for the comment Ebrahim. And thank you for the suggestion on Intel’s Open VINO toolkit. I cannot guarantee if or when I would cover the topic but I’ll certainly give it some thought.

    • Gagandeep November 15, 2018 at 5:26 am #

      Hii Ebrahim

      I am working on project where we are using openvino kit for deep learning solution , but it not able to detect licence plate recognition for specific country , so i want to include that failure inside to the openvino.. you have any idea about that??

  2. david Sastre October 22, 2018 at 11:37 am #

    I am really enjoying this last posts about video trackers. Although I work over C++, I am finding them really helpful. I didn’t know about the possibility of dlib. I am developing a project on a raspberry pi based on video tracking, and the only tracker that can be done real time here on full resolution is the MOSSE tracker. The drawback is that it doesn’t support change in scale, and it is important for me because I am tracking from the air, form a moving UAV. You just solved me this problem on this post, thank you very much!

    By the way, I’ve seen you don’t have any post regarding opencv and gstreamer use. It is a very common combination, and a bit tricky to get working. I personally have it already working, but I am sure a post on this topic would help lot of readers.

    Thank you, keep the good work!

    Regards.

    • Adrian Rosebrock October 22, 2018 at 12:56 pm #

      Thanks David! You’ll be able to use dlib directly from your C++ application which is awesome, I’m glad that helps resolve the problem for you 🙂 Also, thank you for the suggestion on OpenCV and gstreamer. It is a bit tricky. I’ll see if I can cover it in a blog post but I can pretty much guarantee it’s going to be apart of our upcoming Computer Vision + Raspberry Pi book.

  3. Anand C U October 23, 2018 at 1:45 am #

    Hi Adrian. Thanks for the fantastic tutorial.

    I’m currently building a automobile traffic counter with SSD-Mobilenet and MOSSE Tracker.
    The issue I’m facing is to correctly correlate between objects that were tracked on the previous frames and the new objects from the SSD-Mobilenet (I’m running detection module every 5 frames). Currently I am using Non-Maximum Suppression technique to compare the bounding boxes and to decide that a tracked object is same as the one detected in the current frame. However, this fails a lot of times and is leading to erroneous counts. Do you have any suggestions to achieve this in a better way? Thanks

    • Adrian Rosebrock October 23, 2018 at 4:35 am #

      Hey Anand — have you read my previous tutorial on building na OpenCV people counter? That tutorial would give you the starting point for your traffic counter, including how to associate objects between frames.

  4. Jasper H October 23, 2018 at 6:04 pm #

    Hi Adrian,

    Thanks for yet another useful post. How does this dlib correlation tracker compare with those implemented in OpenCV? For example, you had a summary post on those back in July https://www.pyimagesearch.com/2018/07/30/opencv-object-tracking/

  5. Dan Ionescu October 25, 2018 at 4:41 pm #

    Great post again Adrian!

    I’ve read a previous post about opencv trackers and I was wondering how accurate and how fast is this tracker compared to those? I want to use it on a raspberry pi

    Thanks

  6. Chase Lewis October 26, 2018 at 12:13 am #

    Hey Adrian,

    I noticed the update of the tracker returns the ‘Peak to Side-Lobe Ratio’. All the documentation says it is a ‘confidence that the object is within the rect’. Any idea how this value is created / used? I imagine a robust ‘tracker’ would use this value and if it went below a certain threshold return to using the detector.

    For a sample project i’m seeing it start at ~20 and drop to 9 but without more info hard to understand how to construct this value should be used.

    An article about improving the ‘robustness’ and recovering from tracking getting off would be interesting.

    Chase Lewis

  7. Mohamed October 28, 2018 at 9:24 am #

    Hi Adrian. Great post as usual ,Thank you.
    I’m currently building vehicle tracking system ,do you thing using Extended Kalman Filter will be better than centroid and dlib tracker?

    • Adrian Rosebrock October 29, 2018 at 1:23 pm #

      The only way to know is to try 🙂 Run experiments with both and let the empirical results drive your decision making.

  8. Val October 28, 2018 at 11:49 am #

    Hello Adrian!
    Thanks for the awesome books and tutorials!

    I work on detecting additional facial features like forehead wrinkles and the nasolabial fold for better emotion prediction.

    Can you please advise on what the approach to find these features should look like?

    • Adrian Rosebrock October 29, 2018 at 1:23 pm #

      For your project I would suggest first performing facial landmark prediction. Inside the comments of the post you’ll see my recommendations for detecting the forehead region as well.

  9. adi October 29, 2018 at 3:10 am #

    Hey Adrian,

    Do you have an email you can be reached at?

    Adi

  10. zana zakaryaie nejad October 29, 2018 at 5:56 am #

    Hi

    It was a really good news that dlib has implemented multi-scale MOSSE (DSST).

    Is there anybody who have tested it and compared with KCF? which one is faster?

    MOSSE itself is much much faster than KCF but I wanted to know about its multi-scale version.

    • Adrian Rosebrock October 29, 2018 at 1:11 pm #

      I actually provide a full review of MOSSE, KCF, and other trackers in this tutorial.

  11. Falahgs October 29, 2018 at 6:10 am #

    Thank you Adrian for the post
    But I wonder why it was Detection and tracking only one person the first player
    within the video, especially there are several players in the scene
    Thank you so much

    • Adrian Rosebrock October 29, 2018 at 1:11 pm #

      The focus of the tutorial is on object tracking not object detection. Object detection is only performed in the very first frame. Each object is then tracked. He was not detected in that frame and thus he was not tracked. For a full-fledged person tracking application see this tutorial on people tracking.

  12. sophie November 13, 2018 at 2:24 pm #

    Hello Adrian,thanks for your great post. I want to ask if i can use the other deep network, for example YOLOv3.
    Thank you so much.

    • Adrian Rosebrock November 13, 2018 at 4:09 pm #

      Yes, you can swap out the SSD for whatever object detector you would like.

  13. Ibn Ahmad November 14, 2018 at 5:43 am #

    Thanks for the nice tutorials once more. In a situation where we change our Video stream to read video from webcam, what will be the –video argument to pass in the command line?

    • Adrian Rosebrock November 15, 2018 at 12:05 pm #

      You would supply the index of your webcam which is typically “0”:

      vs = VideoStream(src=0)

  14. Gagandeep November 15, 2018 at 5:15 am #

    Hii adraian

    For last 3 sec of video i not able to predict or tracking the location of person.. box start moving away from person.. how we can improve the accuracy of this system

Leave a Reply