Real-time facial landmark detection with OpenCV, Python, and dlib

Over the past few weeks we have been discussing facial landmarks and the role they play in computer vision and image processing.

We’ve started off by learning how to detect facial landmarks in an image.

We then discovered how to label and annotate each of the facial regions, such as eyes, eyebrows, nose, mouth, and jawline.

Today we are going to expand our implementation of facial landmarks to work in real-time video streams, paving the way for more real-world applications, including next week’s tutorial on blink detection.

To learn how to detect facial landmarks in video streams in real-time, just keep reading.

Looking for the source code to this post?
Jump right to the downloads section.

Real-time facial landmark detection with OpenCV, Python, and dlib

The first part of this blog post will provide an implementation of real-time facial landmark detection for usage in video streams utilizing Python, OpenCV, and dlib.

We’ll then test our implementation and use it to detect facial landmarks in videos.

Facial landmarks in video streams

Let’s go ahead and get this facial landmark example started.

Open up a new file, name it , and insert the following code:

Lines 2-9 import our required Python packages.

We’ll be using the face_utils  sub-module of imutils, so if you haven’t installed/upgraded to the latest version, take a second and do so now:

Note: If you are using Python virtual environments, take care to ensure you are installing/upgrading imutils  in your proper environment.

We’ll also be using the VideoStream  implementation inside of imutils , allowing you to access your webcam/USB camera/Raspberry Pi camera module in a more efficientfaster, treaded manner. You can read more about the VideoStream  class and how it accomplishes a higher frame throughout in this blog post.

If you would like to instead work with video files rather than video streams, be sure to reference this blog post on efficient frame polling from a pre-recorded video file, replacing VideoStream  with FileVideoStream .

For our facial landmark implementation we’ll be using the dlib library. You can learn how to install dlib on your system in this tutorial (if you haven’t done so already).

Next, let’s parse our command line arguments:

Our script requires one command line argument, followed by a second optional one, each detailed below:

  • --shape-predictor : The path to dlib’s pre-trained facial landmark detector. Use the “Downloads” section of this blog post to download an archive of the code + facial landmark predictor file.
  • --picamera : An optional command line argument, this switch indicates whether the Raspberry Pi camera module should be used instead of the default webcam/USB camera. Supply a value > 0 to use your Raspberry Pi camera.

Now that our command line arguments have been parsed, we need to initialize dlib’s HOG + Linear SVM-based face detector and then load the facial landmark predictor from disk:

The next code block simply handles initializing our VideoStream  and allowing the camera sensor to warm up:

The heart of our video processing pipeline can be found inside the while  loop below:

On Line 31 we start an infinite loop that we can only break out of if we decide to exit the script by pressing the q  key on our keyboard.

Line 35 grabs the next frame from our video stream.

We then preprocess this frame by resizing it to have a width of 400 pixels and convert it to grayscale (Lines 36 an 37).

Before we can detect facial landmarks in our frame, we first need to localize the face — this is accomplished on Line 40 via the detector  which returns the bounding box (x, y)-coordinates for each face in the image.

Now that we have detected the faces in the video stream, the next step is to apply the facial landmark predictor to each face ROI:

On Line 43 we loop over each of the detected faces.

Line 47 applies the facial landmark detector to the face region, returning a shape  object which we convert to a NumPy array (Line 48).

Lines 52 and 53 then draw a series of circles on the output frame , visualizing each of the facial landmarks. To understand what facial region (i.e., nose, eyes, mouth, etc.) each (x, y)-coordinate maps to, please refer to this blog post.

Lines 56 and 57 display the output frame  to our screen. If the q  key is pressed, we break from the loop and stop the script (Lines 60 and 61).

Finally, Lines 64 and 65 do a bit of cleanup:

As you can see, there are very little differences between detecting facial landmarks in images versus detecting facial landmarks in video streams — the main differences in the code simply involve setting up our video stream pointers and then polling the stream for frames.

The actual process of detecting facial landmarks is the same, only instead of detecting facial landmarks in a single image we are now detecting facial landmarks in a series of frames.

Real-time facial landmark results

To test our real-time facial landmark detector using OpenCV, Python, and dlib, make sure you use the “Downloads” section of this blog post to download an archive of the code, project structure, and facial landmark predictor model.

If you are using a standard webcam/USB camera, you can execute the following command to start the video facial landmark predictor:

Otherwise, if you are on your Raspberry Pi, make sure you append the --picamera 1  switch to the command:

Here is a short GIF of the output where you can see that facial landmarks have been successfully detected on my face in real-time:

Figure 1: A short demo of real-time facial landmark detection with OpenCV, Python, an dlib.

I have included a full video output below as well:


In today’s blog post we extended our previous tutorials on facial landmarks and applied them to the task of real-time detection.

As our results demonstrated, we are fully capable of detecting facial landmarks in a video stream in real-time using a system with a modest CPU.

Now that we understand how to access a video stream and apply facial landmark detection, we can move on to next week’s real-world computer vision application — blink detection.

To be notified when the blink detection tutorial goes live, be sure to enter your email address in the form below — this is a tutorial you won’t want to miss!


If you would like to download the code and images used in this post, please enter your email address in the form below. Not only will you get a .zip of the code, I’ll also send you a FREE 11-page Resource Guide on Computer Vision and Image Search Engines, including exclusive techniques that I don’t post on this blog! Sound good? If so, enter your email address and I’ll send you the code immediately!

, , , , , , ,

26 Responses to Real-time facial landmark detection with OpenCV, Python, and dlib

  1. tony April 17, 2017 at 12:03 pm #

    Thanks for this tutorial . how the face landmarks can be more stable , I tried the tutorial and the points are shaky

  2. Mansoor Nasir April 17, 2017 at 1:00 pm #

    Great work Adrian, my only question is will it work with multiple faces? And will it affect the performance or accuracy?

    • Adrian Rosebrock April 19, 2017 at 12:58 pm #

      I would suggest going back and reading my previous posts facial landmarks. This method will work with multiple faces provided that each face in the image/video stream can be detected.

  3. Shravan Kumar Parunandula April 17, 2017 at 1:05 pm #

    Awaiting for this, thank you so much.

    • Adrian Rosebrock April 19, 2017 at 12:57 pm #

      Thank you Shravan! 🙂

  4. Linus April 17, 2017 at 1:40 pm #

    This one is freaking awesome! Will definitively try it out and install dlib. Thanks Adrian for this row of posts! 🙂

    • Adrian Rosebrock April 19, 2017 at 12:55 pm #

      Thanks Linus — it only gets better from here 🙂

  5. Muhammad April 17, 2017 at 2:34 pm #

    Beautiful! Thanks a lot!

  6. Levi Blaney April 17, 2017 at 9:02 pm #

    Hey this is really great stuff. I can’t wait to try it out. I want to use it for my magic mirror to tell who is standing in front of it. I’m sure I could Google a algorithm up but could you do a blog post on how to some what reliably detect the same person over and over again.

    • Adrian Rosebrock April 19, 2017 at 12:53 pm #

      I wouldn’t recommend using facial landmarks for facial recognition. Algorithms such as LBPs for face recognition, Eigenfaces, and Fisherfaces would work well for a magic mirror application. I cover LBPs for face recognition and Eigenfaces inside the PyImageSearch Gurus course.

  7. Joe April 17, 2017 at 9:53 pm #

    This is awesome! Thanks for another great blog Adrian….keep it up!

    • Adrian Rosebrock April 19, 2017 at 12:51 pm #

      Thanks Joe! 🙂

  8. kunal April 18, 2017 at 3:59 am #

    Really impressed by the way you have done with this coding of real-time facial landmark detection for usage in video streams utilizing Python. Splendid!

  9. Linus April 18, 2017 at 1:59 pm #

    And I can’t understand why you import datetime? Is this just from development?

    And the whole thing worked out just fine BTW 😀

    • Adrian Rosebrock April 19, 2017 at 12:48 pm #

      The import datetime cam be safely removed. I had it imported for a different application I was working on.

  10. David J Axelrod April 19, 2017 at 11:07 am #

    Woah, super cool Adrian! Another awesome article

    • Adrian Rosebrock April 19, 2017 at 12:42 pm #

      Thank you David!

  11. Matt Sandy April 19, 2017 at 11:30 pm #

    I really want to make a game controlled by facial expressions. I think it would be hilarious to get people to play it in public.

  12. Sidharth Patnaik April 20, 2017 at 9:49 pm #

    Just another awesome Tutorial, thanks for sharing! 🙂
    was waiting for this, the whole time.
    can you upload a tutorial based on OpenFace, please ?

    • Adrian Rosebrock April 21, 2017 at 10:51 am #

      Sure, I will certainly consider this for a future tutorial.

  13. tony April 21, 2017 at 4:18 am #

    Thanks for this tutorial , I have asked you this question and I haven’t got reply.

    how the face landmarks can be more stable , I tried the tutorial and the points are shaky

    • Adrian Rosebrock April 21, 2017 at 10:45 am #

      Hi Tony — I’m not sure what you mean by “shaky”. The facial landmark predictor included by dlib is pre-trained. You could try to train your own predictor on your own data to see if that improves your result.

      • tony April 22, 2017 at 12:52 am #

        Thanks , I mean the landmark points are not stable ( shaking) . How can I train new predictor for more than 68 landmarks ?

        • Adrian Rosebrock April 24, 2017 at 9:48 am #

          You would need to use the dlib library. This example demonstrates how to train a custom shape predictor.


  1. Eye blink detection with OpenCV, Python, and dlib - PyImageSearch - April 24, 2017

    […] last week’s blog post, I demonstrated how to perform facial landmark detection in real-time in video […]

Leave a Reply