OpenCV Vehicle Detection, Tracking, and Speed Estimation

In this tutorial, you will learn how to use OpenCV and Deep Learning to detect vehicles in video streams, track them, and apply speed estimation to detect the MPH/KPH of the moving vehicle.

This tutorial is inspired by PyImageSearch readers who have emailed me asking for speed estimation computer vision solutions.

As pedestrians taking the dog for a walk, escorting our kids to school, or marching to our workplace in the morning, we’ve all experienced unsafe, fast-moving vehicles operated by inattentive drivers that nearly mow us down.

Many of us live in apartment complexes or housing neighborhoods where ignorant drivers disregard safety and zoom by, going way too fast.

We feel almost powerless. These drivers disregard speed limits, crosswalk areas, school zones, and “children at play” signs altogether. When there is a speed bump, they speed up almost as if they are trying to catch some air!

Is there anything we can do?

In most cases, the answer is unfortunately “no” — we have to look out for ourselves and our families by being careful as we walk in the neighborhoods we live in.

But what if we could catch these reckless neighborhood miscreants in action and provide video evidence of the vehicle, speed, and time of day to local authorities?

In fact, we can.

In this tutorial, we’ll build an OpenCV project that:

  1. Detects vehicles in video using a MobileNet SSD and Intel Movidius Neural Compute Stick (NCS)
  2. Tracks the vehicles
  3. Estimates the speed of a vehicle and stores the evidence in the cloud (specifically in a Dropbox folder).

Once in the cloud, you can provide the shareable link to anyone you choose. I sincerely hope it will make a difference in your neighborhood.

Let’s take a ride of our own and learn how to estimate vehicle speed using a Raspberry Pi and Intel Movidius NCS.

Note: Today’s tutorial is actually a chapter from my new book, Raspberry Pi for Computer Vision. This book shows you how to push the limits of the Raspberry Pi to build real-world Computer Vision, Deep Learning, and OpenCV Projects. Be sure to pick up a copy of the book if you enjoy today’s tutorial.

Looking for the source code to this post?
Jump right to the downloads section.

OpenCV Vehicle Detection, Tracking, and Speed Estimation

In this tutorial, we will review the concept of VASCAR, a method that police use for measuring the speed of moving objects using distance and timestamps. We’ll also understand how here is a human component that leads to error and how our method can correct the human error.

From there, we’ll design our computer vision system to collect timestamps of cars to measure speed (with a known distance). By eliminating the human component, our system will rely on our knowledge of physics and our software development skills.

Our system relies on a combination of object detection and object tracking to find cars in a video stream at different waypoints. We’ll briefly review these concepts so that we can build out our OpenCV speed estimation driver script.

Finally we’ll deploy and test our system. Calibration is necessary for all speed measurement devices (including RADAR/LIDAR) — ours is no different. We’ll learn to conduct drive tests and how to calibrate our system.

What is VASCAR and how is it used to measure speed?

Figure 1: Vehicle Average Speed Computer and Recorder (VASCAR) devices allow police to measure speed without RADAR or LIDAR, both of which can be detected. We will use a VASCAR-esque approach with OpenCV to detect vehicles, track them, and estimate their speeds without relying on the human component.

Visual Average Speed Computer and Recorder (VASCAR) is a method for calculating the speed of vehicles —  it does not rely on RADAR or LIDAR, but it borrows from those acronyms. Instead, VASCAR is a simple timing device relying on the following equation:

Police use VASCAR where RADAR and LIDAR is illegal or when they don’t want to be detected by RADAR/LIDAR detectors.

In order to utilize the VASCAR method, police must know the distance between two fixed points on the road (such as signs, lines, trees, bridges, or other reference points)

  • When a vehicle passes the first reference point, they press a button to start the timer.
  • When the vehicle passes the second point, the timer is stopped.

The speed is automatically computed as the computer already knows the distance per Equation 1.1.

Speed measured by VASCAR is severely limited by the human factor.

For example, what if the police officer has poor eyesight or poor reaction time

If they press the button late (first reference point) and then early (second reference point), then your speed will be calculated faster than you are actually going since the time component is smaller.

If you are ever issued a ticket by a police officer and it says VASCAR on it, then you have a very good chance of getting out of the ticket in a courtroom. You can (and should) fight it. Be prepared with Equation 1.1 above and to explain how significant the human component is.

Our project relies on a VASCAR approach, but with four reference points. We will average the speed between all four points with a goal of having a better estimate of the speed. Our system is also dependent upon the distance and time components.

For further reading about VASCAR, please refer to the VASCAR Wikipedia article.

Configuring your Raspberry Pi 4 + OpenVINO environment

Figure 2: Configuring OpenVINO on your Raspberry Pi for detecting/tracking vehicles and measuring vehicle speed.

This tutorial requires a Raspberry Pi 4B and Movidius NCS2 (or higher once faster versions are released in the future). Lower Raspberry Pi and NCS models are simply not fast enough. Another option is to use a capable laptop/desktop without OpenVINO altogether.

Configuring your Raspberry Pi 4B + Intel Movidius NCS for this project is admittedly challenging.

I suggest you (1) pick up a copy of Raspberry Pi for Computer Vision, and (2) flash the included pre-configured .img to your microSD. The .img that comes included with the book is worth its weight in gold.

For the stubborn few who wish to configure their Raspberry Pi 4 + OpenVINO on their own, here is a brief guide:

  1. Head to my BusterOS install guide and follow all instructions to create an environment named cv . Ensure that you use a RPi 4B model (either 1GB, 2GB, or 4GB).
  2. Head to my OpenVINO installation guide and create a 2nd environment named openvino . Be sure to download the latest OpenVINO and not an older version.

At this point, your RPi will have both a normal OpenCV environment as well as an OpenVINO-OpenCV environment. You will use the openvino  environment for this tutorial.

Now, simply plug in your NCS2 into a blue USB 3.0 port (for maximum speed) and follow along for the rest of the tutorial.

Caveats:

  • Some versions of OpenVINO struggle to read .mp4 videos. This is a known bug that PyImageSearch has reported to the Intel team. Our preconfigured .img includes a fix — Abhishek Thanki edited the source code and compiled OpenVINO from source. This blog post is long enough as is, so I cannot include the compile-from-source instructions. If you encounter this issue please encourage Intel to fix the problem, and either (A) compile from source, or (B) pick up a copy of Raspberry Pi for Computer Vision and use the pre-configured .img.
  • We will add to this list if we discover other caveats.

Project Structure

Let’s review our project structure:

Our config.json file holds all the project settings — we will review these configurations in the next section. Inside Raspberry Pi for Computer Vision with Python, you’ll find configuration files with most chapters. You can tweak each configuration to your needs. These come in the form of commented JSON or Python files. Using the package, json_minify , comments are parsed out so that the JSON Python module can load the data as a Python dictionary.

We will be taking advantage of both the CentroidTracker and TrackableObject classes in this project. The centroid tracker is identical to previous people/vehicle counting projects in the Hobbyist Bundle (Chapters 19 and 20) and Hacker Bundle (Chapter 13). Our trackable object class, on the other hand, includes additional attributes that we will keep track of including timestamps, positions, and speeds.

A sample video compilation from vehicles passing in front of my colleague Dave Hoffman’s house is included ( cars.mp4).

This video is provided for demo purposes; however, take note that you should not rely on video files for accurate speeds — the FPS of the video, in addition to the speed at which frames are read from the file, will impact speed readouts.

Videos like the one provided are great for ensuring that the program functions as intended, but again, accurate speed readings from video files are not likely — for accurate readings you should be utilizing a live video stream.

The output/ folder will store a log file, log.csv, which includes the timestamps and speeds of vehicles that have passed the camera.

Our pre-trained Caffe MobileNet SSD object detector (used to detect vehicles) files are included in the root of the project.

A testing script is included — speed_estimation_dl_video.py . It is identical to the live script, with the exception that it uses a prerecorded video file. Refer to this note:

Note: OpenCV cannot automatically throttle a video file framerate according to the true framerate. If you use speed_estimation_dl_video.py as well as the supplied cars.mp4  testing file, keep in mind that the speeds reported will be inaccurate. For accurate speeds, you must set up the full experiment with a camera and have real cars drive by. Refer to the next section, “Calibrating for Accuracy”, for a real live demo in which a screencast was recorded of the live system in action.

The driver script, speed_estimation_dl.py, interacts with the live video stream, object detector, and calculates the speeds of vehicles using the VASCAR approach. It is one of the longer scripts we cover in Raspberry Pi for Computer Vision.

Speed Estimation Config file

When I’m working on projects involving many configurable constants, as well as input/output files and directories, I like to create a separate configuration file.

In some cases, I use JSON and other cases Python files. We could argue all day over which is easiest (JSON, YAML, XML, .py, etc.), but for most projects inside of Raspberry Pi for Computer Vision with Python we use either a Python or JSON configuration in place of a lengthy list of command line arguments.

Let’s review config.json, our JSON configuration settings file:

The “max_disappear” and “max_distance” variables are used for centroid tracking and object association:

  • The "max_disappear" frame count signals to our centroid tracker when to mark an object as disappeared (Line 5).
  • The "max_distance" value is the maximum Euclidean distance in pixels for which we’ll associate object centroids (Line 10). If centroids exceed this distance we mark the object as disappeared.

Our "track_object" value represents the number of frames to perform object tracking rather than object detection (Line 14).

Performing detection on every frame would be too computationally expensive for the RPi. Instead, we use an object tracker to lessen the load on the Pi. We’ll then intermittently perform object detection every N frames to re-associate objects and improve our tracking.

The "confidence" value is the probability threshold for object detection with MobileNet SSD. Detect objects (i.e. cars, trucks, buses, etc.) that don’t meet the confidence threshold are ignored (Line 17).

Each input frame will be resized to a "frame_width" of 400 (Line 20).

As mentioned previously, we have four speed estimation zones. Line 23 holds a dictionary of the frame’s columns (i.e. y-pixels) separating the zones. These columns are obviously dependent upon the "frame_width".

Figure 3: The camera’s FOV is measured at the roadside carefully. Oftentimes calibration is required. Refer to the “Calibrating for Accuracy” section to learn about the calibration procedure for neighborhood speed estimation and vehicle tracking with OpenCV.

Line 26 is the most important value in this configuration. You will have to physically measure the "distance" on the road from one side of the frame to the other side.

It will be easier if you have a helper to make the measurement. Have the helper watch the screen and tell you when you are standing at the very edge of the frame. Put the tape down on the ground at that point. Stretch the tape to the other side of the frame until your helper tells you that they see you at the very edge of the frame in the video stream. Take note of the distance in meters — all your calculations will be dependent on this value.

As shown in Figure 3, there are 49 feet between the edges of where cars will travel in the frame relative to the positioning on my camera. The conversion of 49 feet to meters is 14.94 meters.

So why does Line 26 of our configuration reflect "distance": 16?

The value has been tuned for system calibration. See the “Calibrating for Accuracy” section to learn how to test and calibrate your system. Secondly, had the measurement been made at the center of the street (i.e. further from the camera), the distance would have been longer. The measurement was taken next to the street by Dave Hoffman so he would not get run over by a car!

Our speed_limit in this example is 15mph (Line 29). Vehicles traveling less than this speed will not be logged. Vehicles exceeding this speed will be logged. If you need all speeds to be logged, you can set the value to 0.

The remaining configuration settings are for displaying frames to our screen, uploading files to the cloud (i.e., Dropbox), as well as output file paths:

If you set "display" to true on Line 32, an OpenCV window is displayed on your Raspberry Pi desktop.

Lines 35-38 specify our Caffe object detection model and prototxt paths.

If you elect to "use_dropbox", then you must set the value on Line 42 to true and fill in your access token on Line 43. Videos of vehicles passing the camera will be logged to Dropbox. Ensure that you have the quota for the videos!

Lines 46 and 47 specify the "output_path" for the log file.

Camera Positioning and Constants

Figure 4: This OpenCV vehicle speed estimation project assumes the camera is aimed perpendicular to the road. Timestamps of a vehicle are collected at waypoints ABCD or DCBA. From there, our speed = distance / time equation is put to use to calculate 3 speeds among the 4 waypoints. Speeds are averaged together and converted to km/hr and miles/hr. As you can see, the distance measurement is different depending on where (edges or centerline) the tape is laid on the ground/road. We will account for this by calibrating our system in the “Calibrating for Accuracy” section.

Figure 4 shows an overhead view of how the project is laid out. In the case of Dave Hoffman’s house, the RPi and camera are sitting in his road-facing window. The measurement for the "distance" was taken at the side of the road on the far edges of the FOV lines for the camera. Points A, B, C, and D mark the columns in a frame. They should be equally spaced in your video frame (denoted by "speed_estimation_zone"  pixel columns in the configuration).

Cars pass through the FOV in either direction while the MobileNet SSD object detector, combined with an object tracker, assists in grabbing timestamps at points ABCD (left-to-right) or DCBA (right-to-left).

Centroid Tracker

Figure 5: Top-left: To build a simple object tracking algorithm using centroid tracking, the first step is to accept bounding box coordinates from an object detector and use them to compute centroids. Top-right: In the next input frame, three objects are now present. We need to compute the Euclidean distances between each pair of original centroids (circle) and new centroids (square). Bottom-left: Our simple centroid object tracking method has associated objects with minimized object distances. What do we do about the object in the bottom left though? Bottom-right: We have a new object that wasn’t matched with an existing object, so it is registered as object ID #3.

Object tracking via centroid association is a concept we have already covered on PyImageSearch, however, let’s take a moment to review.

A simple object tracking algorithm relies on keeping track of the centroids of objects.

Typically an object tracker works hand-in-hand with a less-efficient object detector. The object detector is responsible for localizing an object. The object tracker is responsible for keeping track of which object is which by assigning and maintaining identification numbers (IDs).

This object tracking algorithm we’re implementing is called centroid tracking as it relies on the Euclidean distance between (1) existing object centroids (i.e., objects the centroid tracker has already seen before) and (2) new object centroids between subsequent frames in a video. The centroid tracking algorithm is a multi-step process. The five steps include:

  1. Step #1: Accept bounding box coordinates and compute centroids
  2. Step #2: Compute Euclidean distance between new bounding boxes and existing objects
  3. Step #3: Update (x, y)-coordinates of existing objects
  4. Step #4: Register new objects
  5. Step #5: Deregister old objects

The CentroidTracker class is covered in the following resources on PyImageSearch:

Tracking Objects for Speed Estimation with OpenCV

In order to track and calculate the speed of objects in a video stream, we need an easy way to store information regarding the object itself, including:

  • Its object ID.
  • Its previous centroids (so we can easily compute the direction the object is moving).
  • A dictionary of timestamps corresponding to each of the four columns in our frame.
  • A dictionary of x-coordinate positions of the object. These positions reflect the actual position in which the timestamp was recorded so speed can accurately be calculated.
  • The last point boolean serves as a flag to indicate that the object has passed the last waypoint (i.e. column) in the frame.
  • The calculated speed in MPH and KMPH. We calculate both and the user can choose which he/she prefers to use by a small modification to the driver script.
  • A boolean to indicate if the speed has been estimated (i.e. calculated) yet.
  • A boolean indicating if the speed has been logged in the .csv log file.
  • The direction through the FOV the object is traveling (left-to-right or right-to-left).

To accomplish all of these goals we can define an instance of TrackableObject — open up the trackableobject.py file and insert the following code:

The TrackableObject constructor accepts an objectID and centroid. The centroids list will contain an object’s centroid location history.

We will have multiple trackable objects — one for each car that is being tracked in the frame. Each object will have the attributes shown on Lines 8-29 (detailed above)

Lines 18 and 19 hold the speed in MPH and KMPH. We need a function to calculate the speed, so let’s define the function now:

Line 33 calculates the speedKMPH attribute as an average of the three estimatedSpeeds between the four points (passed as a parameter to the function).

There are 0.621371 miles in one kilometer (Line 34). Knowing this, Line 35 calculates the speedMPH attribute.

Speed Estimation with Computer Vision and OpenCV

Figure 6: OpenCV vehicle detection, tracking, and speed estimation with the Raspberry Pi.

Before we begin working on our driver script, let’s review our algorithm at a high level:

  • Our speed formula is speed = distance / time (Equation 1.1).
  • We have a known distance constant measured by a tape at the roadside. The camera will face at the road perpendicular to the distance measurement unobstructed by obstacles.
  • Meters per pixel are calculated by dividing the distance constant by the frame width in pixels (Equation 1.2).
  • Distance in pixels is calculated as the difference between the centroids as they pass by the columns for the zone (Equation 1.3). Distance in meters is then calculated for the particular zone (Equation 1.4).
  • Four timestamps (t) will be collected as the car moves through the FOV past four waypoint columns of the video frame.
  • Three pairs of the four timestamps will be used to determine three delta t values.
  • We will calculate three speed values (as shown in the numerator of Equation 1.5) for each of the pairs of timestamps and estimated distances.
  • The three speed estimates will be averaged for an overall speed (Equation 1.5).
  • The speed is converted and made available in the TrackableObject class as speedMPH or speedKMPH. We will display speeds in miles per hour. Minor changes to the script are required if you prefer to have the kilometers per hour logged and displayed — be sure to read the notes as you follow along in the tutorial.

The following equations represent our algorithm:

Now that we understand the methodology for calculating speeds of vehicles and we have defined the CentroidTracker and TrackableObject classes, let’s work on our speed estimation driver script.

Open a new file named speed_estimation_dl.py and insert the following lines:

Lines 2-17 handle our imports including our CentroidTracker and TrackableObject for object tracking. The correlation tracker from Davis King’s dlib is also part of our object tracking method. We’ll use the dropbox API to store data in the cloud in a separate Thread so as not to interrupt the flow of the main thread of execution.

Let’s implement the upload_file function now:

Our upload_file function will run in one or more separate threads. It accepts the tempFile object, Dropbox client object, and imageID as parameters. Using these parameters, it builds a path and then uploads the file to Dropbox (Lines 22 and 23). From there, Line 24 then removes the temporary file from local storage.

Let’s go ahead and load our configuration:

Lines 27-33 parse the --conf command line argument and load the contents of the configuration into the conf  dictionary.

We’ll then initialize our pretrained MobileNet SSD CLASSES and Dropbox client if required:

And from there, we’ll load our object detector and initialize our video stream:

Lines 50-52 load the MobileNet SSD net and set the target processor to the Movidius NCS Myriad.

Using the Movidius NCS coprocessor (Line 52) ensures that our FPS is high enough for accurate speed calculations. In other words, if we have a lag between frame captures, our timestamps can become out of sync and lead to inaccurate speed readouts. If you prefer to use a laptop/desktop for processing (i.e. without OpenVINO and the Movidius NCS), be sure to delete Line 52.

Lines 57-63 initialize the Raspberry Pi video stream and frame dimensions.

We have a handful more initializations to take care of:

For object tracking purposes, Lines 68-71 initialize our CentroidTracker, trackers list, and trackableObjects dictionary.

Line 74 initializes a totalFrames counter which will be incremented each time a frame is captured. We’ll use this value to calculate when to perform object detection versus object tracking.

Our logFile object will be opened later on (Line 77).

Our speed will be based on the ABCD column points in our frame. Line 81 initializes a list of pairs of points for which speeds will be calculated. Given our four points, we can calculate the three estimated speeds and then average them.

Line 84 initializes our FPS counter.

With all of our initializations taken care of, let’s begin looping over frames:

Our frame processing loop begins on Line 87. We begin by grabbing a frame and taking our first timestamp (Lines 90-92).

Lines 99-115 initialize our logFile and write the column headings. Notice that if we are using Dropbox, one additional column is present in the CSV — the image ID.

Note: If you prefer to log speeds in kilometers per hour, be sure to update the CSV column headings on Line 110 and Line 115.

Let’s preprocess our frame and perform a couple of initializations:

Line 118 resizes our frame to a known width directly from the "frame_width" value held in the config file.

Note: If you change "frame_width" in the config, be sure to update the "speed_estimation_zone" columns as well.

Line 119 converts the frame to RGB format for dlib’s correlation tracker.

Lines 122-124 initialize the frame dimensions and calculate meterPerPixel. The meters per pixel value helps to calculate our three estimated speeds among the four points.

Note: If your lens introduces distortion (i.e. a wide area lens or fisheye), you should consider a proper camera calibration via intrinsic/extrinsic camera parameters so that the meterPerPixel value is more accurate. Calibration will be a future PyImageSearch blog topic.

Line 128 initializes an empty list to hold bounding box rectangles returned by either (1) our object detector or (2) the correlation trackers.

At this point, we’re ready to perform object detection to update our trackers:

Object tracking willy only occur on multiples of "track_object" per Line 132. Performing object detection only every N frames reduces the expensive inference operations. We’ll perform object tracking whenever possible to reduce computational load.

Lines 134 initializes our new list of object trackers to update with accurate bounding box rectangles so that correlation tracking can do its job later.

Lines 138-142 perform inference using the Movidius NCS.

Let’s loop over the detections and update our trackers:

Line 145 begins a loop over detections.

Lines 148-159 filter the detection based on the "confidence" threshold and CLASSES type. We only look for the “car” class using our pretrained MobileNet SSD.

Lines 163 and 164 calculate the bounding box of an object.

We then initialize a dlib correlation tracker and begin track the rect ROI found by our object detector (Lines 169-171). Line 175 adds the tracker to our trackers list.

Now let’s handle the event that we’ll be performing object tracking rather than object detection:

Object tracking is less of a computational load on our RPi, so most of the time (i.e. except every N "track_object" frames) we will perform tracking.

Lines 180-185 loop over the available trackers and update the position of each object.

Lines 188-194 add the bounding box coordinates of the object to the rects list.

Line 198 then updates the CentroidTracker’s objects using either the object detection or object tracking rects.

Let’s loop over the objects now and take steps towards calculating speeds:

Each trackable object has an associated objectID. Lines 204-208 create a trackable object (with ID) if necessary.

From here we’ll check if the speed has been estimated for this trackable object yet:

If the speed has not been estimated (Line 212), then we first need to determine the direction in which the object is moving (Lines 215-218).

Positive direction values indicate left-to-right movement and negative values indicate right-to-left movement.

Knowing the direction is important so that we can estimate our speed between the points properly.

With the direction in hand, now let’s collect our timestamps:

Lines 222-267 collect timestamps for cars moving from left-to-right for each of our columns, A, B, C, and D.

Let’s inspect the calculation for column A:

  1. Line 225 checks to see if a timestamp has been made for point A — if not, we’ll proceed to do so.
  2. Line 230 checks to see if the current x-coordinate centroid is greater than column A.
  3. If so, Lines 231 and 232 record a timestamp and the exact x- position of the centroid.
  4. Columns B, C, and D use the same method to collect timestamps and positions with one exception. For column D, the lastPoint is marked as True. We’ll use this flag later to indicate that it is time to perform our speed formula calculations.

Now let’s perform the same timestamp, position, and last point updates for right-to-left traveling cars (i.e. direction < 0):

Lines 271-316 grab timestamps and positions for cars as they pass by columns D, C, B, and A (again, for right-to-left tracking). For A the lastPoint is marked as True.

Now that a car’s lastPoint is True, we can calculate the speed:

When the trackable object’s (1) last point timestamp and position has been recorded, and (2) the speed has not yet been estimated (Line 322) we’ll proceed to estimate speeds.

Line 324 initializes a list to hold three estimatedSpeeds. Let’s calculate the three estimates now.

Line 328 begins a loop over our pairs of points:

We calculate the distanceInPixels using the position values (Lines 330-331). If the distance is 0, we’ll skip this pair (Lines 335 and 336).

Next, we calculate the elapsed time between two points in hours (Lines 339-341). We need the time in hours because we are calculating kilometers per hour and miles per hour.

We then calculate the distance in kilometers by multiplying the pixel distance by the estimated meterPerPixel value (Lines 345 and 346). Recall that meterPerPixel is based on (1) the width of the FOV at the roadside and (2) the width of the frame.

The speed is calculated by Equation 1.1-1.4 (distance over time) and added to the estimatedSpeeds list.

Line 350 makes a call to the TrackableObject class method calculate_speed to average out our three estimatedSpeeds in both miles per hour and kilometers per hour (Equation 1.5).

Line 353 marks the speed as estimated.

Lines 354 and 355 then print the speed in the terminal.

Note: If you prefer to print the speed in km/hr be sure to update both the string to KMPH and the format variable to to.speedKMPH.

Line 358 stores the trackable object to the trackableObjects dicitionary.

Phew! The hard part is out of the way in this script. Let’s wrap up, first by annotating the centroid and ID on the frame:

A small dot is drawn on the centroid of the moving car with the ID number next to it.

Next we’ll go ahead and update our log file and store vehicle images in Dropbox:

At a minimum, every vehicle that exceeds the speed limit will be logged in the CSV file. Optionally Dropbox will be populated with images of the speeding vehicles.

Lines 369-372 check to see if the trackable object has been logged, speed estimated, and if the car was speeding.

If so Lines 374-477 extract the year, month, day, and time from the timestamp.

If an image will be logged in Dropbox, Lines 381-391 store a temporary file and spawn a thread to upload the file to Dropbox.

Using a separate thread for a potentially time-consuming upload is critical so that our main thread isn’t blocked, impacting FPS and speed calculations. The filename will be the imageID on Line 383 so that it can easily be found later if it is associated in the log file.

Lines 394-404 write the CSV data to the logFile. If Dropbox is used, the imageID is the last value.

Note: If you prefer to log the kilometers per hour speed, simply update to.speedMPH to to.speedKMPH on Line 395 and Line 403.

Line 396 marks the trackable object as logged.

Let’s wrap up:

Lines 411-417 display the annotated frame and look for the q keypress in which case we’ll quit ( break).

Lines 421 and 422 increment totalFrames and update our FPS counter.

When we have broken out of the frame processing loop we perform housekeeping including printing FPS stats, closing our log file, destroying GUI windows, and stopping our video stream (Lines 424-438)

Vehicle Speed Estimation Deployment

Now that our code is implemented, we’ll deploy and test our system.

I highly recommend that you conduct a handful of controlled drive-bys and tweak the variables in the config file until you are achieving accurate speed readings.

Prior to any fine-tuning calibration, we’ll just ensure that the program working. Be sure you have met the following requirements prior to trying to run the application:

  • Position and aim your camera perpendicular to the road as per Figure 3.
  • Ensure your camera has a clear line of sight with limited obstructions — our object detector must be able to detect a vehicle at multiple points as it crosses through the camera’s field of view (FOV).
  • It is best if your camera is positioned far from the road. The further points A and D are from each other at the point at which cars pass on the road, the better the distance / time calculations will average out and produce more accurate speed readings. If your camera is close to the road, a wide-angle lens is an option, but then you’ll need to perform camera calibration (a future PyImageSearch blog topic).
  • If you are using Dropbox functionality, ensure that your RPi has a solid WiFi, Ethernet, or even cellular connection.
  • Ensure that you have set all constants in the config file. We may elect to fine-tune the constants in the next section.

Assuming you have met each of the requirements, you are now ready to deploy and run your program. First, we must setup our environment.

Pre-configured Raspbian .img users: Please activate your virtual environment as follows:

Using that script ensures that (1) the virtual environment is activated, and (2) Intel’s environment variables are loaded.

If you installed OpenVINO on your own (i.e. you aren’t using my Pre-configured Raspbian .img): Please source the setupvars.sh  script as follows (adapt the command to where your script lives):

Note: You may have sourced the environment in your ~/.bashrc  file per the OpenVINO installation instructions. In this case, the environment variables are set automatically when you launch a terminal or connect via SSH. That said, you will still need to use the workon  command to activate your virtual environment.

Again, if you are using my Pre-configured Raspbian .img that comes with either Practical Python and OpenCV + Case Studies (Quickstart and Hardcopy) and/or Raspberry Pi for Computer Vision (all bundles), then the ~/start_openvino.sh will call the setupvars.sh  script automatically.

If you do not perform either of the following steps, you will encounter an Illegal Instruction  error.

Enter the following command to start the program and begin logging speeds:

Figure 7: OpenCV vehicle speed estimation deployment. Vehicle speeds are calculated after they leave the viewing frame. Speeds are logged to CSV and images are stored in Dropbox.

As shown in Figure 7 and the video, our OpenCV system is measuring speeds of vehicles traveling in both directions. In the next section, we will perform drive-by tests to ensure our system is reporting accurate speeds.

Note: The video has been post-processed for demo purposes. Keep in mind that we do not know the vehicle speed until after the vehicle has passed through the frame. In the video, the speed of the vehicle is displayed while the vehicle is in the frame a better visualization.

Note: OpenCV cannot automatically throttle a video file framerate according to the true framerate. If you use speed_estimation_dl_video.py as well as the supplied cars.mp4  testing file, keep in mind that the speeds reported will be inaccurate. For accurate speeds, you must set up the full experiment with a camera and have real cars drive by. Refer to the next section, “Calibrating for Accuracy”, for a real live demo in which a screencast was recorded of the live system in action. To use the script, run this command: python speed_estimation_dl_video.py --conf config/config.json --input sample_data/cars.mp4 

On occasions when multiple cars are passing through the frame at one given time, speeds will be reported inaccurately. This can occur when our centroid tracker mixes up centroids. This is a known drawback to our algorithm. To solve the issue additional algorithm engineering will need to be conducted by you as the reader. One suggestion would be to perform instance segmentation to accurate segment each vehicle.

Credits:

Calibrating for Accuracy

Figure 8: Neighborhood vehicle speed estimation and tracking with OpenCV drive test results.

You may find that the system produces slightly inaccurate readouts of the vehicle speeds going by. Do not disregard the project just yet. You can tweak the config file to get closer and closer to accurate readings.

We used the following approach to calibrate our system until our readouts were spot-on:

  • Begin recording a screencast of the RPi desktop showing both the video stream and terminal. This screencast should record throughout testing.
  • Meanwhile, record a voice memo on your smartphone throughout testing of you driving by while stating what your drive-by speed is.
  • Drive by the computer-vision-based VASCAR system in both directions at predetermined speeds. We chose 10mph, 15mph, 20mph, and 25mph to compare our speed to the VASCAR calculated speed. Your neighbors might think you’re weird as you drive back and forth past your house, but just give them a nice smile!
  • Sync the screencast to the audio file so that it can be played back.
  • The speed +/- differences could be jotted down as you playback your video with the synced audio file.
  • With this information, tune the constants:
    • (1) If your speed readouts are a little high, then decrease the "distance" constant
    • (2) Conversely, if your speed readouts are slightly low, then increase the "distance" constant.
  • Rinse and repeat until you are satisfied. Don’t worry, you won’t burn too much fuel in the process.

PyImageSearch colleagues Dave Hoffman and Abhishek Thanki found that Dave needed to increase his distance constant from 14.94m to 16.00m.

Be sure to refer to the following final testing video which corresponds to the timestamps and speeds for the table in Figure 8:

Here are the results of an example calculation with the calibrated constant. Be sure to compare Figure 9 to Figure 4:

Figure 9: An example calculation using our calibrated distance for OpenCV vehicle detection, tracking, and speed estimation.

With a calibrated system, you’re now ready to let it run for a full day. Your system is likely only configured for daytime use unless you have streetlights on your road.

Note: For nighttime use (outside the scope of this tutorial), you may need infrared cameras and infrared lights and/or adjustments to your camera parameters (refer to the Raspberry Pi for Computer Vision Hobbyist Bundle Chapters 6, 12, and 13 for these topics).

Where can I learn more?

If you’re interested in learning more about applying Computer Vision, Deep Learning, and OpenCV to embedded devices such as the:

  • Raspberry Pi
  • Intel Movidus NCS
  • Google Coral
  • NVIDIA Jetson Nano

…then you should take a look at my brand new book, Raspberry Pi for Computer Vision.

This book has over 40 projects (over 60 chapters) on embedded Computer Vision and Deep Learning. You can build upon the projects in the book to solve problems around your home, business, and even for your clients.

Each and every project on the book has an emphasis on:

  • Learning by doing.
  • Rolling up your sleeves.
  • Getting your hands dirty in code and implementation.
  • Building actual, real-world projects using the Raspberry Pi.

A handful of the highlighted projects include:

  • Daytime and nighttime wildlife monitoring
  • Traffic counting and vehicle speed detection
  • Deep Learning classification, object detection, and instance segmentation on resource-constrained devices
  • Hand gesture recognition
  • Basic robot navigation
  • Security applications
  • Classroom attendance
  • …and many more!

The book also covers deep learning using the Google Coral and Intel Movidius NCS coprocessors (Hacker + Complete Bundles). We’ll also bring in the NVIDIA Jetson Nano to the rescue when more deep learning horsepower is needed (Complete Bundle).

Are you ready to join me and learn how to apply Computer Vision and Deep Learning to embedded devices such as the Raspberry Pi, Google Coral, and NVIDIA Jetson Nano?

If so, check out the book and grab your free table of contents!

Grab my free table of contents!

Summary

In this tutorial, we utilized Deep Learning and OpenCV to build a system to monitor the speeds of moving vehicles in video streams.

Rather than relying on expensive RADAR or LIDAR sensors, we used:

  • Timestamps
  • A known distance
  • And a simple physics equation to calculate speeds.

In the police world, this is known as Vehicle Average Speed Computer and Recorder (VASCAR). Police rely on their eyesight and button-pushing reaction time to collect timestamps — a method that barely holds in court in comparison to RADAR and LIDAR.

But of course, we are engineers so our system seeks to eliminate the human error component when calculating vehicle speeds automatically with computer vision.

Using both object detection and object tracking we coded a method to calculate four timestamps via four waypoints. We then let the math do the talking:

We know that speed equals distance over time. Three speeds were calculated among the three pairs of points and averaged for a solid estimate.

One drawback of our automated system is that it is only as good as the key distance constant.

To combat this, we measured carefully and then conducted drive-bys while looking at our speedometer to verify operation. Adjustments to the distance constant were made if needed.

Yes, there is a human component in this verification method. If you have a cop friend that can help you verify with their RADAR gun, that would be even better. Perhaps they will even ask for your data to provide to the city to encourage them to place speed bumps, stop signs, or traffic signals in your area!

Another area that needs further engineering is to ensure that trackable object IDs do not become swapped when vehicles are moving in different directions. This is a challenging problem to solve and I encourage discussion in the comments.

We hope you enjoyed today’s blog post!

To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), just enter your email address in the form below!

Downloads:

If you would like to download the code and images used in this post, please enter your email address in the form below. Not only will you get a .zip of the code, I’ll also send you a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL! Sound good? If so, enter your email address and I’ll send you the code immediately!

, , , , , , , ,

30 Responses to OpenCV Vehicle Detection, Tracking, and Speed Estimation

  1. Girish Emerith December 2, 2019 at 11:51 am #

    how to run script with test video cars.mp4?

    • Adrian Rosebrock December 5, 2019 at 10:07 am #

      Hey Girish — I added a note in the blog post regarding how to run the example script with “cars.mp4”, take a look!

  2. Gimmeserendipity December 2, 2019 at 11:57 am #

    Can you use a similar method to estimate the speed of a moving vehicle from its dashcam video rather than from a still camera filming a vehicle on a road?

    • Adrian Rosebrock December 2, 2019 at 12:17 pm #

      Not easily. This method assumes your camera is fixed and non-moving as it requires specific waypoints to detect speed. If both vehicles are moving you won’t be able to do that.

      • Pawel Rolbiecki December 2, 2019 at 3:17 pm #

        Adrian, what abot this? Let’s assume you are on the highway – 100% flat and ideal. You train proper SSD to detect passing car and set a tracker described above on the object. You draw a line at the bottom of the boundary box of the car.

        Now the trickiest part. In the USA roads in one direction are separated by white lines. Those lines are described in specifications. So you know the length and the gap. You can try to detect the moment when line mentioned by me is crossing line on the road. And measure time between. I can imagine it will work.

        I know that this is not so easy. You need continuous camera calibration in cases of vertical acceleration and movements of front of your car. The image projection will “bum”. But assuming the road is 100% flat. I can imagine it can work.

  3. Girish Emerith December 2, 2019 at 12:42 pm #

    Hello Adrian,
    I have used cv2.VideoCapture(‘cars.mp4’) to test the script with the video cars.mp4. However, I’m getting the unhandled exception ‘tuple’ object has no attribute ‘shape’ at line frame = imutils.resize(frame, width=conf[“frame_width”]). Do you have an idea what might have gone wrong?

    • Adrian Rosebrock December 2, 2019 at 1:16 pm #

      The cv2.VideoCapture object returns a 2-tuple of (1) a boolean, indicating if the frame was successfully read or not and (2) the frame itself. You need the second object in the tuple (i.e., the frame).

  4. Brandon Gilles December 2, 2019 at 1:58 pm #

    Love these tutorials! I’ve thought about building the same thing onto DepthAI, and now I won’t have to write it – as this will run directly on DepthAI for Raspberry Pi.

    So the only difference will be that the calibration step won’t be required, as with the depth information you get real-world XYZ location of the vehicle, so the speed is directly returned without having to calibrated pixel-motion to physical motion and then speed.

    Really excited to take this and implement it on DepthAI. Thanks again for writing it! Have wanted to do so for a long time but, well, just haven’t yet. 🙂

    • Adrian Rosebrock December 2, 2019 at 5:20 pm #

      I’m definitely excited to see it on DepthAI! Having the automatic depth information computed would be fantastic.

  5. Abdallah Eltayeb December 3, 2019 at 4:17 am #

    Hello Adrian,

    How can I make this approach work for vertical detection (camera placed into traffic Lights) instead of horizontal (on the side of the road)?

    For example using this approach on 2 cameras in an intersection to notify (playing a sound) drivers if they are going too fast while another vehicle is also about to cross an intersection.

  6. Shivakumar December 3, 2019 at 4:39 am #

    Hi Adrian,

    How I can run this on Ubuntu 16.04 without RPI board and movidius.
    Please help

  7. Ivo Germann December 5, 2019 at 3:53 pm #

    Hi Adrian,
    Referring to my question on how to get rid of the errors when running your code on Ubuntu I found the solution: Line 54 has to be changed from:
    net.setPreferableTarget(cv2.dnn.DNN_TARGET_MYRIAD)
    to:
    net.setPreferableTarget(cv2.dnn.DNN_TARGET_CPU)
    Now, it already works perfectly fine with my sons toy cars and will soon be tested in reality :-). Thanks again for this great post!

    • Adrian Rosebrock December 9, 2019 at 4:56 am #

      Correct, this example assumes you have an NCS2. If you’re not using an NCS you need to change the target to your CPU has you have done.

  8. Tony Holdroyd December 5, 2019 at 11:06 pm #

    Hello Adrian,
    I’m using the flashed image from your RPi4CV book, but am getting the error message :
    Illegal instruction
    from the line
    net.setPreferableTarget(cv2.dnn.DNN_TARGET_MYRIAD)
    I’m using the RPi 4 with an NCS2 and the openvino virtual environment
    Any ideas, please?
    Thanks
    Tony

    • Adrian Rosebrock December 9, 2019 at 4:57 am #

      Are you using your pre-configured Raspbian .img file? Or your own dev environment? If it’s the latter I would need to know the versions of OpenCV and OpenVINO, otherwise it’s impossible to diagnose.

  9. Simon Bunn December 6, 2019 at 4:29 pm #

    I have two NCS Movidius sticks. Can the program be set to split the workload between multiple NCS sticks?

    • Adrian Rosebrock December 9, 2019 at 4:58 am #

      Not for this example, no.

  10. Daniel Segel December 7, 2019 at 12:24 pm #

    OK, I’m using your raspbian now and I’m getting a hardware error at this line:

    net.setPreferableTarget(cv2.dnn.DNN_TARGET_MYRIAD)

    It just says “illegal hardware instruction python” and exits the program.

    Any ideas? Should I try DNN_TARGET_CPU instead? I’m not clear on what that option does.

    • Adrian Rosebrock December 9, 2019 at 4:58 am #

      See my reply to Tony Holdroyd.

      • Daniel Segel December 9, 2019 at 4:08 pm #

        I’m using your premade image, downloaded this morning from my RPi for CV purchase page (purchased under a different email address). I tried it on both an RPi 3B+ and a 4 and get the same error at the same place on both.

        When I was digging around this past weekend I was running into trouble with the Python versions that OpenVino wanted. Did you test it and have it working with Python 3.7?

        • Adrian Rosebrock December 9, 2019 at 4:22 pm #

          Thanks for the clarification Daniel. I’m going to check with the team and see if we can replicate the issue on our end. I’ll be sure to come back and update the comment once we’ve done some testing.

          UPDATE:

          If you’re using the pre-configured Raspbian .img file, you must run the following command to (1) access the “openvino” virtual environment and (2) run the required OpenVINO environment scripts supplied by Intel:

          $ source ~/start_openvino.sh

          Note that you should be using v1.0.1 of the Raspbian .img file, initially released with the Hacker Bundle of RPi4CV.

  11. Sophia December 7, 2019 at 2:56 pm #

    Adrian, You have a unique and truly amazing style of breaking down and explaining complex models in a very easy-to-understand way!

    It’d be awesome if you could find some time to post some blogs on the deep learning/computer vision technology being used in cashierless stores like Amazon Go, especially how they track multiple items being picked up at once. Looking forward to more blogs from you!

    • Adrian Rosebrock December 9, 2019 at 4:58 am #

      Thanks Sophia.

  12. Tony Holdroyd December 7, 2019 at 4:28 pm #

    Hello Adrian,
    Update:
    Further to my last post, I’ve successfully installed Openvino, using the Intel instructions, which includes OpenCV libs, on my RPi4 on a fresh flash of Raspbian. I validated the install as per Intel instructions – by running a face detection app that uses the NCS2. So I guess my hardware is okay.
    I’m still getting the illegal instruction error when I try to run the speed detection app on a pyimagesearch flashed card as mentioned in my last post though.

    • Adrian Rosebrock December 9, 2019 at 4:59 am #

      Thanks for the update, Tony. I’m going to take a look at this and see what the issue may be.

    • Daniel Segel December 9, 2019 at 4:11 pm #

      Which Intel instructions did you use? I’ve found 2 or 3 different sets of instruction, and I have not yet got one fully working.

      • Adrian Rosebrock December 10, 2019 at 6:39 am #

        Hey Tony and Daniel — just following up from my previous comment.

        If you’re using the pre-configured Raspbian .img file, you must run the following command to (1) access the “openvino” virtual environment and (2) run the required OpenVINO environment scripts supplied by Intel:

        $ source ~/start_openvino.sh

        Note that you should be using v1.0.1 of the Raspbian .img file, initially released with the Hacker Bundle of RPi4CV.

  13. David Hoffman December 11, 2019 at 10:29 am #

    UPDATE: If you are encountering the Illegal Instruction error (indicated by Tony), please review an update that I made to the blog post on Adrian’s behalf in the “Vehicle Speed Estimation Deployment” section. I have inserted instructions to either use our Pre-configured .img “start” script which sets up the environment or to manually source the environment variables. To elaborate, using Intel’s OpenVINO requires that a setupvars.sh script is sourced. We make it dead simple for you with our Pre-configured .img if you choose to use it.

Before you leave a comment...

Hey, Adrian here, author of the PyImageSearch blog. I'd love to hear from you, but before you submit a comment, please follow these guidelines:

  1. If you have a question, read the comments first. You should also search this page (i.e., ctrl + f) for keywords related to your question. It's likely that I have already addressed your question in the comments.
  2. If you are copying and pasting code/terminal output, please don't. Reviewing another programmers’ code is a very time consuming and tedious task, and due to the volume of emails and contact requests I receive, I simply cannot do it.
  3. Be respectful of the space. I put a lot of my own personal time into creating these free weekly tutorials. On average, each tutorial takes me 15-20 hours to put together. I love offering these guides to you and I take pride in the content I create. Therefore, I will not approve comments that include large code blocks/terminal output as it destroys the formatting of the page. Kindly be respectful of this space.
  4. Be patient. I receive 200+ comments and emails per day. Due to spam, and my desire to personally answer as many questions as I can, I hand moderate all new comments (typically once per week). I try to answer as many questions as I can, but I'm only one person. Please don't be offended if I cannot get to your question
  5. Do you need priority support? Consider purchasing one of my books and courses. I place customer questions and emails in a separate, special priority queue and answer them first. If you are a customer of mine you will receive a guaranteed response from me. If there's any time left over, I focus on the community at large and attempt to answer as many of those questions as I possibly can.

Thank you for keeping these guidelines in mind before submitting your comment.

Leave a Reply

[email]
[email]