Real-time object detection on the Raspberry Pi with the Movidius NCS

Real-time object detection on the Raspberry Pi + Movidius NCS GIF

Today’s post is inspired by Danielle, a PyImageSearch reader who emailed me last week and asked:

Hi Adrian,

I’m enjoying your blog and I especially liked last week’s post about image classification with the Intel Movidius NCS.

I’m still considering purchasing an Intel Movidius NCS for a personal project.

My project involves object detection with the Raspberry Pi where I’m using my own custom Caffe model. The benchmark scripts you supplied for applying object detection on the Pi’s CPU were too slow and I need faster speeds.

Would the NCS be a good choice for my project and help me achieve a higher FPS?

Great question, Danielle. Thank you for asking.

The short answer is yes, you can use the Movidius NCS for object detection with your own custom Caffe model. You’ll even achieve high frame rates if you’re processing live or recorded video.

…but there’s a catch.

I told Danielle that she’ll need the full-blown Movidius SDK installed on her (Ubuntu 16.04) machine. I also mentioned that generating graph files from Caffe models isn’t always straightforward.

Inside today’s post you will learn how to:

  • Install the Movidius SDK on your machine
  • Generate an object detection graph file using the SDK
  • Write a real-time object detection script for the Raspberry Pi + NCS

After going through the post you’ll have a good understanding of the Movidius NCS and whether it’s appropriate for your Raspberry Pi + object detection project.

To get started with real-time object detection on the Raspberry Pi, just keep reading.

Looking for the source code to this post?
Jump right to the downloads section.

Real-time object detection on the Raspberry Pi

Today’s blog post is broken into five parts.

First, we’ll install the Movidius SDK and then learn how to use the SDK to generate the Movidius graph files.

From there, we’ll write a script for real time object detection with the Intel Movidius Neural compute stick that can be used with the Pi (or alternative single board computer with minor modifications).

Next, we’ll test the script + compare results.

In a previous post, we learned how to perform real-time object detection in video on the Raspberry Pi using the CPU and the OpenCV DNN module. We achieved approximately 0.9 FPS which serves as our benchmark comparison. Today, we’re going to see how the NCS paired with a Pi performs against the Pi CPU using the same model.

And finally, I’ve captured some Frequently Asked Questions (FAQs). Refer to this section often — I expect it to grow as I receive comments and emails.

Installing the Intel Movidius SDK

Figure 1: The Intel Movidius NCS workflow (image credit: Intel)

Last week, I reviewed the Movidius Workflow. The workflow has four basic steps:

  1. Train a model using a full-size machine
  2. Convert the model to a deployable graph file using the SDK and an NCS
  3. Write a Python script which deploys the graph file and processes the results
  4. Deploy the Python script and graph file to your single board computer equipped with an Intel Movidius NCS

In this section we’ll learn how to install the SDK which includes TensorFlow, Caffe, OpenCV, and the Intel suite of Movidius tools.


  • Stand-alone machine or VM. We’ll install Ubuntu 16.04 LTS on it
  • 30-60 minutes of time depending on download speed and machine capability
  • Movidius NCS USB stick

I highlighted “Stand-alone” as it’s important that this machine only be used for Movidius development.

In other words, don’t install the SDK on a “daily development and productivity use” machine where you might have Python Virtual Environments and OpenCV installed. The install process is not entirely isolated and can/will change existing libraries on your system.

However, there is an alternative:

Use a VirtualBox Virtual Machine (or other virtualization system) and run an isolated Ubuntu 16.04 OS in the VM.

The advantage of a VM is that you can install it on your daily use machine and still keep the SDK isolated. The disadvantage is that you won’t have access to a GPU via the VM.

Danielle wants to use a Mac and VirtualBox works well on macOS, so let’s proceed down that path. Note that you could also run VirtualBox on a Windows or Linux host which may be even easier.

Before we get started, I want to bring attention to non-standard VM settings we’ll be making. We’ll be configuring USB settings which will allow the Movidius NCS to stay connected properly.

As far as I can tell from the forums, these are Mac-specific VM USB settings (but I’m not certain). Please share your experiences in the comments section.

Download Ubuntu and Virtualbox

Let’s get started.

First, download the Ubuntu 16.04 64-bit .iso image from here the official Ubuntu 16.04.3 LTS download page. You can grab the .iso directly or the torrent would also be appropriate for faster downloads.

While Ubuntu is downloading, if you don’t have Oracle VirtualBox, grab the installer that is appropriate for your OS (I’m running macOS). You can download VirtualBox here.

Non-VM users: If you aren’t going to be installing the SDK on a VM, then you can skip downloading/installing Virtualbox. Instead, scroll down to “Install the OS” but ignore the information about the VM and the virtual optical drive — you’ll probably be installing with a USB thumb drive.

After you’ve got VirtualBox downloaded, and while the Ubuntu .iso continues to download, you can install VirtualBox. Installation is incredibly easy via the wizard.

From there, since we’ll be using USB passthrough, we need the Extension Pack.

Install the Extension Pack

Let’s navigate back to the VirtualBox download page and download the Oracle VM Extension Pack if you don’t already have it.

The version of the Extension Pack must match the version of Virtualbox you are using. If you have any VMs running, you’ll want to shut them down in order to install the Extension Pack. Installing the Extension Pack is a breeze.

Create the VM

Once the Ubuntu 16.04 image is downloaded, fire up VirtualBox, and create a new VM:

Figure 2: Creating a VM for the Intel Movidius SDK.

Give your VM reasonable settings:

  • I chose 2048MB of memory for now.
  • I selected 2 virtual CPUs.
  • I set up a 40Gb dynamically allocated VDI (Virtualbox Disk Image).

The first two settings are easy to change later for best performance of your host and guest OSes.

As for the third setting, it is important to give your system enough space for the OS and the SDK. If you run out of space, you could always “connect” another virtual disk and mount it, or you could expand the OS disk (advanced users only).

USB passthrough settings

A VM, by definition, is virtually running as software. Inherently, this means that it does not have access to hardware unless you specifically give it permission. This includes cameras, USB, disks, etc.

This is where I had to do some digging on the intel forms to ensure that the Movidius would work with MacOS (because originally it didn’t work on my setup).

Ramana @ Intel provided “unofficial” instructions on how to set up USB over on the forums. Your mileage may vary.

In order for the VM to access the USB NCS, we need to alter settings.

Go to the “Settings” for your VM and edit “Ports > USB” to reflect a “USB 3.0 (xHCI) Controller”.

You need to set USB2 and USB3 Device Filters for the Movidius to seamlessly stay connected.

To do this, click the “Add new USB Filter” icon as is marked in this image:

Figure 3: Adding a USB Filter in VirtualBox settings to accommodate the Intel Movidius NCS on MacOS.

From there, you need to create two USB Device Filters. Most of the fields can be left blank. I just gave each a Name and provided the Vendor ID.

  1. Name: Movidius1, Vendor ID: 03e7, Other fields: blank
  2. Name: Movidius2, Vendor ID: 040e, Other fields: blank

Here’s an example for the first one:

Figure 4: Two Virtualbox USB device filters are required for the Movidius NCS to work in a VM on MacOS.

Be sure to save these settings.

Install the OS

To install the OS, “insert” the .iso image into the virtual optical drive. To do this, go to “Settings”, then under “Storage” select “Controller: IDE > Empty”, and click the disk icon (marked by the red box).  Then find and select your freshly downloaded Ubuntu .iso.

Figure 5: Inserting an Ubuntu 16.04 .iso file into a Virtualbox VM.

Verify all settings and then boot your machine.

Follow the prompts to “Install Ubuntu”. If you have a fast internet connection, you can select “Download updates while installing Ubuntu”.  I did not select the option to “Install third-party software…”.

The next step is to “Erase disk and install Ubuntu” — this is a safe action because we just created the empty VDI disk. From there, set up system name and a username + password.

Once you’ve been instructed to reboot and removed the virtual optical disk, you’re nearly ready to go.

First, let’s update our system. Open a terminal and type the following to update your system:

Install Guest Additions

Non-VM users: You should skip this section.

From there, since we’re going to be using a USB device (the Intel NCS), let’s install guest additions. Guest additions also allows for bidirectional copy/paste between the VM and the host amongst other nice sharing utilities.

Guest additions can be installed by going to the Devices menu of Virtual box and clicking “Insert Guest Additions CD Image…”:

Figure 6: Virtualbox Guest Additions for Ubuntu has successfully been installed.

Follow the prompt to press “Return to close this window…” which completes the the install.

Take a snapshot

Non-VM users: You can skip this section or make a backup of your system via your preferred method.

From there, I like to reboot followed by taking a “snapshot” of my VM.

Rebooting is important because we just updated and installed a lot of software and want to ensure the changes take effect.

Additionally, a snapshot will allow us to rollback if we make any mistakes or have problems during the install — as we’ll find out, there are some gotchas along the way that can trip you up wih the Movidius SDK, so this is a worthwhile step.

Definitely take the time to snapshot your system. Go to the VirtualBox menubar and press “Machine > Take Snapshot”.

You can give the snapshot a name such as “Installed OS and Guest Additions” as is shown below:

Figure 7: Taking a snapshot of the Movidius SDK VM prior to actually installing the SDK.

Installing the Intel Movidius SDK on Ubuntu

This section assumes that you either (a) followed the instructions above to install Ubuntu 16.04 LTS on a VM, or (b) are working with a fresh install of Ubuntu 16.04 LTS on a Desktop/Laptop.

Intel makes the process of installing the SDK very easy. Cheers to that!

But like I said above, I wish there was an advanced method. I like easy, but I also like to be in control of my computer.

Let’s install Git from a terminal:

From there, let’s follow Intel’s instructions very closely so that there are hopefully no issues.

Open a terminal and follow along:

Now that we’re in the workspace, let’s clone down the NCSDK and the NC App Zoo:

And from there, you should navigate into the ncsdk  directory and install the SDK:

You might want to go outside for some fresh air or grab yourself a cup of coffee (or beer depending on what time it is). This process will take about 15 minutes depending on the capability of your host machine and your download speed.

Figure 8: The Movidius SDK has been successfully installed on our Ubuntu 16.04 VM.

VM Users: Now that the installation is complete, it would be a good time to take another snapshot so we can revert in the future if needed. You can follow the same method as above to take another snapshot (I named mine “SDK installed”). Just remember that snapshots require adequate disk space on the host.

Connect the NCS to a USB port and verify connectivity

Non-VM users: You can skip this step because you’ll likely not have any USB issues. Instead, plug in the NCS and scroll to “Test the SDK”.

First, connect your NCS to the physical USB port on your laptop or desktop.

Note: Given that my Mac has Thunderbolt 3 / USB-C ports, I initially plugged in Apple’s USB-C Digital AV Multiport Adapter which has a USB-A and HDMI port. This didn’t work. Instead, I elected to use a simple adapter, but not a USB hub. Basically you should try to eliminate the need for any additional required drivers if you’re working with a VM.

From there, we need to make the USB stick accessible to the VM. Since we have Guest Additions and the Extension Pack installed, we can do this from the VirtualBox menu. In the VM menubar, click “Devices > USB > ‘Movidius Ltd. Movidius MA2X5X'” (or a device with a similar name). It’s possible that the Movidus already has a checkmark next to it, indicating that it is connected to the VM.

In the VM open a terminal. You can run the following command to verify that the OS knows about the USB device:

You should see that the Movidius is recognized by reading the most recent 3 or 4 log messages as shown below:

Figure 9: Running the dmesg command in a terminal allows us to see that the Movidius NCS is associated with the OS.

If you see the Movidius device then it’s time to test the installation.

Test the SDK

Now that the SDK is installed, you can test the installation by running the pre-built examples:

This may take about five minutes to run and you’ll see a lot of output (not shown in the block above).

If you don’t see error messages while all the examples are running, that is good news. You’ll notice that the Makefile has executed code to go out and download models and weights from Github, and from there it runs mvNCCompile. We’ll learn about mvNCCompile in the next section. I’m impressed with the effort put into the Makefiles by the Movidius team.

Another check (this is the same one we did on the Pi last week):

This test ensures that the links to your API and connectivity to the NCS are working properly.

If you’ve made it this far without too much trouble, then congratulations!

Generating Movidius graph files from your own Caffe models

Generating graph files is made quite easy by Intel’s SDK. In some cases you can actually compute the graph using a Pi. Other times, you’ll need a machine with more memory to accomplish the task.

There’s one main tool that I’d like to share with you: mvNCCompile .

This command line tool supports both TensorFlow and Caffe. It is my hope that Keras will be supported in the future by Intel.

For Caffe, the command line arguments are in the following format (TensorFlow users should refer to the documentation which is similar):

Let’s review the arguments:

  • network.prototxt : path/filename of the network file
  • -w network.caffemodel : path/filename of the caffemodel file
  • -s MaxNumberOfShaves : SHAVEs (1, 2, 4, 8, or 12) to use for network layers (I think the default is 12, but the documentation is unclear)
  • -in InputNodeNodeName : you may optionally specify a specific input layer (it would match the name in the prototxt file)
  • -on OutputNodeName : by default the network is processed through the output tensor and this option allows a user to select an alternative end point in the network
  • -is InputWidth InputHeight : the input shape is very important and should match the design of your network
  • -o OutputGraphFilename : if no file/path is specified this defaults to the very ambiguous filename, graph , in the current working directory

Where’s the batch size argument?

The batch size for the NCS is always 1 and the number of color channels is assumed to be 3.

If you provide command line arguments to mvNCCompile  in the right format with an NCS plugged in, then you’ll be on your way to having a graph file rather quickly.

There’s one caveat (at least from my experience thus far with Caffe files). The mvNCCompile  tool requires that the prototxt be in a specific format.

You might have to modify your prototxt to get the mvNCCompile  tool to work. If you’re having trouble, the Movidius forums may be able to guide you.

Today we’re working with MobileNet Single Shot Detector (SSD) trained with Caffe. The GitHub user, chuanqui305, gets credit for the training the model on the MS-COCO dataset. Thank you chuanqui305!

I have provided chuanqui305’s files in the “Downloads” section. To compile the graph you should execute the following command:

You should expect the Copyright message and possibly additional information or a warning like I encountered above. I procdeded by ignoring the warning without any trouble.

Object detection with the Intel Movidius Neural Compute Stick

Let’s write a real-time object detection script. The script very closely aligns with the non-NCS version that we built in a previous post.

You can find today’s script and associated files in the “Downloads” section of this blog post. I suggest you download the source code and model file if you wish to follow along.

Once you’ve downloaded the files, open :

We import our packages on Lines 2-8, taking note of the mvncapi , which is the the Movidius NCS Python API package.

From there we’ll perform initializations:

Our class labels and associated random colors (one random color per class label) are initialized on Lines 12-16.

Our MobileNet SSD requires dimensions of 300×300, but we’ll be displaying the video stream at 900×900 to better visualize the output (Lines 19 and 20).

Since we’re changing the dimensions of the image, we need to calculate the scalar value to scale our object detection boxes (Line 23).

From there we’ll define a preprocess_image  function:

The actions made in this pre-process function are specific to our MobileNet SSD model. We resize, perform mean subtraction, scale the image, and convert it to float16  format (Lines 27-30).

Then we return the preprocessed  image to the calling function (Line 33).

To learn more about pre-processing for deep learning, be sure to refer to my book, Deep Learning for Computer Vision with Python.

From there we’ll define a predict  function:

This predict  function applies to users of the Movidius NCS and it is largely based on the Movidius NC App Zoo GitHub example — I made a few minor modifications.

The function requires an image  and a graph  object (which we’ll instantiate later).

First we pre-process the image (Line 37).

From there, we run a forward pass through the neural network utilizing the NCS while grabbing the predictions (Lines 41 and 42).

Then we extract the number of valid object predictions ( num_valid_boxes ) and initialize our predictions  list (Lines 46 and 47).

From there, let’s loop over the valid results:

Okay, so the above code might look pretty ugly. Let’s take a step back. The goal of this loop is to append prediction data to our predictions  list in an organized fashion so we can use it later. This loop just extracts and organizes the data for us.

But what in the world is the base_index ?

Basically, all of our data is stored in one long array/list ( output ). Using the box_index , we calculate our base_index  which we’ll then use (with more offsets) to extract prediction data.

I’m guessing that whoever wrote the Python API/bindings is a C/C++ programmer. I might have opted for a different way to organize the data such as a list of tuples like we’re about to construct.

Why are we ensuring values are finite on Lines 55-62?

This ensures that we have valid data. If it’s invalid we continue  back to the top of the loop (Line 63) and try another prediction.

What is the format of the output  list?

The output list has the following format:

  1. output[0] : we extracted this value on Line 46 as num_valid_boxes
  2. output[base_index + 1] : prediction class index
  3. output[base_index + 2] : prediction confidence
  4. output[base_index + 3] : object boxpoint x1 value (it needs to be scaled)
  5. output[base_index + 4] : object boxpoint y1 value (it needs to be scaled)
  6. output[base_index + 5] : object boxpoint x2 value (it needs to be scaled)
  7. output[base_index + 6] : object boxpoint y2 value (it needs to be scaled)

Lines 68-82 handle building up a single prediction tuple. The prediction consists of: (pred_class, pred_conf, pred_boxpts)  and we append the prediction  to the predictions  list on Line 83.

After we’re done looping through the data, we return  the predictions  list to the calling function on Line 86.

From there, let’s parse our command line arguments:

We parse our three command line arguments on Lines 89-96.

We require the path to our graph file. Optionally we can specify a different confidence threshold or display the image to the screen.

Next, we’ll connect to the NCS and load the graph file onto it:

The above block is identical to last week, so I’m not going to review it in detail. Essentially we’re checking that we have an available NCS, connecting, and loading the graph file on it.

The result is a graph  object which we use in the predict function above.

Let’s kick off our video stream:

We start the camera VideoStream , allow our camera to warm up, and our instantiate our FPS counter.

Now let’s process the camera feed frame by frame:

Here we’re reading a frame from the video stream, making a copy (so we can draw on it later), and resizing it (Lines 135-137).

We then send the frame through our object detector which will return predictions  to us.

Let’s loop over the predictions  next:

Looping over the predictions , we first extract the class, confidence and boxpoints for the object (Line 145).

If the confidence  is above the threshold, we print the prediction to the terminal and check if we should display the image on the screen:

If we’re displaying the image, we first build a label  string which will contain the class name and confidence in percentage form (Lines 160-161).

From there we extract the corners of the rectangle and calculate the position for our label  relative to those points (Lines 164-168).

Finally, we display the rectangle and text label on the screen. If there are multiple objects of the same class in the frame, the boxes and labels will have the same color.

From there, let’s display the image and update our FPS counter:

Outside of the prediction loop, we again make a check to see if we should display the frame to the screen. If so, we show the frame (Line 181) and wait for the “q” key to be pressed if the user wants to quit (Lines 182-186).

We update our frames per second counter on Line 189.

From there, we’ll most likely continue to the top of the frame-by-frame loop to complete the process again.

If the user happened to press “ctrl+c” in the terminal or if there’s a problem reading a frame, we break out of the loop.

This last code block handles some housekeeping (Lines 200-211) and finally prints the elapsed time and the frames per second pipeline information to the screen. This information allows us to benchmark our script.

Movidius NCS object detection results

Let’s run our real-time object detector with the NCS using the following command:

Prediction results will be printed in the terminal and the image will be displayed on our Raspberry Pi monitor.

Below I have included an example GIF animation of shooting a video with a smartphone and then post-processing it on the Raspberry Pi:

Real-time object detection on the Raspberry Pi + Movidius NCS GIF

Along with the full example video of clips:

Thank you to David McDuffee for shooting these example clips so I could include it!

Here’s an example video of the system in action recorded with a Raspberry Pi:

A big thank you to David Hoffman for demoing the Raspberry Pi + NCS in action.

Note: As some of you know, this past week I was taking care of a family member who is recovering from emergency surgery. While I was able to get the blog post together, I wasn’t able to shoot the example videos. A big thanks to both David Hoffman and David McDuffee for gathering great examples making today’s post possible!

And here’s a table of results:

Figure 10: Object detection results on the Intel Movidius Neural Comput Stick (NCS) when compared to the Pi CPU. The NCS helps the Raspberry Pi to achieve a ~6.88x speedup.

The Movidius NCS can propel the Pi to a ~6.88x speedup over the standard CPU object detection! That’s progress.

I reported results with the display option being “on” as well as “off”. As you can see, displaying on the screen slows down the FPS by about 1 FPS due to the OpenCV drawing text/boxes as well as highgui overhead. The reason I reported both this week is so that you’ll have a better idea of what to expect if you’re using this platform and performing object detection without the need for a display (such as in a robotics application).

Note: Optimized OpenCV 3.3+ (with the DNN module) installations will have faster FPS on the Pi CPU (I reported 0.9 FPS previously). To install OpenCV with NEON and VFP3 optimizations just read this previous post. I’m not sure if the version of OpenCV 2.4 that gets installed with the Movidius toolchain contains these optimizations which is one reason why I reported the non-optimized 0.49 FPS metric in the table.

I’ll wrap up this section by saying that it is possible give the illusion of faster FPS with threading if you so wish. Check out this previous post and implement the strategy into  that we reviewed today.

Frequently asked questions (FAQs)

In this section I detail the answers to Frequently Asked Questions regarding the NCS.

Why does my Movidius NCS continually disconnect from my VM? It appears to be connected, but then when I run ‘make examples’ as instructed above, I see connectivity error messages. I’m running macOS and using a VM.

You must use the VirtualBox Extension Pack and add two USB device filters specifically for the Movidius. Please refer to the USB passthrough settings above.

No predictions are being made on the video — I can see the video on the screen, but I don’t see any error messages or stacktrace. What might be going wrong?

This is likely due to an error in pre-processing.

Be sure your pre-processing function is correctly performing resizing and normalization.

First, the dimensions of the pre-processed image must match the model exactly. For the MobileNet SSD that I’m working with, it is 300×300.

Second, you must normalize the input via mean subtraction and scaling.

I just bought an NCS and want to run the example on my Pi using my HDMI monitor and a keyboard/mouse. How do I access the USB ports that the NCS is blocking?

It seems a bit of poor design that the NCS blocks adjacent USB ports. The only solution I know of is to buy a short extension cable such as this 6in USB 3.0 compatible cable on Amazon — this will give more space around the other three USB ports.

Of course, you could also take your NCS to a machine shop and mill down the heatsink, but that wouldn’t be good for your warranty or cooling purposes.

How do I install the Python bindings to the NCS SDK API in a virtual environment?

Quite simply: you can’t.

Install the SDK on an isolated computer or VM.

For your Pi, install the SDK API-only mode on a separate micro SD card than the one you currently use for everyday use.

I have errors when running ‘mvNCCompile’ on my models. What do you recommend?

The Movidius graph compiling tool, mvNCCompile, is very particular about the input files. Oftentimes for Caffe, you’ll need to modify the .prototxt file. For TensorFlow I’ve seen that the filenames themselves need to be in a particular format.

Generally it is a simple change that needs to be made, but I don’t want to lead you in the wrong direction. The best resource right now is the Movidius Forums.

In the future, I may update these FAQs and the Generating Movidius graph files from your own Caffe models section with guidelines or a link to Intel documentation.

I’m hoping that the Movidius team at Intel can improve their graph compiler tool as well.

What’s next?

If you’re looking to perform image classification with your NCS, then refer to last week’s blog post.

Let me know what you’re looking to accomplish with a Movidius NCS and maybe I’ll turn the idea into a blog post.

Be sure to check out the Movidius blog and TopCoder Competition as well.

Movidus blog on GitHub

The Movidius team at Intel has a blog where you’ll find additional information:

The GitHub community surrounding the Movidius NCS is growing. I recommend that you search for Movidius projects using the GitHub search feature.

Two official repos that you should watch are (click the “watch” button on to be informed of updates):

TopCoder Competition

Figure 11: Earn up to $8,000 with the Movidius NCS on TopCoder.

Are you interested earning up to $8,000?

Intel is sponsoring a competition on TopCoder.

There are $20,000 in prizes up for grabs (first place wins $8,000)!

Registration and submission closes on February 26, 2018. That is next Monday, so don’t waste any time!

Keep track of the leaderboard and standings!


Today, we answered PyImageSearch reader, Danielle’s questions. We learned how to:

  • Install the SDK in a VM so she can use her Mac.
  • Generate Movidius graph files from Caffe models.
  • Perform object detection with the Raspberry Pi and NCS.

We saw that MobileNet SSD is >6.8x faster on a Raspberry Pi when using the NCS.

The Movidius NCS is capable of running many state-of-the-art networks and is a great value at less than $100 USD. You should consider purchasing one if you want to deploy it in a project or if you’re just yearning for another device to tinker with. I’m loving mine.

There is a learning curve, but the Movidius team at Intel has done a decent job breaking down the barrier to entry with working Makefiles on GitHub.

There is of course room for improvement, but nobody said deep learning was easy.

I’ll wrap today’s post by asking a simple question:

Are you interested in learning the fundamentals of deep learning, how to train state-of-the-art networks from scratch, and discovering my handpicked best practices?

If that sounds good, then you should definitely check out my latest book, Deep Learning for Computer Vision with Python. It is jam-packed with practical information and deep learning code that you can use in your own projects.


If you would like to download the code and images used in this post, please enter your email address in the form below. Not only will you get a .zip of the code, I’ll also send you a FREE 11-page Resource Guide on Computer Vision and Image Search Engines, including exclusive techniques that I don’t post on this blog! Sound good? If so, enter your email address and I’ll send you the code immediately!

, , , , , , , ,

82 Responses to Real-time object detection on the Raspberry Pi with the Movidius NCS

  1. Dougi February 19, 2018 at 10:50 am #

    This looks like a great article and I look forward to digging into it properly.

    I am currently working on a similar thing and am thinking of having a go with DarkNet’s YOLO.

    Do you know if that would work on a Raspberry Pi too? If so, an article on that would be awesome!

    Thanks so much.

    • Adrian Rosebrock February 19, 2018 at 12:26 pm #

      Movidius supports a variation of YOLO based on a Caffe port of the DarkNet method. I would suggest using the Movidius version of YOLO or finding a Caffe version that can be directly imported to OpenCV’s “dnn” module. See this blog post for more information.

    • John Jamieson February 19, 2018 at 9:33 pm #

      Have you had a look at ?

  2. haixun February 19, 2018 at 10:56 am #

    Well done, Adrian! First such detailed post on this. Thanks!

    • Adrian Rosebrock February 19, 2018 at 12:24 pm #

      Thanks haixun!

  3. Steve Cox February 19, 2018 at 1:36 pm #

    Great article. If anyone has hands on experience taking a re-trained tensorflow object detection model and running it on OpenCV 3.3 DNN api just like you do with caffe models I would greatly appreciate the help.

    I can’t seem to find the secret sauce to take a tensorflow model and get it loaded in OpenCV DNN. I realize everyone has great experience with caffe models, but I want to stick with Tensorflow/Karas framework.

    Thanks !!!

    • Adrian Rosebrock February 19, 2018 at 3:10 pm #

      Hey Steve! TensorFlow models can be a real pain to work with when it comes to loading their serialized weights. This is true for both OpenCV DNN and the NCS. If I can figure the “secret sauce” I’ll absolutely be doing a blog post on it.

    • Dmitry February 24, 2018 at 2:39 pm #

      There is a way to create a supportive definition of TensorFlow graph in text. As you know, Caffe models are represented by .caffemodel and .prototxt. Actually they are both protocol buffers but the last one has no weights and easy for editing. For example, you can add extra layers without weights like SoftMax. The script mentioned at a wiki page ( creates a .pbtxt file that can be used in OpenCV to help it import TensorFlow graph. Note that this script works only for SSD-based object detection model.

  4. Ali February 19, 2018 at 2:01 pm #

    Very well detailed tutorial as always, Adrian ! Looking forward to getting my Intel NCS ! Meanwhile I’m trying to use real time object detection along with some opencv image processing for a project, which framework and which model implementation would you suggest can be able to achieve a decent frame rate ( around 20 – 30 fps) on GPU ? Thanks.

    • Adrian Rosebrock February 19, 2018 at 3:11 pm #

      Hey Ali — a Single Shot Detector (SSD) + MobileNet would get you in the range of 30-50 FPS on a GPU.

  5. braca February 19, 2018 at 2:59 pm #

    Thanks Adrian very cool post! I think that you skip the section to install pip3 and one could encounter problems while trying to complete all the steps.

    • Adrian Rosebrock February 19, 2018 at 3:12 pm #

      Hi braca — just to clarify, are you referring to installing pip on the VM or the Raspberry Pi?

    • braca February 19, 2018 at 3:19 pm #

      Ignore previous comment restarted the system and worked!!!

      • braca February 19, 2018 at 3:19 pm #

        I’m using a VM!! Thanks Adrain!!

      • Adrian Rosebrock February 19, 2018 at 5:08 pm #

        Awesome, I’m glad it’s now working for you, Braca 🙂

    • David Hoffman February 19, 2018 at 4:33 pm #

      Hi Braca,

      I used the system PIP. The only packages I installed are imutils and picamera[array].

      $ pip install imutils
      $ pip install “picamera[array]”

      You may need to use pip3.

      • braca February 19, 2018 at 8:12 pm #

        Thanks David!! It helped!!

  6. Alvin February 20, 2018 at 3:41 am #

    Hi adrian, I always have problem when running the make examples on ncsdk. It says syntax error on protobuf. It appear that protobuf 2.6.1 did not compatible on python3 and make examples on movidius is using python3. How did you counter this problem? Thanks

    • Adrian Rosebrock February 22, 2018 at 11:25 am #

      Hi Alvin — I suggest you double check the formatting in your prototxt and then post in the Movidius Forums if you’re still having trouble.

  7. Al Bee February 20, 2018 at 4:29 am #

    Hey Adrian,

    Have you tried YOLO? It uses a totally different approach by applying a single neural network to the full image.

    • Adrian Rosebrock February 20, 2018 at 7:36 am #

      I have used YOLO for different projects, yes. It really depends on the project but I tend to find SSDs provide a better balance between speed and accuracy. Of course, it really depends on the project. I have yet to try YOLO on the NCS though. That will be for another project 🙂

  8. iamtodor February 20, 2018 at 8:30 am #

    Hello Adrian Rosebrock,

    I have found your blog-post very useful for myself. I want to say thank you.
    The only thing I have incomplete is I can’t figure out how to use range-detector:
    I see a lot of people have that problem and you replied you might publish even the whole article about that util. Unfortunately, I don’t find it.
    Can you help me please to run that script? I have an image and specific object – orange. I want to know the upper and lower boundaries.
    Thanks in advance

    • Adrian Rosebrock February 21, 2018 at 9:36 am #

      I had not written the article on it yet, I’ve been busy with a few other projects and writing up these deep learning tutorials. I’ll make a note to write an article on it soon.

  9. Marc A Getter February 20, 2018 at 11:39 am #

    After the make install for ncsdk runs for about 10 minutes, my vm is crashing with a red and purple screen with a combination of characters. Has anyone else encountered this?

    • David Hoffman February 22, 2018 at 11:23 am #

      Which OS is your Host, which OS is your Guest, and which version of VirtualBox are you running? For reference, I’m on macOS OSX 10.13.3, my Guest VM is Ubuntu 16.04, and VirtualBox is 5.2.6.

  10. Jim February 20, 2018 at 2:02 pm #

    I followed all the instructions (I think!) and get:
    (cv) $ python –graph graph –display 1
    Traceback (most recent call last):
    File “”, line 6, in
    from mvnc import mvncapi as mvnc
    ImportError: No module named mvnc

    I did install ncsdk and it talks to my NCS
    $ python
    Hello NCS! Device opened normally.
    Goodbye NCS! Device closed normally.
    NCS device working.

    • Adrian Rosebrock February 22, 2018 at 9:12 am #

      Hey Jim — unfortunately you will not be able to use Python virtual environments with the NCS. I discuss this more in the previous post.

  11. abdbaddude February 21, 2018 at 2:13 am #

    Wondering why you keep using print(“INFO” ….) ? I guess the python logging facility could be used. Or is the a performance gain considered on the raspberryPi.

    • Adrian Rosebrock February 21, 2018 at 9:34 am #

      You could use logging if you wanted, there is no problem with that. I just used “print” as some readers may not be comfortable or used to the logging features with Python. It’s pretty trivial to swap out “print” for “logging” so feel free to use whichever one you are comfortable with.

  12. Prubio February 21, 2018 at 4:26 am #

    I downloaded your code and I had some errors. I share it here in case can help others.

    The first error I suffered was: numpy.float16 cannot be interpreted as integer.
    To solve that, in line 50:

    The second error: DISP_MULTIPLIER is not defined
    In lines 169 and 170 change DISP_MULTIPLIER for DISPLAY_MULTIPLIER.

    Great job Adrian, congrats!
    I am trying to apply ncs to object detection with tensorflow, for my own models, but I have problems. If you have time, I´ll really appreciate an explication about that.


    • Adrian Rosebrock February 21, 2018 at 2:59 pm #

      Hi Prubio — thanks for the catch about the variable name mismatch. This was a mistake while putting the post together and it has now been corrected. I didn’t encounter the same issue with needing to force the type of output[0] to an int, but I’m glad that you got yours working! What is your question about NCS object detection with tensorflow?

      • Prubio February 22, 2018 at 6:42 am #

        I trained ssd_mobilenet_v1 (pre trained in coco dataset) for object detection, with LabelImg and object detection API Tensorflow. I obtained the frozen_inference_graph.pb and it´s working perfectly, but I can´t convert it with mvNC for using with Movidius NCS. I have read that ssd_mobilenet_v1 has not supported. But I have the same problem with ssd_inception_v2.
        How could I do for reach my goal? Train a network with object detection API and obtain the graph to apply inference with Movidius NCS.

        Thanks Adrian.

        • Adrian Rosebrock February 22, 2018 at 11:17 am #

          Prubio — I’ve had success using the models that Movidius has provided. Using models of my own I’ve had limited success and have been referring to the Movidius Forums and searching on GitHub. That’s the first place I’ve been going for support and that seems to be where the “experts” are. Nobody is truly an expert yet (short of the ones that coded the compile tool at Movidius) as this product is still in its infancy. Early adopters have definitely been struggling along, but it will get better I hope.

  13. simon February 21, 2018 at 6:49 pm #


    Thanks for your amazing post. And I have tested the same script on my workstation (i7 Core, 64GB RAM, NVIDIA 980 TI) and the FPS is around 10. I guess it depends on CPU/RAM performance.

    Since it seems to depend on CPU/RAM performance, I wish to monitor system CPU and RAM usage when more than one NCS USB is taking a job; I got the secound NCS USB and wonder how to assign a different work on a second USB while the first USB works on the other script.


    • simon February 21, 2018 at 6:52 pm #

      So this
      “device = mvnc.Device(devices[0])” should be
      “device = mvnc.Device(devices[1])” on the secound USB?


      • simon February 21, 2018 at 7:08 pm #

        I have tested it and it works fine.
        The second one shows the same performance.

        CPU usages on both: 9%
        RAM usage on both: about 130 MB.
        FPS: around 10 FPS.


        • Adrian Rosebrock February 22, 2018 at 8:49 am #

          Excellent. Thanks for sharing.

      • Adrian Rosebrock February 22, 2018 at 8:48 am #

        If you have two NCS devices plugged in, that’s my understanding as to how it would work.

    • Adrian Rosebrock February 22, 2018 at 8:47 am #

      Hi Simon — I’d recommend a threaded approach if you’re putting multiple NCS devices to work. Here’s a relevant blog post to get you started. Populate a queue as frames come in. Then have a thread to preprocess and assign the images to the available NCSs (in separate threads). The trick would be getting the detections back into the right order before displaying since you’ve got multiple worker threads. To accommodate, I’d recommend including additional information with each frame from the start (frame count number). This might be a future blog post idea.

  14. Michael February 21, 2018 at 9:34 pm #


    When I download the code from your email, I get the following error:

    This XML file does not appear to have any style information associated with it. The document tree is shown below.

    Access Denied

    • Adrian Rosebrock February 22, 2018 at 8:57 am #

      Hi Michael — thank you for bringing this to my attention. I uploaded a new version of the NCS code yesterday and forgot to set the permissions to make the file downloadable. It is fixed now and the code can be downloaded.

  15. Jussi February 22, 2018 at 9:47 am #

    VM dont recognize my movidius stick.
    I have installed extension and insert guest additions.

    • David Hoffman February 22, 2018 at 11:05 am #

      Hi Jussi, I definitely struggled with USB too, so you’re not alone. Which OS is your Host, which OS is your Guest, and which version of VirtualBox are you running? For reference, I’m on macOS OSX 10.13.3, my Guest VM is Ubuntu 16.04, and VirtualBox is 5.2.6.

      • Jussi February 23, 2018 at 9:01 am #

        Hi David, I use Ubuntu 17.10, my Guest is 16.04 and VirtualBox is 5.2.6. I am really stack with this. Meaby I try also with my mac.

        • David Hoffman February 23, 2018 at 8:59 pm #

          Hi Jussi, I would suggest that you post in the VirtualBox forums. They’ll be able to help you and their community is very active. Feel free to post a linkback to your VirtualBox forum post so that other PyImageSearch readers can see any potential solutions. I’m also curious — did you experience a VirtualBox error, or was the Stick just not able to stay connected (an NCS error in the terminal of the VM itself)?

          • Jussi February 28, 2018 at 10:44 am #

            Ok. I will ask that from VM forum. Ubuntu did not recognize the usb stick even filters has right setup. I changed to my mac and stick found right away ;).

          • David Hoffman March 1, 2018 at 2:57 pm #

            Thanks for sharing. I’m sorry you’re having trouble with an Ubuntu host — I don’t currently know a solution for you. If you figure it out, then please share to benefit the community.

    • Don February 23, 2018 at 11:38 pm #

      Don’t forget to add the 2 filters described above in “USB passthrough settings”. I struggled with this problem for days, until I came to this article. (thumbs up Adrian) Intel’s team give NO instructions on this, and users in the forums state you must enter “Product ID” and “Vendor ID” in the filter. THIS WILL FAIL!! (at least for me) ONLY enter the “Vendor ID” ‘040e’ and ’03e7′ for each.

      You can see if it’s connected by looking at Devices-USB on the VM window. If there a check next to the Movidius, then ubuntu can see it, and it should auto-reconnect during initialization.

      • Adrian Rosebrock February 26, 2018 at 10:21 am #

        Thanks for sharing Don. I’m glad this blog post was able help you!

  16. Raghvendra Jain February 23, 2018 at 12:56 am #

    Hi, Thank you for the great blog! My question is can we try Caffe2 also on this stick too? If so, it will be very easy to write code using PyTorch and use ONNX to convert a model defined in PyTorch into the ONNX format and then load it into Caffe2. Thank you very much.

    • Adrian Rosebrock February 23, 2018 at 8:55 pm #

      As far as I know, Caffe2 isn’t supported yet — there’s a Movidius Forum topic question and nobody from Intel has responded. I’m sure it is on their roadmap (quite honestly it would be nice if they share their roadmap on the Movidius Blog).

      • Raghvendra Jain February 24, 2018 at 8:06 am #

        Thank you for the reply!

  17. Pongrut February 24, 2018 at 1:58 am #

    Hi, I always appreciate your dedication to your blogs
    I doubt the process of creating a graph. I tried downloading the MobileNet_deploy.caffemodel and MobileNet_deploy.prototxt files from chuanqui305 GitHub, and of course, I failed to generate graph files.

    With your provided files, I can create your graph file as you shown in the blog, so I want to ask what you need to do to generate the graph file successfully.

  18. Andy February 25, 2018 at 12:54 pm #

    Just an update for anyone trying to use the NCS on a VirtualBox VM on Windows 10…
    It looks like there are various issues with setting this up depending on your PC. I’m on an ACER with 2 x USB3.0 and 1 x USB2.0 ports on Windows 10 Home and running a VirtualBox VM with Ubuntu 16.04.
    I’ve gone through various combinations with plugging the NCS into all three different USB ports and the USB2.0 seemed to be the most stable. The device shows up in the device list in the VM USB menu as Movidius MA2X5X with a Vendor ID of 03e7 and Product ID of 2150. It’s worth noting that the movidius forum have indicated that it’s not worth pursuing using the VM route as it does seem fraught with challenges but Adrian has summarised the various steps worth trying.
    For my part, I would add that since I tried the stick in the USB3.0 port first which didn’t work, it seemed to leave residual devices in the system that were being picked up by VirtualBox (I had an unknown device 03e7:2150 and a Movidius LSC (or VSC maybe as it’s now gone) on 03e7:2150 too) .
    The issue of the device being dropped from the VM system is still present (you’ll see this when you run ‘make examples’ in the ncsdk folder ) and there was some traffic about this on the forums but the ncappzoo hello_ncs_py works as a single test so persevere but be patient.

    Thanks again to Adrian for this great write-up with all the supporting detail.

    • Adrian Rosebrock February 26, 2018 at 10:24 am #

      Andy, thanks for sharing your experience the VM in Windows 10.

    • Niklas March 1, 2018 at 7:56 pm #

      Hi Andy,

      quick question is the :
      [Error 7] Toolkit Error: USB Failure. Code: No device found

      what youre refering to here ?
      Got that Error message myself when i try to launch the examples… (Win10)

    • John Jamieson March 5, 2018 at 10:08 pm #

      Hi Andy, I use VirtualBox 5.2.8 on windows 10 FCU. The VM is Ubuntu 16.04.5. The NCS is plugged into a USB3 port. I set the USB in VirtualBox to the USB 3.0 xHCI controller. I then add two USB devices, 1st one with “Vendor 03E7, Product 2150” and the 2nd one with “Vendor 03E7, Product F63B”. Note the Vendor ID for both. I did not test this combo on a USB 2 port yet, but will test it when get a chance to put the VM onto another machine. (laptop only has 3.0). I am able to run every single NCS example (I only have 1 stick) in the ncappzoo without any problems, except maybe the webcam – it struggles somewhat. This includes the python and compiled C examples. I used the minhoolee/install-opencv-3.0.0 script off github for openCV. I dont get any dropouts with the NCS, even when i had it plugged into a TB powered USB hub.

  19. han February 28, 2018 at 2:43 am #

    Thanks for sharing this cool information

    • Adrian Rosebrock March 1, 2018 at 2:53 pm #

      I’m glad you enjoyed it Han. Do you have an NCS or are you considering purchasing one?

  20. Jiang Chuan February 28, 2018 at 4:48 am #

    Hi Adrian,
    I have movidius but I do not have Raspberry Pi, So I am trying to start your sample from my ubuntu host. When I run “python –graph graphs/mobilenetgraph”, It fails as follows:
    [INFO] finding NCS devices…
    [INFO] found 1 devices. device0 will be used. opening device0…
    [INFO] loading the graph file into RPi memory…
    [INFO] allocating the graph on the NCS…
    [INFO] starting the video stream and FPS counter…
    Traceback (most recent call last):
    File “”, line 144, in
    predictions = predict(frame, graph)
    File “”, line 54, in predict
    for box_index in range(num_valid_boxes):
    TypeError: ‘numpy.float16’ object cannot be interpreted as an integer

    Do you have any suggestion about how to fix the error?

    Jiang Chuan.

    • Adrian Rosebrock March 1, 2018 at 2:55 pm #

      Hi Jiang, it looks like num_valid_boxes is being reported as a float16. Try casting it to an int.

  21. Jiang Chuan February 28, 2018 at 4:53 am #

    In order to get video from camera, I changed the following code:

    vs = VideoStream(usePiCamera=True).start()


    vs = VideoStream(0).start()

  22. Jussi February 28, 2018 at 11:58 am #

    Hi, what is (in this case) the best way and line to flip video horizontal? My pi camera is upside down.

    • Adrian Rosebrock March 1, 2018 at 2:41 pm #

      You can use the “cv2.flip” function.

  23. Lee Mewshaw March 1, 2018 at 11:04 am #

    Hi Adrian,
    I’m trying to follow your steps, and I’m not clear on when I switch the Movidius from Ubuntu to the Raspberry Pi. I have successfully made it through “Test the SDK” on the Ubuntu machine, and I’m getting an error on mvNCCompile. I’ll work through the forums to figure that out, but the question for you is after running that command mvNCCompile command, do I move it over to a USB port on the raspberry pi after that, or before? I can’t tell from your steps when you actually are writing something to the Movidius and I should then move it to the pi.

    Thanks in advance for any help!

    • Adrian Rosebrock March 1, 2018 at 2:59 pm #

      Hi Lee. I’m sorry if this was unclear. Please review the workflow image above. You need the NCS to generate the graph. You also need the NCS to deploy the graph. Does this make sense?

  24. Christoph Viehoff March 3, 2018 at 10:02 pm #

    I ran follwed the VM installation and the SDK installed sucessfully . All tests pass when I run the mvNCComple script from the Real-time-object-detection folder on my VM I get the following error:

    nvNCComplie V02.00, Copyright @ Movidius Ltd 2016

    Error importing caffe

    • Adrian Rosebrock March 9, 2018 at 10:29 am #

      Hi Christoph — try opening a fresh terminal and/or check your PYTHONPATH environment variable. For further information, please see the response from Tome at Intel on this direct forum link. Let me know if that works.

  25. Zimeng March 5, 2018 at 1:02 pm #

    Hi Adrian, after building CV env on Rasp pi (jessie+cv3.3) , I wanna update system to Strech (recommended in your tutorials on using Movidius NCS ) ,will it influence my current cv env?

    • Adrian Rosebrock March 7, 2018 at 9:21 am #

      Updating the actual OS is notorious for breaking development environments. I do not recommend it. But if you would like to try, backup your .img file on your desktop and then try the upgrade.

  26. Kevin March 8, 2018 at 1:28 pm #

    Hi Adrian. Thank you for this amazing tutorial. Everything worked perfectly. Now I’m trying to make a gender detection in real-time with the Movidius. Do you have any recommendations of a model to use? I’ve found a classification model in, but I would like to make a detection. Can the classification be used inside this detection code?

    Thank You,

    • Adrian Rosebrock March 9, 2018 at 9:02 am #

      I don’t think there is a need for a detection model. Use a face detector to detect the face. Extract the ROI. Then pass the ROI into a classification model.

      • Kevin March 9, 2018 at 3:40 pm #

        Ohhh. Thank you Adrian 😀

  27. schamarti March 9, 2018 at 1:34 am #

    Hi Adrian, I have installed sdk and I am able to run example graph provided in the downloaded section. To convert model on raspberry pi, I am getting error mvNCCompile command not found. Any hints

    • Adrian Rosebrock March 9, 2018 at 10:32 am #

      Hey Schmarti — run mvNCCompile on the Ubuntu machine or VM where you installed the full SDK. The tool isn’t available on the Pi.

  28. Hein March 12, 2018 at 1:26 am #

    Hi Adrian, I am confused about this article.

    Is the tutorial you provided to run on Ubantu or Raspberry?

    • Adrian Rosebrock March 14, 2018 at 1:09 pm #

      The Raspberry Pi runs Raspbian but I needed an Ubuntu VM to develop the code and create the deep learning model that later runs on the Pi.

  29. monsour March 17, 2018 at 11:08 pm #

    hai sir Adrian can i ask some help? do u have any xml file for garbage detection? because it is a very long process when i trained my own haar_cascade.

    thank you for the help

    • Adrian Rosebrock March 19, 2018 at 5:18 pm #

      I do not have any pre-trained models for garbage detection. You would need to train your own.

Leave a Reply