Raspberry Pi: Deep learning object detection with OpenCV

A few weeks ago I demonstrated how to perform real-time object detection using deep learning and OpenCV on a standard laptop/desktop.

After the post was published I received a number of emails from PyImageSearch readers who were curious if the Raspberry Pi could also be used for real-time object detection.

The short answer is “kind of”…

…but only if you set your expectations accordingly.

Even when applying our optimized OpenCV + Raspberry Pi install the Pi is only capable of getting up to ~0.9 frames per second when applying deep learning for object detection with Python and OpenCV.

Is that fast enough?

Well, that depends on your application.

If you’re attempting to detect objects that are quickly moving through your field of view, likely
not.

But if you’re monitoring a low traffic environment with slower moving objects, the Raspberry Pi could indeed be fast enough.

In the remainder of today’s blog post we’ll be reviewing two methods to perform deep learning-based object detection on the Raspberry Pi.

Looking for the source code to this post?
Jump right to the downloads section.

Raspberry Pi: Deep learning object detection with OpenCV

Today’s blog post is broken down into two parts.

In the first part, we’ll benchmark the Raspberry Pi for real-time object detection using OpenCV and Python. This benchmark will come from the exact code we used for our laptop/desktop deep learning object detector from a few weeks ago.

I’ll then demonstrate how to use multiprocessing to create an alternate method to object detection using the Raspberry Pi. This method may or may not be useful for your particular application, but at the very least it will give you an idea on different methods to approach the problem.

Object detection and OpenCV benchmark on the Raspberry Pi

The code we’ll discuss in this section is is identical to our previous post on Real-time object detection with deep learning and OpenCV; therefore, I will not be reviewing the code exhaustively.

For a deep dive into the code, please see the original post.

Instead, we’ll simply be using this code to benchmark the Raspberry Pi for deep learning-based object detection.

To get started, open up a new file, name it real_time_object_detection.py , and insert the following code:

We then need to parse our command line arguments:

Followed by performing some initializations:

We initialize CLASSES , our class labels, and corresponding COLORS , for on-frame text and bounding boxes (Lines 22-26), followed by loading the serialized neural network model (Line 30).

Next, we’ll initialize the video stream object and frames per second counter:

Wwe initialize the video stream and allow the camera warm up for 2.0 seconds (Lines 35-37).

On Line 35 we initialize our VideoStream  using a USB camera If you are using the Raspberry Pi camera module you’ll want to comment out Line 35 and uncomment Line 36 (which will enable you to access the Raspberry Pi camera module via the VideoStream  class).

From there we start our fps  counter on Line 38.

We are now ready to loop over frames from our input video stream:

Lines 41-55 simply grab and resize a frame , convert it to a blob , and pass the blob  through the neural network, obtaining the detections  and bounding box predictions.

From there we need to loop over the detections  to see what objects were detected in the frame :

On Lines 58-80, we loop over our detections . For each detection we examine the confidence  and ensure the corresponding probability of the detection is above a predefined threshold. If it is, then we extract the class label and compute (x ,y) bounding box coordinates. These coordinates will enable us to draw a bounding box around the object in the image along with the associated class label.

From there we’ll finish out the loop and do some cleanup:

Lines 82-91 close out the loop — we show each frame, break  if ‘q’ key is pressed, and update our fps  counter.

The final terminal message output and cleanup is handled on Lines 94-100.

Now that our brief explanation of real_time_object_detection.py  is finished, let’s examine the results of this approach to obtain a baseline.

Go ahead and use the “Downloads” section of this post to download the source code and pre-trained models.

From there, execute the following command:

As you can see from my results we are obtaining ~0.9 frames per second throughput using this method and the Raspberry Pi.

Compared to the 6-7 frames per second using our laptop/desktop we can see that the Raspberry Pi is substantially slower.

That’s not to say that the Raspberry Pi is unusable when applying deep learning object detection, but you need to set your expectations on what’s realistic (even when applying our OpenCV + Raspberry Pi optimizations).

Note: For what it’s worth, I could only obtain 0.49 FPS when NOT using our optimized OpenCV + Raspberry Pi install — that just goes to show you how much of a difference NEON and VFPV3 can make.

A different approach to object detection on the Raspberry Pi

Using the example from the previous section we see that calling net.forward()  is a blocking operation — the rest of the code in the while  loop is not allowed to complete until net.forward()  returns the detections .

So, what if net.forward()  was not a blocking operation?

Would we able to obtain a faster frames per second throughput?

Well, that’s a loaded question.

No matter what, it will take approximately a little over a second for net.forward()  to complete using the Raspberry Pi and this particular architecture — that cannot change.

But what we can do is create a separate process that is solely responsible for applying the deep learning object detector, thereby unblocking the main thread of execution and allow our while  loop to continue.

Moving the predictions to separate process will give the illusion that our Raspberry Pi object detector is running faster than it actually is, when in reality the net.forward()  computation is still taking a little over one second.

The only problem here is that our output object detection predictions will lag behind what is currently being displayed on our screen. If you detecting fast-moving objects you may miss the detection entirely, or at the very least, the object will be out of the frame before you obtain your detections from the neural network.

Therefore, this approach should only be used for slow-moving objects where we can tolerate lag.

To see how this multiprocessing method works, open up a new file, name it pi_object_detection.py , and insert the following code:

For the code walkthrough in this section, I’ll be pointing out and explaining the differences (there are quite a few) compared to our non-multprocessing method.

Our imports on Lines 2-10 are mostly the same, but notice the imports of Process  and Queue  from Python’s multiprocessing package.

Next, I’d like to draw your attention to a new function, classify_frame :

Our new classify_frame  function is responsible for our multiprocessing — later on we’ll set it up to run in a child process.

The classify_frame  function takes three parameters:

  • net : the neural network object.
  • inputQueue : our FIFO (first in first out) queue of frames for object detection.
  • outputQueue: our FIFO queue of detections which will be processed in the main thread.

This child process will loop continuously until the parent exits and effectively terminates the child.

In the loop, if the inputQueue  contains a frame , we grab it, and then pre-process it and create a blob  (Lines 16-22), just as we have done in the previous script.

From there, we send the blob  through the neural network (Lines 26-27) and place the detections  in an outputQueue  for processing by the parent.

Now let’s parse our command line arguments:

There is no difference here — we are simply parsing the same command line arguments on Lines 33-40.

Next we initialize some variables just as in our previous script:

This code is the same — we initialize class labels, colors, and load our model.

Here’s where things get different:

On Lines 56-58 we initialize an inputQueue  of frames, an outputQueue  of detections, and a detections  list.

Our inputQueue  will be populated by the parent and processed by the child — it is the input to the child process.  Our outputQueue  will be populated by the child, and processed by the parent — it is output from the child process. Both of these queues trivially have a size of one as our neural network will only be applying object detections to one frame at a time.

Let’s initialize and start the child process:

It is very easy to construct a child process with Python’s multiprocessing module — simply specify the target  function and args  to the function as we have done on Lines 63 and 64.

Line 65 specifies that p  is a daemon process, and Line 66 kicks the process off.

From there we’ll see some more familiar code:

Don’t forget to change your video stream object to use the PiCamera if you desire by switching which line is commented (Lines 71 and 72).

Once our vs  object and fps  counters are initialized, we can loop over the video frames:

On Lines 80-82, we read a frame, resize it, and extract the width and height.

Next, we’ll work our our queues into the flow:

First we check if the inputQueue  is empty — if it is empty, we put a frame in the inputQueue  for processing by the child (Lines 86 and 87). Remember, the child process is running in an infinite loop, so it will be processing the inputQueue  in the background.

Then we check if the outputQueue  is not empty — if it is not empty (something is in it), we grab the detections  for processing here in the parent (Lines 90 and 91). When we call get()  on the outputQueue , the detections are returned and the outputQueue  is now momentarily empty.

If you are unfamiliar with Queues or if you want a refresher, see this documentation.

Let’s process our detections:

If our detections  list is populated (it is not None ), we loop over the detections as we have done in the previous section’s code.

In the loop, we extract and check the confidence  against the threshold (Lines 100-105),  extract the class label index (Line 110), and draw a box and label on the frame (Lines 111-122).

From there in the while loop we’ll complete a few remaining steps, followed by printing some statistics to the terminal, and performing cleanup:

In the remainder of the loop, we display the frame to the screen (Line 125) and capture a key press and check if it is the quit key at which point we break out of the loop (Lines 126-130). We also update our fps  counter.

To finish out, we stop the fps  counter, print our time/FPS statistics, and finally close windows and stop the video stream (Lines 136-142).

Now that we’re done walking through our new multiprocessing code, let’s compare the method to the single thread approach from the previous section.

Be sure to use the “Downloads” section of this blog post to download the source code + pre-trained MobileNet SSD neural network. From there, execute the following command:

Here you can see that our while  loop is capable of processing 27 frames per second. However, this throughput rate is an illusion — the neural network running in the background is still only capable of processing 0.9 frames per second.

Note: I also tested this code on the Raspberry Pi camera module and was able to obtain 60.92 frames per second over 35 elapsed seconds.

The difference here is that we can obtain real-time throughput by displaying each new input frame in real-time and then drawing any previous detections  on the current frame.

Once we have a new set of detections  we then draw the new ones on the frame.

This process repeats until we exit the script. The downside is that we see substantial lag. There are clips in the above video where we can see that all objects have clearly left the field of view…

…however, our script still reports the objects as being present.

Therefore, you should consider only using this approach when:

  1. Objects are slow moving and the previous detections can be used as an approximation to the new location.
  2. Displaying the actual frames themselves in real-time is paramount to user experience.

Summary

In today’s blog post we examined using the Raspberry Pi for object detection using deep learning, OpenCV, and Python.

As our results demonstrated we were able to get up to 0.9 frames per second, which is not fast enough to constitute real-time detection. That said, given the limited processing power of the Pi, 0.9 frames per second is still reasonable for some applications.

We then wrapped up this blog post by examining an alternate method to deep learning object detection on the Raspberry Pi by using multiprocessing. Whether or not this second approach is suitable for you is again highly dependent on your application.

If your use case involves low traffic object detection where the objects are slow moving through the frame, then you can certainly consider using the Raspberry Pi for deep learning object detection. However, if you are developing an application that involves many objects that are fast moving, you should instead consider faster hardware.

Thanks for reading and enjoy!

And if you’re interested in studying deep learning in more depth, be sure to take a look at my new book, Deep Learning for Computer Vision with Python. Whether this is the first time you’ve worked with machine learning and neural networks or you’re already a seasoned deep learning practitioner, my new book is engineered from the ground up to help you reach expert status.

Just click here to start your journey to deep learning mastery.

Downloads:

If you would like to download the code and images used in this post, please enter your email address in the form below. Not only will you get a .zip of the code, I’ll also send you a FREE 11-page Resource Guide on Computer Vision and Image Search Engines, including exclusive techniques that I don’t post on this blog! Sound good? If so, enter your email address and I’ll send you the code immediately!

, , , , , , , , , , ,

225 Responses to Raspberry Pi: Deep learning object detection with OpenCV

  1. Mr.Odeh October 16, 2017 at 11:16 am #

    Thanks, dr.adrian for this great article!
    It is working fine with me, but I have a small question.
    I want to detect insects in real time using raspberry, do you recommend any pre-trained module that can do the object detection for insects not just persons, dogs, sofa …etc? if there isn’t what should u do in general to achieve my aim?
    thanks again

    • Adrian Rosebrock October 16, 2017 at 12:14 pm #

      I’m not aware of a pre-trained model that specifically detects insects. I would suggest fine-tuning an existing, pre-trained neural network. I discussing fine-tuning inside Deep Learning for Computer Vision with Python.

  2. JBeale October 16, 2017 at 11:55 am #

    Very impressive! My own experiments on RPi was about 3 or 4 seconds per frame. So almost 1 fps is quite an improvement from that. With a moderately wide-angle lens that could already be useful, unless you must have the object almost completely fill the frame. Does this code fully utilize all 4 cores on the RPi 3, or is there potentially some additional parallelization possible?

    • Adrian Rosebrock October 16, 2017 at 12:13 pm #

      There are always more optimizations that can be made, it’s just a matter of if it’s worth it. The fully utilize all cores to their maximum potential we would need OpenCL (which to my knowledge) The Raspberry Pi does not support.

      • JBeale October 17, 2017 at 1:46 pm #

        Thanks. I have yet to try your code, but if (for example) it only uses 2 cores and it’s CPU-bound rather than memory or I/O bound, you ought to get a speedup by simply instantiating two separate processes, one looking at odd frames and one doing even frames. But maybe it’s not that easy; you might run out of memory.

      • JBeale October 18, 2017 at 12:14 pm #

        Speaking of OpenCL on Raspbery Pi: it is not 100% complete, but:
        09-OCT-2017 : I present to you VC4CL (VideoCore IV OpenCL):
        https://www.raspberrypi.org/forums/viewtopic.php?t=194952

        • Adrian Rosebrock October 18, 2017 at 3:55 pm #

          Awesome, thanks for sharing.

          • Jay October 27, 2017 at 1:53 am #

            Came across this article by accident. Great site. I realise this is a python based site but what are the speed improvements were this to be implemented in C++ for comparison sake?

          • Adrian Rosebrock October 27, 2017 at 11:15 am #

            Hi Jay — I’m glad you enjoyed the blog post. Python is just a wrapper around the original C/C++ code for OpenCV. So the speed will be very similar.

      • Belen March 21, 2018 at 1:20 am #

        My pi3 runs this script at 98% CPU load, and core temperature reaching it’s peak. How do I bring this down?

        • Adrian Rosebrock March 22, 2018 at 10:05 am #

          In short, you can’t without reducing the frames per second processing pipeline. If you are concerned about peak temperature you should only process one frame every 10-15 seconds.

          • Belen March 26, 2018 at 12:07 am #

            Hi Adrian,

            Thanks for the reply, so how do i reduce the frames per second processing pipeline?

          • Adrian Rosebrock March 27, 2018 at 6:19 am #

            I would recommend inserting a time.sleep call at the end of each iteration of the loop.

  3. Dayle October 16, 2017 at 12:18 pm #

    Hi Adrian,

    Thanks a ton for remembering us Pi enthusiasts. I first got interested in image analysis after someone stole your beer, but was afraid you would lose interest in the Pi after purchasing the beast.

    Looking forward to diving in to this post and reading the new book.

    Cheers,
    Dayle

    • Adrian Rosebrock October 16, 2017 at 12:35 pm #

      Hi Dayle — I certainly have not lost interested in the Raspberry Pi, I’ve just primarily been focusing on deep learning tutorials lately 🙂

  4. Anish October 16, 2017 at 2:25 pm #

    How is this method different from using Squezenet for object detection on a raspberry pi?
    The one you posted a couple of weeks ago?
    Also what are the pros and cons of using squezenet over this method?

    • Adrian Rosebrock October 17, 2017 at 9:37 am #

      SqueezeNet is an image classifier. It takes an entire image and returns a single class label. It does no object detection or localization.

      The SSD and Faster R-CNN frameworks can be used for object detection. It requires that you take an architecture (SqueezeNet, VGGNet, etc.) and then train it using the object detection framework. This will minimize the joint loss between class label prediction AND localization.

      The gist is that vanilla SqueezeNet and SSD are two totally different frameworks.

      If you’re interested in learning more about deep learning (and how these architectures differ), I would definitely suggest working through Deep Learning for Computer Vision with Python where I cover these methods in detail.

  5. Ahmad October 16, 2017 at 6:33 pm #

    i have this error , there is a problem here -> args = vars(ap.parse_args())

    usage: real_time_object_detection.py [-h] -p PROTOTXT -m MODEL [-c CONFIDENCE]
    real_time_object_detection.py: error: argument -p/–prototxt is required

    • Adrian Rosebrock October 17, 2017 at 9:34 am #

      Please read up on command line arguments.

      • aman January 6, 2018 at 4:27 am #

        I am also getting same error as above mentioned.

        usage: pi_object_detection.py [-h] -p PROTOTXT -m MODEL [-c CONFIDENCE]
        pi_object_detection.py: error: argument -p/–prototxt is required

        I read out your suggestion of “command line argument” but didn’t able to understand how to use it.

        So, help me ,how would i remove this error?

        • Adrian Rosebrock January 8, 2018 at 2:54 pm #

          Hey Aman — this blog assumes some basic knowledge of working the command line. If you’re new to the command line that’s okay — we all start somewhere! Take the time to familiarize yourself with command line arguments before continuing. Otherwise, be sure to Google “command line name of your operating system” and and read up on how to use the command line for your OS.

        • Ahmed March 24, 2018 at 11:23 am #

          hey aman did you find a solution to this error please if you have a solution tell me

  6. Komoriii October 16, 2017 at 9:38 pm #

    Impressive tutorial.This article helped me a lot,thank you!

    • Adrian Rosebrock October 17, 2017 at 9:34 am #

      Fantastic, I’m glad to hear it Komoriii! 🙂

  7. Sachin October 17, 2017 at 1:36 am #

    Great post as always, Adrian! I have learned a lot about computer vision from the content on your site.

    Regarding doing AI on the Pi, I would personally not do detection and recognition on an edge device. At least not until they ship a Pi with an AI chip and a decent GPU! And maybe not even then, due to the high power (electricity) consumption of AI. I’d much rather use the Pi as a sensor + basic signal processor, WiFi over all the video / sensor signals to a CPU box, and run all the algorithms on that box.
    So I guess I agree with your conclusions.

    • Adrian Rosebrock October 17, 2017 at 9:33 am #

      Hi Sachin — thanks for the comment. I actually discuss the tradeoffs of using the Raspberry Pi for deep learning in this post. In general, I do agree with you that a Raspberry Pi should not be used for deep learning unless under very specific circumstances.

      • Peter November 30, 2017 at 9:53 pm #

        Any specific CPU box that you think is a good (relatively cheap) option for doing the post-processing?

        • Adrian Rosebrock December 2, 2017 at 7:31 am #

          What type of post-processing are you referring to? The type of post-processing you are doing would impact my suggestion.

  8. Abhishek October 17, 2017 at 3:35 am #

    Hi Adrian, i love ur work, Sir can you please tell me how i can compute :the (x, y)-coordinates of the bounding box for the object if i’m using Squeezenet instead of MobileNet SSD caffe Model on my raspberry pi 3…..what i supposed to change in “box = detections[0, 0, i, 3:7] * np.array([w, h, w, h])” so that it will work with detecting object in squeezenet with highest probablity (I’m able to find the index of object with highest probability till now with your previous post on deep learning) any help is appreciated 🙂 [i have raspberrian stretch with opencv3.30 -dev installed(neon optimized)]

    • Adrian Rosebrock October 17, 2017 at 9:31 am #

      Are you using your own custom trained SqueezeNet using the SSD framework? Keep in mind that you cannot swap out networks trained for image classification and use them for object detection.

      • Abhishek October 17, 2017 at 11:16 pm #

        Thanks for reply :), i figured that out eventually (Previously i was using SqueezeNet v1.1 imageclassifier instead of SSD framework) but i found another great SqueezeNet-SSD (based on Squeezenet v1.1)

        https://github.com/chuanqi305/SqueezeNet-SSD

        I benchmarked it using this script but see merely any difference from MobileNetSSD …..both have same FPS around 1.72FPS(opencv optimized) on normal script and 29.4FPS(opencv optimized) using this Multithreaded script…..Through Squeezenet v1.1 (around 0.4 ms on raspberry pi 3) is way faster than any other image classifier, why this Squeezenet-SSD is slower ? I’m totally confused :\

        • Adrian Rosebrock October 19, 2017 at 4:56 pm #

          SqueezeNet v1.1 is slower because it utilizes ResNet-like modules. These increase accuracy, but slow the network down a bit.

          • cweihang December 29, 2017 at 3:30 am #

            I think that’s not the point. It’s the difference between SqueezeNet-SSD and SqueezeNet that causes the speed difference.

          • Adrian Rosebrock December 31, 2017 at 9:53 am #

            Thanks for the note, I must have misunderstood the original comment when I read it the first time. In general, yes, image classification networks will tend to run faster than object detection networks.

  9. David Killen October 17, 2017 at 4:46 am #

    trivial point, no need to publish, but you say ‘net.forwad()’ vice ‘net.forward()’ at least twice

    • Adrian Rosebrock October 17, 2017 at 9:30 am #

      Thanks for letting me know, David! I have updated the post.

  10. M.Komijani October 17, 2017 at 8:37 am #

    Hello Adrian,

    Thanks for this great article!
    Actually, I’m a starter in deep learning, but I want to use the Raspberry Pi for deep learning object detection.
    One question: Does x-nor net improves the speed results?
    (https://pjreddie.com/media/files/XNOR_ECCV2.pdf)

    • Adrian Rosebrock October 17, 2017 at 9:27 am #

      I haven’t used XNOR net to benchmark it, but from the paper the argument is that you can use XNOR net to speedup the network. You end up saving (approximately) 32x memory and 58x faster convolutional operations.

  11. Ying October 18, 2017 at 5:09 am #

    hi Adrian,

    Can we only detect people or car (i.e. specific class) by changing the python code?

    • Adrian Rosebrock October 19, 2017 at 4:53 pm #

      Yes. Check the idx of the predicted class and filter out the ones you are uninterested in.

  12. jsmith October 18, 2017 at 11:27 am #

    Hi Adrian,

    I’ve been wanting to do this for months, and it was this that got me to your website, so thank you!

    I have been trying to tweak the code so that I can grab the frame when an object is detected and save that as a .jpg in a folder as:

    /Pictures/{label}/{label}_{confidence}.jpg

    following the ‘Accessing the Raspberry Pi Camera with OpenCV and Python’ tutorial, so I can have my own dataset by using the Pi to do all the hard work.

    However I keep getting an mmalError message.

    How would you go about taking a frame from the Pi when it detects an object and saving that frame in a folder with that object’s class so you can have a dataset to work with?

    Thanks!

    • Adrian Rosebrock October 19, 2017 at 4:51 pm #

      I would suggest debugging this line-by-line. Try to determine what line is throwing the error by inserting “print” statements. If you can provide that, I can try to point you in the right direction.

      From there, you can use the cv2.imwrite to save your image to disk. You can format your filename using the detected label and associated probability returned by net.forward.

  13. Roald October 18, 2017 at 3:52 pm #

    Hi Adrian,

    I’m having issues that net.forward() seems to return inconsistent results. For example, I have two frames. One frame with my cat, one frame without. If I process frame-without-cat, the cat is not found. If I process frame-with-cat, it is correctly found. However, if I do this:

    detect frame-without-cat
    detect frame-with-cat
    detect frame-without-cat

    I get the following results:
    cat not detected
    cat detected
    cat detected

    Which is inconsistent, as the third and first frame should have the same result. However, if for each detection I reload the model, this issue does not occur. It looks as though the net retains previous detections?

    Do you have any idea what this could be? If need be, I can provide example data and source.

    • Adrian Rosebrock October 19, 2017 at 4:47 pm #

      Hi Roald — this is indeed strange; however, I would double-check your images and 100% verify that you are passing in the correct images as you expect. The network should not be retaining any type of memory from previous detections. Secondly, check the confidence (i.e., probability) of your false-positives and see if you can increase the confidence to filter out these weak detections.

  14. Noble October 20, 2017 at 12:03 am #

    Hi Adrian,

    Downloaded the example for this post and ran it:

    $ $ python3 real_time_object_detection.py
    usage: real_time_object_detection.py [-h] -p PROTOTXT -m MODEL [-c CONFIDENCE]
    real_time_object_detection.py: error: the following arguments are required: -p/–prototxt, -m/–model

    How do I specify the path for the protext file and the model file to ap.add_argument. They are all in the same folder.

    • Adrian Rosebrock October 22, 2017 at 8:46 am #

      Please read up on command line arguments. This will enable you to learn more about command line basics. Furthermore, I also present examples on how to run the Python script via the command line inside this blog post.

    • aman January 8, 2018 at 3:22 am #

      Hi, i also got the same problem while running the python program.
      Have you solved this or still stucked in this problem

    • Rituparna Das January 24, 2018 at 7:16 pm #

      Did you solve
      the problem?

  15. Human October 20, 2017 at 5:15 pm #

    i want to track a ball is that code reliable to do the task

    • Adrian Rosebrock October 22, 2017 at 8:37 am #

      I cover ball/object tracking inside this post.

  16. Apramey October 21, 2017 at 7:45 am #

    Hello Adrian Sir,
    I’m great fan of all your articles and I ought learn more from you.
    I’m presently running on ubuntu mate on raspberry pi 3, I even optimized pi, the way you told in previous post. I removed the unnecessary applications of ubuntu mate which I’m not using. The code runs without any error. But the problem is GPU rendering, I get the frame, but I can’t visualize the video it’s recording. it continuously lags after code starts running.

    • Adrian Rosebrock October 22, 2017 at 8:35 am #

      The Raspberry Pi will only be able to process ~1 frame per second using this deep learning-based object detection method so the lag is entirely normal. Is there another type of lag you are referring to?

  17. Reed October 29, 2017 at 1:01 am #

    Hi Adrian
    I tried to run the codes above, but the result was
    $ python pi_object_detection.py –prototxt MobileNetSSD_deploy.prototxt.txt –model MobileNetSSD_deploy.caffemodel
    [INFO] loading model…
    [INFO] starting process…
    [INFO] starting video stream…

    (h, w) = image.shape[:2]
    AttributeError: ‘NoneType’ object has no attribute ‘shape’

    and I also read the post on 26th Dec 2016, still have no clue
    what should I do?

    • Adrian Rosebrock October 30, 2017 at 3:19 pm #

      Hi Reed — this error usually occurs when the webcam didn’t properly grab an image. You could put the following between lines 80 and 81:

      if frame is None:
      continue

      • JayEf November 25, 2017 at 3:23 pm #

        now it gives me:
        NameError: name ‘frame’ is not defined

    • Mitul Goswami February 21, 2018 at 4:29 am #

      hello reed

      you only have to uncomment the line no. 76 and comment line no. 75 in your ”pi_object_detection.py” file.

      nothing any more to add anything !

      enjoy !!

  18. chetan k j November 8, 2017 at 12:40 am #

    hi,
    will u please tel me how to detect human(person) within a fraction of second.

    i used this coding technique, found idx values and compared with threshold but its having one to two second delay.

    if idx == 15 and threshold>0.2:
    print ‘human detected’

    please suggest how to detect human within fraction of second.

  19. chetan k j November 8, 2017 at 7:48 am #

    Hi,

    please help to find human within fraction of second using raspberry pi-3.

    • Adrian Rosebrock November 9, 2017 at 6:42 am #

      I would suggest taking a look at this blog post to start. Even with an optimized OpenCV install you are not going to be able to detect objects in a fraction of a second on the Raspberry Pi, it’s simply too slow.

      • Chetan K j November 9, 2017 at 1:47 pm #

        Thanks for ur reply,
        Will u please give me suggestions, which board is used to do this operation means with a fraction of second detecting human…. Which board is better to use

        • Adrian Rosebrock November 13, 2017 at 2:21 pm #

          I would suggest trying the Jetson TX1 and TX2.

  20. sachin November 9, 2017 at 1:52 am #

    hi,
    nice work

    • Adrian Rosebrock November 9, 2017 at 6:14 am #

      Thanks Sachin!

  21. Dharshan November 9, 2017 at 7:47 am #

    Hi Adrian,

    Fantastic Work. Thanks a lot for sharing the code. I used an IP camera that streams h264/Jpeg camera over RTSP and was able to see fps of 0.6 as compared to 0.9 you see. Not Bad I guess. Exploring avenues to increase the fps count.

    • Adrian Rosebrock November 13, 2017 at 2:28 pm #

      Nice job, Dharshan!

    • Mr Sky January 26, 2018 at 8:33 am #

      Hi, can you share code to RTSP input?

  22. hamed November 9, 2017 at 2:10 pm #

    hi does it have any image processing in matlab soft

    • Adrian Rosebrock November 13, 2017 at 2:20 pm #

      Are you referring to the Raspberry Pi? The Raspberry Pi does not include any out-of-the-box image processing software for MATLAB.

  23. Andres November 10, 2017 at 11:37 pm #

    Hi Adrian! I wanted to know if it is needed to have opencv 3.3 to run this programme since everytime i try to intall it my rasp gets frozen. i have installed previous versions of opencv without problem.
    Nice work by the way 🙂

    • Adrian Rosebrock November 13, 2017 at 2:09 pm #

      Hi Andres — yes, OpenCV 3.3 is required for this blog post. If you’re having trouble getting OpenCV installed on your Pi, please see this tutorial.

  24. Paul November 19, 2017 at 7:16 am #

    Thx for this great blog, I got it working on my robot with raspPI3&arduino. I am streaming the output frames to a webclient, because the robot has no monitor attached. However, it is very slow given all the other processes that are running in my robot script. So it takes 10 seconds to identify the objects…. Is it possible to limit the number of known objects to look for, for example to look only for persons, sofa, potted plant and chair? Would this speed up the detection process? (since i don’t have airplanes or cars in my living room where the robot is supposed to manouvre..

    • Adrian Rosebrock November 20, 2017 at 4:05 pm #

      The speed of the object detection has nothing to do with the number of classes — it has to do with the depth/complexity of the network (within reason). The deeper/more complex the network, the slower it will run. I have a tutorial on optimizing the Raspberry Pi but in general, if you want to deploy a deep learning object detector, the Raspberry Pi is not the right hardware. I would recommend an NVIDIA Jetson.

  25. Abuthahir November 20, 2017 at 6:41 am #

    Hey. That was a great tutorial. And eventually worked for me. I want to make this thing to run at my raspberry pi boots. I tried it in linked way (https://pastebin.com/zjyEq99c) but I failed. Is there any way to do it? I’m looking for the help.

    • Adrian Rosebrock November 20, 2017 at 3:46 pm #

      I would recommend following this tutorial where I demonstrate how to run the script from boot. Be sure to take a look at the comments as well where we discuss a few other alternative methods.

  26. Sharath November 26, 2017 at 10:59 am #

    Hi Adrian,
    Really Nice. Keep up the good work.
    I am using Pi & open cv for the first time in my life and i simply followed your mentioned steps from the installation tutorial and it really worked.
    Then I integrated this py code on my stretch os with external Logitech webcam (C920-C) and it did work for the first time and i was really happy to see it working.

    Later, I restarted Pi to check the behaviour again and all of sudden it stopped working and started throwing error on command window.Now i am stuck..

    Error message :
    “VIDEOIO ERROR: V4L2: Pixel format of incoming image is unsupported by OpenCV
    Unable to stop the stream: Device or resource busy”.

    Then i started researching on google, found many blogs and its a common issue, but unfortunately i could not crack it.

    Do i need to change the setting in ” /home/pi/opencv-3.3.0/CMakeLists.txt” from OFF to ON in ” OCV_OPTION(WITH_LIBV4L “Use libv4l for Video 4 Linux support” ON ”
    and recompile the openCV.
    If yes, should i use the same commands from #Step5 to #Step7 of compile and Install openCv, the whole process will take 10h again to do it. 🙁

    Do you have any other solution for this to solve it easily.

    Thanks!

  27. Lenny December 5, 2017 at 9:26 am #

    Dear Adrian,
    Thank for this website it helps me to understand how to do new things. however I am running into this problem and I cannot figure it out:

    (cv)pi@raspberry:~/real-time-object-detection $ python real_time_object_detection.py –prototxt MobileNetSSD_deploy.prototxt.txt –model MobileNetSSD_deploy.caffemodel
    [INFO] loading model…
    [INFO] starting video stream…
    OK
    Traceback (most recent call last):
    File “real_time_object_detection.py”, line 48, in
    frame = imutils.resize(frame, width=400)

    (h, w) = image.shape[:2]
    AttributeError: ‘NoneType’ object has no attribute ‘shape’

    this file exists /home/pi/.virtualenvs/cv/local/lib/python2.7/site-packages/imutils/convenience.py
    and I am pretty sure I did not messed up the installation.

    Do you have any advice?

    Best regards,

    • Adrian Rosebrock December 8, 2017 at 5:18 pm #

      It sounds like your Raspberry Pi is unable to read the frame from your camera sensor as vs.read() is returning None. Are you using the Raspberry Pi camera module? Or a USB camera? I would also suggest reading up on NoneType errors as I discuss here.

    • Zack Inventor May 13, 2018 at 12:30 pm #

      How did you solve it? I have same issue.

  28. Benni Joel December 9, 2017 at 3:34 am #

    Thank You Adrian. But I wonder if I can detect the cube. Is there any packages like Caffe that can detect the cube of a particular dimension. Thank You in advance.

    • Adrian Rosebrock December 9, 2017 at 7:25 am #

      Hey Benni — are you trying to detect just a cube? Or the dimensions associated with a cube?

  29. Gaurav December 12, 2017 at 12:32 am #

    Hi Adrian, thanks for the great tutorial. I am following your tutorials from long time. I followed this tutorial and it worked as expected, Thanks a lot.

    Does the detection time depend on image resolution. Can we reduce the detection time if connect any low resolution USB camera to raspberry pi.

    • Adrian Rosebrock December 12, 2017 at 9:06 am #

      The input resolution to the neural network is fixed in this case (300x300px). Reducing the resolution from the USB camera to the Pi will speedup the frame acquisition rate but keep in mind that the bottleneck here isn’t frame I/O — it’s the speed of the SSD performing object detection.

  30. Gaurav Wable December 12, 2017 at 1:42 am #

    How can i track a detected object? E.g. I want to track the movement of first detected person among all people present in front of camera.

    • Adrian Rosebrock December 12, 2017 at 9:03 am #

      I would suggest taking a look at tracking algorithms, in particular “correlation tracking”. The dlib library has an implementation of correlation tracking.

  31. raghav December 21, 2017 at 9:23 am #

    hey, error = ‘module object has no attribute

  32. Dom December 21, 2017 at 7:36 pm #

    Hi Adrian. Maybe you know how to stream this real time object detection to the local area website (web browser)?

    • Adrian Rosebrock December 22, 2017 at 6:45 am #

      Hi Dom — I don’t have any tutorials on streaming the output frames to a web browser but I’ll add this to my queue for 2018. Thank you for the suggestion!

  33. Gary December 22, 2017 at 3:08 pm #

    How to stream object detection in web? Maybe i can make it with flask or django? Thanks.

    • Adrian Rosebrock December 26, 2017 at 4:38 pm #

      Hey Gary — I do not have any tutorials for RTSP or MJPEG streaming but I’ll add it to my queue!

      • Yin April 28, 2018 at 1:51 am #

        Hi, Adrian. How is the recent progress in MJPEG research? I am looking forward to viewing your frames in the form of MJPEG stream!

        • Adrian Rosebrock April 28, 2018 at 6:01 am #

          Hey Yin, thanks for following up. To be honest I haven’t had a chance to dive into it yet. I’ve been wrapping up a series of other posts on deep learning but I will try to do it sometime in 2018.

  34. seyed December 28, 2017 at 4:16 pm #

    Hi Adrian .thanks for the great tutorial. I download the code and run ‘pi_object_detection.py’ on raspberry pi and I have an error…”no module named ‘imutils’ ” then I open a terminal and type ‘pip install imutils’
    and recieve ‘Successfully installed imutils-0.4.5’ but there is also same error
    can you help me please

    • Adrian Rosebrock December 31, 2017 at 9:58 am #

      Are you using a Python virtual environments? If so, make sure you use the “workon” command to access your Python virtual environment, install imutils, and then execute the script:

  35. wally December 30, 2017 at 3:25 pm #

    Adrian,

    Have you considered the possibilities of the new Google AIY Vision kit?

    Seems to be some kind of co-processor/microcontroller add on board (vision bonnet) that is supposed to let a PiZero-W run deep learning object detection and classification models.

    I got one, but I need to return it as the connector for the flat cable broke when I tried to open it, so I didn’t get to first base 🙁

    Presumably it could work with the Pi3 as well.

    Google provides a “compiler” that takes a tensorflow “frozen model” and produces code to run on the vision bonnet.

    Obviously my hope here would be to get a higher frame rate than Pi3 can provide. The cost of the vision bonnet kit (~$45) and a PiZero-W (~$10) would not be too much above the price of a Pi3 if we can increase the frame rate significantly.

    • Adrian Rosebrock December 31, 2017 at 9:34 am #

      I am interested in playing around with the Google AIY Vision Kit but I haven’t purchased one yet. I’m still a bit concerned with using a Pi Zero. With only a single core responsible for all system + user operations your frame rate will suffer even if you are using threading to decrease I/O latency.

  36. Mark Z January 1, 2018 at 2:07 pm #

    I am using your Raspberry Pi image from your course, and the above code, and get the following error. Googling the error doesn’t give me a clear path to resolving this, can you please advise?

    File “pi_object_detection.py”, line 55, in
    net = cv2.dnn.readNetFromCaffe(args[“prototxt”], args[“model”])
    AttributeError: ‘module’ object has no attribute ‘dnn’

    • Adrian Rosebrock January 3, 2018 at 1:12 pm #

      Hey Mark — you need OpenCV 3.3 or greater to access the “dnn” module that contains the deep learning features. Any OpenCV version prior to OpenCV 3.3. does not include them.

      • J February 20, 2018 at 12:00 pm #

        I’m getting this error as well. I thought we downloaded OpenCV3.4 but when I do the version check it’s 3.1.0. I used your instructions on installing OpenCV on Raspbian and that was also the version that was displayed in your tutorial. Am I missing something?

        • Adrian Rosebrock February 22, 2018 at 9:14 am #

          Unfortunately, it sounds like you may have downloaded OpenCV 3.1 rather than OpenCV 3.3 or OpenCV 3.4. I would go back to the OpenCV install tutorial you used and ensure you download the source code to OpenCV 3.3 or OpenCV 3.4 and then recompile. I hope that helps!

  37. Paul January 2, 2018 at 6:25 am #

    Good morning and Happy New Year,
    i am getting this error message:

    [INFO] loading model…
    Traceback (most recent call last):
    File “pi_object_detection.py”, line 56, in
    net = cv2.dnn.readNetFromCaffe(args[“prototxt”], args[“model”])
    AttributeError: ‘module’ object has no attribute ‘dnn’

    Any idea ? Thanks in advance!

    • Adrian Rosebrock January 3, 2018 at 1:03 pm #

      Hi Paul — you need to have OpenCV 3.3 installed on your system. Anything older than OpenCV 3.3 does not have the “dnn” module.

  38. Theethat January 8, 2018 at 6:32 am #

    Can I use readNetFromTensorflow(.pb, .prototxt) instread readNetFromCaffe? It can detect object same Caffe or not?

    • Adrian Rosebrock January 8, 2018 at 2:36 pm #

      The readNetFromTensorflow function still doesn’t work in all cases. Whether or not it detects the same objects as the Caffe example in this blog post depends on what your TensorFlow network was trained on.

  39. Theethat January 9, 2018 at 1:16 pm #

    Thank so much Adrian. I have weight.pb file from trained my own dataset in Tensorflow. But I don’t know how to train dataset with Caffe SSD_Mobilenet Model. Do you have plan to post tutorial train own images image dataset with Caffe?

    • Adrian Rosebrock January 10, 2018 at 12:53 pm #

      You can try importing your TensorFlow weights via cv2.dnn.readNetFromTensorflow but it’s very likely that it will fail. The TensorFlow import methods are not (currently) as reliable and robust as the Caffe ones. As for training your own custom object detectors, take a look at Deep Learning for Computer Vision with Python where I cover how to train your own custom deep learning-based object detectors in detail.

      • Sergei January 23, 2018 at 9:32 am #

        https://github.com/opencv/opencv/issues/10043
        as I can see there is one tested ssd mobilenet TF model that can be imported without issues but be careful with the version of the model.

        • Adrian Rosebrock January 23, 2018 at 1:55 pm #

          Thank you for sharing this, Sergei!

  40. hardik shekhawat January 18, 2018 at 2:58 am #

    hello Adrian,
    i have been following your tutorial & i really liked it , i want to detect a street pole with sign on it.. can you tell me how to do edit the above pole accordingly, thank you

    • Adrian Rosebrock January 18, 2018 at 8:50 am #

      The model provided with this example was trained on the COCO dataset. You would need to train your own custom model to recognize street poles and signs. To start, use this tutorial to gather example images of what you’re trying to detect. I then demonstrate how to train your own custom object detectors inside my book, Deep Learning for Computer Vision with Python — recognizing street signs is even one of the included tutorials!

  41. huiyan January 18, 2018 at 5:19 am #

    hi,Adrian. i’m a chinese boy learning computer science. I read a lot of posts you wrote. I don’t know why my raspberrypi run the code in this post is very slow(about 3-4s one frame). does the video quality affect the speed? and can i transmit the video frame from pi to my pc to compute and post the result to pi?
    thanks a lot

    • Adrian Rosebrock January 18, 2018 at 8:43 am #

      As I discuss in this blog post the Raspberry Pi can be very slow for deep learning. If you are using a deeper, larger model the inference time will also be larger.

      As for transmitting a frame from your Pi to a PC you could use SFTP, Dropbox API, or even dedicated messaging passing libraries such ZeroMQ or RabbitMQ.

  42. Rituparna Das January 19, 2018 at 11:39 pm #

    I m getting an error like module object has no attribute dnn
    in the line net=cv2.dnn…

    • Rituparna Das January 20, 2018 at 12:25 am #

      And i have open cv version 3.3.1

      • Adrian Rosebrock January 20, 2018 at 8:07 am #

        That is indeed strange. Can you fire up a Python shell and verify your OpenCV version just to be safe?

        Additionally, are you using a Python virtual environment? Do you have any other OpenCV versions installed on your Raspberry Pi? And how did you install OpenCV on your Raspberry Pi — did you use one of my install tutorials on PyImageSearch?

  43. Rituparna Das January 22, 2018 at 9:13 pm #

    Well it seems i didn’t have open cv installed in pi.I did it by your method and it is successfully done.One more problem is i created the real_time_obeject_detection file and downloaded the caffemodel etc.when i m executing i m getting error no module named imutils.
    Also the mobilenet ssd prototxt and caffemodel are saved in my downloads so should i make another file in pi3 to access them?I mean how the program will access that without knowing where it is saved?

    • Adrian Rosebrock January 23, 2018 at 2:06 pm #

      Make sure you install the “imutils” library:

      $ pip install imutils

      You can supply the paths to your prototxt and caffemodel files via the command line arguments, exactly as I do in the blog post. When you execute the script simply pass the paths to the prototxt/caffemodel files where they live on your system.

  44. Kelvin January 23, 2018 at 9:48 am #

    Hi,Adrian. I use OpenCV and Tensorflow (Object Detetion API) on Raspberry Pi. But when I run code it have error gtk2.0 and gtk3.0 duplicate in same project. Do you know how to resolve it?

    • Adrian Rosebrock January 23, 2018 at 1:53 pm #

      Hey Kelvin, what is the exact error message? I would need to see the exact error to provide a suggestion.

  45. khai January 23, 2018 at 11:44 pm #

    Hello Adrian, have you looked into neural compute stick developed by movidius intel. It is USB3 interfaced and works on raspberry pi 3. It has support for caffe and tensorflow too and claim to greatly speed up object detection using deep learning. Perhaps, you can make a tutorial on that soon. Cheers

    • Adrian Rosebrock January 24, 2018 at 5:04 pm #

      I have! Look for a few posts on the Movidius Neural Compute Stick in February 🙂

      • Mike February 5, 2018 at 3:30 pm #

        Any info to share yet on the Movidius Neural Compute Stick yet?

        • Adrian Rosebrock February 6, 2018 at 10:08 am #

          I’ll be doing a blog post on the NCS either this coming Monday (Feb. 12th) or the following Monday (Feb. 19th).

  46. Rituparna Das January 24, 2018 at 7:20 pm #

    $pi_object_detection.py [-h] -p PROTOTXT -m MODEL [-c CONFIDENCE]
    pi_object_detection.py: error: argument -p/–prototxt is required

    I am getting this error!!.
    Please kindly tell me the steps after i download the code in my pc how to execute it on raspberrypi3.
    Also i observed in video
    you are in opencv virtual env and then inside pi-object-detection directory from where you executed files how can i do that?

    • Adrian Rosebrock January 25, 2018 at 3:51 pm #

      You need to supply the command line arguments as I do in the blog post:

      Notice how I have supplied values for --prototxt and --model via command line arguments. If you’re new to command line arguments make sure you read up on them.

      • Rima January 26, 2018 at 1:19 am #

        I gave the same commands and was getting the error.After i downloaded the files it is saved in F: drive if i provide the path like –prototxt f:\MobileNetSSD_depploy.prototxt.txt in raspberrypi terminal i am still getting error messages .My doubt is can raspberry access the files from system f or any drives or we need to make a directory in raspberrypi first and copy all this files and then provide the path like home/pi/dir??

        • Adrian Rosebrock January 26, 2018 at 10:07 am #

          I’m a bit confused here. You downloaded the .prototxt and .caffemodel files to your F: drive on your Windows machine and now you’re trying to execute the code on your Raspberry Pi? You need to download/transfer the .prototxt and .caffemodel files to your Raspberry Pi before you can run the script on your Raspberry Pi. Your Raspberry Pi cannot access the hard drive on your laptop/desktop.

  47. Sanjaya Kumar Das January 26, 2018 at 8:46 pm #

    To use picamera i changed 35 and 36th lines but getting these errors

    File “pi_object_detection.py”, line 75, in

    from picamera.array import PiRGBArray
    ImportError: No module named ‘picamera’

    • Adrian Rosebrock January 30, 2018 at 10:41 am #

      You need to install the “picamera” library:

      $ pip install "picamera[array]"

      Additionally, read this guide on accessing your Raspberry Pi camera module.

  48. imran February 1, 2018 at 5:31 am #

    Beside detection can this process count an object like if I want to use it a surveillance cam which not only saves video but can do a accounting of numbers of defined object going in and out.

    • Adrian Rosebrock February 3, 2018 at 11:07 am #

      Yes. Just create a counter for each object in your class labels and in the “for” loop on Line 58 you will increment the counter for the respective class label.

      The problem becomes tracking each object and not re-counting it at each iteration. For that you should use a dedicated object tracking algorithm such as correlation trackers or centroid trackers.

  49. Kawsur February 12, 2018 at 10:51 am #

    Hello Sir,
    this is a great project and now I’m working over it. Good demonstration. But I’m facing some problem . Like at 1st stacked at ….

    [INFO] loading model…
    [INFO] starting process…
    [INFO] starting video stream…

    (h, w) = image.shape[:2]
    AttributeError: ‘NoneType’ object has no attribute ‘shape’

    Then I added ..

    if frame is None:
    continue

    between line 80 and 81 .

    But now facing another problem. And it is
    [INFO] loading model…
    [INFO] starting process…
    [INFO] starting video stream…

    and it’s not showing any error even. Raspberry pi is in working state and cpu usage showing around 56 to 60%. One thing is raspberry pi is working normally. This will be very helpful if you tell me the problem and that will do lot to me. Thank you sir.

    • Adrian Rosebrock February 12, 2018 at 6:09 pm #

      Hey Kawsur — make sure you take a look at the comments section. I’ve addressed the NoneType error a handful of times in the comments. Additionally, make sure you see this blog post where I discuss NoneType errors and how to solve them in more detail.

      • Kawsur February 13, 2018 at 12:47 am #

        I fixed that none type error. But as I said it’s not showing me any error now. Just loading and loading. I checked my pi camera and it’s also working. Will you please tell me what I need to add to take frame from pi camera. I checked your pi_object_detection code and there is nothing about frame input. Maybe I need to add some line about taking frame from pi camera .

      • Kawsur February 13, 2018 at 12:56 am #

        Hello sir,
        Good Morning . It’s been running now. But one problem is now it’s showing a warning like ** (Frame:1162): WARNING **: Error retrieving accessibility bus address: org.freedesktop.DBus.Error.ServiceUnknown: The name org.a11y.Bus was not provided by any .service files
        then camera star to detect object. Is there any problem with it?

        • Adrian Rosebrock February 13, 2018 at 9:31 am #

          This is a warning, not an error, and can be safely ignored.

        • Dragos February 27, 2018 at 3:05 pm #

          How did you fixed it?

        • fensius April 10, 2018 at 9:41 pm #

          Hey Kawsur how did you fixed it ?

  50. Omar February 20, 2018 at 5:21 pm #

    Hi Adrian,

    Thank you very much for creating this amazing blog, I have learned a lot from you and implemented most of your projects.

    I have a question, Could you please tell me if there’s more predefined classes that I could add like making pi able to detect objects like geometrical shapes and is it possible to detect text from live stream?

    I am concerned because I am now working on a license plate recognition using opencv since I can’t get openalpr to work with live streaming.

    Thanks for your help.
    Greetings from Egypt.

    • Adrian Rosebrock February 22, 2018 at 9:09 am #

      I would suggest taking a look at the “Caffe Model Zoo” for more pre-trained models and seeing if any of them match your use case. More likely than not, you will likely need to train your own model.

  51. rajnish February 23, 2018 at 6:53 am #

    Hi Adrian,

    i am getting the error

    usage: real_time_object_detection.py [-h] -p PROTOTXT -m MODEL [-c CONFIDENCE]
    real_time_object_detection.py: error: the following arguments are required: -p/-
    -prototxt, -m/–model

    • Adrian Rosebrock February 26, 2018 at 2:09 pm #

      I’ve addressed this question a handful of times (please read the comments or ctrl + f the page and search for your error). See my replies to “Ahmad”, “aman”, and “Noble”. Thank you!

  52. Hemkesh February 27, 2018 at 11:41 am #

    Hi Adrian Thanks for the amazing blog.

    I downloaded the code from your website and tried to run it, but every time I run the code, my Pi Reboots. I am using Pi camera so I commented line 35 and uncommented line 36.

    • Adrian Rosebrock March 2, 2018 at 11:09 am #

      Your Pi Reboots? Wow, that is odd behavior! I haven’t encountered this before, but it sounds like your Pi might have a firmware issue or it’s immediately getting overheated and dying. I would suggest trying with a fresh install of Raspbian.

  53. Phil February 27, 2018 at 6:17 pm #

    Hey, I am getting a error and I have no clue how to fix it, could you take a look?
    When I run the py file I get the following error

    [INFO] loading model…
    Traceback (most recent call last):
    File “real_time_object_detection.py”, line 31, in
    net = cv2.dnn.readNetFromCaffe(args[“prototxt”], args[“model”])
    AttributeError: ‘module’ object has no attribute ‘dnn’

    Thank you

    • Adrian Rosebrock March 2, 2018 at 11:03 am #

      You need OpenCV 3.3 or greater for this post. OpenCV 3.3+ contains the “dnn” module. Previous versions of OpenCV do not.

  54. Roger February 28, 2018 at 10:40 pm #

    Amazing work!
    Just a few questions,
    1. How do I add more classes in the mobileNet SSD? (I wanted to check for counterfeit currency in our country and hence would need to add my images to the trained dataset)

    2. CPU load is at 98% while running the object detection program, will the pi cam help reducing that in any way? if not, how do I bring it down?

    Thanks Adrian!

    • Adrian Rosebrock March 2, 2018 at 10:50 am #

      1. You would need to either train the model from scratch or fine-tune it (assuming you have already gathered your dataset of counterfeit currency, of course). I cover how to train your own custom deep learning object detectors inside Deep Learning for Computer Vision with Python. I would suggest starting there.

      2. Unfortunately no, deep learning models are very computationally intensive. It does not have anything to do with the Pi camera. It’s simply the amount of computation required by the network for the detection.

  55. Shriram March 2, 2018 at 7:53 am #

    Hi Adrian, thanks for the excellent guide on object detection. I’ve got everything right, but when I run this code
    spberry Pi: Deep learning object detection with OpenCVShell

    $ python pi_object_detection.py \
    –prototxt MobileNetSSD_deploy.prototxt.txt \
    –model MobileNetSSD_deploy.caffemodel

    I’m getting the following error

    [INFO] loading model…
    [INFO] starting process…
    [INFO] starting video stream…
    Illegal Instructor

    Upto starting video stream is going right, but then I’m getting illegal instructor. I’ve searched the internet for a solution I couldn’t find the answer. Could you look into this error. I have a Raspberry Pi Model B+ V1.2 and Ras Pi camera module v2 noir and running raspbian stretch.

    • Adrian Rosebrock March 2, 2018 at 10:24 am #

      Hey Shriram — I would suggest inserting “print” statements throughout the code to debug what line is causing the issue. That will help determine what the problem is.

      • Shriram March 3, 2018 at 1:40 pm #

        The code is running till line no. 80

        80 frame = vs.read()

        When I include a print statement after the above line of code, it isn’t getting printed and throws Illegal Instructor error. I believe the following line of code throws the error

        81 frame = imutils.resize(frame, width=400)

        Could you help out with the problem?

        • Adrian Rosebrock March 7, 2018 at 9:44 am #

          Great job diagnosing the problem, Shriram. I wonder if it could be an issue related to the VideoStream class used in conjunction with your camera. Perhaps a threading issue of some sort. Just to clarify — are you using a Raspberry Pi camera module or a USB webcam?

          • Shriram March 8, 2018 at 12:56 pm #

            I’m using a Raspberry Pi camera module V2 (NoIR) and Raspberry Pi Model B+ V1.2 .

          • Adrian Rosebrock March 9, 2018 at 9:06 am #

            Try starting with this tutorial and see if you can access the NoIR camera via the raw “picamera” library first. Once you debug that move on to VideoStream.

          • Shriram March 10, 2018 at 2:31 pm #

            I tried the tutorial you mentioned and the NoIR is working perfectly for ‘picamera’ library and VideoStream was also good. I found that when I remove the code,

            frame = imutils.resize(frame, width=400)

            the programs runs, but in the video output, object detection isn’t being done. imutils.resize is the one causing the ‘Illegal instructor’. Can you help me out with this error? I tried it with some other values for width, but it didn’t execute.
            I even tried using cv2.resize, but then found that it can resize only images.

          • Adrian Rosebrock March 14, 2018 at 1:21 pm #

            I’m not sure what the exact reason is for this error. Does cv2.resize produce the same “illegal instruction” error as well?

  56. hakimhakim March 2, 2018 at 3:35 pm #

    Hi Adrian.I followed your tutorial, and it works so well. Thanks.
    But, my video stream is upside down. How can I rotate it 180 degrees/ flip it vertically?
    I tried to put these near the beginning of the loop, but it doesnt work…
    I’m using Pi camera version 2

    (code removed due to formatting reasons)

    • Adrian Rosebrock March 7, 2018 at 9:48 am #

      If you are using the Raspberry Pi camera module you should take a look at the documentation and use either .hflip or .vflip. If you want a pure OpenCV approach, just use cv2.flip.

  57. Jibola March 3, 2018 at 7:22 pm #

    Hi Adrian

    i keep getting a
    cv2.error:(-2) Failed:fs.is_open()
    can’t open “MobileSSD_deploy.caffemodel” in function ReadProtoFromBinaryFile

    • Jibola March 3, 2018 at 7:34 pm #

      never mind Adrian, fixed it, thanks so much for your site, it’s being really helpful for my final year project

      • Adrian Rosebrock March 7, 2018 at 9:41 am #

        Congrats on resolving the issue, Jibola. Would you mind share your solution? I imagine it was that your path to the .caffemodel was invalid but I know a lot of other readers new to the command line have the same error and your solution, expressed in your own words, would help them.

  58. chopin March 9, 2018 at 2:45 am #

    Hi,Adrian.I need your help. I use yolo v2 to train my datasets,and i got the tiny-yolo-voc_final.weights finally,but in this passage use .caffemodel,what shold i do? I have used darkflow transforming the.weights to .h5,if i want use your programme on my pi,I can apply it derectly or need to train my datasets useing caffe again? thank you very,much! I’m stupid and crazy.

    • Adrian Rosebrock March 9, 2018 at 8:48 am #

      Hey Chopin — it’s been awhile since I’ve used DarkNet to train a network so I’m not sure on the best path to transform the resulting models to a Caffe model. I simply haven’t done it so I’m unfortunately not the right person to ask. In the worst case scenario you would need to re-train using Caffe rather than DarkNet, but you should spend some time researching the DarkNet to Caffe conversion and if it’s possible. I’m sorry I couldn’t be of more help here!

      • chopin March 9, 2018 at 7:41 pm #

        okay. thank you again.you are a gentlman.

  59. haidar jaafar March 10, 2018 at 12:13 pm #

    How can i make raspberry pi say name to the object

    • Adrian Rosebrock March 14, 2018 at 1:22 pm #

      You would need a “text to speech” Python library. I’m not sure which is the best one for that.

      • haidar jaafar March 21, 2018 at 8:47 am #

        please you can change your code for say the name to object

        • Adrian Rosebrock March 21, 2018 at 9:52 am #

          I am happy to provide these tutorials for free to you but any additional functionality will need to be implemented in you. Do your research, experiment, and invest the time and I am confident that you’ll be able to do it 🙂

  60. sunil March 12, 2018 at 10:35 am #

    Is there any way to live stream the output frame window to a local webpage??

  61. Guru March 15, 2018 at 6:36 am #

    Hi Adrian,

    Can you please let me know, the process to deploy this python code in Rasberry Pi … So that I can perform the object detection on the go …

    Thanks
    Guru

    • Adrian Rosebrock March 19, 2018 at 5:55 pm #

      What specifically do you mean by “deploy” in this context? Can you elaborate?

  62. Nishant March 16, 2018 at 8:16 pm #

    Great work on this too, Adrian. I must say you have a good amount of patience to write this much with minute details.

    I just have one question. The scrip you have written here in the article, does it run on RasPi to achieve real time object detection or it runs on a computer using RasPi for just sending video stream?

    • Adrian Rosebrock March 19, 2018 at 5:27 pm #

      Thank you for the kind words, Nishant.

      The script I used in this post actually executes on the Raspberry Pi. It does not send the video anywhere. All video + object detection is performed on the Raspberry Pi.

  63. Sunil March 20, 2018 at 1:21 pm #

    Your post helped me a lot..Thank you
    Also, I have a question.
    Can i live stream the output window frame of this object detection code to a webpage hosted locally on my raspberry pi. If there any way to achieve this?

    • Adrian Rosebrock March 22, 2018 at 10:14 am #

      Hey Sunil — I don’t have any posts on real-time streaming to a webpage but I’ll try to cover this in the future. Thank you for the suggestion.

  64. Ray Jefferson March 21, 2018 at 1:46 pm #

    I keep on getting this error everytime i run the py file. please help me

    OpenCV Error: Unspecified error (FAILED: fs.is_open(). Can’t open “MobileNetSSD_deploy.prototxt.txt”) in ReadProtoFromTextFile, file /home/pi/opencv-3.3.0/modules/dnn/src/caffe/caffe_io.cpp, line 1113
    Traceback (most recent call last):
    File “pi_object_detection.py”, line 54, in
    net = cv2.dnn.readNetFromCaffe(args[“prototxt”], args[“model”])
    cv2.error: /home/pi/opencv-3.3.0/modules/dnn/src/caffe/caffe_io.cpp:1113: error: (-2) FAILED: fs.is_open(). Can’t open “MobileNetSSD_deploy.prototxt.txt” in function ReadProtoFromTextFile

    • Ray Jefferson Paje March 21, 2018 at 9:30 pm #

      Nvm i solved this issue

      • Adrian Rosebrock March 22, 2018 at 9:40 am #

        Yep, you would need to either train your own model or fine-tune an existing one.

        • Jack April 24, 2018 at 3:18 pm #

          I am getting this same exact error. I was wondering if you could go into greater detail on how to resolve it?

          • Adrian Rosebrock April 25, 2018 at 5:33 am #

            Hey Jack — this still sounds like a path issue. Be sure to read up on command line arguments before continuing. I think you may have supplied an invalid path to the file.

      • mengwang April 19, 2018 at 9:36 pm #

        would you like to share your manner that you solve this probelm? i just encounter the same question.i use the vgg model for fine-tune and use the opencv loading this model.

  65. Ray Jefferson Paje March 21, 2018 at 9:30 pm #

    Hi Adrian! I need to detect other things like (crumpled papers, candy wrapper, and other trash) can I just add it to my Classes or do I need to use other pre trained model? Thank you

  66. Huzzi March 29, 2018 at 5:04 pm #

    Do you know of any pre-trained models for traffic symbols i.e Start, Stop signs that I can use instead of the model you used?

  67. Arun George April 1, 2018 at 2:28 am #

    I am getting this error-

    File “pi_object_detection.py”, line 55, in
    net = cv2.dnn.readNetFromCaffe(args[“prototxt”], args[“model”])
    AttributeError: ‘module’ object has no attribute ‘dnn’

    • Adrian Rosebrock April 4, 2018 at 12:35 pm #

      Make sure you are running OpenCV 3.3+. The “dnn” module was not added until OpenCV 3.3+. You may need to compile and reinstall OpenCV.

  68. Shrirang Khare April 5, 2018 at 12:21 am #

    Hello Adrian,
    Thanks for this blog and valuable information!

    I tried to run forward pass to get detection using net.forward() . It works perfectly fine when there are object(s) trained on the image, although, when there isn’t an object, the net returns to me the detection of the last image that had the object. Has somebody stumbled on this issue?

    • Adrian Rosebrock April 6, 2018 at 9:00 am #

      It’s hard to say what could be causing that issue. My guess would be a latency between the frame read, the network predicting the object, and the frame being displayed to your thread. Keep in mind that the Pi is really slow for object detection so there may be a number of issues going on.

  69. Aabhas Mathur April 8, 2018 at 2:18 am #

    Hi Adrian sir,
    I want to ask you a question that what should i do to make my rasberry pi cam focus on one object at a time.for example if I want to make a Mechanical arm to pick utensils like plate,cup,bottle ,etc.but want it to pick one object at a time so I want it to focus on one object at a time the what modification should i make the above code or what library should i use to do that.

    • Adrian Rosebrock April 10, 2018 at 12:33 pm #

      There are a huge amount of modifications you would make, unfortunately far too many for me to list out in a single comment. You should do some research on robotics, in particular depth cameras, servos, and associated components. Detecting the actual object is just one small piece of this puzzle.

  70. Sachin April 8, 2018 at 7:08 am #

    Can we detect specific class, capture the detected frame and upload to dropbox API?

    • Adrian Rosebrock April 10, 2018 at 12:27 pm #

      Yes. You can check to see if the class is one you are interested in and if so, write the specific frame to disk via “cv2.imwrite”. You can then upload the file to Dropbox. See this blog post for an example of using the Dropbox API.

  71. Mauricio April 9, 2018 at 11:26 am #

    [INFO] loading model . . .
    [INFO] starting process . . .
    [INFO] starting video stream . . .
    Illegal instruction

    Help please 🙁

    i’ve a USB webcam

    • Adrian Rosebrock April 10, 2018 at 12:05 pm #

      Try inserting “print” statements to debug exactly where the “Illegal instruction” line is being thrown. Unfortunately it’s impossible to know based on your output.

  72. James April 19, 2018 at 4:19 pm #

    Hi Adrian,

    Awesome series of posts. This allowed me to get OpenCV all up and running on my Pi. Thanks!

    Only question I have is will it be quicker if it’s only detecting people? If so I’m assuming I have to redo the model so it’s only got person detection? The reason I’m asking is because I removed some of the class names in the CLASSES array and it didn’t work as expected (labelled me as a sheep!) which lead me to believe you have to use all the classes the model was trained for (and they have to be defined in a specific order?)

    Can I just tweak the model files so that it only have the person trained for?

    I’m hoping this all makes sense!

    James.

    • Adrian Rosebrock April 20, 2018 at 9:53 am #

      Hey James — you cannot add or remove labels from the CLASSES list. That will not change how the network behaves. This has become a question often asked in the object detection posts here on PyImageSearch blog so I’m writing a blog post on the very topic. It will release in early May so please keep an eye out for it!

  73. usup April 21, 2018 at 1:48 pm #

    i have error object detection.py: error: the following arguments are required: -p/–prototxt, -m/–model
    What is wrong?

    • Adrian Rosebrock April 25, 2018 at 6:15 am #

      If you’re new to command line arguments that’s okay, but you will need to read up on them before continuing. It’s a very quick fix but you need to read the post I just linked to first.

  74. Ashutosh April 22, 2018 at 7:02 am #

    Hi,
    I am getting this error

    Traceback (most recent call last):

    AttributeError: ‘module’ object has no attribute ‘dnn’

    Can you please help me with this?
    Thanks.

    • Adrian Rosebrock April 25, 2018 at 6:12 am #

      You need at least OpenCV 3.3 to utilize the “dnn” module. The “dnn” module is not included in previous versions of OpenCV. Check your OpenCV version and upgrade if necessary.

  75. yang April 24, 2018 at 6:20 am #

    Awesome posts.thank you for the nice work!
    I have a problem here, when I test the file of ‘A different approach to object detection on the Raspberry Pi ‘, which is ‘the pi_object_detection.py’. the result shows that:
    TypeError: can’t pickle cv2.dnn_Net objects
    Do you know how to fix that.

    • Adrian Rosebrock April 25, 2018 at 5:44 am #

      Can you clarify what your OpenCV version is? I haven’t ran into that error before.

  76. Jibola April 24, 2018 at 12:21 pm #

    Can this code run any Net trained on SSD. I trained a shufflenet SSD on the same VOC dataset but I get an error saying OpenCV doesnt have the shufflenet_chanel_param

    • Adrian Rosebrock April 25, 2018 at 5:40 am #

      No, this code will not work for “any” network as OpenCV must have the equivalent layer implemented in its library. OpenCV would need to have the equivalent “shuffnet layer” implemented in the “dnn” module. It does not, hence the error.

      • jibola April 25, 2018 at 7:40 am #

        thanks for the reply, i changed to squeezenet since the layers exist in the open cv library but everything takes too long, the Frame box opens after 30seconds and the video is stuck on a frame for over a minute, was the MobilSSD used here optimised in some way or is it that the pi can’t handle Squeezenet

        • Adrian Rosebrock April 25, 2018 at 10:18 am #

          The only optimizations done were to use my optimized OpenCV install. No further optimizations were done.

  77. Yin April 27, 2018 at 11:42 pm #

    Hi, Adrian. I use command below:
    #./mjpg_streamer -i “./input_raspicam.so” -o “./output_http.so -w ./www”
    to transfer video collected by Raspberry Pi to web page,
    so I can view picamera video on web page under my WLAN .
    But now, I wanna view your “frames” result on web page, how can I export your results to web pages? I think that will be cool !

    • Adrian Rosebrock April 28, 2018 at 6:03 am #

      Hi Yin, I replied to your other comment on this post. Please see it.

  78. ghiz April 28, 2018 at 10:52 pm #

    hi adrian , please help
    i tried ur code but i have problem

    et = cv2.dnn.readNetFromCaffe(args[“prototxt”], args[“model”])
    AttributeError: ‘module’ object has no attribute ‘dnn’

    i dont know what do to no have this problem , please help me

    • Adrian Rosebrock May 3, 2018 at 10:26 am #

      You need to ensure you have at least OpenCV 3.3 installed. You can follow my OpenCV install tutorials to install a new, updated version of OpenCV on your system.

  79. James April 29, 2018 at 7:43 pm #

    hi Adrain, i’m obtained a tiny yolo caffemodel from here
    https://github.com/cinastanbean/caffe-yolo-1

    but opencv gives the error Assertionfailed blobs.size() >= 3 when i run it with your code,
    i’m assuming this error comes from the cv2.dnn.BlobfromImage . do you have any idea on the parameter i could change from your code to support tiny yolo. I already changed the size to (224,224)

    • James April 29, 2018 at 8:53 pm #

      Changing my caffemodel and prototxt file solves this but a new problem has occured. it detects evrything has background ,it’s not runing through the array , if i take away background from the list of classes ,it just detects everything has an aeroplane which is the next item on the list . Yolo isnt the only net that ive had this issue with using your code

      • Adrian Rosebrock April 30, 2018 at 12:42 pm #

        Hi James — the list of classes, is just a list of strings. Do not modify it because what’s really important is the index in the list of each of those strings. In other words, you cannot simply remove elements from the classes list and expect the object detector to work properly. If you change the ordering of the list, then the indexes won’t match up. There’s a blog post coming out on 5/14/2018 explaining this.

    • Adrian Rosebrock May 3, 2018 at 10:16 am #

      Hey James — I’m not sure what the error is here. Hopefully in the future I can write a tutorial on YOLO + OpenCV, I would really like to.

  80. Recepsergen May 1, 2018 at 8:06 pm #

    Hello adrian
    I ran the code. And the project worked. When a human object is detected, I can capture and record a photo. But what I want to do is how to record a 3-second video when a human object is detected. I could not … could you help …

    Thank you …

    • Adrian Rosebrock May 3, 2018 at 10:06 am #

      You’ll want to refer to this blog post on saving video clips with OpenCV.

  81. KK May 9, 2018 at 1:41 pm #

    Hi,
    I am getting this error

    Traceback (most recent call last):

    (h,w)=image.shape[:2]
    AttributeError: ‘NoneType’ object has no attribute ‘shape’

    Can you please help me with this?
    Thanks.

    • Adrian Rosebrock May 14, 2018 at 12:24 pm #

      See this blog post on “NoneType” errors with OpenCV and how to resolve them.

  82. Ashraf Idris May 13, 2018 at 2:31 am #

    hi Adrian, can i use this to detect potholes on the road? if cant, could u help me to detect it. thank you

    • Adrian Rosebrock May 14, 2018 at 12:01 pm #

      If you would like to use this method to detect potholes in a road you would first need to train a deep learning detector on pothole images. Do you have such example images?

  83. Zack Inventor May 13, 2018 at 12:00 pm #

    Hi, I have completed the opencv install and am now wondering where to I execute the command. I tried to do it in the cv environment but got no such file or directory.

    I have the script in the file in the downloads. Do I have to put it into the opencv thing somehow or can I locate it from the cv environment and then execute the command?

    thanks!

  84. Zack Inventor May 13, 2018 at 5:17 pm #

    I fixed both issues but I have one more. It has to do with the imutils module. Even though I imported it and checked the version in the (cv) environment. It still says no module. I have tried almost everything on the internet and nothing works. Do you know the problem?

    thanks!

    please respond ASAP.

    • Adrian Rosebrock May 14, 2018 at 11:56 am #

      You’ll want to make sure you install the “imutils” into your Python virtual environment:

  85. Zack Inventor May 15, 2018 at 10:02 pm #

    Hi, I have one more question. All I want to detect is bird because if it detects other things beside bird my if statement will not work and it could possibly increase my FPS. I am now only getting 0.31 FPS.

    I tried deleting all of the classes beside bird but I got errors. Do you know a quick and easy way to do this?

    thanks

    Please respond ASAP

    • Adrian Rosebrock May 17, 2018 at 7:03 am #

      This blog post will help you learn the fundamentals of deep learning object detection, including adding or removing classes.

Trackbacks/Pingbacks

  1. Real-time object detection on the Raspberry Pi with the Movidius NCS - PyImageSearch - February 19, 2018

    […] involves object detection with the Raspberry Pi where I’m using my own custom Caffe model. The benchmark scripts you supplied for applying object detection on the Pi’s CPU were too slow and I need faster […]

Leave a Reply