Taking screenshots with OpenCV and Python

Happy New Year!

It’s now officially 2018…which also means that PyImageSearch is (almost) four years old!

I published the very first blog post on Monday, January 12th 2014. Since then over 230 posts have been published, along with two books and a full-fledged course.

At the beginning of every New Year I take some quiet time to reflect.

I grab my notebook + a couple pens (leaving my laptop and phone at home; no distractions) and head to the local cafe in my neighborhood. I then sit there and reflect on the past year and ask myself the following questions:

  • What went well and gave me life?
  • What went poorly and sucked life from me?
  • How can I double-down on the positive, life-giving aspects?
  • How can I get the negative aspects off my plate (or at least minimize their impact on my life)?

These four questions (and my thoughts on them) ultimately shape the upcoming year.

But most of all, the past four years running PyImageSearch has always been at the top of my list for “life-giving”.

Thank you for making PyImageSearch possible. Running this blog is truly the best part of my day.

Without you PyImageSearch would not be possible.

And in honor of that, today I am going to answer a question I received from Shelby, a PyImageSearch reader:

Hi Adrian, I’ve been reading PyImageSearch for the past couple of years. One topic I’m curious about is taking screenshots with OpenCV. Is this possible?

I’d like to build an app that can automatically control the user’s screen and it requires screenshots. But I’m not sure how to go about it.

Shelby’s question is a great one.

Building a computer vision system to automatically control or analyze what is on a user’s screen is a great project.

Once we have the screenshot we can identify elements on a screen using template matching, keypoint matching, or local invariant descriptors.

The problem is actually obtaining the screenshot in the first place.

We call this data acquisition — and in some cases, acquiring the data is actually harder than
applying the computer vision or machine learning itself.

To learn how to take screenshots with OpenCV and Python, just keep reading.

Looking for the source code to this post?
Jump right to the downloads section.

Taking screenshots with OpenCV and Python

Today’s blog post is broken down into two sections.

In the first section, we’ll learn how to install the PyAutoGUI library. This library is responsible for actually capturing our screenshots to disk or directly to memory.

From there we’ll learn how to use PyAutoGUI and OpenCV together to obtain our screenshots.

Installing PyAutoGUI for screenshots

You can find instructions for installing PyAutoGUI in their install documentation; however, as a matter of completeness, I have included the instructions below.

highly recommend that you install the PyAutoGUI into your Python virtual environment for computer vision (as we have done for all other install tutorials here on PyImageSearch).

Discussing virtual environments in detail is beyond the scope of this blog post; however, I encourage you to set up an environment for computer vision (including OpenCV and other tools) by following the installation instructions for your system available here.

macOS

Installing PyAutoGUI for macOS is very straightforward. As stated above, you’ll want to be sure you’re “inside” your virtual environment prior to executing the following pip commands:

Ubuntu or Raspbian

To install PyAutoGUI for Ubuntu (or Raspbian), you’ll need to make use of both Aptitude and pip. Again, before the pip commands, be sure that you’re working on your Python virtual environment:

Screenshots and screen captures with OpenCV and Python

Now that PyAutoGUI is installed, let’s take our first screenshot with OpenCV and Python.

Open up a new file, name it take_screenshot.py , and insert the following code:

On Lines 2-5 we’re importing our required packages, notably pyautogui .

From there we’ll take a screenshot via two different methods.

In the first method, we take the screenshot and store it in memory for immediate use:

Line 10 shows that we’re grabbing a screenshot with pyautogui.screenshot and storing it as image  (again, this image is stored in memory it is not saved to disk).

Easy, huh?

Not so fast!

PyAutoGUI actually stores the image as a PIL/Pillow image, so we need to perform an additional step before the image can be used with OpenCV.

On Line 11 we convert the image to a NumPy array and swap the color channels from RGB ordering (what PIL/Pillow uses) to BGR (what OpenCV expects). That’s all that’s required for making our screenshot image OpenCV-compatible.

From here the sky is the limit with what you can do. You could detect buttons displayed on the screen or even determine the coordinates of where the mouse is on the screen.

We won’t do either of those tasks today. Instead, let’s just write the image to disk with cv2.imwrite  to ensure the process worked correctly (Line 12).

The second method (where we write the screenshot to disk) is even easier:

As shown, this one-liner writes the image straight to disk. Enough said.

We could stop there, but for a sanity check, let’s make sure that OpenCV can also open + display the screenshot:

Here, we read the image from disk. Then we resize and display it on the screen until a key is pressed.

That’s it!

As you can tell, PyAutoGui is dead simple thanks to the hard work of Al Sweigart.

Let’s see if it worked.

To test this script, open up a terminal and execute the following command:

And here’s our desktop screenshot shown within our desktop…proving that the screenshot was taken and displayed:

Figure 1: Taking a screenshot with Python, OpenCV, and PyAutoGUI on macOS.

Notice how in the terminal the Python script is running (implying that the screenshot is currently being taken).

After the script exits, I have two new files in my working directory: in_memory_to_disk.png  and straight_to_disk.png .

Let’s list contents of the directory:

As you can see, I’ve got my take_screenshot.py  script and both screenshot PNG images

Now that we have our screenshot in OpenCV format, we can apply any “standard” computer vision or image processing operation that we wish, including edge detection, template matching, keypoint matching, object detection, etc.

In a future blog post, I’ll be demonstrating how to detect elements on a screen followed by controlling the entire GUI from the PyAutoGUI library based on what our computer vision algorithms detect.

Stay tuned for this post in early 2018!

Summary

In today’s blog post we learned how to take screenshots using OpenCV, Python, and the PyAutoGUI library.

Using PyAutoGUI we can easily capture screenshots directly to disk or to memory, which we can then convert to OpenCV/NumPy format.

Screenshots are an important first step when creating computer vision software that can automatically control GUI operations on the screen, including automatically moving the mouse, clicking the mouse, and registering keyboard events.

In future blog posts, we’ll learn how we can automatically control our entire computer via computer vision and the PyAutoGUI.

To be notified when future blog posts are published here on PyImageSearch, just enter your email address in the form below!

Downloads:

If you would like to download the code and images used in this post, please enter your email address in the form below. Not only will you get a .zip of the code, I’ll also send you a FREE 11-page Resource Guide on Computer Vision and Image Search Engines, including exclusive techniques that I don’t post on this blog! Sound good? If so, enter your email address and I’ll send you the code immediately!

, ,

12 Responses to Taking screenshots with OpenCV and Python

  1. satinder singh January 1, 2018 at 11:30 am #

    hello sir I really can’t explain how good your blogs are and i need your help.I have working on a project to make snapchat like filters.I have used DLIB to get the facial features.I am unable to draw the filter at specific coordinates. I have seen your post on drawing overlays but i am unable to do the same with a png image.

  2. Victor Ramamoorthy January 1, 2018 at 12:27 pm #

    scrot can also dump a screen shot as a png file which you can read onto opencv. Agreed that pyautogui is elegant. Thanks.

  3. Suhas January 1, 2018 at 4:56 pm #

    Hey Adrian,

    Happy New Year!

    All your posts that I’ve read so far are just great! The passion in you to make genuine research about everything small is heartwarming. I’ve been quite observant about the time you make to reply to every question and comment posted here, something I rarely see on any other site. With equal eagerness and excitement, I always wait, to hear from you back again. I think it’s a good idea to email the replies whenever there’s a new reply. Now I need to check this thread regularly to see what you think of this idea. 😛

    • Adrian Rosebrock January 3, 2018 at 1:09 pm #

      Thank you for the kind words, Suhas 🙂 Comments like these really make my day. A Happy New Year to you too.

  4. Abkul January 5, 2018 at 7:08 am #

    Hi Adrian,

    I have no question today but just to wish you and the crew at pyimagesearch every best in 2018 and continue with your great evangelism of computer vision/OpenCV/ML/AI agenda.

    I look forward to every Monday to read your blog posts.

    • Adrian Rosebrock January 5, 2018 at 1:25 pm #

      Thank you Abkul, I really appreciate that 🙂 Have a wonderful 2018 as well.

  5. Ricardo January 31, 2018 at 10:21 pm #

    Thank you, sir. This post was really helpful.

    • Adrian Rosebrock February 3, 2018 at 11:16 am #

      Thanks Ricardo — I’m glad you found it helpful!

  6. Tarul Vyas April 16, 2018 at 12:50 pm #

    well done. i have a question. what if i want to take the screenshot of the active window.

    • Adrian Rosebrock April 16, 2018 at 2:00 pm #

      I don’t think OpenCV and Python directly allow this. You could first capture a screenshot and then use OpenCV’s GUI functions to determine the location of the window and then crop it from the output image.

  7. Brendan April 17, 2018 at 2:30 pm #

    Would this work to capture a picture that my raspberry pi is projecting onto a web interface??

    • Adrian Rosebrock April 18, 2018 at 3:04 pm #

      Hey Brendan, I’m not sure I follow. Could you share a screenshot or illustration of what you are trying to accomplish?

Leave a Reply