Taking screenshots with OpenCV and Python

Happy New Year!

It’s now officially 2018…which also means that PyImageSearch is (almost) four years old!

I published the very first blog post on Monday, January 12th 2014. Since then over 230 posts have been published, along with two books and a full-fledged course.

At the beginning of every New Year I take some quiet time to reflect.

I grab my notebook + a couple pens (leaving my laptop and phone at home; no distractions) and head to the local cafe in my neighborhood. I then sit there and reflect on the past year and ask myself the following questions:

  • What went well and gave me life?
  • What went poorly and sucked life from me?
  • How can I double-down on the positive, life-giving aspects?
  • How can I get the negative aspects off my plate (or at least minimize their impact on my life)?

These four questions (and my thoughts on them) ultimately shape the upcoming year.

But most of all, the past four years running PyImageSearch has always been at the top of my list for “life-giving”.

Thank you for making PyImageSearch possible. Running this blog is truly the best part of my day.

Without you PyImageSearch would not be possible.

And in honor of that, today I am going to answer a question I received from Shelby, a PyImageSearch reader:

Hi Adrian, I’ve been reading PyImageSearch for the past couple of years. One topic I’m curious about is taking screenshots with OpenCV. Is this possible?

I’d like to build an app that can automatically control the user’s screen and it requires screenshots. But I’m not sure how to go about it.

Shelby’s question is a great one.

Building a computer vision system to automatically control or analyze what is on a user’s screen is a great project.

Once we have the screenshot we can identify elements on a screen using template matching, keypoint matching, or local invariant descriptors.

The problem is actually obtaining the screenshot in the first place.

We call this data acquisition — and in some cases, acquiring the data is actually harder than
applying the computer vision or machine learning itself.

To learn how to take screenshots with OpenCV and Python, just keep reading.

Looking for the source code to this post?
Jump right to the downloads section.

Taking screenshots with OpenCV and Python

Today’s blog post is broken down into two sections.

In the first section, we’ll learn how to install the PyAutoGUI library. This library is responsible for actually capturing our screenshots to disk or directly to memory.

From there we’ll learn how to use PyAutoGUI and OpenCV together to obtain our screenshots.

Installing PyAutoGUI for screenshots

You can find instructions for installing PyAutoGUI in their install documentation; however, as a matter of completeness, I have included the instructions below.

highly recommend that you install the PyAutoGUI into your Python virtual environment for computer vision (as we have done for all other install tutorials here on PyImageSearch).

Discussing virtual environments in detail is beyond the scope of this blog post; however, I encourage you to set up an environment for computer vision (including OpenCV and other tools) by following the installation instructions for your system available here.

macOS

Installing PyAutoGUI for macOS is very straightforward. As stated above, you’ll want to be sure you’re “inside” your virtual environment prior to executing the following pip commands:

Ubuntu or Raspbian

To install PyAutoGUI for Ubuntu (or Raspbian), you’ll need to make use of both Aptitude and pip. Again, before the pip commands, be sure that you’re working on your Python virtual environment:

Screenshots and screen captures with OpenCV and Python

Now that PyAutoGUI is installed, let’s take our first screenshot with OpenCV and Python.

Open up a new file, name it take_screenshot.py , and insert the following code:

On Lines 2-5 we’re importing our required packages, notably pyautogui .

From there we’ll take a screenshot via two different methods.

In the first method, we take the screenshot and store it in memory for immediate use:

Line 10 shows that we’re grabbing a screenshot with pyautogui.screenshot and storing it as image  (again, this image is stored in memory it is not saved to disk).

Easy, huh?

Not so fast!

PyAutoGUI actually stores the image as a PIL/Pillow image, so we need to perform an additional step before the image can be used with OpenCV.

On Line 11 we convert the image to a NumPy array and swap the color channels from RGB ordering (what PIL/Pillow uses) to BGR (what OpenCV expects). That’s all that’s required for making our screenshot image OpenCV-compatible.

From here the sky is the limit with what you can do. You could detect buttons displayed on the screen or even determine the coordinates of where the mouse is on the screen.

We won’t do either of those tasks today. Instead, let’s just write the image to disk with cv2.imwrite  to ensure the process worked correctly (Line 12).

The second method (where we write the screenshot to disk) is even easier:

As shown, this one-liner writes the image straight to disk. Enough said.

We could stop there, but for a sanity check, let’s make sure that OpenCV can also open + display the screenshot:

Here, we read the image from disk. Then we resize and display it on the screen until a key is pressed.

That’s it!

As you can tell, PyAutoGui is dead simple thanks to the hard work of Al Sweigart.

Let’s see if it worked.

To test this script, open up a terminal and execute the following command:

And here’s our desktop screenshot shown within our desktop…proving that the screenshot was taken and displayed:

Figure 1: Taking a screenshot with Python, OpenCV, and PyAutoGUI on macOS.

Notice how in the terminal the Python script is running (implying that the screenshot is currently being taken).

After the script exits, I have two new files in my working directory: in_memory_to_disk.png  and straight_to_disk.png .

Let’s list contents of the directory:

As you can see, I’ve got my take_screenshot.py  script and both screenshot PNG images

Now that we have our screenshot in OpenCV format, we can apply any “standard” computer vision or image processing operation that we wish, including edge detection, template matching, keypoint matching, object detection, etc.

In a future blog post, I’ll be demonstrating how to detect elements on a screen followed by controlling the entire GUI from the PyAutoGUI library based on what our computer vision algorithms detect.

Stay tuned for this post in early 2018!

Summary

In today’s blog post we learned how to take screenshots using OpenCV, Python, and the PyAutoGUI library.

Using PyAutoGUI we can easily capture screenshots directly to disk or to memory, which we can then convert to OpenCV/NumPy format.

Screenshots are an important first step when creating computer vision software that can automatically control GUI operations on the screen, including automatically moving the mouse, clicking the mouse, and registering keyboard events.

In future blog posts, we’ll learn how we can automatically control our entire computer via computer vision and the PyAutoGUI.

To be notified when future blog posts are published here on PyImageSearch, just enter your email address in the form below!

Downloads:

If you would like to download the code and images used in this post, please enter your email address in the form below. Not only will you get a .zip of the code, I’ll also send you a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL! Sound good? If so, enter your email address and I’ll send you the code immediately!

, ,

20 Responses to Taking screenshots with OpenCV and Python

  1. satinder singh January 1, 2018 at 11:30 am #

    hello sir I really can’t explain how good your blogs are and i need your help.I have working on a project to make snapchat like filters.I have used DLIB to get the facial features.I am unable to draw the filter at specific coordinates. I have seen your post on drawing overlays but i am unable to do the same with a png image.

  2. Victor Ramamoorthy January 1, 2018 at 12:27 pm #

    scrot can also dump a screen shot as a png file which you can read onto opencv. Agreed that pyautogui is elegant. Thanks.

  3. Suhas January 1, 2018 at 4:56 pm #

    Hey Adrian,

    Happy New Year!

    All your posts that I’ve read so far are just great! The passion in you to make genuine research about everything small is heartwarming. I’ve been quite observant about the time you make to reply to every question and comment posted here, something I rarely see on any other site. With equal eagerness and excitement, I always wait, to hear from you back again. I think it’s a good idea to email the replies whenever there’s a new reply. Now I need to check this thread regularly to see what you think of this idea. 😛

    • Adrian Rosebrock January 3, 2018 at 1:09 pm #

      Thank you for the kind words, Suhas 🙂 Comments like these really make my day. A Happy New Year to you too.

  4. Abkul January 5, 2018 at 7:08 am #

    Hi Adrian,

    I have no question today but just to wish you and the crew at pyimagesearch every best in 2018 and continue with your great evangelism of computer vision/OpenCV/ML/AI agenda.

    I look forward to every Monday to read your blog posts.

    • Adrian Rosebrock January 5, 2018 at 1:25 pm #

      Thank you Abkul, I really appreciate that 🙂 Have a wonderful 2018 as well.

  5. Ricardo January 31, 2018 at 10:21 pm #

    Thank you, sir. This post was really helpful.

    • Adrian Rosebrock February 3, 2018 at 11:16 am #

      Thanks Ricardo — I’m glad you found it helpful!

  6. Tarul Vyas April 16, 2018 at 12:50 pm #

    well done. i have a question. what if i want to take the screenshot of the active window.

    • Adrian Rosebrock April 16, 2018 at 2:00 pm #

      I don’t think OpenCV and Python directly allow this. You could first capture a screenshot and then use OpenCV’s GUI functions to determine the location of the window and then crop it from the output image.

  7. Brendan April 17, 2018 at 2:30 pm #

    Would this work to capture a picture that my raspberry pi is projecting onto a web interface??

    • Adrian Rosebrock April 18, 2018 at 3:04 pm #

      Hey Brendan, I’m not sure I follow. Could you share a screenshot or illustration of what you are trying to accomplish?

  8. Gauthier April 28, 2018 at 9:49 am #

    Fantastic man

  9. Gauthier April 28, 2018 at 9:52 am #

    But where is the next post ?

    “In future blog posts, we’ll learn how we can automatically control our entire computer via computer vision and the PyAutoGUI.”

    • Adrian Rosebrock April 28, 2018 at 12:02 pm #

      Thanks Gauthier, I’m glad you liked the post. To be honest I haven’t had a chance to write the next post yet. I’ve been busy writing some new deep learning tutorials and simply haven’t had a chance to get to it yet.

  10. Tommy June 8, 2018 at 2:38 pm #

    Looks like you can directly use xlib (if in linux):
    So you can directly construct opencv image. Below example does it for PIL.

    from Xlib import display, X
    import Image #PIL

    W,H = 200,200
    dsp = display.Display()
    root = dsp.screen().root
    raw = root.get_image(0, 0, W,H, X.ZPixmap, 0xffffffff)
    image = Image.fromstring(“RGB”, (W, H), raw.data, “raw”, “BGRX”)

    • Adrian Rosebrock June 13, 2018 at 6:12 am #

      Thanks for sharing, Tommy!

    • Const October 5, 2018 at 10:13 am #

      Thanks for the great tip! Just a quick note:

      image = Image.fromstring(“RGB”, (W, H), raw.data, “raw”, “BGRX”)

      For Debian 9.5, had to be:

      image = Image.fromstring(“RGB”, (W, H), raw.data, “raw”, “RGBX”)

      since original code had inverse Red and Blue channels. Thanks for sharing!

  11. Susan July 16, 2018 at 2:50 am #

    Hello Adrian.
    I am just a beginner in programming stuff.
    I have a video of a traffic junction and i need to capture screenshot from this video at the press of a key (basically when any vehicle goes towards parking). These images are then stored in a specified location for further processing.

    Do you have any related tutorials for this?

    Thanks

    • Adrian Rosebrock July 17, 2018 at 7:22 am #

      Hey Susan — all you would need is the “cv2.imwrite” function to write frames to disk. If you are new to Python and OpenCV I would recommend that you work through Practical Python and OpenCV to learn the fundamentals. The contents of the book would help you solve the project very quickly, I am absolutely confident of that.

Leave a Reply