Capturing mouse click events with Python and OpenCV

Capturing mouse click events with Python and OpenCV

Being raised on the east coast of the United States I guess I should be used to the snow by now — but I’m not. As a kid, I loved the snow. I loved sledding, snowboarding, and snowball fights.

But now, as an adult, snow just means an inconvenience. It means travel is going to be terrible. It means not being able to drive to the gym. And it means being stuck in the apartment.

That said, I’m taking this snow day on the east coast and writing a blog post on capturing mouse click events with Python and OpenCV.

In this example we’ll click and drag a rectangular Region of Interest (ROI) and crop it from our image. This technique is especially helpful if you are labeling data as input to an image classification algorithm.

So if you’re trying to understand how mouse click events work with Python and OpenCV, then look no further! This blog post will show you exactly that!

Looking for the source code to this post?
Jump right to the downloads section.

OpenCV and Python versions:
In order to run this example, you’ll need Python 2.7 and OpenCV 2.4.X.

Capturing mouse click events with Python and OpenCV

Let’s go ahead and get this example started. Open up a new file, name it click_and_crop.py , and we’ll get to work:

We’ll start by importing our two necessary packages: argparse  for parsing command line arguments and cv2  for our OpenCV bindings.

We also define two global variables on Lines 7 and 8: refPt , which is a list of two (x, y)-coordinates specifying  the rectangular region we are going to crop from our image, and cropping , a boolean indicating whether we are in cropping mode or not.

To process the mouse click events we define the click_and_crop  callback function on Line 10. Anytime a mouse event happens, OpenCV will relay the pertinent details to our click_and_crop  function.

In order for our function to handle the relay, we need to accept 5 arguments:

  • event: The event that took place (left mouse button pressed, left mouse button released, mouse movement, etc).
  • x: The x-coordinate of the event.
  • y: The y-coordinate of the event.
  • flags: Any relevant flags passed by OpenCV.
  • params: Any extra parameters supplied by OpenCV.

From there we grab reference to our refPt  list and cropping  flag on Line 12.

We then make a check on Line 17 to see if our left mouse button was pressed. If it was, we record the (x, y)-coordinates of the event and indicate that we are now in “cropping mode”.

At this point we would drag out the rectangular region of the image that we want to crop. After we are done dragging out the region, we release the left mouse button — Line 22 handles when the left mouse button is released and updates the list of points containing our ROI. We also draw the rectangle representing the ROI on Lines 29 and 30.

Now, let’s see how we can use this function to crop an image:

We start by parsing our command line arguments on Lines 33-35. We need only a single switch here, --image , which is the path to the image we want to crop. This image is then loaded and cloned on Lines 38 and 39. We make a deep copy of the image so we can draw the rectangular bounding box on Line 29 without destroying the original image.

Lines 40 and 41 handle registering our click_and_crop  callback function. We first call cv2.namedWindow  to create a window named “image”. And we then set the mouse callback by calling cv2.setMouseCallback , supplying our named “image” window and our click_and_crop  callback function.

From here, the rest of this example just ties the pieces together.

We start a loop on Line 44 which (1) displays our image on screen and (2) waits for a keypress.

This is the point where we select the rectangular region we want to crop from the image.

If we press the r  key after selecting the region to crop, we reset our cropping

And if we press the c  key, then we break from the loop and perform the actual cropping on Lines 59-62.

Finally, we cleanup and close all open windows on Line 65.

OpenCV mouse events in action

Now that we have our example coded up, let’s try it out. Open up a terminal and execute the following command:

You’ll first be presented with the image on your screen:

Figure 1: Our original image displayed on screen.

Figure 1: Our original image displayed on screen.

Select the region you want to crop by clicking, dragging, and releasing:

Figure 2: Clicking and dragging the region of the image we want to crop.

Figure 2: Clicking and dragging the region of the image we want to crop.

And finally press the c  key to perform the crop:

Figure 3: Cropping the actual image.

Figure 3: Cropping the actual image.

As you can see, we have successfully cropped Tim’s face from the image.

Here is another example where we crop the Velociraptor:

Figure 4: Another example of cropping our image, this time extracting the Velociraptor head from the image.

Figure 4: Another example of cropping our image, this time extracting the Velociraptor head from the image.

So there you have it — a simple method to capture mouse events to crop an image using Python and OpenCV.

Summary

In this blog post we learned how to capture mouse events using OpenCV and Python. While this post did not provide a complete and exhaustive overview of all the mouse events you can capture, it laid the groundwork for what is possible.

This post also demonstrated how you can quickly crop and extract regions of an image, which is especially useful when creating training data for your own custom object detectors.

Downloads:

If you would like to download the code and images used in this post, please enter your email address in the form below. Not only will you get a .zip of the code, I’ll also send you a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL! Sound good? If so, enter your email address and I’ll send you the code immediately!

, , , ,

61 Responses to Capturing mouse click events with Python and OpenCV

  1. MJ March 22, 2015 at 6:58 pm #

    Another great post… and amazingly timely for me as you often are! This was a huge help for my current work. I did note, that if the bounding box was drawn starting from the right, moving left; it would not work. To eliminate this, with a little research, I added a line to draw the rectangle using the coordinates first ordered by the minimum x,y values- which would be the top left of the box, and got the lower right by subtraction. Now it doesn’t matter how it is drawn. The line is below.

    “refPt = (min(ix,x), min(iy,y), abs(ix-x), abs(iy-y)) #set bounding box by mouse move”

    Feeling emboldened by your challenge, I also added a cv2.EVENT_MOUSEMOVE (where that line sits), in hopes of seeing the box grow as it is drawn, I did this on a copy of a copy of the original image (didn’t want to use the original for any work). The best I could get here was to see a number of boxes growing larger as I moved the mouse… none of which disappeared, so I had a bunch of boxes on the finished image. The number of boxes depended on the speed I moved the mouse. The cropping still worked great, so the multiple boxes are a riddle for another day! I am working with the opencv grabcut code… which uses mouse movements to capture coordinates- and it draws a box nice and smooth. I cannot see what the difference is… so far!

    Thanks again Adrian!

    • Adrian Rosebrock March 23, 2015 at 7:38 am #

      Hi MJ, thanks for the comment! And your code on setting the top-left coordinate of the box is fantastic — great addition! As for the multiple drawings of the bounding box, I ran into the same issue. I wanted to draw a nice smooth rectangle around the region, but I ended up with multiple rectangles drawn on the image, exactly like in your case. This problem will probably need some more investigation…

      • Steve July 27, 2015 at 4:20 pm #

        Hey Adrian, I have a workaround for this issue. Create a new global, in my case I called it sel_rect_endpoint. Then add this after the mouseup event:

        To store the values as the mouse moves.

        Then, change the imshow line in the while True loop to:

        And you will see the rectangle as the mouse moves!

        • Adrian Rosebrock July 28, 2015 at 6:36 am #

          Awesome, thanks so much for the workaround Steve, that’s a great trick!

        • Stephen September 15, 2015 at 8:33 pm #

          Hi Steve and Adrian,

          Thanks for sharing such a great example to real-time boxing in OpenCV.
          I’ve successfully run the combination of your 2 codes.

          So, now I am trying to implement it as a function saved in a separate .py file and then import it/call it to a main program. However, when I do this the “real-time” drawing of the box did not perform as before. The box still get drawn after I release my mouse-click, but it is not as cool as a real-time drawing of the box.

          My question is if such a limitation is to be expected when using a function saved in a different file, or is there a way around it?

          Thank you very much.
          Stephen

        • saad June 3, 2017 at 6:32 am #

          how does this work it keeps on giving me an error

        • Danial February 22, 2018 at 11:46 pm #

          It raise error (NameError: name ‘roi’ is not defined)

          • Brent Griffin April 20, 2018 at 12:17 pm #

            Here are two changes that I made to Steve’s useful add-on to get it working:

            1) Replace ‘roi[0]’ with ‘refPt[0]’ in ‘cv2.rectangle(rect_cpy, refPt[0], sel_rect_endpoint[0], (0, 255,0), 1)’.

            2) Also note that ‘rect_cp’ should be ‘rect_cpy’ in ‘cv2.imshow(‘image’, rect_cp)’.

            Thanks for the great post!

  2. Fabio G April 4, 2015 at 3:01 pm #

    Just used this for my grab-cut algorithm. Thanks a lot!

    Thanks a lot!

    • Adrian Rosebrock April 4, 2015 at 6:45 pm #

      Awesome, glad to hear that it was helpful for you! 🙂

    • Darwin April 13, 2015 at 7:49 am #

      The grab cut does not work properly when you set the flag cv2.WINDOW_OPENGL (in cv2.namedWindow)

      • Adrian Rosebrock April 13, 2015 at 7:58 am #

        Thanks for the tip Darwin!

  3. begueradj April 9, 2015 at 6:56 am #

    Your blog is one of the rare really useful ones I found on internet.
    Tricky problems with simple,straightforward but efficient solutions.
    Without mentioning your precious article sent for free: Tthe ultimate barcode detection guide

    Wish you all the best, Man.

    Regards from BEGUERADJ

    • Adrian Rosebrock April 9, 2015 at 7:49 am #

      Thanks so much Begueradj, I really appreciate your kind compliment 😀

  4. NoahZ April 28, 2015 at 6:35 pm #

    How would you format this so you could import the cropper tool as a module to be used in a seperate script? I’m confused as to how you “load” the picture you want to crop into the click_and_crop function? Basically, if you want to do this:

    import click_and_crop as cc
    image = cv2.imread(….)

    and then somehow call
    cc.click_and_crop(image)

    how would you go about making this kind of thing work?

    Thanks!

    • Adrian Rosebrock May 1, 2015 at 7:00 pm #

      Hey Noah, I would consider reading up on constructing Python functions and classes. This isn’t so much a computer vision problem as a code organization problem.

  5. Rick G June 29, 2015 at 11:33 pm #

    Adrian as everyone here says, your blog is great and very usefull. I am using bits and pieces of your code for a seven segment display reader. I can get the mouse events to work great on a static image, but I am using the raspberry Pi 2 with the picam. When ever the mouse is over a window the code seems to freeze and get stuck in the mouse event loop. Once you remove the mouse from the window everything starts running nicely again.

    Is there a way to keep the main code moving while the mouse is hovering over a window?

    • Adrian Rosebrock June 30, 2015 at 6:37 am #

      Hey Rick, that certainly is odd behavior. I have not tested this code out on a Pi 2 (only on my laptop), so I don’t have an answer for that off hand. The code should definitely not freeze like that, it makes me wonder if there is a problem with GTK and how the Pi is handling the window events. Can you try the code on your laptop and see if you get the same behavior? If not, then we’ll know the problem is related to the Pi.

  6. mizz1205 July 2, 2015 at 1:23 am #

    I am very beginner of openCV and python, from Japan. I would appreciate your advice.

    When I run this click_and_crop.py code, I have error message :
    >>>
    usage: click_and_crop.py [-h] -i IMAGE
    click_and_crop.py: error: argument -i/–image is required
    >>>

    I have installed correct version:Python 2.7 and OpenCV 2.4.11
    jurassic picture placed in the same directory.

    • Adrian Rosebrock July 2, 2015 at 6:43 am #

      Please execute the Python script via command line, not IDLE:

      $ python click_and_crop.py --image jurassic_park_kitchen.jpg

      There is no need to first launch the python interpreter and then execute the code. Just use the command above and the program will work.

      • mizz1205 July 3, 2015 at 1:57 am #

        Thank you for your advice. I have succeeded running this code on raspberry pi.(^^♪

        • Adrian Rosebrock July 3, 2015 at 6:18 am #

          Congrats! 🙂

  7. rajesh October 14, 2015 at 8:14 am #

    Hi Adrian,
    I found this tutorial very helpful. But for my project ,i should be implementing the click and crop technique for a real time video and return the cropped frame image after the mouse event.
    Please help ..

    • Adrian Rosebrock October 14, 2015 at 9:46 am #

      Unfortunately you can’t really do click and crop for a real-time video. Basically, you would have the video playing, press a key to pause the stream, and then do your clicking and cropping. I actually detail how to do that in this post.

      • John March 7, 2017 at 9:37 am #

        Adrian, Can you add a line to a live video feed with cv2.line() and re-engineer the code above?

        • Adrian Rosebrock March 8, 2017 at 1:08 pm #

          Hey John — while I’m happy to provide code and explanations via blog posts for free, I cannot write custom code for you on a case-by-case basis. I hope you understand. If you need help getting started with computer vision I would suggest going through Practical Python and OpenCV.

  8. Manu December 2, 2015 at 1:53 pm #

    Hi Adrian,

    You made my day with this tutorial. Do you know how can i show the rectangle during the clic and not only after the drop?

    Thank you very much!

  9. Prabha March 10, 2016 at 7:49 am #

    Hi Adrian!
    This post is really helpful. I am new to opencv and trying out few things with it. I have two seperate images and i want to display both on the same window and when i click on one image it should be superimposed on the second image. Can you help me?

    Thank you in advance!

    • Adrian Rosebrock March 10, 2016 at 12:00 pm #

      Normally I wouldn’t recommend using OpenCV for complex GUIs. The GUI toolkit provided with OpenCV is mainly for debugging and creating minimal applications. For more complex apps, you should look into an actual GUI framework, such as Tkinter or Qt. As for superimposing two images, you should read up on transparent overlays.

  10. Vedika Agarwal April 15, 2016 at 2:10 pm #

    Hi Adrian,
    I need to mark cars in the image. There are multiple cars in the image, which I need to mark as rectangles. How shall I update the code?

    • Adrian Rosebrock April 17, 2016 at 3:37 pm #

      If you’re trying to mark and annotate your image, I think using dlib’s imglab tool is a better option.

  11. Niki May 26, 2016 at 2:09 pm #

    Hi Adrian,

    Thank you for the tutorial. I have a question: when I run this code although it works, I see the following error appearing continuously in the console :

    cv2.rectangle(image, refPt[0], refPt[1], (0, 255, 0), 2)
    IndexError: list index out of range

    I was wondering if it’s normal or there is something wrong. Also, when I press “r”, the window does not restore, even when I press the “r” key several times until I choose a point in the image, and only after that pressing “r”will restore image, which is kind of annoying.

    Another question: I try to do cropping several times in a loop, to get different regions of interest coordinates. So, basically each time a message appears on the console, asking the user to choose certain area of interest. After the area has been chosen, the refPt content is first stored in a variable, then it becomes empty again for the next round. But, for some reason it gets stuck there, the window doesn’t change and I cannot choose any new rectangle in the image. I even try to destroy the window after each round, and create a new one for the next round, still no success. I would appreciate if you could give me a hint on what I might be probably doing wrong here.

    Update:
    I got my code running in the loop, but I still have the issue of receiving the same error continuously, plus having the same problem with the “r” key.

    • Adrian Rosebrock May 27, 2016 at 1:33 pm #

      That’s quite strange regarding the “r” key. Are you using an US-based keyboard layout? If not, the “r” key may be mapped to a different ordinal number.

      • Niki May 27, 2016 at 2:04 pm #

        Thank you for your reply Adrian. I got to fix the errors and make my code run smoothly through the loop. It was just some indentation thing, which is weird as I didn’t get any indentation error.

        • Adrian Rosebrock May 27, 2016 at 2:08 pm #

          Congrats on resolving the error!

  12. Jose C November 2, 2016 at 7:08 am #

    Hi Sir Adrian How could add code the solution of “MJ” and Steve to cut the region in reverse? I’ve been testing and I can not find the solution. Thanks in advance.

    • Adrian Rosebrock November 3, 2016 at 9:42 am #

      Can you elaborate on what you mean by “cut the region in reverse”? I’m not quite sure I understand what you mean.

      • Jose C November 3, 2016 at 2:04 pm #

        Hello Adrian. Thank you very much for answering so quickly.

        Your code works for me perfectly, again thanks for these tutorials fantanticos, it has started perform them less than a month ago and the truth I have learned well, fast and I’m gone really well.

          Reviews companions “MJ” and “Steve” add some more to this code so useful. Could you indicate how they have done it if it is so friendly and will not involve much hassle? Because I have tried to add and I could not … Again many thanks.

  13. ludovica November 2, 2016 at 9:16 am #

    thank you, very useful.

  14. amit February 27, 2017 at 5:36 am #

    hello Adrian Rosebrock
    how extract the coordinates of selected rectangle area and add a label to them.

    • Adrian Rosebrock February 27, 2017 at 11:04 am #

      What type of label are you referring to?

  15. Kush February 28, 2017 at 12:49 am #

    Hi Adrian…this was really helpful…but i just wanted to know if this could be done without the argument parser..i.e just by calling the click_and_crop function for a loaded image…selecting the coordinates and returning them

    Also if i want to do multiple cropping on the same image and create a new image with the extracted cropped parts at the same coordinates location off my original image with every other part that was not selected as white.

    Thanks for this tutorial…it was extremely helpful

    • Adrian Rosebrock February 28, 2017 at 6:53 am #

      You don’t need the argument parser. If you wanted to skip this step, just hardcode the path to cv2.imread.

  16. john March 6, 2017 at 10:36 am #

    Your posts are extremely helpful!

    typo on line 58

  17. Daniella Solomon May 27, 2017 at 4:42 am #

    Hi!
    I’m trying to do something similar with zoom, and I’m using EVENT_MOUSEWHEEL.
    (scrolling mouse)
    it seems that this function doesn’t work on Linux (Ubuntu).

    Did you try to use this event?
    I can’t find any information on the net.

    • Adrian Rosebrock May 28, 2017 at 1:00 am #

      Hi Daniella — I’m honestly not sure about that event as I’ve never used it before. Sorry I couldn’t be of more help here!

  18. mukesh July 29, 2017 at 7:07 am #

    how to save ROI in folder
    (How to SAVE ROI image in folder)

    • Adrian Rosebrock August 1, 2017 at 9:50 am #

      You would just need to use the cv2.imwrite function:

      cv2.imwrite("path/to/output.jpg", roi)

  19. jessica August 3, 2017 at 4:35 am #

    Hi Adrian,

    Really helpful tutorial. But when I try to crop the same image again, the program stops and quits the image screen. do you know how I can fix this? I need to be able to display several portions of the cropped image.

    • Adrian Rosebrock August 4, 2017 at 6:55 am #

      You would need to modify the functionality of the code. Right now this script assumes you’re interested in just one crop. You’ll want to include the if len(refPt) == 2: statement in your while loop and give each ROI window a unique name (otherwise the window names will overwrite each other).

  20. alkasm August 22, 2017 at 5:46 am #

    Quick note: the global variable cropping isn’t actually used in this script. I imagine it was used in an earlier trial where it was used for a while loop or if statement condition, but in the current state, it is simply set and unused.

  21. esradeniz October 3, 2017 at 9:03 am #

    I didnt work this program. please help me. how do I working this program in python Idle?

    Sample:

    import click_and_crop as cc
    image = cv2.imread(….)
    cc.click_and_crop(image)

    • Adrian Rosebrock October 3, 2017 at 11:04 am #

      This script cannot be easily executed using IDLE. Please save the program as a script and execute it via the command line.

  22. Nemi Bhattarai November 7, 2017 at 10:40 pm #

    Thank you Adrain for the post. How can we modify the code such that.. Every picture in the folder gets looped and cropping multiple parts of an single image and saving it into different file .

    Example: Lets say I have two folders named “Players” and “CroppedPlayers”. There are 10 images of football being played inside “Players” folder and “CroppedPlayers” folder is empty. From each image I want to crop multiple players and save in side CroppedPlayers folder as separate image.

    Suppose each image has 11 players in it and I have 10 such images I want to have 110 images as result saved in different folder.

    (Sincerely looking forward)

    • Adrian Rosebrock November 9, 2017 at 6:47 am #

      There are multiple ways to accomplish this. You can use the tool as-is to mark ROIs, loop over each ROI, extract it, and write it to disk via cv2.imwrite. I cover the fundamentals of OpenCV and image processing inside Practical Python and OpenCV.

      Otherwise, it might be easier to use a dedicated annotation tool such as dlib’s “imglab”, then process the resulting XML file and extract the ROIs.

  23. aditya November 14, 2017 at 1:36 am #

    Hi, Adrian.
    A very nice post, Enjoyed it. However, is it possible to retain the contents inside a box, but remove the rectangle around it?
    Suppose, I draw a 100’s of Green rectangle in the image, but now I wish to remove few(say 10) of these & retain the rest (other 90) of them. How do I do that?

    • Adrian Rosebrock November 15, 2017 at 1:06 pm #

      I would suggest maintaining a Python list of rectangles and drawing the rectangles on a copy of the input image. You can then add/remove the rectangles list as you please.

  24. Abdullah Husein February 3, 2018 at 10:03 am #

    Hi Adrian.
    This post is simply amazing. However, is it possible to crop directly from the os screen, not from OpenCV window? Something like rectangular-snip in screen capturer program.

    • Adrian Rosebrock February 3, 2018 at 10:29 am #

      OpenCV’s GUI functions are limited. They are primarily meant for debugging or for extremely simple applications. You’ll want to look into the GUI libraries for your respective operating system to obtain the functionality you are describing.

  25. Yash b February 6, 2018 at 7:14 am #

    Hi I found this code very helpful. I want the functionality of uploading a file via directory and then using the click and crop function on the image rather than typing name of file from the command line. can anybody help me with that ? I’m having a hard time figuring that out.

Leave a Reply