Image Stitching with OpenCV and Python

Click here to download the source code to this post.

In this tutorial, you will learn how to perform image stitching using Python, OpenCV, and the cv2.createStitcher  and cv2.Stitcher_create  functions. Using today’s code you’ll be able to stitch multiple images together, creating a panorama of stitched images.

Just under two years ago I published two guides on image stitching and panorama construction:

  1. Fundamentals of image stitching
  2. Real-time panorama and image stitching

Both of these tutorials covered the fundamentals of the typical image stitching algorithm, which, at a bare minimum, require four key steps:

  1. Detecting keypoints (DoG, Harris, etc.) and extracting local invariant descriptors (SIFT, SURF, etc.) from two input images
  2. Matching the descriptors between the images
  3. Using the RANSAC algorithm to estimate a homography matrix using our matched feature vectors
  4. Applying a warping transformation using the homography matrix obtained from Step #3

However, the biggest problem with my original implementations is that they were not capable of handling more than two input images.

In today’s tutorial, we’ll be revisiting image stitching with OpenCV, including how to stitch more than two images together into a panoramic image.

To learn how to stitch images with OpenCV and Python, just keep reading!

Looking for the source code to this post?
Jump right to the downloads section.

Image Stitching with OpenCV and Python

In the first part of today’s tutorial, we’ll briefly review OpenCV’s image stitching algorithm that is baked into the OpenCV library itself via cv2.createStitcher  and cv2.Stitcher_create  functions.

From there we’ll review our project structure and implement a Python script that can be used for image stitching.

We’ll review the results of this first script, note its limitations, and then implement a second Python script that can be used for more aesthetically pleasing image stitching results.

Finally, we’ll review the results of our second script and again note any limitations or drawbacks.

OpenCV’s image stitching algorithm

Figure 1: The stitching module pipeline implemented in the Stitcher class (source).

The algorithm we’ll be using here today is similar to the method proposed by Brown and Lowe in their 2017 paper, Automatic Panoramic Image Stitching with Invariant Features.

Unlike previous image stitching algorithms which are sensitive to the ordering of input images, the Brown and Lowe method is more robust, making it insensitive to:

  • Ordering of images
  • Orientation of images
  • Illumination changes
  • Noisy images that are not actually part of the panorama

Furthermore, their image stitching method is capable of producing more aesthetically pleasing output panorama images through the use of gain compensation and image blending.

A complete, detailed review of the algorithm is outside the scope of this post, so if you’re interested in learning more, please refer to the original publication.

Project structure

Let’s see how this project is organized with the tree  command:

The input images go in the images/  folder. I opted to make a subfolder for my scottsdale/  set of images in case I wanted to add additional subfolders here later.

Today we’ll be reviewing two Python scripts:

  • image_stitching_simple.py : Our simple version of image stitching can be completed in less than 50 lines of Python code!
  • image_stitching.py : This script includes my hack to extract an ROI of the stitched image for an aesthetically pleasing result.

The last file, output.png , is the name of the resulting stitched image. Using command line arguments, you can easily change the filename + path of the output image.

The cv2.createStitcher and cv2.Stitcher_create functions

Figure 2: The constructor signature for creating a Stitcher class object with OpenCV.

OpenCV has already implemented a method similar to Brown and Lowe’s paper via the cv2.createStitcher  (OpenCV 3.x) and cv2.Stitcher_create  (OpenCV 4) functions.

Assuming you have OpenCV properly configured and installed you’ll be able to investigate the function signature of cv2.createStitcher  for OpenCV 3.x:

Notice how this function has only a single parameter, try_gpu  which can be used to improve your the throughout of your image stitching pipeline. OpenCV’s GPU support is limited and I’ve never been able to get this parameter to work so I recommend always leaving it as False .

The cv2.Stitcher_create  function for OpenCV 4 has a similar signature:

To perform the actual image stitching we’ll need to call the .stitch  method:

This method accepts a list of input images , and then attempts to stitch them into a panorama, returning the output panorama image to the calling function.

The status  variable indicates whether or not the image stitching was a success and can be one of four variables:

  • OK = 0 : The image stitching was a success.
  • ERR_NEED_MORE_IMGS = 1 : In the event you receive this status code, you will need more input images to construct your panorama. Typically this error occurs if there are not enough keypoints detected in your input images.
  • ERR_HOMOGRAPHY_EST_FAIL = 2 : This error occurs when the RANSAC homography estimation fails. Again, you may need more images or your images don’t have enough distinguishing, unique texture/objects for keypoints to be accurately matched.
  • ERR_CAMERA_PARAMS_ADJUST_FAIL = 3 : I have never encountered this error before so I don’t have much knowledge about it, but the gist is that it is related to failing to properly estimate camera intrinsics/extrinsics from the input images. If you encounter this error you may need to refer to the OpenCV documentation or even dive into the OpenCV C++ code.

Now that we’ve reviewed the cv2.createStitcher , cv2.Stitcher_create , and .stitch  methods, let’s move on to actually implementing image stitching with OpenCV and Python.

Implementing image stitching with Python

Let’s go ahead and get started implementing our image stitching algorithm!

Open up the image_stitching_simple.py  file and insert the following code:

Our required packages are imported on Lines 2-6. Notably, we’ll be using OpenCV and imutils. If you haven’t already, go ahead and install them:

  • To install OpenCV, just follow one of my OpenCV installation guides.
  • The imutils package can be installed/updated with pip: pip install --upgrade imutils . Be sure to upgrade it as new features are often added.

From there we’ll parse two command line arguments on Lines 9-14:

  • --images : The path to the directory of input images to stitch.
  • --output : The path to the output image where the result will be saved.

If you aren’t familiar with the concepts of argparse  and command line arguments then read this blog post.

Let’s load our input images:

Here we grab our imagePaths  (Line 18).

Then for each imagePath , we’ll load the image  and add it to the images  list (Lines 19-25).

Now that the images  are in memory, let’s go ahead and stitch them together into a panorama using OpenCV’s built-in capability:

The stitcher  object is created on Line 30. Notice that depending on whether you’re using OpenCV 3 or 4, a different constructor is called.

Subsequently, we can pass our images  to the .stitch  method (Line 31). The call to .stitch  returns both a status  and our stitched  image (assuming the stitching was successful).

Finally, we’ll both (1) write the stitched image to disk and (2) display it on the screen:

Assuming our status  flag indicates success (Line 35), we write the stitched  image to disk (Line 37) and display it until a key is pressed (Lines 40 and 41).

Otherwise, we’ll simply print a failure message (Lines 45 and 46).

Basic image stitching results

To give our image stitching script a try, make sure you use the “Downloads” section of the tutorial to download the source code and example images.

Inside the images/scottsdale/  directory you will find three photos that I took when visiting Frank Lloyd Wright’s famous Taliesin West house in Scottsdale, AZ:

Figure 3: Three photos to test OpenCV image stitching with. These images were taken by me in Scottsdale, AZ at Frank Lloyd Wright’s famous Taliesin West house.

Our goal is to stitch these three images into a single panoramic image. To perform the stitching, open up a terminal, navigate to where you downloaded the code + images, and execute the following command:

Figure 4: Image stitching performed with OpenCV. This image has undergone stitching but has yet to be cropped.

Notice how we have successfully performed image stitching!

But what about those black regions surrounding the panorama? What are those?

Those regions are from performing the perspective warps required to construct the panorama.

There is a way to get rid of them…but we’ll need to implement some additional logic in the next section.

A better image stitcher with OpenCV and Python

Figure 5: In this section, we’ll learn how to improve image stitching with OpenCV by cropping out the region of the panorama inside the red-dash border shown in the figure.

Our first image stitching script was a good start but those black regions surrounding the panorama itself are not something we would call “aesthetically pleasing”.

And more to the point, you wouldn’t see such an output image from popular image stitching applications built into iOS, Android, etc.

Therefore, we’re going to hack our script a bit and include some additional logic to create more aesthetically pleasing panoramas.

I’m going to again reiterate that this method is a hack.

We’ll be reviewing basic image processing operations including threshold, contour extraction, morphological operations, etc. in order to obtain our desired result.

To my knowledge, OpenCV’s Python bindings do not provide us with the required information to manually extract the maximum inner rectangular region of the panorama. If OpenCV does, please let me know in the comments as I would love to know.

Let’s go ahead and get started — open up the image_stitching.py  script and insert the following code:

All of this code is identical to our previous script with one exception.

The --crop  command line argument has been added. When a 1  is provided for this argument in the terminal, we’ll go ahead and perform our cropping hack.

The next step is where we start implementing additional functionality:

Notice how I’ve made a new block for when the --crop  flag is set on Line 40. Let’s begin going through this block:

  • First, we’ll add a 10  pixel border to all sides of our stitched  image (Lines 43 and 44), ensuring we’ll be able to find contours of the complete panorama outline later in this section.
  • Then we’re going to create a gray  version of our stitched  image (Line 49).
  • And from there we threshold the gray  image (Line 50).

Here is the result ( thresh ) of those three steps:

Figure 6: After thresholding, we’re presented with this threshold mask highlighting where the OpenCV stitched + warped image resides.

We now have a binary image of our panorama where white pixels (255) are the foreground and black pixels (0) are the background.

Given our thresholded image we can apply contour extraction, compute the bounding box of the largest contour (i.e., the outline of the panorama itself), and draw the bounding box:

Contours are extracted and parsed on Lines 55-57Line 58 then grabs the contour with the largest area (i.e., the outline of the stitched image itself).

Note: The imutils.grab_contours  function is new in imutils==0.5.2  to accommodate OpenCV 2.4, OpenCV 3, and OpenCV 4 and their different return signatures for cv2.findContours .

Line 62 allocates memory for our new rectangular mask. Line 63 then calculates the bounding box of our largest contour. Using the bounding rectangle information, on Line 64, we draw a solid white rectangle on the mask.

The output of the above code block would look like the following:

Figure 7: The smallest rectangular region that the entire OpenCV panorama can fit in.

This bounding box is the smallest rectangular region that the entire panorama can fit in.

Now, here comes one of the biggest hacks I’ve ever put together for a blog post:

On Lines 70 and 71 we create two copies of our mask  image:

  1. The first mask, minMask , will be slowly reduced in size until it can fit inside the inner part of the panorama (see Figure 5 at the top of this section).
  2. The second mask, sub , will be used to determine if we need to keep reducing the size of minMask .

Line 75 starts a while  loop that will continue looping until there are no more foreground pixels in sub .

Line 79 performs an erosion morphological operation to reduce the size of minRect .

Line 80 then subtracts thresh  from minRect  — once there are no more foreground pixels in minRect  then we can break from the loop.

I have included an animation of the hack below:

Figure 8: An animation of the hack I came up with to extract the minRect region of the OpenCV panorama image, making for an aesthetically pleasing stitched image

On the top, we have our sub  image and on the bottom we have the minRect  image.

Notice how the size of minRect  is progressively reduced until there are no more foreground pixels left in sub  — at this point we know we have found the smallest rectangular mask that can fit into the largest rectangular region of the panorama.

Given the minimum inner rectangle we can again find contours and compute the bounding box, but this time we’ll simply extract the ROI from the stitched  image:

Here we have:

  • Found contours in minRect  (Lines 84 and 85).
  • Handled parsing contours for multiple OpenCV versions (Line 86). You’ll need imutils>=0.5.2  to use this function.
  • Grabbed the largest contour (Line 87).
  • Computed the bounding box of the largest contour (Line 88).
  • Extracted the ROI from our stitched using the bounding box information (Line 92).

The final stitched  image can be displayed to our screen and then saved to disk:

Lines 95-99 handle saving and displaying the image regardless of whether or not our cropping hack is performed.

Just as before, if the status  flag didn’t come back as a success, we’ll print an error message (Lines 103 and 104).

Let’s go ahead and check out the results of our improved image stitching + OpenCV pipeline.

Improved image stitching results

Again, make sure you have used the “Downloads” section of today’s tutorial to download the source code and example images.

From there, open up a terminal and execute the following command:

Figure 8: The result of our multiple image stitching with OpenCV and Python.

Notice how this time we have removed the black regions from the output stitched images (caused by the warping transformations) by applying our hack detailed in the section above.

Limitations and drawbacks

In a previous tutorial, I demonstrated how you could build a real-time panorama and image stitching algorithm — this tutorial hinged on the fact that we were manually performing keypoint detection, feature extraction, and keypoint matching, giving us access to the homography matrix used to warp our two input images into a panorama.

And while OpenCV’s built-in cv2.createStitcher  and cv2.Stitcher_create  functions are certainly capable of constructing accurate, aesthetically pleasing panoramas, one of the primary drawbacks of the method is that it abstracts away any access to the homography matrices.

One of the assumptions of real-time panorama construction is that the scene itself is not changing much in terms of content.

Once we compute the initial homography estimation we should only have to occasionally recompute the matrix.

Not having to perform a full-blown keypoint matching and RANSAC estimation gives us a tremendous boost of speed when building our panorama, so without access to the raw homography matrices, it would be challenging to take OpenCV’s built-in image stitching algorithm and convert it to real-time.

Running into errors when performing image stitching using OpenCV?

It is possible that you may run into errors when trying to use either the cv2.createStitcher  function or cv2.Stitcher_create  functions.

The two “easy to resolve” errors I see people encounter is forgetting what version of OpenCV they are using.

For example, if you are using OpenCV 4 but try to call cv2.createSticher  you will encounter the following error message:

You should instead be using the cv2.Stitcher_create  function.

Similarly, if you are using OpenCV 3 and you try to call cv2.Sticher_create  you will receive this error:

Instead, use the cv2.createSticher  function.

If you are unsure which OpenCV version you are using you can check using cv2.__version__ :

Here you can see that I am using OpenCV 4.0.0.

You can perform the same check on your system.

The final error that you can encounter, and arguably the most common, is related to OpenCV (1) not having contrib support and (2) being compiled without the OPENCV_ENABLE_NONFREE=ON  option enabled.

To resolve this error you must have the opencv_contrib  modules installed along with the OPENCV_ENABLE_NONFREE  option set to ON .

If you are encountering an error related to OpenCV’s non-free and contrib modules, make sure you refer to my OpenCV install guides to ensure you have the full install of OpenCV.

Note: Please note that I cannot help debug your own OpenCV install if you did not follow one of my install guides so please make sure you’re using my OpenCV install guides when configuring your system.

Summary

In today’s tutorial you learned how to perform multiple image stitching using OpenCV and Python.

Using both OpenCV and Python we were able to stitch multiple images together and create panoramic images.

Our output panoramic images were not only accurate in their stitching placement but also aesthetically pleasing as well.

However, one of the biggest drawbacks of using OpenCV’s built-in image stitching class is that it abstracts away much of the internal computation, including the resulting homography matrices themselves.

If you are trying to perform real-time image stitching, as we did in a previous post, you may find it beneficial to cache the homography matrix and only occasionally perform keypoint detection, feature extraction, and feature matching.

Skipping these steps and using the cached matrix to perform perspective warping can reduce the computational burden of your pipeline and ultimately speed-up the real-time image stitching algorithm, but unfortunately, OpenCV’s cv2.createStitcher  Python bindings do not provide us with access to the raw matrices.

If you are interested in learning more about real-time panorama construction, please refer to my previous post.

I hope you enjoyed today’s tutorial on image stitching!

To download the source code to today’s post, and be notified tutorials are published here on PyImageSearch, just enter your email address in the form below!

Downloads:

If you would like to download the code and images used in this post, please enter your email address in the form below. Not only will you get a .zip of the code, I’ll also send you a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL! Sound good? If so, enter your email address and I’ll send you the code immediately!

, , , , ,

25 Responses to Image Stitching with OpenCV and Python

  1. Manmohan Bishnoi December 17, 2018 at 12:05 pm #

    Hello Adrian,

    Thanks for a great tutorial once again.
    Typical steps for panorama creation from multiple images are:

    1. Detect Features
    2. Compute Descriptors
    3. Match features
    4. Remove false matches
    5. Calculate Homography
    6. Stitch images
    7. Detect seams
    8. Multi-band blend for final panorama
    9. Crop for aesthetic final image

    I am trying to generate real-time panorama from images taken using burst shots from a mobile camera rotated in horizontal circular direction, similar to most Apps in App store.
    But doing all these steps for two adjacent images takes about 10 seconds with OpenCV Stitcher pipeline.

    So if I take 32 images at 11.25 degrees apart and try to stitch them in real-time, it is way too slow.
    To speed up things I tried doing some tasks in parallel using multi-threading.

    Any tips how to speedup this and reduce time for stitching two adjacent images to about 2-4 seconds?

    I am trying to generate panorama incrementally by stitching images in sequence.

    Thanks

    • Adrian Rosebrock December 17, 2018 at 12:45 pm #

      Hi Manmohan — I would suggest looking at the GPU functionality I hinted at in the post. That should ideally help you speedup the pipeline but it may require you going down the rabbit hole and playing with the raw C++ code to get it to work (I haven’t been able to).

    • Tim December 17, 2018 at 7:44 pm #

      Thanks @Manmohan. I was wondering if I could stitch the frames from 3 cameras together a deliver a panoramic video stream. (I currently use a high-res fisheye camera but even with de-warping the image is not ‘right’, and I’d like better resolution for zoom in)

      Your comment leads me to believe I’ll never get ~15 frames per sec via stitching.

      (I’ve been playing inference accelerators (Movidius) but that is no help here)

      @Adrian, thoughts?
      My use case is a video of a wilderness; ~165 deg FoV. Boars, deer, mnt lion might be, rarely, anywhere, and when they are I like to zoom in.
      Note, the object detection is to alert me when something interesting appears. Also this is a personal ‘fun’ project so time and resources are scarce.

      • Adrian Rosebrock December 18, 2018 at 8:53 am #

        Is your wildlife camera stationary and non-moving? If so, yes, you can absolutely achieve 15+ FPS. This tutorial will show you how. You’ll need to update the code to work with more than two images or hack the OpenCV C++ source to access the raw homography matrix.

  2. Cenk December 17, 2018 at 1:52 pm #

    Many thanks for the detailed demo.

    Do you think Cython would help to speed up the stitching process?

    Thanks

    • Adrian Rosebrock December 17, 2018 at 2:22 pm #

      No, mainly because we’re calling OpenCV’s functions which are C++ compiled functions. There is a small overhead between the Python call and the compiled bindings but that overhead is essentially nothing.

  3. Agus Ambarwari December 17, 2018 at 11:54 pm #

    Thank you Mr. Adrian, I am from Indonesia and interest in computer vision.
    Can stitching image use more than two images and not only from right to left? May, from top to bottom.

    • Adrian Rosebrock December 18, 2018 at 8:49 am #

      Technically yes, but I haven’t tried with this particular implementation. Be sure to give it a try!

  4. Ankit Agrawal December 18, 2018 at 4:13 am #

    Hello Adrian,

    Thank you for you post, this has been a lot of help for me.

    Also, I wanted to ask are you coming up with the post for realtime panorama stitching with more than two cameras, precisely 4 cameras?

    • Adrian Rosebrock December 18, 2018 at 8:46 am #

      As I mentioned in the post, the method used here realistically cannot be used for real-time panorama stitching. You would need to hack the OpenCV C++ code to access the homography matrix and only apply new feature matching once every N frames.

      • Younus December 19, 2018 at 12:23 am #

        Hello Adrian i am also trying to achieve the same as Ankit mentioned, isn’t it possble by combining your this tutorial and real time stitching tutorial by saving cache and applying it to every frame possible? and if not then what are you telling about hacking opencv c++ code can you guide a little about it. Thank you.

        • Adrian Rosebrock December 19, 2018 at 1:52 pm #

          No, make sure you re-read this tutorial as I explain why you cannot cache the homography matrix. I personally have not worked with the C++ code. My suggestion was that you would need to do your own research with the code and so if you could hack the code and compile your own bindings that exposes the matrix. It would be a challenging, non-trivial process.

  5. Rahul Ragesh December 19, 2018 at 8:00 am #

    Hello Adrian,

    I have 3 retinal images which have warped using Homography manually (I couldn’t use the built-in function because image qualities were bad). Now I want to blend these images together. I couldn’t find any opencv python functions for blending. Am I missing something?

    • Adrian Rosebrock December 19, 2018 at 1:47 pm #

      Are you referring to the specific blending functions used by the algorithm in this post? If so, those functions are abstracted by the C++ API. You should refer to the OpenCV docs and source code for the stitching module.

  6. mjbordalo December 19, 2018 at 2:39 pm #

    Hello, thanks for another great tutorial.
    I would like to do a stitching but with a top-view camera.
    Like photos taken from a drone and sequentially stritch this photos.
    Which steps should i change?
    tnhks in advance

    • Adrian Rosebrock December 20, 2018 at 5:21 am #

      Have you tried using the code as-is? If so, what problems did you run into?

      • mjbordalo January 3, 2019 at 12:14 pm #

        yes. I tried to take a bunch of photos with my phone pointing downward while i was turning around. (so that the last picture would be similiar to the first one) The output resolt tries to put the images in a landscape mode like all are side by side

  7. William Stevenson December 19, 2018 at 3:28 pm #

    Hello Adrian

    I used pip install opencv_contrib_python, which fetched opencv_contrib_python-3.4.4.19-cp37-cp37m-win_amd64.whl for 64 bit python 3.7. It does not show errors during installation. cv2.__version__ shows 3.4.4. Release date is 27 Nov 2018.

    Set OPENCV_ENABLE_NONFREE CMake option and rebuild the library in function ‘cv::xfeatures2d::SURF::create’

    I thought that opencv_contrib_modules would contain all the contributed modules, or is this not guaranteed?

    • Adrian Rosebrock December 20, 2018 at 5:20 am #

      No, the pip install of opencv-contrib-python does not include the NONFREE modules. To enable them you would need to compile OpenCV from source. You should follow one of my OpenCV install guides to compile from source.

  8. Hure December 19, 2018 at 7:16 pm #

    Can this sample code, perhaps with some parameter change, also be used to do what I’ll call pixel exact stiching? For example stitch together 5 screen captures of parts of a Google map with some overlap between each screen capture (and same zoom level and so on of course) into a larger map image? In other words find vertical or horizontal one pixel thick lines that are identical and join two images at each such seam. If this code won’t do that, do you know of some other opencv commands to use for that?

    • Adrian Rosebrock December 20, 2018 at 5:16 am #

      That is a pretty specific use case. You can try it and see but I don’t think you’ll be able to obtain that level of accuracy. It’s certainly worth a test though!

  9. Sebastian December 20, 2018 at 10:19 am #

    Great tutorial as usual, but I want to use image stitching for commercial use. But the NONFREE OpenCV functions are unfortionally patented. I’d like (my boss) to pay for them, but I’m without a clue where to ask for permission or how to buy a license for commercial usage. Do you by any chance know how I can obtain a license or use it comercially without breaking the law?

    • Adrian Rosebrock December 27, 2018 at 11:07 am #

      You would want to reach out to the patent holders. You can find them via Google’s patent search.

  10. Emilio January 10, 2019 at 12:34 pm #

    Hi Adrian,

    Great tutorial as always! I was wondering, back in your OpenCV panorama stitching from 2016, you made a stitcher script and you were able to compute matches yourself. Have you found any way to retrieve matches from this method? I would like to store all the matches computed while stitching.

    Thanks!

    • Adrian Rosebrock January 11, 2019 at 9:35 am #

      Hi Emilio — could you clarify a bit more what you mean by “retrieve matches from this method”? What specifically are you trying to accomplish?

Leave a Reply