OpenCV panorama stitching

bryce_result_02

In today’s blog post, I’ll demonstrate how to perform image stitching and panorama construction using Python and OpenCV. Given two images, we’ll “stitch” them together to create a simple panorama, as seen in the example above.

To construct our image panorama, we’ll utilize computer vision and image processing techniques such as: keypoint detection and local invariant descriptors; keypoint matching; RANSAC; and perspective warping.

Since there are major differences in how OpenCV 2.4.X and OpenCV 3.X handle keypoint detection and local invariant descriptors (such as SIFT and SURF), I’ve taken special care to provide code that is compatible with both versions (provided that you compiled OpenCV 3 with opencv_contrib  support, of course).

In future blog posts we’ll extend our panorama stitching code to work with multiple images rather than just two.

Read on to find out how panorama stitching with OpenCV is done.

Looking for the source code to this post?
Jump right to the downloads section.

OpenCV panorama stitching

Our panorama stitching algorithm consists of four steps:

  • Step #1: Detect keypoints (DoG, Harris, etc.) and extract local invariant descriptors (SIFT, SURF, etc.) from the two input images.
  • Step #2: Match the descriptors between the two images.
  • Step #3: Use the RANSAC algorithm to estimate a homography matrix using our matched feature vectors.
  • Step #4: Apply a warping transformation using the homography matrix obtained from Step #3.

We’ll encapsulate all four of these steps inside panorama.py , where we’ll define a Stitcher  class used to construct our panoramas.

The Stitcher  class will rely on the imutils Python package, so if you don’t already have it installed on your system, you’ll want to go ahead and do that now:

Let’s go ahead and get started by reviewing panorama.py :

We start off on Lines 2-4 by importing our necessary packages. We’ll be using NumPy for matrix/array operations, imutils  for a set of OpenCV convenience methods, and finally cv2  for our OpenCV bindings.

From there, we define the Stitcher  class on Line 6. The constructor to Stitcher  simply checks which version of OpenCV we are using by making a call to the is_cv3  method. Since there are major differences in how OpenCV 2.4 and OpenCV 3 handle keypoint detection and local invariant descriptors, it’s important that we determine the version of OpenCV that we are using.

Next up, let’s start working on the stitch  method:

The stitch  method requires only a single parameter, images , which is the list of (two) images that we are going to stitch together to form the panorama.

We can also optionally supply ratio , used for David Lowe’s ratio test when matching features (more on this ratio test later in the tutorial), reprojThresh  which is the maximum pixel “wiggle room” allowed by the RANSAC algorithm, and finally showMatches , a boolean used to indicate if the keypoint matches should be visualized or not.

Line 15 unpacks the images  list (which again, we presume to contain only two images). The ordering to the images  list is important: we expect images to be supplied in left-to-right order. If images are not supplied in this order, then our code will still run — but our output panorama will only contain one image, not both.

Once we have unpacked the images  list, we make a call to the detectAndDescribe  method on Lines 16 and 17. This method simply detects keypoints and extracts local invariant descriptors (i.e., SIFT) from the two images.

Given the keypoints and features, we use matchKeypoints  (Lines 20 and 21) to match the features in the two images. We’ll define this method later in the lesson.

If the returned matches M  are None , then not enough keypoints were matched to create a panorama, so we simply return to the calling function (Lines 25 and 26).

Otherwise, we are now ready to apply the perspective transform:

Provided that M  is not None , we unpack the tuple on Line 30, giving us a list of keypoint matches , the homography matrix H  derived from the RANSAC algorithm, and finally status , a list of indexes to indicate which keypoints in matches  were successfully spatially verified using RANSAC.

Given our homography matrix H , we are now ready to stitch the two images together. First, we make a call to cv2.warpPerspective  which requires three arguments: the image we want to warp (in this case, the right image), the 3 x 3 transformation matrix ( H ), and finally the shape out of the output image. We derive the shape out of the output image by taking the sum of the widths of both images and then using the height of the second image.

Line 30 makes a check to see if we should visualize the keypoint matches, and if so, we make a call to drawMatches  and return a tuple of both the panorama and visualization to the calling method (Lines 37-42).

Otherwise, we simply returned the stitched image (Line 45).

Now that the stitch  method has been defined, let’s look into some of the helper methods that it calls. We’ll start with detectAndDescribe :

As the name suggests, the detectAndDescribe  method accepts an image, then detects keypoints and extracts local invariant descriptors. In our implementation we use the Difference of Gaussian (DoG) keypoint detector and the SIFT feature extractor.

On Line 52 we check to see if we are using OpenCV 3.X. If we are, then we use the cv2.xfeatures2d.SIFT_create  function to instantiate both our DoG keypoint detector and SIFT feature extractor. A call to detectAndCompute  handles extracting the keypoints and features (Lines 54 and 55).

It’s important to note that you must have compiled OpenCV 3.X with opencv_contrib support enabled. If you did not, you’ll get an error such as AttributeError: 'module' object has no attribute 'xfeatures2d' . If that’s the case, head over to my OpenCV 3 tutorials page where I detail how to install OpenCV 3 with opencv_contrib  support enabled for a variety of operating systems and Python versions.

Lines 58-65 handle if we are using OpenCV 2.4. The cv2.FeatureDetector_create  function instantiates our keypoint detector (DoG). A call to detect  returns our set of keypoints.

From there, we need to initialize cv2.DescriptorExtractor_create  using the SIFT  keyword to setup our SIFT feature extractor . Calling the compute  method of the extractor  returns a set of feature vectors which quantify the region surrounding each of the detected keypoints in the image.

Finally, our keypoints are converted from KeyPoint  objects to a NumPy array (Line 69) and returned to the calling method (Line 72).

Next up, let’s look at the matchKeypoints  method:

The matchKeypoints  function requires four arguments: the keypoints and feature vectors associated with the first image, followed by the keypoints and feature vectors associated with the second image. David Lowe’s ratio  test variable and RANSAC re-projection threshold are also be supplied.

Matching features together is actually a fairly straightforward process. We simply loop over the descriptors from both images, compute the distances, and find the smallest distance for each pair of descriptors. Since this is a very common practice in computer vision, OpenCV has a built-in function called cv2.DescriptorMatcher_create  that constructs the feature matcher for us. The BruteForce  value indicates that we are going to exhaustively compute the Euclidean distance between all feature vectors from both images and find the pairs of descriptors that have the smallest distance.

A call to knnMatch  on Line 79 performs k-NN matching between the two feature vector sets using k=2 (indicating the top two matches for each feature vector are returned).

The reason we want the top two matches rather than just the top one match is because we need to apply David Lowe’s ratio test for false-positive match pruning.

Again, Line 79 computes the rawMatches  for each pair of descriptors — but there is a chance that some of these pairs are false positives, meaning that the image patches are not actually true matches. In an attempt to prune these false-positive matches, we can loop over each of the rawMatches  individually (Line 83) and apply Lowe’s ratio test, which is used to determine high-quality feature matches. Typical values for Lowe’s ratio are normally in the range [0.7, 0.8].

Once we have obtained the matches  using Lowe’s ratio test, we can compute the homography between the two sets of keypoints:

Computing a homography between two sets of points requires at a bare minimum an initial set of four matches. For a more reliable homography estimation, we should have substantially more than just four matched points.

Finally, the last method in our Stitcher  method, drawMatches  is used to visualize keypoint correspondences between two images:

This method requires that we pass in the two original images, the set of keypoints associated with each image, the initial matches after applying Lowe’s ratio test, and finally the status  list provided by the homography calculation. Using these variables, we can visualize the “inlier” keypoints by drawing a straight line from keypoint N in the first image to keypoint M in the second image.

Now that we have our Stitcher  class defined, let’s move on to creating the stitch.py  driver script:

We start off by importing our required packages on Lines 2-5. Notice how we’ve placed the panorama.py  and Stitcher  class into the pyimagesearch  module just to keep our code tidy.

Note: If you are following along with this post and having trouble organizing your code, please be sure to download the source code using the form at the bottom of this post. The .zip of the code download will run out of the box without any errors.

From there, Lines 8-14 parse our command line arguments: --first , which is the path to the first image in our panorama (the left-most image), and --second , the path to the second image in the panorama (the right-most image).

Remember, these image paths need to be suppled in left-to-right order!

The rest of the stitch.py  driver script simply handles loading our images, resizing them (so they can fit on our screen), and constructing our panorama:

Once our images are loaded and resized, we initialize our Stitcher  class on Line 23. We then call the stitch  method, passing in our two images (again, in left-to-right order) and indicate that we would like to visualize the keypoint matches between the two images.

Finally, Lines 27-31 display our output images to our screen.

Panorama stitching results

In mid-2014 I took a trip out to Arizona and Utah to enjoy the national parks. Along the way I stopped at many locations, including Bryce Canyon, Grand Canyon, and Sedona. Given that these areas contain beautiful scenic views, I naturally took a bunch of photos — some of which are perfect for constructing panoramas. I’ve included a sample of these images in today’s blog to demonstrate panorama stitching.

All that said, let’s give our OpenCV panorama stitcher a try. Open up a terminal and issue the following command:

Figure 1: (Top) The two input images from Bryce canyon (in left-to-right order). (Bottom) The matched keypoint correspondences between the two images.

Figure 1: (Top) The two input images from Bryce canyon (in left-to-right order). (Bottom) The matched keypoint correspondences between the two images.

At the top of this figure, we can see two input images (resized to fit on my screen, the raw .jpg files are a much higher resolution). And on the bottom, we can see the matched keypoints between the two images.

Using these matched keypoints, we can apply a perspective transform and obtain the final panorama:

Figure 2: Constructing a panorama from our two input images.

Figure 2: Constructing a panorama from our two input images.

As we can see, the two images have been successfully stitched together!

Note: On many of these example images, you’ll often see a visible “seam” running through the center of the stitched images. This is because I shot many of photos using either my iPhone or a digital camera with autofocus turned on, thus the focus is slightly different between each shot. Image stitching and panorama construction work best when you use the same focus for every photo. I never intended to use these vacation photos for image stitching, otherwise I would have taken care to adjust the camera sensors. In either case, just keep in mind the seam is due to varying sensor properties at the time I took the photo and was not intentional.

Let’s give another set of images a try:

Figure 3: Another successful application of image stitching with OpenCV.

Figure 3: Another successful application of image stitching with OpenCV.

Again, our Stitcher  class was able to construct a panorama from the two input images.

Now, let’s move on to the Grand Canyon:

Figure 4: Applying image stitching and panorama construction using OpenCV.

Figure 4: Applying image stitching and panorama construction using OpenCV.

In the above input images we can see heavy overlap between the two input images. The main addition to the panorama is towards the right side of the stitched images where we can see more of the “ledge” is added to the output.

Here’s another example from the Grand Canyon:

Figure 5: Using image stitching to build a panorama using OpenCV and Python.

Figure 5: Using image stitching to build a panorama using OpenCV and Python.

From this example, we can see that more of the huge expanse of the Grand Canyon has been added to the panorama.

Finally, let’s wrap up this blog post with an example image stitching from Sedona, AZ:

Figure 6: One final example of applying image stitching.

Figure 6: One final example of applying image stitching.

Personally, I find the red rock country of Sedona to be one of the most beautiful areas I’ve ever visited. If you ever have a chance, definitely stop by — you won’t be disappointed.

So there you have it, image stitching and panorama construction using Python and OpenCV!

Summary

In this blog post we learned how to perform image stitching and panorama construction using OpenCV. Source code was provided for image stitching for both OpenCV 2.4 and OpenCV 3.

Our image stitching algorithm requires four steps: (1) detecting keypoints and extracting local invariant descriptors; (2) matching descriptors between images; (3) applying RANSAC to estimate the homography matrix; and (4) applying a warping transformation using the homography matrix.

While simple, this algorithm works well in practice when constructing panoramas for two images. In a future blog post, we’ll review how to construct panoramas and perform image stitching for more than two images.

Anyway, I hope you enjoyed this post! Be sure to use the form below to download the source code and give it a try.

Downloads:

If you would like to download the code and images used in this post, please enter your email address in the form below. Not only will you get a .zip of the code, I’ll also send you a FREE 11-page Resource Guide on Computer Vision and Image Search Engines, including exclusive techniques that I don’t post on this blog! Sound good? If so, enter your email address and I’ll send you the code immediately!

, , , , , , ,

130 Responses to OpenCV panorama stitching

  1. Tito Luyo Murata January 11, 2016 at 1:25 pm #

    Still don’t finish to read but looks amazing.
    One question: I suppose it can be used to complete a map using different pics of aerial photos

    • Adrian Rosebrock January 11, 2016 at 3:04 pm #

      Certainly! Provided that there are enough keypoints matched between each photos, you can absolutely use it for aerial images.

      • Waladi January 25, 2016 at 11:42 pm #

        Great topic Adrian.
        As for stitching Aerial View imagery – mainly for mapping purpose, can it be used to stitch second image that “overlap” on below part of the first image (not the left-to-right, but bottom-to-upper part of image) ?

        • Adrian Rosebrock January 26, 2016 at 5:57 pm #

          Yes, you can use it to stitch bottom-to-top images as well, but you’ll need to change Lines 31-33 to handle allocating an image that is tall rather than wide and then update the array slices to stack the images on top of each other. But again, yes, it’s totally possible.

        • Jakob Kirkegaard February 11, 2016 at 4:09 am #

          Unfortunately, the stitcher functionality in OpenCV 3.1 or more precisely the HomographyBasedEstimator and the BunderAdjuster used by the Stitcher class will only estimate camera rotations (no camera translations), i.e. it assumes that all camera centers are approximately equal.

          If the camera experience translations (like aerial shots) or translations in general, the obtained results are usually not that great – even though the images can be matched given good keypoints.

          • Manuel August 5, 2016 at 1:16 am #

            Hi Jakob, could you please point me out how what approach could I follow to handle the no-camera-translations problem?

            Thanks

  2. Ankit Dixit January 12, 2016 at 1:00 am #

    Hi adrian,

    A very good topic you have covered in this post, thanks for the description, i have a question regarding an OCR problem, i have first version of your book where you have described digit recognition using HOG features, that algorithm works on contour detection (blob based), my question is what may be the other way to approach the problem where i cant get individual contours for each digit or character (Segmentation is not possible), thanks for your suggestion in advance. and a big thank you for writing a very easy to understand book.

    • Adrian Rosebrock January 12, 2016 at 6:25 am #

      Are you referring to cursive handwriting where the characters are not individually segment-able? If so, I think R-CNNs are a likely candidate if you have enough training data. Otherwise, you might want to look at the Street View House Numbers Dataset and the relevant papers associated with high accuracy results.

      • Ankit Dixit January 13, 2016 at 5:12 am #

        Hi Adrian thanks for your reply, actually i am working something like vehicle registration data extraction through registration card into a json file, where there are some fixed fields like name and address and their respective variables. i have captured a few images of registration card from a mobile camera so scale varies a lot and in some cases minor orientation changes also there, a big advantage here is there is no hand written letters or digits so variability of data is less, and all alphabets are in upper case, but at the time of segmentation(image Thresholding) some letters got merged in a single blob (or contour) so i cant extract each letter individually. so i cant apply blob based analysis, i have tried few pre-processing steps to separate the blobs but it results in some useful structural information loss, what should i do here.

        • Adrian Rosebrock January 13, 2016 at 6:29 am #

          I would suggest sending me an email so we can chat more offline about it. This post is about panorama construction, not digit extraction, so it would be great if we could keep the comments thread on topic.

  3. Sean McLeod January 12, 2016 at 4:31 am #

    Great post. I’ve been wanting to try use OpenCV for orthophoto stitching of aerial photos from drones.

    In terms of the seam that’s surely to do with different exposures and not focusing. Typically for landscape photos the focus will be on infinity anyway. Some of the panoramic software I’ve used in the past has a feature to try and equalize the exposures so that the seam isn’t visible. Sounds like potentially a topic for another OpenCV blog post 😉

    • Adrian Rosebrock January 12, 2016 at 6:22 am #

      Great point Sean! I ran a few tests just using images around the apartment captured with my iPhone with a fixed exposure. This got rid of the seam. But of course, it requires that your environment doesn’t change dramatically through the Panorama.

      • Bruno January 17, 2016 at 6:00 pm #

        An alternative is to apply a blending mask along the seam, this way you avoid having to adjust exposure between images. If the difference in exposure is small between the neighbouring images, it hides the seam nicely.

        • Adrian Rosebrock January 18, 2016 at 3:22 pm #

          Great point Bruno! Perhaps that will be a subject for a future blog post 🙂

  4. tonyv January 20, 2016 at 8:56 am #

    Why, in stitcher.stich(), line 15 is (imageB, imageA) = images
    when it is called with stitcher.stitch([imageA, imageB]). i.e reversed?

    I note that showMatches displays the left and right images exchanged, and correcting line 15 fixes that, but renders the main panorama incorrectly.

    Any comments?

    • Adrian Rosebrock January 20, 2016 at 1:41 pm #

      It’s more intuitive for us to think of a panorama being rendered from left-to-right (which is an assumption this code makes). We reverse the unpacking in the stitch method for the actual matching of the keypoint detection and local invariant descriptors. Again, order does matter when it comes to the stitching of the images.

  5. VIJAYA KUMAR February 5, 2016 at 10:21 am #

    hi adrian i’m vijay i’ve configured the opencv and python in windows and i’ve some doubt….! still to do some project does we need the any ide like eclipes ………? i’ve just stucked in here will you please help me………..
    thanks in advance………..

    • Adrian Rosebrock February 6, 2016 at 10:00 am #

      Hey Vijay, I honestly haven’t used a Windows system in 9+ years and I don’t do development with Eclipse, so I’m not the right person to ask about this. Best of luck with your project!

  6. Fadil February 23, 2016 at 9:54 am #

    Hey Adrian,
    First thanks for your blogpost, really well explained! As for my problem, I am trying to make your code work for three images and I can’t seem to obtain a good result (the right side of the image gets deformed and pixelized).
    I did not get to creative I just conduct the exact same operation you did on 2 pictures. First I stitch picture A and B(call the result R1), then picture B and C (R2)and finally I stitch R1 and R2.
    You said you’ll post how to do panorama with +2 images, do you have a better algorithm in mind ?

    • Adrian Rosebrock February 23, 2016 at 3:20 pm #

      Constructing panoramas > 2 images is a substantially harder problem. I would suggest giving this thread a read for a more detailed discussion on the problem.

  7. Massimo March 9, 2016 at 9:40 pm #

    Hi Adrian,

    great post!
    I was wondering if you can help me with this:

    http://nbviewer.jupyter.org/gist/anonymous/443728eef41cca0648f1

    My images has a black border (results from camera calibration)
    The mosaic works great, but I’m struggling in understanding how to get rid of the black borders.
    Have you any clue?

    Thanks!!!
    Massimo.

    • Adrian Rosebrock March 10, 2016 at 12:04 pm #

      When you say “get rid of” the black borders, do you mean simply setting the border to white? If so, first create the result image using np.ones and fill them with (255, 255, 255) (i.e., white) rather than 0 (black).

      • Massimo March 13, 2016 at 4:17 am #

        Hi Adrian,

        both original images (A,B) have black pixels at the border (before stitching them together)

        See: http://epinux.com/panorama.png

        The resulting mosaic has the black border (from image A) overlaying pixels with data from image B.

        What I’m looking for is a way to set some sort of transparency
        (or perhaps set to null? the black borders )
        see the image labeled “desired result” at the link above.

        • Adrian Rosebrock March 13, 2016 at 10:17 am #

          Keep in mind that an image is always rectangular. You can’t set the pixels to “null” since they are part of the image matrix. You can change their actual color (such as making them black or white), but you can’t “remove” the pixels from the image.

          • Massimo March 13, 2016 at 12:56 pm #

            Sorry,what I menart is: do you know a way to add an alpha channel to the resulting image ?
            If possible, I can then try to set the transparency to 100% where the black borders are.

          • Adrian Rosebrock March 14, 2016 at 3:31 pm #

            I would suggest creating an alpha channel that has the same shape as the other channels and then merging all four together:

            I haven’t tested this myself, but it should work.

  8. Koshy George March 11, 2016 at 4:47 am #

    Dear Adrian,

    A brilliant tutorial explained with such such simplicity. It really helped me understand and appreciate the Python-OpenCV combo. Thank you

    • Adrian Rosebrock March 13, 2016 at 10:27 am #

      No problem, I’m happy the tutorial could help Koshy! 🙂

  9. Vanessa April 17, 2016 at 5:58 am #

    Hi Adrian,
    Is it possible to save the output image without the black excess? I know it’s because it computed based on the width of the two images, is there a way to save the image without it? If so, how do i do this? Thank You~

    • Adrian Rosebrock April 17, 2016 at 3:28 pm #

      I personally haven’t done this, but yes, it is possible. You basically need to find where the black excess is surrounding the image. A hacky way to do this would be to apply thresholding and find the contour of the image itself. A better approach would be to examine the homography/warping matrix and figure out the coordinates of where the valid stitched image is. This will result in some cropping of actual image content, but will remove the black excess. I’ll try to do a blog post on this topic in the future.

  10. Jonjo May 3, 2016 at 2:21 pm #

    When are you going to release a blog about stitching multiple images? You mentioned in this article that you would do it soon.

    • Adrian Rosebrock May 3, 2016 at 5:41 pm #

      I would love to do it, but I simply haven’t had enough time as I’ve been working on other projects. I honestly cannot commit to a timeframe for the stitching multiple images.

  11. Kyle May 6, 2016 at 4:07 pm #

    This is awesome. If it’s trivial, how could I change the code in order to only take the intersection of the two images?

    Thanks

    • Adrian Rosebrock May 6, 2016 at 4:29 pm #

      Hey Kyle — what you do mean by “intersection of the two images”? Could you please clarify?

      • Kyle May 6, 2016 at 5:20 pm #

        By intersection I mean that I only want the parts image portion that is present in both images. I tried using bitwise_and (with the result of your cv2.warpPerspective in the stitch function and with the other image from the panorama), but the image it outputs has it’s colors all messed up. I don’t really know how to create a mask correctly in order to achieve what I want.

        • Adrian Rosebrock May 7, 2016 at 12:38 pm #

          Instead of creating a mask, the best option is to explore the (x, y)-coordinates of the matched feature vectors. Find the (x, y)-coordinates of the matched keypoints that correspond to the top-left, top-right, bottom-right, and bottom-left corners. From there, you can crop out the overlapping ROI.

  12. Wayne Anderson May 9, 2016 at 4:27 am #

    Hello, really enjoying your tutorials but I’ve run into a little snag. I have been trying to get this one to work but I keep getting a typeerror: Nonetype for this line: (result, vis) = stitcher.stitch([imageA, imageB], showMatches=True). I even downloaded the source files too and they don’t seem to work. Pretty new to python so I am not sure why this is happening. Any ideas?

    • Adrian Rosebrock May 9, 2016 at 6:58 pm #

      The stitch method will return None if not enough keypoints are matched to compute the homography. What images are you using? Are they the same images as in this post or ones of your own? If they are ones of your own, then there are not enough matched keypoints to stitch the images together. In that case, I would try different combinations of keypoint detectors and feature descriptors. It could also be the case that the images simply cannot be stitched together.

      • Wayne Anderson May 10, 2016 at 11:06 am #

        Ah okay, that’s good to know! I guess that the images don’t have enough keypoints then. And I did get the posts pictures to work, I was using two of the left images instead of a left and right… Looks like it was mainly human error. Thanks for the help and information though!

  13. Mr. Pink May 9, 2016 at 3:21 pm #

    What do you think would have to be done to make the stitching not care whether you gave the left photo first or the right photo first?

    • Adrian Rosebrock May 9, 2016 at 6:46 pm #

      This becomes very problematic because at least one of the images needs to act as a “reference point”. This is why panorama apps normally tend to make sure you move from left-to-right or down-to-up. Without setting an initial reference point, you have to resort to heuristics, which often fail.

      • Mr. Pink May 10, 2016 at 11:09 am #

        So I’ve being looking at this for a bit now, and have managed to find something that you may be interested in.
        I plotted all the matched points (called “matches” from line 30 in your code) of when images are inputted left to right and when inputted right to left. I found out that these points make roughly a line, and it is possible to calculate the slope of such a line.
        The slope of the left to right instance should always be smaller than the right to left instance. Therefore it’s possible to throw in a simple if statement and make a swap of variables.
        I have tested this theory with many images, and it seems to work very well. I have a couple others in my code for special cases (like being equal for example). Just thought you or some reader would be interested to know.

        • Adrian Rosebrock May 10, 2016 at 6:20 pm #

          Thanks for sharing, and great investigative work! This is certainly an interesting heuristic — however, it’s a heuristic that is easily broken. For certain situations though, this might work well enough.

  14. najib June 8, 2016 at 4:30 am #

    you can do the same program in c ++

  15. Amey June 15, 2016 at 6:23 am #

    Hello Adrian

    Thanks for a wonderful post on image stitching. This is kind of faster then
    C++ example I had tried earlier.

    I had a query though on the topic. How can you stitch 3 or 4 images to get one panrama image? What changes are needed in this code?

    Regards
    Amey

    • Adrian Rosebrock June 15, 2016 at 12:27 pm #

      Stitching > 2 images together is substantially harder than stitching 2 images together. The code has to change dramatically. I will have to write a separate blog post on this, but I’m honestly not sure when I’ll be able to.

  16. thenuka June 22, 2016 at 4:00 am #

    How to remove the black portion in stitching if you are going to stitch about 10 images together when kindda creating an aerial map

  17. Abdulkader July 23, 2016 at 4:59 am #

    Hi Adrian, I’m so grateful for that brilliant Tutorial … but I have a problem !
    when I try to run the code from Terminal nothing will be shown on screen although it gives NO error and –first/–second parameters are set perfectly .. I have the the latest version of OpenCV, is there any suggestion to help me, please my project should be finished in less than a week … wainting for ur reply … thanks again 🙂

    • Adrian Rosebrock July 27, 2016 at 2:48 pm #

      If nothing is getting displayed to your screen, then it sounds like the homography matrix isn’t being computed (in which case there are not enough keypoint matches). I would suggest inserting some print statements to help you debug where the code is existing.

  18. Sri July 25, 2016 at 4:29 pm #

    Awesome Mr. Adrian. Love you blog.
    I have a question. I am trying to run this program. Getting this error line 33
    result[0:imageB.shape[0],0:imageB.shape[1]] = imageB

    ValueError: could not broadcast input array from shape (302,400,3) into shape (300,400,3)

    • Adrian Rosebrock July 27, 2016 at 2:30 pm #

      Hey Sri — based on the error message, it looks like imageB has a different height than the result image. Try changing the code to this and see if it helps:

  19. Manuel August 4, 2016 at 8:25 pm #

    Hi Adrian,

    You are calling stitch.py with the left image as first argument, that becomes imageA, and you warp it with warpPerspective. But what you say in the text is that warpPerspective gets as input the image we want to warp: the right image.

    So I don’t know what I’m missing here.

    Thanks.

    PD: I’m going send you my previous comment tomorrow.

  20. Ahmad August 18, 2016 at 8:06 am #

    Very informative post. Thanks for it. I just wanted to know can you direct me to some post which is about spherical stitching. Actually i have 7 images from gopros (Including top and bottom) I want to create a spherical panorama from it. Any idea how that can be done.

    • Adrian Rosebrock August 18, 2016 at 9:24 am #

      I don’t know of any tutorials related to spherical/fish eye image stitching, but if I come across any, I’ll be sure to post them here.

      • Ahmad August 29, 2016 at 11:48 am #

        Thanks Adrian. That would be of great help.

      • Tirumaleswararao Gudivada March 1, 2017 at 3:12 pm #

        Hi Adrian,

        Great blog. Thank you for your valuable time. I learned a lot.
        I am also looking for algos to stitch fisheye images.
        It would be great if you can share any info you find relevant to this.

        Thanks,
        Tiru

  21. Felipe September 24, 2016 at 1:01 pm #

    Hi Adrian, excellent work with this example.
    I’m trying to implement image stitching from a video using this example , but I cannot make it work, none of images retrieves from video have successfuly stitched together. I’ve searched a lot of examples using OpenCV with Java, C# and Python. Please if you can send me some highlights to accomplish image stitching from a video it would be great!! thanks in advance 😉

    • Adrian Rosebrock September 27, 2016 at 8:51 am #

      Have you tried using my example on image stitching from video?

      • Felipe October 1, 2016 at 12:04 pm #

        Thank for your quickly response! 😉 I’m trying to do something like this: https://www.youtube.com/watch?v=93jOLlObfuE

        • Adrian Rosebrock October 2, 2016 at 8:59 am #

          Thanks for sharing. I’ll certainly use that video as inspiration when I create the future tutorial on > 2 image/frame stitching.

          • Felipe October 3, 2016 at 11:25 am #

            Awesome! I’ll be waiting for your tutorial to test it! thanks again! 😉

  22. Devin Willis October 6, 2016 at 9:59 am #

    Hey Adrian,
    I was wondering if it would be possible to take multiple images of a slide and stitch these together to create a very high resolution image of the whole slide. If so I was wondering why when I run the code it says segmentation fault (core dumped)

    • Adrian Rosebrock October 7, 2016 at 7:38 am #

      You’re likely trying to stitch together very large images and thus detecting TON of keypoints and local invariant descriptors (hence the memory error). Instead, load your large images into memory, but then resize them to be be a maximum of 600 pixels along their largest dimension. Perform keypoint detection and matching on the smaller images (we rarely process images larger than 600 pixels along their maximum dimension).

      Once you have your homography matrix, apply it to your original high resolution images.

      • Glenn January 7, 2017 at 9:42 pm #

        I am getting this error as well. It happens even when I use smaller images. It seems to be related to this line:
        kps = detector.detect(gray)

        • Adrian Rosebrock January 9, 2017 at 9:15 am #

          How small are your “smaller” images in terms of width and height? Also, which version of OpenCV are you using?

    • Lorena March 31, 2017 at 3:49 am #

      hello felipe you can see your code.
      thank you

  23. Chris October 6, 2016 at 3:04 pm #

    Great Tutorial this is awesome! I had an interesting idea. I was wondering if it was possible to combine photos of different sizes based on your code. As a project I want to use a video to create a panorama. Using OpenCV to parse through the frames I would stitch one photo to the combined strip. This would mean that the left/first photo would be a lot wider than the right/second photo. Just looking for your opinion. Also your book is great I have been using it for my research.

    • Adrian Rosebrock October 7, 2016 at 7:30 am #

      Your images certainly don’t have to be the same size or from the same camera sensor to be stitched together provided enough keypoints match; however, I think you’re going to get mixed results that may not be aesthetically pleasing. If one image has a different exposure than the other then you’ll need to correct the final image by applying image blending.

  24. sam October 21, 2016 at 4:47 am #

    Hay Andrian,

    Please post the multiple image stitching it would be great please…

  25. Metalim November 1, 2016 at 12:16 pm #

    hi Adrian,

    How can I evaluate quantitatively different feature descriptors and extractors performance on the same image pair? Which parameters can tell me about the accuracy of feature matching using different descriptors and extractors? Is Lowe ratio or repError doing that also?

    • Adrian Rosebrock November 3, 2016 at 9:56 am #

      If I understand your question right you are trying to determine which set of keypoint detectors and local invariant descriptors perform best for a given set of image? The Lowe ratio test is used to avoid false positive matches. On the other hand, the RANSAC method is used to actually spatially verify these matches.

  26. Chris November 17, 2016 at 3:40 pm #

    Hi Adrian,

    I was wondering if you happened to do the other blog post where you stitched multiple images together? I modified ur code from this example to linearly stitch images but am struggling to find a way to stitch images regardless of orientation. Any help is appreciated.

    Also If you know of any tutorials on how to build the opencv_conrib modules for opencv 3 on windows that would be a god send.

    Thanks for your help,

    Cheers.

    • Adrian Rosebrock November 18, 2016 at 8:53 am #

      Hey Chris, I haven’t had a chance to write up another blog post on image stitching. Most of my current posts have been related to deep learning and updated install tutorials for Ubuntu and Mac (I don’t support Windows on this blog). I’ll try to circle back to the image stitching post, but I honestly can’t say when that might be.

      • Chris November 20, 2016 at 1:31 pm #

        Thanks for getting back to me:)

        Cheers,

  27. Chris November 20, 2016 at 2:05 pm #

    Hi Adrian,

    Sorry for all the questions. I was curious as to why you used the imutils package to resize your images instead of the built in resize function in OpenCV? are there any advantages or tradeoffs?

    Thanks in advance.

    Cheers,

    • Adrian Rosebrock November 21, 2016 at 12:30 pm #

      My imutils.resize function automatically takes into account the aspect ratio of the image whereas the cv2.resize function does not.

  28. Hugo November 30, 2016 at 12:42 pm #

    Hi Adrian!

    First of all, thanks A LOT for that and all the other contents I’ve been using a lot since I got an interest in image processing with Python!

    I enjoy a lot both the quality and the pedagogy of your guides & solutions 🙂

    I have a question about this one I can’t find the answer of on my own: I’m trying to get a “reduction factor” of the homographies I compute.
    Which would mean something like: if a segment measures 10 pixels before warping, how long is it after warping. Because of deformation, there’s no unique value but I guess it could be possible to have the value range?
    I was thinking maybe calculate the total distance between all matching key points on image A then the total distance between all matching key points on image B and calculate the ratio of those 2 values? If there’s enough and well-reparted matching points that should give me an average reduction factor shouldn’t it?
    Unless there’s something built-in and more reliable with a cv2 function?

    Thanks again for everything 🙂

    • Adrian Rosebrock December 1, 2016 at 7:30 am #

      Hmmm, this is a good question. I’m not sure there is a built-in OpenCV function to do this. By definition applying a perspective transform is going to cause the images to be warped. I guess my question is what are you trying to accomplish by computing this value?

  29. Julio December 14, 2016 at 12:35 pm #

    Hello, could you explain how to use SURF with RANSAC but without using the cv2.findHomography () function because I want to use the cv2.getaffinetransform

    Thanks

  30. Christopher Daniel December 20, 2016 at 11:14 am #

    hello,

    I think we don’t have to use imutils I executed code without imutils and it works fine and quality is also good compared to imutils substituted input image

    • Adrian Rosebrock December 21, 2016 at 10:23 am #

      In this example imutils is used to (1) determine your OpenCV version and (2) resize your image. You can modify this code to not use imutils, but it is highly recommended.

      • Christopher Daniel December 25, 2016 at 2:41 am #

        thank you if images are resized only by width I got broadcast error. So I removed the lines
        imageA = imutils.resize(imageA, width=400)
        imageB = imutils.resize(imageB, width=400)
        and worked fine

        then I tried
        imageA = imutils.resize(imageA, width=400,height=350)
        imageB = imutils.resize(imageB, width=400,height=350) and worked fine

        it worked fine.May be you can add this change to your tutorial

  31. Chris January 15, 2017 at 1:36 pm #

    Thanks again for all your effort into building this OpenCV community. I had a question regrading the photos themselves. You implementation requires a certain order for the images to be piped into your program. TO my understanding this is due to the warp perspective and the way you link the images from line 31-33 in panorama.py. I was wondering if there is a way to modify that section of code to allow for any order of images to be passed through (I’ve been trying my own thing to no avail)?

    Thanks for any help in advance.

    Chris

    • Adrian Rosebrock January 16, 2017 at 8:13 am #

      You are correct, we assume a specific order passed to the function — this is by far the easiest method. For multiple images you actually keypoint matching on all the frames and then define a “path finding” algorithm that can order the frames. But for two images, I think it’s much easier to define a function that expects the images in a specific order.

      • Chris January 16, 2017 at 2:34 pm #

        Do you know where I might be able to find out more information regarding these “path finding” algorithms to allow me to input an unordered image set?

        Again thank you for all your help.

        • Adrian Rosebrock January 17, 2017 at 8:50 am #

          Take a look at Dijkstra’s algorithm and dynamic programming to start. I don’t know of any examples off the top of my head that implement this explicitly for image stitching. But again, this will (ideally) be a topic that I’ll cover in a future PyImageSearch post, I’m just not sure when.

  32. Saswati January 30, 2017 at 7:02 am #

    Hi,
    I am new to opencv and image processing. Please anybody help me to solve this error. I am even unable to figure out its meaning.

    usage: stitch.py [-h] -f FIRST -s SECOND
    stitch.py: error: argument -f/–first is required

  33. Judy February 1, 2017 at 6:45 am #

    Would it be possible to create a solution with this that stitches stereo video? Working on a project for school and looking for some insight. Thanks.

    • Adrian Rosebrock February 1, 2017 at 12:45 pm #

      Yes, absolutely. In fact, I’ve already done a blog post on the topic.

      • Judy February 3, 2017 at 3:14 pm #

        Thank you for your response! I actually did find that blog post later, and you’ll notice that I made a comment on there as well, I’m curious about how one might go about streaming the stitched video to something like a VR headset, or just streaming it in general. Just looking for some ideas.

        Thanks again!

        • Adrian Rosebrock February 4, 2017 at 9:26 am #

          I don’t do any work with VR headsets; however, there are a number of different streaming protocols. I would suggest doing research on your particular headset and see if it’s possible to stream the frames from the headset itself. Regarding streaming in general, I plan to cover that in a future blog post, although I’m not entirely sure when that will be.

  34. JJolles February 6, 2017 at 9:56 am #

    Thanks Adrian for a very clear tutorial. I was wondering if you could help me a bit further how to stitch four images together (2×2 grid), or guide me in the right direction. Basically I have four cameras in a grid filming a large space and which overlap by 1/5th of the space on the sides and top/bottom. I got your script working on two side-by-side images, but how could I adapt your script to stitch all four images together? Is there an easy way to adapt your script by first stitching the top 2 and then the bottom 2 and then stitching those new top and bottom images together? Any help is appreciated. I will then actually try and use the stiching parameters to use in stitching frames from the video together.
    Thanks again!

    • Adrian Rosebrock February 7, 2017 at 9:11 am #

      Are these cameras fixed and non-moving? If so it might be easier to simply calibrate the cameras and perform projection that way.

      • gwhiz February 14, 2017 at 2:03 am #

        Hi Adrian,

        We are also looking for a way to stitch video from multiple cameras together. IN our arrangment, the cameras are pointing from the edge of the space towards the center of the space, the opposite from most rigs today. This arrangment is basically bullet time. If we align our cameras to have the same centers (for example put one on each of four walls in a room at the same x/y on each wall) can we stitch the video together?

      • JJolles February 20, 2017 at 12:02 pm #

        Thanks, I hadn’t seen you reply hence the delay. Yes the cameras are fixed and non-moving. I calibrated them to account for any distortion of the lenses. But then using your script on just the top two cameras it does warp the right camera based on the left. I would like all four to be stitched with all having the ability to be warped. Do you have any further tips, also what you mean regarding calibrating the cameras? I appreciate it!

        • Adrian Rosebrock February 22, 2017 at 1:45 pm #

          I don’t have any tutorials on camera calibration here on the PyImageSearch blog, but I would suggest starting here.

  35. Nilavro February 10, 2017 at 5:34 pm #

    Hi Adrain, thanks for the code and guide.

    I was wondering how may I perform a cylindrical/inverse cylindrical projection before of the candidate images to be stitched together. This will help me to stitch multiple images taken from a rotating base.

    Thanks again.

    • Adrian Rosebrock February 13, 2017 at 1:53 pm #

      Absolutely. I will try to cover cylindrical projections in a future blog post.

  36. Daniel March 8, 2017 at 10:20 pm #

    Hi Adrian,
    Thanks for your sharing! Now the output panorama is left photo overlaps right photo. I am wondering which part of code should I change to make right photo overlaps left photo? Thanks!

    • Adrian Rosebrock March 10, 2017 at 3:55 pm #

      You would want to swap the left and right images on Lines 31-33.

  37. Mathew Orman March 16, 2017 at 10:19 am #

    Does not work on Raspberry Pi 3 but it works on PC windows 10
    Any ideas?

  38. Mathew Orman March 16, 2017 at 10:58 am #

    AT first it complained about an extra agument showMatches in stitch call.
    Then I removed it and now it say “Segmentation fault” on 320 x 240 images…
    I use python 2.7…

    • Adrian Rosebrock March 17, 2017 at 6:43 am #

      The segmentation fault is helpful, it’s likely that this is a local invariant descriptor issue. I would start to debug the problem by inserting print statements throughout your code until you can narrow down exactly which line is causing the seg-fault.

      Also, make sure you use the “Downloads” section of the post to download the code (if you haven’t done so already) rather than copying and pasting. This will ensure you are using the same codebase and project structure.

  39. Mathew Orman March 16, 2017 at 11:13 am #

    I am using Python v. 2.7 and cv2 v. 2.4.9.1

  40. Carol April 8, 2017 at 12:02 pm #

    Hey Adrian,
    I’ve see your tutorial page that shows how to install OpenCV 3 with opencv_contrib support enabled, but I didn’t see the way for windows, can you please upload one?
    Thank you so much!

    • Adrian Rosebrock April 8, 2017 at 12:33 pm #

      Hi Carol — I actually don’t have any Windows install tutorials. Sorry about that! When it comes to computer vision and OpenCV, I highly recommend that you use a Unix-based environment.

  41. Enkhbold Munkhbold April 11, 2017 at 4:25 am #

    Hello Adrian,
    I’m having this error message:

    usage: stitch.py [-h] -f FIRST -s SECOND
    stitch.py: error: argument -f/–first is required

    When i try and run the stitch.py
    I extracted the files to a folder and just ran it on Python2.7.13
    Did i do something wrong ? it’s not working.

    • Adrian Rosebrock April 12, 2017 at 1:08 pm #

      You need to read up on command line arguments before you continue.

      • Enkhbold Munkhbold April 21, 2017 at 2:14 am #

        I still don’t quite get it.
        Can you tell me what to do step by step after downloading the zip file?
        When and if you get a chance. It would be really helpful.
        I personally do not know much of python since i was taught on Java and a little bit of C++.

        I got on the command line and went to the file (i put it on my desktop and did “cd desktop => panorama-stitching” and tried to run stitch.py) and had that trouble.
        and doing “stitch.py bryce_left_02.png bryce_right_02.png” had the same result.
        Thanks.

        • Adrian Rosebrock April 21, 2017 at 10:48 am #

          Hi Enkhbold — while I’m happy to help point readers in the right direction, the PyImageSearch blog does assume you’ve had some basic programming experience and are comfortable working with the command line. I can’t provide step-by-step instructions on how to execute a Python script. I would recommend you go through my list of recommended Python resources to help you learn the basics first.

          • Enkhbold Munkhbold April 28, 2017 at 1:37 am #

            Was able to make it work (found the comment on top of the stitch.py) i was trying “python stitch.py –first images/bryce_left_01.png \ –second images/bryce_right_01.png” from the picture above and what made it not work was that backslash.
            Now i want to stitch multiple images can you provide the page you talked about that if you made it ?
            Me and my teacher are trying to make a program that constantly reads images from a video and saves it into a big panoramic picture to make it easier for watchers to see what was on a few secs or way back whenever they want.(the idea came from MIT lectures and because the video can only show so much, or it would make things impossible to read since its too small)
            So again, Is there a multiple image stitching method you made that i can take a look at ?

          • Adrian Rosebrock April 28, 2017 at 9:23 am #

            Hi Enkhbold — I have not written a tutorial on stitching multiple images together yet. I’m not sure when I will get to it, but I will try to cover it in the future.

  42. Daniella Solomon April 17, 2017 at 7:13 am #

    Hi,
    There is a reason that you sending in stitcher line 55 the RGB image and not the gray one? (OpenCV3 option)

    attached:
    (kps, features) = descriptor.detectAndCompute(image, None)

    instead of
    (kps, features) = descriptor.detectAndCompute(gray, None)

    I’m trying to do something similar with videos, I have two cameras, one PTZ and one wide and I’m drawing a rectangular on the wide one of what the PTZ is showing and it’s really slow, I tried to use threads but still not close to real time. any suggestion?

    • Adrian Rosebrock April 17, 2017 at 10:50 am #

      Thank you for pointing out this typo — we typically detect keypoints and extract local invariant descriptors from grayscale rather than multi-channel images. The gray image should be used instead, but in most cases you won’t notice any changes in performance.

  43. Esraa Elbasha April 28, 2017 at 11:20 am #

    I want to control a servo depending on the overlap between two pictures .. is no overlap I will rotate the servo till there is .. I tried to push two pictures of different places and it still finds keypoints .. how can I prevent that or at least increase the occuracy

  44. Esraa Elbasha April 29, 2017 at 1:51 am #

    is there anyway to know the overlap % between two pictures !?

    • Adrian Rosebrock May 1, 2017 at 1:43 pm #

      Yes, absolutely. A good way to approximate the overlap is to match the keypoints. Once you have the (x, y)-coordinates of the matched keypoints in both images you can determine how much of image #1 overlaps with image #2.

  45. Mohsen khoshnazar May 21, 2017 at 1:34 am #

    Hello Adrian and thanks for your post.
    Is that necessary using Homograpy? Or it just use for better resaults?
    Thanks.

    • Adrian Rosebrock May 21, 2017 at 5:06 am #

      It is necessary to compute the homography as this allows us to estimate the affine transformation.

      • Mohsen khoshnazar May 21, 2017 at 5:18 am #

        Thanks for your answer. Yes its right, because we have more than 4 points. Thanks again Adrian.

  46. Cristian June 12, 2017 at 8:55 pm #

    Hi Adrian, very good job. I want to know how to make the mosaic but without reducing the quality of the images and the resulting mosaic. Greetings.

    • Adrian Rosebrock June 13, 2017 at 10:55 am #

      Apply keypoint detection, local invariant descriptors, and RANSAC to the resized images, but then apply the homography transfer to the original, not resized images.

      • Cristian June 13, 2017 at 11:47 am #

        Thank you very much, it worked for me but half of the mosaic is presented (I work with four aerial images). In other words, I do not work with a horizontal panorama, rather a square panorama.

  47. Daniel June 17, 2017 at 10:55 am #

    Hi Adrian, Thank you for the tutorial. I wonder how to warp the left image to match the right image instead of warping the rigtht one to match the left one. Thank you.

    • Daniel June 17, 2017 at 10:59 am #

      I tried to change the parameter of warpPerspective from imageA to imageB and cover the columns 400:800 with imageA, but the left image does not stitch to the right one

      • Adrian Rosebrock June 20, 2017 at 11:12 am #

        Make sure you update Line 33 where the images are actually stitched together.

  48. Lorena June 18, 2017 at 11:16 am #

    Line 33 what does it represent me ??

    • Adrian Rosebrock June 20, 2017 at 11:06 am #

      Line 33 is responsible for stitching the actual images together. It accomplishes this via NumPy array slicing.

Leave a Reply