Detecting machine-readable zones in passport images

mrz_output_04

Today’s blog post wouldn’t be possible without PyImageSearch Gurus member, Hans Boone. Hans is working on a computer vision project to automatically detect Machine-readable Zones (MRZs) in passport images — much like the region detected in the image above.

The MRZ region in passports or travel cards fall into two classes: Type 1 and Type 3. Type 1 MRZs are three lines, with each line containing 30 characters. The Type 3 MRZ only has two lines, but each line contains 44 characters. In either case, the MRZ encodes identifying information of a given citizen, including the type of passport, passport ID, issuing country, name, nationality, expiration date, etc.

Inside the PyImageSearch Gurus course, Hans showed me his progress on the project and I immediately became interested. I’ve always wanted to apply computer vision algorithms to passport images (mainly just for fun), but lacked the dataset to do so. Given the personal identifying information a passport contains, I obviously couldn’t write a blog post on the subject and share the images I used to develop the algorithm.

Luckily, Hans agreed to share some of the sample/specimen passport images he has access to — and I jumped at the chance to play with these images.

Now, before we get to far, it’s important to note that these passports are not “real” in the sense that they can be linked to an actual human being. But they are genuine passports that were generated using fake names, addresses, etc. for developers to work with.

You might think that in order to detect the MRZ region of a passport that you need a bit of machine learning, perhaps using the Linear SVM + HOG framework to construct an “MRZ detector” — but that would be overkill.

Instead, we can perform MRZ detection using only basic image processing techniques such as thresholdingmorphological operations, and contour properties. In the remainder of this blog post, I’ll detail my own take on how to apply these methods to detect the MRZ region of a passport.

Looking for the source code to this post?
Jump right to the downloads section.

Detecting machine-readable zones in passport images

Let’s go ahead and get this project started. Open up a new file, name it detect_mrz.py , and insert the following code:

Lines 2-6 import our necessary packages. I’ll assume you already have OpenCV installed. You’ll also need imutils, my collection of convenience functions to make basic image processing operations with OpenCV easier. You can install imutils  using pip :

From there, Lines 9-11 handle parsing our command line argument. We only need a single switch here, --images , which is the path to the directory containing the passport images we are going to process.

Finally, Lines 14 and 15 initialize two kernels which we’ll later use when applying morphological operations, specifically the closing operation. For the time being, simply note that the first kernel is rectangular with a width approximately 3x larger than the height. The second kernel is square. These kernels will allow us to close gaps between MRZ characters and openings between MRZ lines.

Now that our command line arguments are parsed, we can start looping over each of the images in our dataset and process them:

Lines 20 and 21 loads our original image from disk and resizes it to have a maximum height of 600 pixels. You can see an example of an original image below:

Figure 1: Our original passport image that we are trying to detect the MRZ in.

Figure 1: Our original passport image that we are trying to detect the MRZ in.

Gaussian blurring is applied on Line 26 to reduce high frequency noise. We then apply a blackhat morphological operation to the blurred, grayscale image on Line 27.

A blackhat operator is used to reveal dark regions (i.e., MRZ text) against light backgrounds (i.e., the background of the passport itself). Since the passport text is always black on a light background (at least in terms of this dataset), a blackhat operation is appropriate. Below you can see the output of applying the blackhat operator:

Figure 2: Applying the blackhat morphological operator reveals the black MRZ text against the light passport background.

Figure 2: Applying the blackhat morphological operator reveals the black MRZ text against the light passport background.

The next step in MRZ detection is to compute the gradient magnitude representation of the blackhat image using the Scharr operator:

Here we compute the Scharr gradient along the x-axis of the blackhat image, revealing regions of the image that are not only dark against a light background, but also contain vertical changes in the gradient, such as the MRZ text region. We then take this gradient image and scale it back into the range [0, 255] using min/max scaling:

Figure 3: Applying Scharr operator to our blackhat image reveals regions that contain strong vertical changes in gradient.

Figure 3: Applying Scharr operator to our blackhat image reveals regions that contain strong vertical changes in gradient.

While it isn’t entirely obvious why we apply this step, I will say that it’s extremely helpful in reducing false-positive MRZ detections. Without it, we can accidentally mark embellished or designed regions of the passport as the MRZ. I will leave this as an exercise to you to verify that computing the gradient of the blackhat image can improve MRZ detection accuracy.

The next step is to try to detect the actual lines of the MRZ:

First, we apply a closing operation using our rectangular kernel. This closing operation is meant to close gaps in between MRZ characters. We then apply thresholding using Otsu’s method to automatically threshold the image:

Figure 4: Applying a closing operation using a rectangular kernel (that is wider than it is tall) to close gaps in between the MRZ characters

Figure 4: Applying a closing operation using a rectangular kernel (that is wider than it is tall) to close gaps in between the MRZ characters

As we can see from the figure above, each of the MRZ lines is present in our threshold map.

The next step is to close the gaps between the actual lines, giving us one large rectangular region that corresponds to the MRZ:

Here we perform another closing operation, this time using our square kernel. This kernel is used to close gaps between the individual lines of the MRZ, giving us one large region that corresponds to the MRZ. A series of erosions  are then performed to break apart connected components that may have been joined during the closing operation. These erosions are also helpful in removing small blobs that are irrelevant to the MRZ.

Figure 5: A second closing operation is performed, this time using a square kernel to close the gaps in between individual MRZ lines.

Figure 5: A second closing operation is performed, this time using a square kernel to close the gaps in between individual MRZ lines.

For some passport scans, the border of the passport may have become attached to the MRZ region during the closing operations. To remedy this, we set 5% of the left and right borders of the image to zero (i.e., black):

You can see the output of our border removal below.

Figure 6: Setting 5% of the left and right border pixels to zero, ensuring that the MRZ region is not attached to the scanned margin of the passport.

Figure 6: Setting 5% of the left and right border pixels to zero, ensuring that the MRZ region is not attached to the scanned margin of the passport.

Compared to Figure 5 above, you can now see that the border has been removed.

The last step is to find the contours in our thresholded image and use contour properties to identify the MRZ:

On Line 56-58 we compute the contours (i.e., outlines) of our thresholded image. We then take these contours and sort them based on their size in descending order on Line 59 (implying that the largest contours are first in the list).

On Line 62 we start looping over our sorted list of contours. For each of these contours, we’ll compute the bounding box (Line 66) and use it to compute two properties: the aspect ratio and the coverage ratio. The aspect ratio is simply the width of the bounding box divided by the height. The coverage ratio is the width of the bounding box divided by the width of the actual image.

Using these two properties we can make a check on Line 72 to see if we are examining the MRZ region. The MRZ is rectangular, with a width that is much larger than the height. The MRZ should also span at least 75% of the input image.

Provided these two cases hold, Lines 75-84 use the (x, y)-coordinates of the bounding box to extract the MRZ and draw the bounding box on our input image.

Finally, Lines 87-89 display our results.

Results

To see our MRZ detector in action, just execute the following command:

Below you can see of an example of a successful MRZ detection, with the MRZ outlined in green:

Figure 7: On the left, we have our input image. And on the right, we have the MRZ region that has been successfully detected.

Figure 7: On the left, we have our input image. And on the right, we have the MRZ region that has been successfully detected.

Here is another example of detecting the Machine-readable Zone in a passport image using Python and OpenCV:

Figure 8: Applying MRZ detection to a scanned passport.

Figure 8: Applying MRZ detection to a scanned passport.

It doesn’t matter if the MRZ region is at the top or the bottom of the image. By applying morphological operations, extracting contours, and computing contour properties, we are able to extract the MRZ without a problem.

The same is true for the following image:

Figure 9: Detecting machine-readable zones in images using computer vision.

Figure 9: Detecting machine-readable zones in images using computer vision.

Let’s give another image a try:

Figure 10: Again, we are able to detect the MRZ in the passport scan using basic image processing techniques.

Figure 10: Again, we are able to detect the MRZ in the passport scan using basic image processing techniques.

Up until now we have only seen Type 1 MRZs that contain three lines. However, our method works just as well with Type 3 MRZs that contain only two lines:

Figure 11: Detecting the MRZ in a Type 3 passport image using Python and OpenCV.

Figure 11: Detecting the MRZ in a Type 3 passport image using Python and OpenCV.

Here’s another example of detecting a Type 3 MRZ:

Figure 12: Applying computer vision and image processing to detect machine-readable zones in images.

Figure 12: Applying computer vision and image processing to detect machine-readable zones in images.

Summary

In this blog post we learned how to detect Machine-readable Zones (MRZs) in passport scans using only basic image processing techniques, namely:

  • Thresholding.
  • Gradients.
  • Morphological operations (specifically, closings and erosions).
  • Contour properties.

These operations, while simple, allowed us to detect the MRZ regions in images without having to rely on more advanced feature extraction and machine learning methods such as Linear SVM + HOG for object detection.

Remember, when faced with a challenging computer vision problem — always consider the problem and your assumptions! As this blog post demonstrates, you might be surprised what basic image processing functions used in tandem can accomplish.

Once again, a big thanks to PyImageSearch Gurus member, Hans Boone, who supplied us with these example passport images! Thanks Hans!

Downloads:

If you would like to download the code and images used in this post, please enter your email address in the form below. Not only will you get a .zip of the code, I’ll also send you a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL! Sound good? If so, enter your email address and I’ll send you the code immediately!

, , , , ,

56 Responses to Detecting machine-readable zones in passport images

  1. Johnny Johnson December 8, 2015 at 9:51 am #

    Nicely done sir,. thank you for sharing.

  2. Rameses December 10, 2015 at 10:52 am #

    Many thanks for your excellent blog, Adrian. Regarding this post, will it work on OpenCV3 and Python 3, or is the source code for OpenCV2.x and Python 2.7?

    • Adrian Rosebrock December 10, 2015 at 2:24 pm #

      This code will work for both OpenCV 2.4 and OpenCV 3, along with both Python 2.7 and Python 3.

  3. Kenny January 2, 2016 at 2:23 pm #

    Awesome Adrian! Thanks for sharing!

  4. Ghassan February 16, 2016 at 3:50 am #

    Is there is away to unencrypt the MRZ via python ?
    like getting name , surname , passport NO … etc

    • Adrian Rosebrock February 16, 2016 at 3:39 pm #

      I personally do not know of any, but at the same time I don’t do a lot of work with passports.

      • Don January 5, 2017 at 5:27 pm #

        There is a Python package called PassportEye that will recognize the MRZ, then OCR and parse the fields. The code does reference this blog post, but some of the image samples here do not work with his code.

        https://pypi.python.org/pypi/PassportEye

        • Adrian Rosebrock January 7, 2017 at 9:35 am #

          Thank you for sharing Don!

  5. CWard March 22, 2016 at 1:36 pm #

    Just a simple question : who is your ID specimen provider ?
    Thank you by advance

  6. chourng November 24, 2016 at 11:31 pm #

    Thank for sharing.

    Please convert it in c++ code.

  7. yang2472 March 15, 2017 at 10:42 am #

    hello i am very new, but i cant run this code. I tried to type: python detect_mrz.py –images examples in the command line but it says invalid syntax. I saved all the images to the folder named example and put it under my project path. plz help

    • Adrian Rosebrock March 17, 2017 at 9:37 am #

      Make sure you use the “Downloads” section of this post to download the source code + example images instead of trying to copy and paste the code and download the images yourself. This will ensure your project structure is the same as mine.

  8. dan May 9, 2017 at 7:07 pm #

    Hi Adrian,

    please tell me what is the purpose of this line:

    image = imutils.resize(image, height=600)

    Thank You

    • Adrian Rosebrock May 11, 2017 at 8:50 am #

      That line of code resizes the image to have a height of 600 pixels, maintaining the aspect ratio.

  9. Gabriel May 13, 2017 at 3:53 pm #

    Hi Adrian,

    I got lost in the ” gradX = (255 * ((gradX – minVal) / (maxVal – minVal))).astype(“uint8”) “.
    If you cv2.imshow gradX after the np.absolute, you’ll see that in the MRZ zone, the letters (and almost everything) are just white spots without too much shape.
    How was that operation able to create the result on figure 3?

    Thank you.

    PS: your OpenCV + case studies book is very nice and this blog is amazing!

    • Adrian Rosebrock May 15, 2017 at 8:45 am #

      After computing the gradient representation of the image, we need to scale it back to the range [0, 255] by applying “min-max scaling”. We then convert the data type from a float to an unsigned 8-bit integer which is what images are typically represented as when displaying them to our screen. Other OpenCV functions also assume an 8-bit unsigned integer data type.

  10. Andrey May 15, 2017 at 8:39 am #

    Hello Adrian! Great Work!

    Can you tell me what means ‘None’ in:
    thresh = cv2.erode(thresh, None, iterations=4) line.
    It must be kernel or some but what kernel will be passed if its argument is ‘None’ ?

    • Adrian Rosebrock May 15, 2017 at 9:07 am #

      The None here indicates that a default 3×3 structuring kernel should be used.

  11. Andrey May 16, 2017 at 11:54 pm #

    Adrian, can you tell me why image needs to be resized to 600 height?

    • Adrian Rosebrock May 17, 2017 at 9:50 am #

      When processing an input image, we rarely work with images that are larger than 600-800 pixels along their maximum dimension. The extra detail in high resolution images may look appeasing to the human eye, but the only “confuse” computer vision and image processing algorithms. The less detail there is, the easier it is for these algorithms to focus on the “structural” components of the image.

  12. Deepak joshi June 22, 2017 at 12:51 pm #

    hello, I’m working on similar project and need your help to sort out somethings.

    can you get in touch with me?

    i would really be glad.

  13. eleman September 27, 2017 at 1:22 pm #

    Hey Adrian am wondering if you can make a tutorial about this Extracting personal information on a passport such as owner’s photo, first / last name, birth date, document ID number, issue / expiration date, place of issue

    • Adrian Rosebrock September 28, 2017 at 9:11 am #

      I will consider it. In the mean time, I would suggest you read this post on OCR.

  14. Irma October 6, 2017 at 9:36 am #

    Hi Adrian,

    What are the arguments “-i” and “–image” that I need to execute ap.add_argument(“-i”, “–images”, required=True, help=”path to images directory”)?

  15. Felipe October 10, 2017 at 4:44 pm #

    Hello, good article.
    I have questions, why the rectKernet size is (13, 5) and sqKernel(21,21)?. how you calculate that?
    Does it depends on input image size?

    • Adrian Rosebrock October 13, 2017 at 9:03 am #

      It depends on your image size. All images hear are resized to have a height of 600 pixels. I fiddled with the kernel sizes experimentally after the image was resized to 600 pixels. This is common practice when building basic image processing workflows.

  16. Dimitris December 7, 2017 at 4:35 am #

    I have seen this in other people’s implementations, also in the openCV tutorials :
    thresh = cv2.threshold(gradX, 0, 255, cv2.THRESH_BINARY | cv2.THRESH_OTSU)[1]

    Is there a specific reason for ORing the threshold parameters :
    cv2.THRESH_BINARY | cv2.THRESH_OTSU ?

    • Adrian Rosebrock December 8, 2017 at 4:52 pm #

      Yes, we take the bitwise OR on the two flags to indicate that we are doing both binary thresholding using Otsu’s method.

  17. Alex January 20, 2018 at 11:11 am #

    Hi Adrian,
    Can you explain me please these lines?

    # during thresholding, it’s possible that border pixels were
    # included in the thresholding, so let’s set 5% of the left and
    # right borders to zero
    p = int(image.shape[1] * 0.05)
    thresh[:, 0:p] = 0
    thresh[:, image.shape[1] – p:] = 0

    Thanks, Alex

    • Adrian Rosebrock January 22, 2018 at 6:31 pm #

      To start we compute 5% of the width. Then set all pixels that are within 5% of the width on either side to zero. We do this because we know the MRZ cannot be along the width borders of the image.

  18. Akhtar Ali April 24, 2018 at 7:33 am #

    Please provide for Android MRZ scanner

    • Adrian Rosebrock April 25, 2018 at 5:42 am #

      I don’t have any plans on building an Android MRZ scanner but if you implement this method in Java + OpenCV I am confident that you can do it 🙂

  19. Jeremiah May 19, 2018 at 9:39 am #

    Nice stuff.

    • Adrian Rosebrock May 22, 2018 at 6:21 am #

      Thanks Jeremiah, I’m glad you liked it.

  20. lizard July 14, 2018 at 12:46 pm #

    facing error
    File “../../Downloads/mrz/detect_mrz.py”, line 90, in
    cv2.imshow(“ROI”, roi)
    NameError: name ‘roi’ is not defined

    • Adrian Rosebrock July 17, 2018 at 7:34 am #

      Make sure you are using the “Downloads” section of this blog post to download the source code and example images. This error was likely introduced through a copy and paste issue.

  21. ahmad August 12, 2018 at 9:34 am #

    Many thanks for your excellent this post, Adrian.
    This script works very well.
    But there are some problems with Iraqi and Yemeni passports.
    In the Iraqi passport, there is a barcode box above mrz. Which makes a mistake when calculating
    Do you have a solution?
    The Iraqi passport sample is at the following link:
    https://www.oaths.ca/BPx2ah1CcAALs9X-1.jpg

    Thanks

    • Adrian Rosebrock August 15, 2018 at 8:57 am #

      What specifically are the “mistakes” the script is making? You’ll likely need to modify the heuristics of the script to crop the location the MRZ is supposed to be.

      • ahmad August 17, 2018 at 11:52 pm #

        I want to just select mrz.
        However, the barcode box (top box mrz) and mrz are chosen together!!!

        • Adrian Rosebrock August 22, 2018 at 10:20 am #

          What is the “top box MRZ”? Perhaps you could share an image of your output?

  22. Benedict August 16, 2018 at 9:13 pm #

    If I am not mistaken, this method works for MRZ is essentially due to two features. (1) the MRZ is black while the background is white, and (2) MRZ text is packed in a relatively large rectangular shape.

    So how about credit card number? Those numbers are “3D” and the background of the card varies a lot depending on the issuing bank. I have tried out your credit card sample code, but those works for template card only.

    Are there any suggestions on what kinds of features we should look into, so that we can readout the credit card number (or at least find the ROI). Thanks.

    • Adrian Rosebrock August 17, 2018 at 7:16 am #

      Hey Benedict — great question, thanks for asking. This coming Monday I’ll be publishing a blog post on “text detection” in natural scene images. Keep an eye out for it as it will address your exact question.

  23. Saurabh August 31, 2018 at 5:52 am #

    How can i apply the same technique to find machine readable zones in a document image.
    One solution maybe to compare the height and width of the contour to detect paragraphs and lines but also detects a few unwanted contours.

  24. Leks September 12, 2018 at 5:19 am #

    Great article. How can we apply perspective transform to the ROI rectangle?

    • Adrian Rosebrock September 12, 2018 at 1:55 pm #

      Follow the steps in this post.

  25. Leks September 13, 2018 at 5:07 am #

    Hi Adrian, thanks for the link , the four_point_transform method takes points as parameter but here we have ROI rectangle. Do i have to process the ROI with all the steps in the suggested post?

    • Adrian Rosebrock September 14, 2018 at 9:38 am #

      That depends on what the contents of your ROI is. What specifically is in your ROI?

  26. Leks September 14, 2018 at 9:41 am #

    My ROI is a MAT obtained using frame.submat(rectangle).

    • Adrian Rosebrock September 14, 2018 at 9:54 am #

      But what does the ROI actually contain? What are the visual contents of the ROI?

  27. Leks September 14, 2018 at 10:15 am #

    The visual content is the MRZ part. as described in your post. I’ve adapted your code to java.

    • Adrian Rosebrock September 17, 2018 at 3:03 pm #

      I think you may want to go back and refactor how you’re extracting the ROI. If you can find the original ROI, assuming you’re doing so programmatically, just compute the rotated bounding box of the ROI (as we did in the original tutorial I linked you to), and then perform the perspective transform. There really isn’t a reason to apply a perspective transform after you’ve already extracted the ROI, it just makes the pipeline more complicated.

  28. Leks October 29, 2018 at 10:27 am #

    Hi Adrian,

    I’m wondering if you plan to make a tutorial to OCR the MRZ. Preprocessing the MRZ to get good results. I’ve tried with Tesseract but the result varies since I use live camera and i guess i need to thin the character too.

    • Adrian Rosebrock October 29, 2018 at 1:03 pm #

      Hey Leks — I don’t have any plans to cover OCR’ing the MRZ but I will consider it for a future tutorial (I don’t know if/when I will cover it though).

  29. M Ahsan November 12, 2018 at 2:08 pm #

    Thanks, Adrian
    The code is running fine but it is only getting the first line, not both.Need help please

Leave a Reply