My Top 9 Favorite Python Libraries for Building Image Search Engines

When I first became interested in computer vision and image search engines over eight years ago, I had no idea where to start. I didn’t know which language to use, I didn’t know which libraries to install, and the libraries I found I didn’t know how to use. I WISH there had been a list like this, detailing the best libraries to use for image processing, computer vision, and image search engines.

This list is by no means complete or exhaustive. It’s just my favorite Python libraries that I use each and everyday for computer vision and image search engines. If you think I’ve left an important one out, please leave me a note in the comments or send me an email.

For Starters:

1. NumPy

NumPy is a library for the Python programming language that (among other things) provides support for large, multi-dimensional arrays. Why is that important? Using NumPy, we can express images as multi-dimensional arrays. For example, let’s say we downloaded our favorite grumpy cat image from Google and we  now want to represent it as an array using NumPy. This image is 452×589 pixels in the RGB color space, where each pixel is consists of three components: a red value, a green value, and a blue value. Each pixel value p is in the range 0 <= p <= 255. We then have a matrix of 452×589 such pixels. Using NumPy, we can conveniently store this image in a (452, 593, 3) array, with 452 rows, 593 columns, and 3 values, one for each pixel in the RGB colorspace. Representing images as NumPy arrays is not only computational and resource efficient, but many other image processing and machine learning libraries use NumPy array representations as well. Furthermore, by using NumPy’s built-in high-level mathematical functions, we can quickly perform numerical analysis on an image.

2. SciPy

Going hand-in-hand with NumPy, we also have SciPy. SciPy adds further support for scientific and technical computing. One of my favorite sub-packages of SciPy is the spatial package which includes a vast amount of distance functions and a kd-tree implementation. Why are distance functions important? When we “describe” an image, we perform feature extraction. Normally after feature extraction an image is represented by a vector (a list) of numbers. In order to compare two images, we rely on distance functions, such as the Euclidean distance. To compare two arbitrary feature vectors, we simply compute the distance between their feature vectors. In the case of the Euclidean distance, the smaller the distance the more “similar” the two images are.

3. matpotlib

Simply put, matplotlib is a plotting library. If you’ve ever used MATLAB before, you’ll probably feel very comfortable in the matplotlib environment. When analyzing images, we’ll make use of matplotlib, whether plotting the overall accuracy of search systems or simply viewing the image itself, matplotlib is a great tool to have in your toolbox.

4. PIL and Pillow

I don’t have anything against PIL or Pillow, don’t get me wrong, they are very good at what to do: simple image manipulations, such as resizing, rotation, etc. Overall, I just find the syntax clunky. That being said, many non-scientific Python projects utilize PIL or Pillow. For example, the Python web-framework Django uses PIL to represent an ImageField in a database. PIL and Pillow have their place if you need to do some quick and dirty image manipulations, but if you’re serious about learning about image processing, computer vision, and image search engines, I would highly recommend that you spend your time playing with OpenCV and SimpleCV instead.

My Go-To’s:

5. OpenCV

If NumPy’s main goal is large, efficient, multi-dimensional array representations, then, by far, the main goal of OpenCV is real-time image processing. This library has been around since 1999, but it wasn’t until the 2.0 release in 2009 did we see the incredible NumPy support. The library itself is written in C/C++, but Python bindings are provided when running the installer. OpenCV is hands down my favorite computer vision library, but it does have a learning curve. Be prepared to spend a fair amount of time learning the intricacies of the library and browsing the docs (which have gotten substantially better now that NumPy support has been added). If you are still testing the computer vision waters, you might want to check out the SimpleCV library mentioned below, which has a substantially smaller learning curve.

6. SimpleCV

The goal of SimpleCV is to get you involved in image processing and computer vision as soon as possible. And they do a great job at it. The learning curve is substantially smaller than that of OpenCV, and as their tagline says, “it’s computer vision made easy”. That all said, because the learning curve is smaller, you don’t have access to as many of the raw, powerful techniques supplied by OpenCV. If you’re just testing the waters, definitely try this library out. However, as fair warning, over 95% of my code examples in this blog will be with OpenCV. Don’t worry though, I’m absolutely meticulous when it comes to documentation and I’ll provide you with complete, yet concise, explantations of the code.

7. mahotas

Mahotas, just as OpenCV and SimpleCV, rely on NumPy arrays. Much of the functionality implemented in Mahotas can be found in OpenCV and/or SimpleCV, but in some cases, the Mahotas interface is just easier to use, especially when it comes to their features package.

8. scikit-learn

Alright, you got me, Scikit-learn isn’t an image processing or computer vision library — it’s a machine learning library. That said, you can’t have advanced computer vision techniques without some sort of machine learning, whether it be clustering, vector quantization, classification models, etc. Scikit-learn also includes a handful of image feature extraction functions as well.

9. ilastik

I’ll be honest. I’ve never used ilastik. But through my experiences at computer vision conferences, I’ve met a fair amount of people who do, so I felt compelled to put it in this list. Ilastik is mainly for image segmentation and classification and is especially geared towards the scientific community.

Practical Python and OpenCV

BONUS:

I couldn’t stop at just nine. Here are three more bonus libraries that I use all the time.

10. pprocess

Extracting features from images is inherently a parallelizable task. You can reduce the amount of time it takes to extract features from an entire dataset by using a multiprocessing library. My favorite is pprocess, due to the simple nature I need it for, but you can use your favorite.

11. h5py

The h5py library is the de-facto standard in Python to store large numerical datasets. The best part? It provides support for NumPy arrays. So, if you have a large dataset represented as a NumPy array, and it won’t fit into memory, or if you want efficient, persistent storage of NumPy arrays, then h5py is the way to go. One of my favorite techniques is to store my extracted features in a h5py dataset and then apply scikit-learn’s MiniBatchKMeans to cluster the features. The entire dataset never has to be entirely loaded off disk at once and the memory footprint is extremely small, even for thousands of feature vectors.

12. scikit-image

On the first draft of this blog post, I completely forgot about scikit-image. Silly me. Anyway, scikit-image is fantastic, but you have to know what you are doing to effectively use this library — and I don’t mean this in a “there is a steep learning curve” type of way. The learning curve is actually quite low, especially if you check out their gallery. The algorithms included in scikit-image (I would argue) follow closer to the state-of-the-art in computer vision. New algorithms right from academic papers can be found in scikit-image, but in order to (effectively) use these algorithms, you need to have developed some rigor and understanding in the computer vision field. If you already have some experience in computer vision and image processing, definitely check out scikit-image; otherwise, I would continue working with OpenCV and SimpleCV to start.

In Summary:

NumPy provides you with a way to represent images as a multi-dimensional array. Many other image processing, computer vision, and machine learning libraries utilize NumPy so it’s paramount to have it (and SciPy) installed. While PIL and Pillow are great for simple image processing tasks, if you are serious about testing the computer vision waters, your time is better spent playing with SimpleCV. Once you’ve convinced yourself that computer vision is awesome, install OpenCV and re-learn what you did in SimpleCV. Over 95% of the code examples I will show you in this blog will be in OpenCV. Finally, install scikit-learn and h5py. You won’t need them just yet. But you’ll love them once I show you what they can do.

 

, , ,

16 Responses to My Top 9 Favorite Python Libraries for Building Image Search Engines

  1. Juan Morales January 16, 2014 at 2:47 pm #

    Thanks for the compilation. Very comprehensive, although I miss ITK.

    When I started with image segmentation in 2009 I chose ITK (using the python bindings) over opencv. It has also a steep learning curve but has everything I need for medical image segmentation.
    ¿Do you have any comments on this library?

    • Adrian Rosebrock January 16, 2014 at 3:30 pm #

      I’m glad you liked the compilation!

      So here’s the thing with ITK — it has its uses. You mentioned you were using it for medical image segmentation, and I would say the medical image processing community benefits from it the most. In fact, part (if not all of it) is integrated into MedPy. I haven’t used ITK that much, but when I have, I found the learning curve to very steep, more steep that OpenCV.

      The reason I tend towards OpenCV + Python is that I get the “power tools” of OpenCV with the rapid prototyping and ease of coding of Python. Sure, I could write code using OpenCV + C/C++ and get some performance gains, but the code wouldn’t be as flexible and maintainable. Furthermore, as an entrepreneur, I love being able to deploy projects in Python using Django or Flask.

      All libraries have their uses and their audience. I think ITK is starting to show its age now, especially with libraries like ilastik becoming more popular, but it will still be around. If you’re still working with medical image segmentation, definitely look into ilastik (#9 in the list above).

      • Juan Morales January 16, 2014 at 6:00 pm #

        I agree, python-itk is not pythonic at all, with all those badly wrapped C++ templates. I hope that simpleitk (http://www.simpleitk.org/) keeps improving so we can finally get an easy to use ITK.

        I’ll definitely have a look to ilastitk. Thank you a lot!

  2. Jorge Carrillo January 21, 2015 at 2:39 pm #

    Thank you very much for this. For me, your first paragraph says it all, except I’m starting today. I was expecting to spend a bit of time to arrive at a list that you’ve so generously contributed. My primary interest computer vision and astro imaging and this looks to be a great starting point.

    My question is about ImageMagick. It has Python bindings and it’s widely used around the internet. Is it worth looking at?

    • Adrian Rosebrock January 21, 2015 at 8:42 pm #

      ImageMagick is really awesome for what it does. But it’s not a true computer vision library. It’s great for batch processing images, but if you want to get into the nuts and bolts of computer vision, or build advanced systems like object classification, you won’t be able to use ImageMagick for that.

  3. Yu January 21, 2015 at 7:24 pm #

    Very good article. Thanks for your hard work. BTW, there’s a typo in ‘3.Matpotlib’. It should be ‘matplotlib’.

  4. Oliver Fjellvang July 18, 2016 at 10:10 am #

    Very interessting article.

    I’m looking for an image recogniction API that can extract images from pictures and tag it afterwards.

    Does anybody know where I can find this?

    • Adrian Rosebrock July 18, 2016 at 5:09 pm #

      That sounds like a perfect fit for the Google Vision API.

      • asteriod February 6, 2017 at 2:33 am #

        you could try clarifai’s API >> https://www.clarifai.com/

        They include real-time object recognition; this means objects in a video recording are identified.

  5. Manuel August 15, 2016 at 6:07 pm #

    Hi, does SimpleCV, regarding to OpenCV, acts as python bindings? Does it use the already existent python bindings in OpenCV or have they created their own?

    • Adrian Rosebrock August 16, 2016 at 1:00 pm #

      SimpleCV is essentially around basic OpenCV and PIL/Pillow functionality that aims at making computer vision easier.

  6. Chris September 18, 2016 at 11:21 pm #

    Hey Adrian, sorry about my english.
    Can you explain me understand about scikit learn and scikit image. what different and if same, what same?

    • Adrian Rosebrock September 19, 2016 at 1:04 pm #

      The scikit-learn library focuses on machine learning whereas the scikit-image library focuses on computer vision and image processing.

  7. Ali Raza December 20, 2017 at 4:27 pm #

    Thank you so much sir Adrian for such a great information.

Trackbacks/Pingbacks

  1. The Best Python Books of 2014 - PyImageSearch - May 12, 2014

    […] Think back to my top 9 favorite Python libraries for building image search engines post. […]

  2. Taking screenshots with OpenCV and Python - PyImageSearch - January 1, 2018

    […] published the very first blog post on Monday, January 12th 2014. Since then over 230 posts have been published, along with two books and a full-fledged […]

Leave a Reply