Installing boost and boost-python on OSX with Homebrew

logos_header

I really, really hope that someone finds this resource useful. The amount of time I have wasted over the past few months (passively) trying to get boost and boost-python to install on my OSX machine via Homebrew has been nothing short of excruciating.

Don’t get me wrong, I love Homebrew. And if you are on an OSX machine and aren’t using Homebrew, then I suggest you stop reading this post and install it right now.

Anyway, like I said, I hope that this post saves other people some time and hassle. And while this post isn’t entirely dedicated to computer vision, it is still very relevant if you are developing computer vision based applications using Python and OpenCV.

Packages such as Spotify’s Annoy for Approximate Nearest Neighbor search have direct applications in the Content-Based Image Retrieval (CBIR)/image search engine space.

Update 4 May 2015:  Erik Bernhardsson has released an update to Annoy that removes the dependency of Boost and Boost.Python from Annoy. You can now simply install Annoy using pip: pip install annoy without any extra dependencies.

And libraries such as dlib provide Python bindings so you can leverage the power of dlib from your Python shell.

Both Annoy and dlib are just two examples of packages that require the use of boost (and boost-python if you want Python bindings).

Anyway, let’s go ahead and get this tutorial started — I’ve definitely wasted enough of my time working with this problem and I don’t want to waste any of yours either!

What is Homebrew?

Homebrew is “the missing package manager for OSX”. It makes installing and managing packages not installed by the default Apple installation a breeze, in the same manner that Debian apt-get  does.

Note: Comparing Homebrew to apt-get is not entirely fair, but if this is the first time you are hearing of Homebrew, this comparison should suffice.

What is boost and boost-python?

Boost is a collection of peer-reviewed (i.e. very high quality) C++ libraries that help programmers and developers not get caught up in reinventing the wheel. Boost provides implementations for linear algebra, multithreading, basic image processing, and unit testing, just to name a few.

Again, these libraries are peer-reviewed and very high quality. A very large number of C++ applications, especially in the scientific space, rely on the Boost libraries in some way or another.

We also have boost-python, which provides interoperability between the C++ and Python programming languages.

Why is this useful?

Well let’s say you are implementing an Approximate Nearest Neighbor algorithm (like Spotify’s Annoy) and you want to provide pure, vanilla Python support.

However, you want to milk every last little bit of memory and CPU performance out of the library, so you decide to implement performance critical sections in C++.

To do this, you would code these critical tasks in C++ using boost — and then interface with the Python programming language with boost-python.

In fact, this is exactly what the Annoy package does. While the package is pip-installable, the package requires boost and boost-python so that it can be compiled and installed.

Installing boost and boost-python on OSX with Homebrew

Now that we have some basic terminology down, let’s go ahead and install our packages.

Step 1: Install Homebrew

Installing Homebrew could not be easier.

Just head to the Homebrew homepage and copy and paste the following code into your terminal:

Note: This blog post was written in January 2015. Definitely head to the Homebrew homepage and use the latest install script provided by the Homebrew community.

Step 2: Update Homebrew

Now that you have Homebrew installed, you need to update it and grab the latest package (i.e. “formula”) definitions. These formula are simply instructions on how to install a given library or package.

To update Homebrew, simply do:

Step 3: Install Python

It’s bad form to use the system Python as your main interpreter. And this is especially true if you intend on using virtualenv.

Before we go any further, let’s install Python via brew:

Step 4: Installing boost

So far so good. But now it’s time to install boost.

And this is where you really need to start paying attention.

To install boost, execute the following command:

You see that --with-python  flag?

Yeah, don’t forget that — it’s important.

In my case, I figured that boost-python would already be installed, given the --with-python  flag.

Apparently that’s not the case. You need to explicitly install boost-python as well. Otherwise, you’ll get the dreaded segfault error when you try to call a package from within Python that expects to find boost bindings.

Also, you might want to go take a nice little walk while boost downloads, compiles, and installs. It’s a large library and if you are keen on optimizing your time throughout the work-day (like I am), then I highly suggest that you context switch and get some other work done.

Step 5: Installing boost-python

Now that boost is installed, we can get boost-python installed as well:

The boost-python package should install a lot faster than boost, but you still might want to make yourself a cup of coffee, especially if your system is slow.

Step 6: Confirm boost and boost-python is installed

Make sure that both boost and boost-python are installed:

As you can see from my terminal output, both boost and boost-python have been successfully installed (provided that you didn’t get any errors from the above steps, of course).

Already using Python + virtualenv? Keep reading.

Oh, you thought we were done?

So did I. And boy, that was a mistake.

Because guess what? If you already have Python installed and are using virtualenv (and in my case, virtualenvwrapper), you’ve still got some work to do.

Note: If you are not already using virtualenv  and virtualenvwrapper  to manage your Python packages, this is something that you should really look into. It makes your life substantially easier — trust me.

New virtualenvs:

If you are creating a new virtualenv, you’ll be good to go. No extra work is required, everything will work smoothly out of the box.

Existing virtualenvs:

So let me tell you something you already know: When we construct a virtual environment, our Python executable, along with relevant libraries, includes, and site-packages are cloned and sequestered into their own independent environment.

And let me tell you something you might not know: If you already have your virtualenv setup before compiling and installing boost and boost-python (like I did), then you will not have access to your boost bindings.

So what’s the best way to solve this problem?

Honestly, I’m not sure what the “best” way is. There has to be a more elegant method than what I’m proposing. But here’s how I fixed the problem:

  1. Generated a requirements.txt  for my virtualenv
  2. Deactivated and deleted my virtualenv
  3. Recreated my virtualenv
  4. pip install -r requirements.txt  that shit and be done with it

After you’ve performed these steps your new virtualenv will have the boost-python bindings in place. And hopefully you won’t have wasted as much time as I have.

An Annoy Example

Now that we have boost and boost-python installed, let’s take them for a test drive using the Annoy package.

Update 4 May 2015:  As I mentioned at the top of this post, Erik Bernhardsson has released an update to Annoy that removes the dependency of Boost and Boost.Python from Annoy. You can now simply install Annoy using pip without a having Boost or Boost.Python installed.

Let’s start by creating our virtualenv using virtualenvwrapper:

Now that our packages are installed, let’s create 1,000 random vectors with 128-D. We’ll pass these vectors into Annoy and construct our embedding using 10 trees:

Now that our embedding is created, let’s find the 10 (approximate) nearest neighbors to the first vector in our list:

We can also find nearest neighbors that are not already part of the index:

So what would happen if you tried to execute this code without boost and boost-python installed?

Your code would segfault during the get_nns_by_item  and get_nns_by_vector  functions. And if you were using dlib, then you would segfault during the import. Definitely something to keep in mind. If you are segfault-ing during these functions, then something is funky with your boost and boost-python install.

Summary

In this blog post I reviewed how to install boost and boost-python on OSX using Homebrew.

Personally, I wasted a ridiculous amount of time while passively working on this problem over a few month period — the goal of this article was to (hopefully) help you save time and avoid any heartache and frustration.

And if you know of a more elegant way to solve this problem, please let me know in the comments or shoot me an email!

, , , , , ,

30 Responses to Installing boost and boost-python on OSX with Homebrew

  1. Arno April 27, 2015 at 10:35 am #

    Please keep ’em coming! I find these OS X focused, step-by-step install and configure posts to be extremely useful.

    • Adrian Rosebrock May 1, 2015 at 7:06 pm #

      Thanks Arno! 🙂

  2. Tim June 2, 2015 at 7:50 pm #

    Hello! The –with-python switch to boost no longer does anything. Running just brew install boost-python will install both boost and Boost.Python for your hacking enjoyment, and they should run quite quickly since they’ll give you precompiled binary packages by default now.

    • Adrian Rosebrock June 3, 2015 at 6:43 am #

      Hi Tim, thanks for the tip!

      So just to confirm, running brew install python is no longer necessary? Just doing brew install boost-python will take care of install boost and boost.python?

      • Matt Fletcher November 20, 2017 at 5:34 am #

        Tim’s correct, it’s now just brew install boost-python. Please edit the article to reflect this 🙂

  3. vin July 24, 2015 at 7:56 pm #

    hey adrian!

    ive run into this problem several times and recommend rpforest: https://github.com/lyst/rpforest

    good luck!

    • Adrian Rosebrock July 25, 2015 at 7:00 am #

      rpforest is really awesome 🙂 Although now annoy is 100% pip-installable. I think Eric (the creator of annoy) is planning on adding rpforest to his benchmark. My only gripe with rpforest is that it only allows for the cosine distance, not Euclidean which I find a little frustrating.

  4. RJDan December 1, 2015 at 11:19 am #

    Very useful, ‘specially for a noob like me.

    • Adrian Rosebrock December 1, 2015 at 5:27 pm #

      I’m glad the post could help!

  5. Rajan January 5, 2016 at 11:50 pm #

    Fantastic post!! Do you have any idea how to install dlib on ubuntu?

    • Adrian Rosebrock January 6, 2016 at 6:24 am #

      Installing dlib on Ubuntu is pretty straightforward, you should be able to follow the dlib install instructions. In any case, I’ll try to do a dlib installation tutorial in the future.

  6. Lea March 1, 2016 at 10:21 pm #

    Please please please do a dlib installation tutorial! The whole process was so frustrating….and your posts literally made my life easier when dealing with python vision projects. Merci beaucoup!

    • Adrian Rosebrock March 3, 2016 at 7:11 am #

      I have one drafted up. I’ll try to push it up in the queue 🙂

      • Ankush Tripathi February 5, 2017 at 11:57 pm #

        hi Adrian!
        your posts are very easy to understand and are very helpful.
        did you post dlib installation tutorial. I couldnt find it. please help us.

        • Adrian Rosebrock February 7, 2017 at 9:17 am #

          Thank you for the suggestion Ankush, I will certainly consider this.

  7. Pawandeep Singh March 7, 2016 at 5:35 am #

    I installed Boost and Boost Python. But I don’t know how to use them with Xcode. Can you write a tutorial explaining how to use this with Xcode. Thank You

    • Adrian Rosebrock March 7, 2016 at 4:10 pm #

      I personally don’t use Boost or Boost.Python directly. I’ve only used them in context of other libraries, such as annoy and dlib. That said, I’ll see if I can write a post on this topic in the future if there is enough interest.

  8. Siya April 2, 2016 at 6:04 pm #

    Hi , I would like to know how does one link boost libraries after you have managed to install them using Homebrew on MAC OS. I have attempted to install a radio astronomy data package but I need to link the boost libraries first before installing the package I am interested in. Please help

  9. Abder-Rahman May 15, 2016 at 1:41 am #

    Very helpful, thanks for your kind efforts on that. Keep it up!

  10. Bogdan July 19, 2016 at 1:22 am #

    I have Ubuntu 16.04 LTS with (default) python interpreter 2.7.12. I have also created a virtualenv (OpenCV 3 + Python 3.5) using PyImageSearch tutorial (http://goo.gl/xMTMK7), in which I have installed dlib.

    I can import dlib in my virtualenv (“cv”), without errors, but when I run a dlib example I get the following errors:

    ///////

    Traceback (most recent call last):
    File “python_examples/face_detector.py”, line 47, in
    import dlib
    ImportError: /usr/lib/x86_64-linux-gnu/libboost_python-py27.so.1.58.0: undefined symbol: PyClass_Type

    ///////

    I have tried to follow Adrian’s advice on deleting and recreating the virtualenv, but because virtualenv is configured to use Python 3.5 (instead of Ubuntu Python executable), the new virtualenv is always created with python 3.5. So the new virtualenv gets the bindings for boost-python, but it’s using python 3.5 and dlib is expecting python 2.7 site-packages.

    Any ideeas?

    • Adrian Rosebrock July 19, 2016 at 12:31 pm #

      Hey Bogdan — Based on the error message, it looks like dlib was (likely) compiled against a different Python version than what is being used to import it. I would double-check your CMake command for dlib and ensure that you are compiling against the correct version of Python (i.e., the same one in your Python virtual environment).

      • Bogdan July 20, 2016 at 9:31 am #

        thanks for the hint. I am not familiar with CMake, but is it possible to pass a parameter to cmake like you did in your tutorial on how to install OpenCV 3 with Python 3;

        //////
        cmake -D CMAKE_BUILD_TYPE=RELEASE \
        -D CMAKE_INSTALL_PREFIX=/usr/local \
        -D INSTALL_C_EXAMPLES=ON \
        -D INSTALL_PYTHON_EXAMPLES=ON \
        -D OPENCV_EXTRA_MODULES_PATH=~/opencv_contrib/modules \
        -D BUILD_EXAMPLES=ON ..

        //////

        I have installed dlib via “python setup.py install” without modifying the cmake file

        • Adrian Rosebrock July 20, 2016 at 2:31 pm #

          Yep, it’s totally possible. For the dlib compile, you would use something like this:

          But you’ll want to modify the PYTHON_LIBRARY to point to your version.

          • Bogdan July 28, 2016 at 9:06 am #

            somehow I don’t have a python3.5.dylib file. I only have the following

            ~/.virtualenvs/cv/lib/python3.5/config-3.5m-x86_64-linux-gnu/libpython3.5.m.so

            I have installed python3.5 in my virtualenv following your tutorial. Might be that python3.5 doesn’t use a *.dylib file anymore?

          • Adrian Rosebrock July 29, 2016 at 8:35 am #

            Hey Bogdan — it looks like you’re on a Linux system. This tutorial is for OSX. Linux does not use .dylib files.

  11. shannon November 24, 2016 at 10:52 am #

    Great tutorial, it saved me loads of time–thanks!

    • Adrian Rosebrock November 28, 2016 at 10:44 am #

      Glad to hear it Shannon! 🙂

  12. proycon March 7, 2018 at 5:40 am #

    I think currently (2018) a brew install boost boost-python3 should do the trick

    • WOO SEOK June 26, 2019 at 3:12 am #

      I think that’s totally correct.

Trackbacks/Pingbacks

  1. Annoy – now without Boost dependencies and with Python 3 Support | Erik Bernhardsson - May 3, 2015

    […] ironic thing is the day after, PyImageSearch featured a blog post complaining about Annoy’s Boost […]

Before you leave a comment...

Hey, Adrian here, author of the PyImageSearch blog. I'd love to hear from you, but before you submit a comment, please follow these guidelines:

  1. If you have a question, read the comments first. You should also search this page (i.e., ctrl + f) for keywords related to your question. It's likely that I have already addressed your question in the comments.
  2. If you are copying and pasting code/terminal output, please don't. Reviewing another programmers’ code is a very time consuming and tedious task, and due to the volume of emails and contact requests I receive, I simply cannot do it.
  3. Be respectful of the space. I put a lot of my own personal time into creating these free weekly tutorials. On average, each tutorial takes me 15-20 hours to put together. I love offering these guides to you and I take pride in the content I create. Therefore, I will not approve comments that include large code blocks/terminal output as it destroys the formatting of the page. Kindly be respectful of this space.
  4. Be patient. I receive 200+ comments and emails per day. Due to spam, and my desire to personally answer as many questions as I can, I hand moderate all new comments (typically once per week). I try to answer as many questions as I can, but I'm only one person. Please don't be offended if I cannot get to your question
  5. Do you need priority support? Consider purchasing one of my books and courses. I place customer questions and emails in a separate, special priority queue and answer them first. If you are a customer of mine you will receive a guaranteed response from me. If there's any time left over, I focus on the community at large and attempt to answer as many of those questions as I possibly can.

Thank you for keeping these guidelines in mind before submitting your comment.

Leave a Reply

[email]
[email]