Labeling superpixel colorfulness with OpenCV and Python

After our previous post on computing image colorfulness was published, Stephan, a PyImageSearch reader, left a comment on the tutorial asking if there was a method to compute the colorfulness of specific regions of an image (rather than the entire image).

There are multiple ways of attacking this problem. The first could be to apply a sliding window to loop over the image and compute the colorfulness score for each ROI. An image pyramid could even be applied if the colorfulness of a specific region needed to be computed at multiple scales.

However, a better approach would be to use superpixels. Superpixels are extracted via a segmentation algorithm that groups pixels into (non-rectangular) regions based on their local color/texture. In the case of the popular SLIC superpixel algorithm, image regions are grouped based on a local version of k-means clustering algorithm in the L*a*b* color space.

Given that superpixels will give us a much more natural segmentation of the input image than sliding windows, we can compute the colorfulness of specific regions in an image by:

  1. Applying superpixel segmentation to the input image.
  2. Looping over each of the superpixels individually and computing their respective colorfulness scores.
  3. Maintaining a mask that contains the colorfulness score for each superpixel.

Based on this mask we can then visualize the most colorful regions of the image. Regions of the image that are more colorful will have larger colorful metric scores, while regions that are less colorful will smaller values.

To learn more about superpixels and computing image colorfulness, just keep reading.

Looking for the source code to this post?
Jump right to the downloads section.

Labeling superpixel colorfulness with OpenCV and Python

In the first part of this blog post we will learn how to apply the SLIC algorithm to extract superpixels from our input image. The original 2010 publication by Achanta et al., SLIC Superpixels, goes into the details of the methodology and technique. We also briefly covered SLIC superpixels in this blog post for readers who want a more concise overview of the algorithm.

Given these superpixels, we’ll loop over them individually and compute their colorfulness score, taking care to compute the colorfulness metric for the specific region and not the entire image (as we did in our previous post).

After we implement our script, we’ll apply our combination of superpixel + image colorfulness to a set of input images.

Using superpixels for segmentation

Let’s get started by opening up a new file in your favorite editor or IDE, name it colorful_regions.py , and insert the following code:

The first Lines 1-8 handle our imports — as you can see we make heavy use of several scikit-image functions in this tutorial.

The slic  function will be used to compute superpixels (scikit-image documentation).

Next, we will define our colorfulness metric function with a minor modification from the previous post where it was introduced:

Lines 10-31 represent our colorfulness metric function, which has been adapted to compute the colorfulness for a specific region of an image.

The region can be any shape as we take advantage of NumPy masked arrays — only pixels part of the mask will be included in the computation.

For the specified mask  region of a particular image , the segment_colorfulness  function performs the following tasks:

  1. Splits the image into RGB component channels (Line 14).
  2. Masks the image  using mask  (for each channel) so that the colorfulness is only performed on the area specified — in this case the region will be our superpixel (Lines 15-17).
  3. Uses the R  and G  components to compute rg  (Line 20).
  4. Uses the RGB components to compute yb  (Lines 23).
  5. Computes the mean and standard deviation of rg  and yb  whilst combining them (Lines 27 and 28).
  6. Does the final calculation of the metric and returns (Line 31) it to the calling function.

Now that our key colorfulness function is defined, the next step is to parse our command line arguments:

On Lines 34-39 we make use of argparse  to define two arguments:

  1. --image : The path to our input image.
  2. --segments : The number of superpixels. The SLIC Superpixels paper shows examples of breaking an image up into different numbers of superpixels. This parameter is fun to experiment with (as it controls the level of granularity of your resulting superpixels); however we’ll be working with a default=100 . The smaller the value, the fewer and larger the superpixels, allowing the algorithm running faster. The larger the number of segments, the more fine-grained the segmentation, but SLIC will take longer to run (due to more clusters needing to be computed).

Now it’s time to load the image into memory, allocate space for our visualization, and compute SLIC superpixel segmentation:

On Line 43 we load our command line argument --image  into memory as orig  (OpenCV format).

We follow this step by allocating memory with the same shape (width and height) as the original input image for our visualization image, vis .

Next, we load the command line argument --image  into memory as image , this time in scikit-image format. The reason we use scikit-image’s io.imread  here is because OpenCV loads images in BGR order rather than RGB format (which scikit-image does). The slic  function will convert our input image  to the L*a*b* color space during the superpixel generation process assuming our image is in RGB format.

Therefore we have two choices:

  1. Load the image with OpenCV, clone it, and then swap the ordering of the channels.
  2. Simply load a copy of the original image using scikit-image.

Either approach is valid and will result in the same output.

Superpixels are calculated by a call to slic  where we specify image , n_segments , and the  slic_zero switch. Specifying slic_zero=True  indicates that we want to use the zero parameter version of SLIC, an extension to the original algorithm that does not require us to manually tune parameters to the algorithm. We refer to the superpixels as segments  for the rest of the script.

Now let’s compute the colorfulness of each superpixel:

We start by looping over each of the individual segments  on Line 52.

Lines 56 and 57 are responsible for constructing a mask  for the current superpixel. The mask  will have the same width and height as our input image and will be filled (initially) with an array of ones (Line 56).

Keep in mind that when using NumPy masked arrays, that a given entry in an array is only included in a computation if the corresponding mask  value is set to zero (implying that the pixel is unmasked). If the value in the mask  is one, then the value is assumed to be masked and is hence ignored.

Here we initially set all pixels to masked, then set only the pixels part of the current superpixel to unmasked (Line 57).

Using our orig  image and our mask  as parameters to segment_colorfulness , we can compute C , which is the colorfulness of the superpixel (Line 61).

Then, we update our visualization array, vis , with the value of C (Line 62).

At this point, we have answered PyImageSearch reader, Stephan’s question — we have computed the colorfulness for different regions of an image.

Naturally we will want to see our results, so let’s continue by constructing a transparent overlay visualization for the most/least colorful regions in our input image:

Since vis  is currently a floating point array, it is necessary to re-scale it to a typical 8-bit unsigned integer [0-255] array. This is important so that we can display the output image to our screen with OpenCV. We accomplish this by using the rescale_intensity  function (from skimage.exposure ) on Line 67.

Now we’ll overlay the superpixel colorfulness visualization on top of the original image. We’ve already discussed transparent overlays and the cv2.addWeighted  (and associated parameters), so please refer to this blog post for more details on how transparent overlays are constructed.

Finally, let’s display images to the screen and close out this script:

We will display three images to the screen using cv2.imshow , including:

  1. orig : Our input image.
  2. vis : Our visualization image (i.e., level of colorfulness for each of the superpixel regions).
  3. output : Our output image.

Superpixel and colorfulness metric results

Let’s see our Python script in action — open a terminal, workon  your virtual environment if you are using one (highly recommended), and enter the following command:

Figure 1: Computing a region-based colorfulness score using superpixel segmentation.

On the left you can see the original input image, a photo of myself exploring Antelope Canyon, arguably the most beautiful slot canyon in the United States. Here we can see a mixture of colors due to the light filtering in from above.

In the middle we have our computed visualization for each of the 100 superpixels. Dark regions in this visualization refer to less colorful regions while light regions indicate more colorful.

Here we can see the least colorful regions are around the walls of the canyon, closest to the camera — this is where the least light is researching.

The most colorful regions of the input image are found where the light is directly reaching inside the canyon, illuminating part of the wall like candlelight.

Finally, on the right we have our original input image overlaid with the colorfulness visualization — this image allows us to more easily identify the most/least colorful regions of the image.

The following image is a photo of myself in Boston by the iconic Citgo sign overlooking Kenmore square:

Figure 2: Using superpixels, we can first segment our image, and then compute a colorfulness score for each region.

Here we can see the least colorful regions of the image are towards the bottom where the shadow is obscuring much of the sidewalk. The more colorful regions can be found towards the sign and sky itself.

Finally, here is a photo from Rainbow Point, the highest elevation in Bryce Canyon:

Figure 3: Labeling individual superpixels in an image based on how “colorful” each region is.

Notice here that my black hoodie and shorts are the least colorful regions of the image, while the sky and foliage towards the center of the photo are the most colorful.

Summary

In today’s blog post we learned how to use the SLIC segmentation algorithm to compute superpixels for an input image.

We then accessed each of the individual superpixels and applied our colorfulness metric.

The colorfulness scores for each region were combined into a mask, revealing the most colorful and least colorful regions of the input image.

Given this computation, we were able to visualize the colorfulness of each region in two ways:

  1. By examining the raw vis  mask.
  2. Creating a transparent overlay that laid vis  on top of the original image.

In practice, we could use this technique to threshold the mask and extract only the most/least colorful regions.

To be notified when future tutorials are published here on PyImageSearch, be sure to enter your email address in the form below!

Downloads:

If you would like to download the code and images used in this post, please enter your email address in the form below. Not only will you get a .zip of the code, I’ll also send you a FREE 11-page Resource Guide on Computer Vision and Image Search Engines, including exclusive techniques that I don’t post on this blog! Sound good? If so, enter your email address and I’ll send you the code immediately!

, , , , ,

11 Responses to Labeling superpixel colorfulness with OpenCV and Python

  1. Stephan June 26, 2017 at 1:01 pm #

    you are awesome !
    Stephan says THANKS

    • Adrian Rosebrock June 27, 2017 at 6:21 am #

      Thanks Stephan 🙂

  2. Stefan van der Walt June 26, 2017 at 5:03 pm #

    Thank you for the fun post!

    This example could be made even simpler by using the regionprops functionality in skimage.measure. Masked arrays are not very fast, unfortunately, so I tend to avoid them.

    Here’s the suggested implementation and output:

    https://gist.github.com/stefanv/05c57153a544bad6f3dae06c2c69abd2

    • Adrian Rosebrock June 27, 2017 at 6:19 am #

      Thanks for the suggestion, Stefan! Part of the reason I used NumPy’s masked arrays was simply re-familiarize myself with them. But you are right, regionprops is also a good choice.

  3. André Pilastri June 26, 2017 at 5:33 pm #

    Hi Adrian,

    I very enjoy your posts!
    Just one question: Do you know deep learning work using graphs?
    I think this is an interesting topic.

    • Adrian Rosebrock June 27, 2017 at 6:16 am #

      Hi André — what do you mean by “deep learning work using graphs”? Are you trying to build a graph of superpixel-like structures using deep learning? Or are you trying to apply deep learning to actual graphs (i.e., figures and plots)?

      • André Pilastri June 27, 2017 at 9:01 am #

        I build a similarity graph through superpixels and would like to apply in deep learning. Taking as input the graph. Do you know anything about this?

        • Adrian Rosebrock June 30, 2017 at 8:29 am #

          What would the end goal be of supplying a machine learning model with the graph? What are you trying to accomplish?

  4. Tom June 27, 2017 at 5:58 am #

    I like how you go with explanations practically line-by-line in your tutorials. Can you elaborate a bit about what is going on and the technique/method you’re using in line 57 “mask[segments == v] = 0” ? Specifically, what does the double equal sign do in square brackets? Thanks

    • Adrian Rosebrock June 27, 2017 at 6:08 am #

      The segments variable is a NumPy array containing our superpixels. Each superpixel has a unique integer value. Therefore, we can find all (x, y)-coordinates in mask that have segments == v (implying that we are extracting all pixels part of the superpixel with value v). There is accomplished by NumPy array indexing.

  5. PBS June 29, 2017 at 9:08 am #

    Another great blog post! – Thanks very much.

Leave a Reply