Labeling superpixel colorfulness with OpenCV and Python

After our previous post on computing image colorfulness was published, Stephan, a PyImageSearch reader, left a comment on the tutorial asking if there was a method to compute the colorfulness of specific regions of an image (rather than the entire image).

There are multiple ways of attacking this problem. The first could be to apply a sliding window to loop over the image and compute the colorfulness score for each ROI. An image pyramid could even be applied if the colorfulness of a specific region needed to be computed at multiple scales.

However, a better approach would be to use superpixels. Superpixels are extracted via a segmentation algorithm that groups pixels into (non-rectangular) regions based on their local color/texture. In the case of the popular SLIC superpixel algorithm, image regions are grouped based on a local version of k-means clustering algorithm in the L*a*b* color space.

Given that superpixels will give us a much more natural segmentation of the input image than sliding windows, we can compute the colorfulness of specific regions in an image by:

  1. Applying superpixel segmentation to the input image.
  2. Looping over each of the superpixels individually and computing their respective colorfulness scores.
  3. Maintaining a mask that contains the colorfulness score for each superpixel.

Based on this mask we can then visualize the most colorful regions of the image. Regions of the image that are more colorful will have larger colorful metric scores, while regions that are less colorful will smaller values.

To learn more about superpixels and computing image colorfulness, just keep reading.

Looking for the source code to this post?
Jump right to the downloads section.

Labeling superpixel colorfulness with OpenCV and Python

In the first part of this blog post we will learn how to apply the SLIC algorithm to extract superpixels from our input image. The original 2010 publication by Achanta et al., SLIC Superpixels, goes into the details of the methodology and technique. We also briefly covered SLIC superpixels in this blog post for readers who want a more concise overview of the algorithm.

Given these superpixels, we’ll loop over them individually and compute their colorfulness score, taking care to compute the colorfulness metric for the specific region and not the entire image (as we did in our previous post).

After we implement our script, we’ll apply our combination of superpixel + image colorfulness to a set of input images.

Using superpixels for segmentation

Let’s get started by opening up a new file in your favorite editor or IDE, name it , and insert the following code:

The first Lines 1-8 handle our imports — as you can see we make heavy use of several scikit-image functions in this tutorial.

The slic  function will be used to compute superpixels (scikit-image documentation).

Next, we will define our colorfulness metric function with a minor modification from the previous post where it was introduced:

Lines 10-31 represent our colorfulness metric function, which has been adapted to compute the colorfulness for a specific region of an image.

The region can be any shape as we take advantage of NumPy masked arrays — only pixels part of the mask will be included in the computation.

For the specified mask  region of a particular image , the segment_colorfulness  function performs the following tasks:

  1. Splits the image into RGB component channels (Line 14).
  2. Masks the image  using mask  (for each channel) so that the colorfulness is only performed on the area specified — in this case the region will be our superpixel (Lines 15-17).
  3. Uses the R  and G  components to compute rg  (Line 20).
  4. Uses the RGB components to compute yb  (Lines 23).
  5. Computes the mean and standard deviation of rg  and yb  whilst combining them (Lines 27 and 28).
  6. Does the final calculation of the metric and returns (Line 31) it to the calling function.

Now that our key colorfulness function is defined, the next step is to parse our command line arguments:

On Lines 34-39 we make use of argparse  to define two arguments:

  1. --image : The path to our input image.
  2. --segments : The number of superpixels. The SLIC Superpixels paper shows examples of breaking an image up into different numbers of superpixels. This parameter is fun to experiment with (as it controls the level of granularity of your resulting superpixels); however we’ll be working with a default=100 . The smaller the value, the fewer and larger the superpixels, allowing the algorithm running faster. The larger the number of segments, the more fine-grained the segmentation, but SLIC will take longer to run (due to more clusters needing to be computed).

Now it’s time to load the image into memory, allocate space for our visualization, and compute SLIC superpixel segmentation:

On Line 43 we load our command line argument --image  into memory as orig  (OpenCV format).

We follow this step by allocating memory with the same shape (width and height) as the original input image for our visualization image, vis .

Next, we load the command line argument --image  into memory as image , this time in scikit-image format. The reason we use scikit-image’s io.imread  here is because OpenCV loads images in BGR order rather than RGB format (which scikit-image does). The slic  function will convert our input image  to the L*a*b* color space during the superpixel generation process assuming our image is in RGB format.

Therefore we have two choices:

  1. Load the image with OpenCV, clone it, and then swap the ordering of the channels.
  2. Simply load a copy of the original image using scikit-image.

Either approach is valid and will result in the same output.

Superpixels are calculated by a call to slic  where we specify image , n_segments , and the  slic_zero switch. Specifying slic_zero=True  indicates that we want to use the zero parameter version of SLIC, an extension to the original algorithm that does not require us to manually tune parameters to the algorithm. We refer to the superpixels as segments  for the rest of the script.

Now let’s compute the colorfulness of each superpixel:

We start by looping over each of the individual segments  on Line 52.

Lines 56 and 57 are responsible for constructing a mask  for the current superpixel. The mask  will have the same width and height as our input image and will be filled (initially) with an array of ones (Line 56).

Keep in mind that when using NumPy masked arrays, that a given entry in an array is only included in a computation if the corresponding mask  value is set to zero (implying that the pixel is unmasked). If the value in the mask  is one, then the value is assumed to be masked and is hence ignored.

Here we initially set all pixels to masked, then set only the pixels part of the current superpixel to unmasked (Line 57).

Using our orig  image and our mask  as parameters to segment_colorfulness , we can compute C , which is the colorfulness of the superpixel (Line 61).

Then, we update our visualization array, vis , with the value of C (Line 62).

At this point, we have answered PyImageSearch reader, Stephan’s question — we have computed the colorfulness for different regions of an image.

Naturally we will want to see our results, so let’s continue by constructing a transparent overlay visualization for the most/least colorful regions in our input image:

Since vis  is currently a floating point array, it is necessary to re-scale it to a typical 8-bit unsigned integer [0-255] array. This is important so that we can display the output image to our screen with OpenCV. We accomplish this by using the rescale_intensity  function (from skimage.exposure ) on Line 67.

Now we’ll overlay the superpixel colorfulness visualization on top of the original image. We’ve already discussed transparent overlays and the cv2.addWeighted  (and associated parameters), so please refer to this blog post for more details on how transparent overlays are constructed.

Finally, let’s display images to the screen and close out this script:

We will display three images to the screen using cv2.imshow , including:

  1. orig : Our input image.
  2. vis : Our visualization image (i.e., level of colorfulness for each of the superpixel regions).
  3. output : Our output image.

Superpixel and colorfulness metric results

Let’s see our Python script in action — open a terminal, workon  your virtual environment if you are using one (highly recommended), and enter the following command:

Figure 1: Computing a region-based colorfulness score using superpixel segmentation.

On the left you can see the original input image, a photo of myself exploring Antelope Canyon, arguably the most beautiful slot canyon in the United States. Here we can see a mixture of colors due to the light filtering in from above.

In the middle we have our computed visualization for each of the 100 superpixels. Dark regions in this visualization refer to less colorful regions while light regions indicate more colorful.

Here we can see the least colorful regions are around the walls of the canyon, closest to the camera — this is where the least light is researching.

The most colorful regions of the input image are found where the light is directly reaching inside the canyon, illuminating part of the wall like candlelight.

Finally, on the right we have our original input image overlaid with the colorfulness visualization — this image allows us to more easily identify the most/least colorful regions of the image.

The following image is a photo of myself in Boston by the iconic Citgo sign overlooking Kenmore square:

Figure 2: Using superpixels, we can first segment our image, and then compute a colorfulness score for each region.

Here we can see the least colorful regions of the image are towards the bottom where the shadow is obscuring much of the sidewalk. The more colorful regions can be found towards the sign and sky itself.

Finally, here is a photo from Rainbow Point, the highest elevation in Bryce Canyon:

Figure 3: Labeling individual superpixels in an image based on how “colorful” each region is.

Notice here that my black hoodie and shorts are the least colorful regions of the image, while the sky and foliage towards the center of the photo are the most colorful.


In today’s blog post we learned how to use the SLIC segmentation algorithm to compute superpixels for an input image.

We then accessed each of the individual superpixels and applied our colorfulness metric.

The colorfulness scores for each region were combined into a mask, revealing the most colorful and least colorful regions of the input image.

Given this computation, we were able to visualize the colorfulness of each region in two ways:

  1. By examining the raw vis  mask.
  2. Creating a transparent overlay that laid vis  on top of the original image.

In practice, we could use this technique to threshold the mask and extract only the most/least colorful regions.

To be notified when future tutorials are published here on PyImageSearch, be sure to enter your email address in the form below!


If you would like to download the code and images used in this post, please enter your email address in the form below. Not only will you get a .zip of the code, I’ll also send you a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL! Sound good? If so, enter your email address and I’ll send you the code immediately!

, , , , ,

25 Responses to Labeling superpixel colorfulness with OpenCV and Python

  1. Stephan June 26, 2017 at 1:01 pm #

    you are awesome !
    Stephan says THANKS

    • Adrian Rosebrock June 27, 2017 at 6:21 am #

      Thanks Stephan 🙂

  2. Stefan van der Walt June 26, 2017 at 5:03 pm #

    Thank you for the fun post!

    This example could be made even simpler by using the regionprops functionality in skimage.measure. Masked arrays are not very fast, unfortunately, so I tend to avoid them.

    Here’s the suggested implementation and output:

    • Adrian Rosebrock June 27, 2017 at 6:19 am #

      Thanks for the suggestion, Stefan! Part of the reason I used NumPy’s masked arrays was simply re-familiarize myself with them. But you are right, regionprops is also a good choice.

  3. André Pilastri June 26, 2017 at 5:33 pm #

    Hi Adrian,

    I very enjoy your posts!
    Just one question: Do you know deep learning work using graphs?
    I think this is an interesting topic.

    • Adrian Rosebrock June 27, 2017 at 6:16 am #

      Hi André — what do you mean by “deep learning work using graphs”? Are you trying to build a graph of superpixel-like structures using deep learning? Or are you trying to apply deep learning to actual graphs (i.e., figures and plots)?

      • André Pilastri June 27, 2017 at 9:01 am #

        I build a similarity graph through superpixels and would like to apply in deep learning. Taking as input the graph. Do you know anything about this?

        • Adrian Rosebrock June 30, 2017 at 8:29 am #

          What would the end goal be of supplying a machine learning model with the graph? What are you trying to accomplish?

  4. Tom June 27, 2017 at 5:58 am #

    I like how you go with explanations practically line-by-line in your tutorials. Can you elaborate a bit about what is going on and the technique/method you’re using in line 57 “mask[segments == v] = 0” ? Specifically, what does the double equal sign do in square brackets? Thanks

    • Adrian Rosebrock June 27, 2017 at 6:08 am #

      The segments variable is a NumPy array containing our superpixels. Each superpixel has a unique integer value. Therefore, we can find all (x, y)-coordinates in mask that have segments == v (implying that we are extracting all pixels part of the superpixel with value v). There is accomplished by NumPy array indexing.

  5. PBS June 29, 2017 at 9:08 am #

    Another great blog post! – Thanks very much.

  6. stephan schulz September 5, 2017 at 6:05 pm #

    Again, thanks for taking the time to show us how to do such thing.

    I implemented your code for the toolchain.

    • Adrian Rosebrock September 7, 2017 at 7:13 am #

      Very cool, thanks for sharing Stephan.

  7. Stephan Schulz November 3, 2017 at 4:35 pm #

    I am wondering if there is a better way to derive the mean color from each super pixel cluster.
    Currently is adding all R,G,B values together and dividing them by the amount in each array.

    I find this makes for a very muddy color.

    Since this code does evaluate the colourfulness, is there a way to get a color repesentation of this color? My eye catches the brighter more intense colors, but the mean does not give me this feeling.

    Thanks for any hints.

    • Adrian Rosebrock November 6, 2017 at 10:44 am #

      Hi Stephan — I’m not sure what you mean by a “color representation of this color”? Do you mean the name of a color?

  8. stephan schulz November 9, 2017 at 11:31 am #

    In your example you have one set where you show the source image with each super pixel cluster filled with a single color. This color seems to be the mean color of all pixels inside this cluster.

    I would like this color to be the most colourful one in the cluster, not the mean. Is there a way to extract that information?
    thx. 🙂

    • Adrian Rosebrock November 13, 2017 at 2:25 pm #

      Hi Stephan — can you elaborate on “I would like this color to be the most colorful one in the cluster”? The most colorful pixel?

  9. stephan schulz November 16, 2017 at 1:08 pm #

    if one of the pixel cluster has a lot of black or white in it, but also has nice yellow dot in the middle then it might still get a high colourfulness score.
    how do I now extract the information about this choice?
    if I average the pixels in that cluster than the black/white will muddy the resulting mean color.
    how can I query your code to know more about which of the colours present in this specific cluster allowed it to receive a high score?
    in the end I want to represent this cluster by it’s most “beautiful” / most noticeable colour, not the mean.

    • Adrian Rosebrock November 18, 2017 at 8:19 am #

      The most “beautiful/noticeable” color seems a bit subjective here. The colorfulness score was developed to work on aggregate (i.e., means) and are correlated with the authors’ test subjects. You can’t really query a single pixel using this method.

  10. vs December 6, 2017 at 11:55 am #

    Hi, I have created the superpixel by using SLIC. I want to pass superpixels to the classifier. For this, I have to label the superpixel by manually. It will be great, if you suggest the way.

    • Adrian Rosebrock December 8, 2017 at 5:07 pm #

      I think it would be beneficial to learn what the end goal of the project is. Is there a particular reason you need to label the superpixel manually?

  11. Amy March 6, 2018 at 5:39 am #

    Hi Adrian, do u have any suggestion on how to use superpixel as backgroud removal? I mean, after we have superpixel, I want to extract the object presence in the image. (Maybe by comparing individual superpixel accordingly?)

    • Adrian Rosebrock March 7, 2018 at 9:14 am #

      This is potentially possible but you should instead use a pixel-wise segmentation algorithm. Segmentation embodies a large amount of literature but recent approaches use deep learning. Take a look at UNet and DeepMask.

      • Amy March 8, 2018 at 9:27 pm #

        Thanks Adrian for your help! 🙂

  12. PJS February 20, 2019 at 3:01 pm #

    Thanks for all your great posts. For this one, shouldn’t line16 be changed from:

    G =, mask=mask)


    G =, mask=mask)

Before you leave a comment...

Hey, Adrian here, author of the PyImageSearch blog. I'd love to hear from you, but before you submit a comment, please follow these guidelines:

  1. If you have a question, read the comments first. You should also search this page (i.e., ctrl + f) for keywords related to your question. It's likely that I have already addressed your question in the comments.
  2. If you are copying and pasting code/terminal output, please don't. Reviewing another programmers’ code is a very time consuming and tedious task, and due to the volume of emails and contact requests I receive, I simply cannot do it.
  3. Be respectful of the space. I put a lot of my own personal time into creating these free weekly tutorials. On average, each tutorial takes me 15-20 hours to put together. I love offering these guides to you and I take pride in the content I create. Therefore, I will not approve comments that include large code blocks/terminal output as it destroys the formatting of the page. Kindly be respectful of this space.
  4. Be patient. I receive 200+ comments and emails per day. Due to spam, and my desire to personally answer as many questions as I can, I hand moderate all new comments (typically once per week). I try to answer as many questions as I can, but I'm only one person. Please don't be offended if I cannot get to your question
  5. Do you need priority support? Consider purchasing one of my books and courses. I place customer questions and emails in a separate, special priority queue and answer them first. If you are a customer of mine you will receive a guaranteed response from me. If there's any time left over, I focus on the community at large and attempt to answer as many of those questions as I possibly can.

Thank you for keeping these guidelines in mind before submitting your comment.

Leave a Reply