Black and white image colorization with OpenCV and Deep Learning

In this tutorial, you will learn how to colorize black and white images using OpenCV, Deep Learning, and Python.

Image colorization is the process of taking an input grayscale (black and white) image and then producing an output colorized image that represents the semantic colors and tones of the input (for example, an ocean on a clear sunny day must be plausibly “blue” — it can’t be colored “hot pink” by the model).

Previous methods for image colorization either:

  1. Relied on significant human interaction and annotation
  2. Produced desaturated colorization

The novel approach we are going to use here today instead relies on deep learning. We will utilize a Convolutional Neural Network capable of colorizing black and white images with results that can even “fool” humans!

To learn how to perform black and white image coloration with OpenCV, just keep reading!

Looking for the source code to this post?
Jump right to the downloads section.

Black and white image colorization with OpenCV and Deep Learning

In the first part of this tutorial, we’ll discuss how deep learning can be utilized to colorize black and white images.

From there we’ll utilize OpenCV to colorize black and white images for both:

  1. Images
  2. Video streams

We’ll then explore some examples and demos of our work.

How can we colorize black and white images with deep learning?

Figure 1: Zhang et al.’s architecture for colorization of black and white images with deep learning.

The technique we’ll be covering here today is from Zhang et al.’s 2016 ECCV paper, Colorful Image Colorization.

Previous approaches to black and white image colorization relied on manual human annotation and often produced desaturated results that were not “believable” as true colorizations.

Zhang et al. decided to attack the problem of image colorization by using Convolutional Neural Networks to “hallucinate” what an input grayscale image would look like when colorized.

To train the network Zhang et al. started with the ImageNet dataset and converted all images from the RGB color space to the Lab color space.

Similar to the RGB color space, the Lab color space has three channels. But unlike the RGB color space, Lab encodes color information differently:

  • The L channel encodes lightness intensity only
  • The a channel encodes green-red.
  • And the b channel encodes blue-yellow

A full review of the Lab color space is outside the scope of this post (see this guide for more information on Lab), but the gist here is that Lab does a better job representing how humans see color.

Since the L channel encodes only the intensity, we can use the L channel as our grayscale input to the network.

From there the network must learn to predict the a and b channels. Given the input L channel and the predicted ab channels we can then form our final output image.

The entire (simplified) process can be summarized as:

  1. Convert all training images from the RGB color space to the Lab color space.
  2. Use the L channel as the input to the network and train the network to predict the ab channels.
  3. Combine the input L channel with the predicted ab channels.
  4. Convert the Lab image back to RGB.

To produce more plausible black and white image colorizations the authors also utilize a few additional techniques including mean annealing and a specialized loss function for color rebalancing (both of which are outside the scope of this post).

For more details on the image colorization algorithm and deep learning model, be sure to refer to the official publication of Zhang et al.

Project structure

Go ahead and download the source code, model, and example images using the “Downloads” section of this post.

Once you’ve extracted the zip, you should navigate into the project directory.

From there, let’s use the tree  command to inspect the project structure:

We have four sample black and white images in the images/  directory.

Our Caffe model and prototxt are inside the model/  directory along with the cluster points NumPy file.

We’ll be reviewing two scripts today:

  • bw2color_image.py
  • bw2color_video.py

The image script can process any black and white (also known as grayscale) image you pass in.

Our video script will either use your webcam or accept an input video file and then perform colorization.

Colorizing black and white images with OpenCV

Let’s go ahead and implement black and white image colorization script with OpenCV.

Open up the bw2color_image.py file and insert the following code:

Our colorizer script only requires three imports: NumPy, OpenCV, and argparse .

Let’s go ahead and use argparse to parse command line arguments. This script requires that these four arguments be passed to the script directly from the terminal:

  • --image : The path to our input black/white image.
  • --prototxt : Our path to the Caffe prototxt file.
  • --model . Our path to the Caffe pre-trained model.
  • --points : The path to a NumPy cluster center points file.

With the above four flags and corresponding arguments, the script will be able to run with different inputs without changing any code.

Let’s go ahead and load our model and cluster centers into memory:

Line 21 loads our Caffe model directly from the command line argument values. OpenCV can read Caffe models via the  cv2.dnn.readNetFromCaffe function.

Line 22 then loads the cluster center points directly from the command line argument path to the points file. This file is in NumPy format so we’re using np.load .

From there, Lines 25-29:

  • Load centers for ab channel quantization used for rebalancing.
  • Treat each of the points as 1×1 convolutions and add them to the model.

Now let’s load, scale, and convert our image:

To load our input image from the file path, we use cv2.imread  on Line 34.

Preprocessing steps include:

  • Scaling pixel intensities to the range [0, 1] (Line 35).
  • Converting from BGR to Lab color space (Line 36).

Let’s continue with our preprocessing:

We’ll go ahead and resize the input image to 224×224 (Line 41), the required input dimensions for the network.

Then we grab the L  channel only (i.e., the input) and perform mean subtraction (Lines 42 and 43).

Now we can pass the input L channel through the network to predict the ab channels:

A forward pass of the L  channel through the network takes place on Lines 48 and 49 (here is a refresher on OpenCV’s blobFromImage if you need it).

Notice that after we called net.forward , on the same line, we went ahead and extracted the predicted ab  volume. I make it look easy here, but refer to the Zhang et al. documentation and demo on GitHub if you would like more details.

From there, we resize the predicted ab  volume to be the same dimensions as our input image (Line 53).

Now comes the time for post-processing. Stay with me here as we essentially go in reverse for some of our previous steps:

Post processing includes:

  • Grabbing the L  channel from the original input image (Line 58) and concatenating the original L  channel and predicted ab  channels together forming colorized  (Line 59).
  • Converting the colorized image from the Lab color space to RGB (Line 63).
  • Clipping any pixel intensities that fall outside the range [0, 1] (Line 64).
  • Bringing the pixel intensities back into the range [0, 255] (Line 69). During the preprocessing steps (Line 35) we divided by 255  and now we are multiplying by 255 . I’ve also found that this scaling and "uint8"  conversion isn’t a requirement but that it helps the code work between OpenCV 3.4.x and 4.x versions.

Finally, both our original image  and colorized  images are displayed on the screen!

Image colorization results

Now that we’ve implemented our image colorization script, let’s give it a try.

Make sure you’ve used the “Downloads” section of this blog post to download the source code, colorization model, and example images.

From there, open up a terminal, navigate to where you downloaded the source code, and execute the following command:

Figure 2: Grayscale image colorization with OpenCV and deep learning. This is a picture of famous late actor, Robin Williams.

On the left, you can see the original input image of Robin Williams, a famous actor and comedian who passed away ~5 years ago.

On the right, you can see the output of the black and white colorization model.

Let’s try another image, this one of Albert Einstein:

Figure 3: Image colorization using deep learning and OpenCV. This is an image of Albert Einstein.

I’m particularly impressed by this image colorization.

Notice how the water is an appropriate shade of blue while Einstein’s shirt is white and his pants are khaki — all of these are plausible colorizations.

Here is another example image, this one of Mark Twain, one of my all-time favorite authors:

Figure 4: A black/white image of Mark Twain has undergone colorization via OpenCV and deep learning.

Here we can see that the grass and foliage are correctly colored a shade of green, although you can see these shades of green blending into Twain’s shoes and hands.

The final image demonstrates a not-so-great black and white image colorization with OpenCV:

Figure 5: Janie is the puppers we recently adopted into our family. This is her first snow day. Black and white cameras/images are great for snow, but I wanted to see how image colorization would turn out with OpenCV and deep learning.

This photo is of myself and Janie, my beagle puppy, during a snowstorm a few weeks ago.

Here you can see that while the snow, Janie, my jacket, and even the gazebo in the background are correctly colored, my blue jeans are actually red.

Not all image colorizations will be perfect but the results here today do demonstrate the plausibility of the Zhang et al. approach.

Real-time black and white video colorization with OpenCV

We’ve already seen how we can apply black and white image colorization to images — but can we do the same with video streams?

You bet we can.

This script follows the same process as above except we’ll be processing frames of a video stream. I’ll be reviewing it in less detail and focusing on the frame grabbing + processing aspects.

Open up the bw2color_video.py and insert the following code:

Our video script requires two additional imports:

  • VideoStream allows us to grab frames from a webcam or video file
  • time  will be used to pause to allow a webcam to warm up

Let’s initialize our VideoStream  now:

Depending on whether we’re working with a webcam  or video file, we’ll create our vs  (i.e., “video stream”) object here.

From there, we’ll load the colorizer deep learning model and cluster centers (the same way we did in our previous script):

Now we’ll start an infinite while  loop over incoming frames. We’ll process the frames directly in the loop:

Each frame from our vs  is grabbed on Lines 55 and 56. A check is made for a None  type frame  — when this occurs, we’ve reached the end of a video file (if we’re processing a video file) and we can break  from the loop (Lines 60 and 61).

Preprocessing (just as before) is conducted on Lines 66-75. This is where we resize, scale, and convert to Lab. Then we grab the L  channel, and perform mean subtraction.

Let’s now apply deep learning colorization and post-process the result:

Our deep learning forward pass of L through the network results in the predicted ab  channel.

Then we’ll post-process the result to from our colorized  image (Lines 86-95). This is where we resize, grab our original L , and concatenate our predicted ab . From there, we convert from Lab to RGB, clip, and scale.

If you followed along closely above, you’ll remember that all we do next is display the results:

Our original webcam frame  is shown along with our grayscale image and colorized  result.

If the "q"  key  is pressed, we’ll break  from the loop and cleanup.

That’s all there is to it!

Video colorization results

Let’s go ahead and give our video black and white colorization script a try.

Make sure you use the “Downloads” section of this tutorial to download the source code and colorization model.

From there, open up a terminal and execute the following command to have the colorizer run on your webcam:

Figure 6: Black and white image colorization in video with OpenCV and deep learning demo.

If you want to run the colorizer on a video file you can use the following command:

Credits:

The model here is running in close to real-time on my 3Ghz Intel Xeon W.

With a GPU, real-time performance could certainly be obtained; however, keep in mind that GPU support for OpenCV’s “dnn” module is currently a bit limited and it, unfortunately, does not yet support NVIDIA GPUs.

Summary

In today’s tutorial, you learned how to colorize black and white images using OpenCV and Deep Learning.

The image colorization model we used here today was first introduced by Zhang et al. in their 2016 publication, Colorful Image Colorization.

Using this model, we were able to colorize both:

  1. Black and white images
  2. Black and white videos

Our results, while not perfect, demonstrated the plausibility of automatically colorizing black and white images and videos.

According to Zhang et al., their approach was able to “fool” humans 32% of the time!

To download the source code to this post, and be notified when future tutorials are published here on PyImageSearch, just enter your email address in the form below!

Downloads:

If you would like to download the code and images used in this post, please enter your email address in the form below. Not only will you get a .zip of the code, I’ll also send you a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL! Sound good? If so, enter your email address and I’ll send you the code immediately!

61 Responses to Black and white image colorization with OpenCV and Deep Learning

  1. Billal BEGUERADJ February 25, 2019 at 11:47 am #

    I worked on an images colorization project but I tackled with a traditional way: first I had to segment the images. This is a new interesting method to do it. Thank you for sharing.

    • Adrian Rosebrock February 25, 2019 at 2:32 pm #

      You’re welcome Billal, enjoy it!

  2. Bob February 25, 2019 at 12:36 pm #

    Wow, that’s pretty cool!

    • Adrian Rosebrock February 25, 2019 at 2:32 pm #

      Thanks Bob!

  3. seraj alhamidi February 25, 2019 at 3:07 pm #

    pretty n cool , u r a awesome

    • Adrian Rosebrock February 26, 2019 at 6:26 am #

      Thanks Seraj, you are very kind.

  4. Said February 25, 2019 at 3:24 pm #

    Adrian, thanks for such a great posts in your blog.
    Your work is a big inspiration for many people.

    • Adrian Rosebrock February 26, 2019 at 6:25 am #

      Thanks Said!

  5. David Bonn February 25, 2019 at 3:46 pm #

    Very cool Adrian!

    I’m continually impressed by newer applications of Deep Learning beyond basic classification, regression, and object detection. There is so much more here to learn!

    • Adrian Rosebrock February 26, 2019 at 6:25 am #

      Absolutely! Deep learning is being applied not only to the field of computer vision but also many sub-fields and niches of the computer vision field. It’s a great time to be studying computer vision!

  6. Ed Freeman February 25, 2019 at 4:25 pm #

    How well does it work on black people? Asian people?

    • Adrian Rosebrock February 26, 2019 at 6:25 am #

      Hey Ed — I suggest you try it and see!

    • Plaban Das February 26, 2019 at 12:48 pm #

      That would be really interesting to look on.

  7. Rahat February 25, 2019 at 5:25 pm #

    Figure 6 video shows temporal inconsistency. Do you know any algorithm that can solve this issue and make the generated video look like the original one (without any glitch / flicker) ?

    • Taha March 3, 2019 at 4:38 pm #

      The current approach cannot solve temporal inconsistency so there was a paper in which the authors developed a color transfer method for videos in which they solved temporal flickering in generated art videos by using some form of optical flow….i guess some modified version can be used to solve this prob too…so if u end up finding an open source implementation of such alog tell me here

  8. Denis Rothman February 25, 2019 at 5:28 pm #

    Adrian, Your blog is absolutely marvelous! I’m a huge collector of AI algorithms for my bibliography background work. You’re unique in the way you explain and implement. Bravo!

    • Adrian Rosebrock February 26, 2019 at 6:24 am #

      Thanks Denis, I really appreciate that. It’s my pleasure to teach and share.

  9. Denis Rothman February 25, 2019 at 5:52 pm #

    The multiple layer network is pretty effective with the L lighting and 2 color channels coming in separately. That’s ingenious.
    So I gather the model is a caffe model that was trained on a dataset of images providing some pretty efficient probabilities?

    Good work!

    • Adrian Rosebrock February 26, 2019 at 6:24 am #

      The model itself was trained on ImageNet. The authors converted the ImageNet dataset to the Lab color space and then used the “L” channel as the input with the “ab” as the target output. Really ingenious and it worked quite well.

  10. Ninan February 25, 2019 at 9:25 pm #

    Adrian, you are simply remarkable!

    • Adrian Rosebrock February 26, 2019 at 6:23 am #

      Thanks Ninan, I really appreciate the kind words 🙂

  11. Daniel Trujillo February 25, 2019 at 10:35 pm #

    My webcam keeps loading before it gets to the –video function. Any advice?

    python bw2color_video.py \
    –prototxt model/colorization_deploy_v2.prototxt \
    –model model/colorization_release_v2.caffemodel \
    –points model/pts_in_hull.npy
    –video videos/blackandwhiteskatefooty.mp4

    • Daniel Trujillo February 25, 2019 at 10:45 pm #

      I of course mean when I am trying to run colorization on video, and not my webcam.

      • Daniel Trujillo February 25, 2019 at 11:13 pm #

        supposed to –input not –video

        • Adrian Rosebrock February 26, 2019 at 6:27 am #

          Thanks for pointing that out Daniel, I’ve updated the example execution of the script to use --input.

          • Aiden February 28, 2019 at 2:51 am #

            you also need a back-slash after the pts_in_hull.npy bit for colorizing a video and not the webcam i.e.

            python bw2color_video.py \
            — prototxt model/colorization_deploy_v2.prototxt \
            — model model/colorization_release_v2.caffemodel \
            — points model/pts_in_hull.npy \
            — video videos/blackandwhiteskatefooty.mp4

          • Adrian Rosebrock February 28, 2019 at 1:40 pm #

            Thanks, updated as well!

  12. mohd zamri February 25, 2019 at 11:41 pm #

    Cool. very cool. thanks. always informative and great reading.

    • Adrian Rosebrock February 26, 2019 at 6:23 am #

      Thanks Mohd, I’m glad you enjoyed it!

  13. monika February 26, 2019 at 1:00 am #

    sir, can you provide me the guide to create video defogging algorithm, its being already done in some papers but not fully explored……i am working in a project where i need clear defog image from the hazy environment. if u can suggest anything , help will be appreciated

    thank you

    • Adrian Rosebrock February 26, 2019 at 6:22 am #

      Can you link to the paper/algorithm you are referring to? I’m not familiar with it.

  14. Alex Choi February 26, 2019 at 5:56 am #

    Hi, Adrian,

    It’s a really cool tutorial.

    Last year I saw an article that showed grayscale to color, but the method wasn’t revealed.

    You’ve resolved my curiosity!

    Thanks a lot.
    Alex.

    • Adrian Rosebrock February 26, 2019 at 6:22 am #

      Thanks Alex, I’m glad you enjoyed it!

  15. Dave February 26, 2019 at 9:16 am #

    Hi Adrian,
    this is really an awesome tutorial!
    just one question, can the same method be applied to colourize a huge dataset of grayscale images(facial images)?
    Thanks alot for great tutorials!

    • Adrian Rosebrock February 27, 2019 at 5:38 am #

      Yes, there is nothing preventing you from trying to colorize a set of facial images, but as I said in the post, some images may not be “believable” in their colorizations. Give it a try and see!

  16. Plaban Das February 26, 2019 at 12:45 pm #

    Adrian, excellent work. Thanks for sharing.

    • Adrian Rosebrock February 27, 2019 at 5:33 am #

      Thanks Plaban!

  17. WG February 26, 2019 at 11:59 pm #

    I’m impressed overall.

    I tried running the script against this Dorthea Lange test photo and the results were impressive. There is quite a bit of bleed though from her shirt to the boys jacket; which is somewhat similar to what you’ve been experiencing.

    You can try it out here as your iput photo:

    https://en.wikipedia.org/wiki/Image_editing

    I’m going to keep watching this space because I think the calculations and learning algorythms are going to continue improving.

    Thanks.

  18. Chanbin Park February 27, 2019 at 7:29 am #

    Hi Adrian, thanks for your wonderful work!

    Just have one question.

    When you make your projects utilizing deep learning, do you build your algorithm from scratch or manipulate other people’s algorithms?

    • Adrian Rosebrock February 28, 2019 at 1:54 pm #

      I do both. In some tutorials I build my own datasets and train my own networks from scratch. Otherwise times I perform transfer learning/fine-tuning. And other times I utilize existing models for other purposes.

  19. Chanbin Park February 27, 2019 at 7:31 am #

    Oh and by the way, what do you mean by clustering points in the article?

    Thanks.

  20. Aiden February 27, 2019 at 5:52 pm #

    Hi Adrian,

    This is awesome!
    I’ve tried it on some old footage of my favourite football team from the 1950’s and the results are staggering! It’s like going back in time!
    Just wondering how I can save the colorized version as a file?

    • Adrian Rosebrock February 28, 2019 at 1:44 pm #

      You can use the “cv2.imwrite” function to write the output image to disk.

  21. Marcelo Rovai March 1, 2019 at 1:42 pm #

    Great tutorial, Dr. Adrian. I tried it on several photos and on a very old footage shoot in Rio in 1932. Here the result: https://youtu.be/17IfJr6C9fI

    Thanks a lot
    Marcelo

    • Adrian Rosebrock March 5, 2019 at 9:04 am #

      Very nice, thanks for sharing Marcelo!

  22. Taha Anwar March 7, 2019 at 6:37 am #

    Hey Adrian, I’ve looked at your implementation, satya’s implementation, Opnecv’s dnn official code on github and briefly gone over the original research paper but still can’t figure out what are these numpy cluster points and what role are these playing here. Can you elaborate a little on that.
    Note: I understand that these point points are achieved by k-means alog.
    Thanks

  23. Kevin Braun March 11, 2019 at 3:18 pm #

    Anyone else get this? Ubuntu 16.04/Python 2.7

    AttributeError: ‘module’ object has no attribute ‘dnn’

    Thank you.

    • Adrian Rosebrock March 13, 2019 at 3:29 pm #

      It sounds like you are using a very old version of OpenCV. The “dnn” module was added in OpenCV 3.3. You should be using OpenCV 3.4.3+ for this tutorial.

  24. Sesha Sai SIngaraju March 17, 2019 at 9:22 am #

    Hey Adrian!!
    This guide certainly sparked some idea in me man. I’ve been just watching a black and white movie released in 1957, which was colored and re-released in 2010. This got me an idea….

    Since we’ve been able to color black and white images and videos, can we apply the same logic to larger videos, I mean like full length movies?

    I know it’s gonna take very long, but if we can, we’ll be able to recolor black and white images of our choice and have fun, right?

    • Adrian Rosebrock March 19, 2019 at 10:09 am #

      Yes. It will take longer to run, but you can recolor full-length movies.

  25. Gnork March 29, 2019 at 7:03 am #

    Is there anything similar to restoring an old video, removing noise, scratches, etc.?

  26. Will April 22, 2019 at 11:06 am #

    I’m using Python 3.6.8 and OpenCV 4.1.0 and even though my code successfully compiles however i’m not getting the desired outputs.I obtain 3 images consecutively as Black ,Yellow and Dark blue -(bluish black ) on the three different stages.

    Please Help.

    • Adrian Rosebrock April 25, 2019 at 8:56 am #

      That is very, very odd. I wonder if something changed with OpenCV 4.1. Could you try with OpenCV 3.4.4 or OpenCV 4.0 to see if you obtain the same result?

      • 1111alberto July 19, 2019 at 7:17 am #

        I tested the code with python 3.7.4 and opencv 4.1 in windows , it,s work but i change the output from mp4 to mp4v ., And download openh264.dll version 2 and put it in the same folder .

  27. Deepthi May 22, 2019 at 3:05 pm #

    Hi Adrian,

    Great work!!

    I got colorized images, but could not get colorized video. It doesn’t show any error but exit after [INFO] loading model…can you please help me.

    • Adrian Rosebrock May 23, 2019 at 9:30 am #

      Double-check the path to your input video is correct (it sounds like it may not be).

  28. Deepthi June 6, 2019 at 3:44 am #

    Hi,

    Able to save image to a folder using cv2.imwrite.
    How to save video? Have seen your blog on saving usb camera or Raspberry Pi camera video to save, tried a process to save a local video.mp4 , but couldn’t get output.
    Is there anyway to save local .mp4 video to a folder.

    • Adrian Rosebrock June 6, 2019 at 6:38 am #

      Hi Deepthi — can you let me know how much prior experience you have with OpenCV and computer vision? I get the impression that you may be new to it (which is perfectly okay), but I would strongly encourage you to read Practical Python and OpenCV first. That book will teach you how to use basic functions such as “cv2.imwrite”.

  29. Nguyen Tien Su June 20, 2019 at 11:08 pm #

    Dear Adrian Rosebrock!

    Thank you very much for your lessons! I learned a lot from them!
    In this post, I want to know how can I train the model from my own dataset?
    Can you help me, please?

  30. Graham June 23, 2019 at 5:36 am #

    Hi, I successfully ran this for images and a short video within Anaconda, however, I did need to install ‘imutils’ (pip install imutils).

    The next step is how do you save the video into a new mp4 file? Help!

    Thanks

Before you leave a comment...

Hey, Adrian here, author of the PyImageSearch blog. I'd love to hear from you, but before you submit a comment, please follow these guidelines:

  1. If you have a question, read the comments first. You should also search this page (i.e., ctrl + f) for keywords related to your question. It's likely that I have already addressed your question in the comments.
  2. If you are copying and pasting code/terminal output, please don't. Reviewing another programmers’ code is a very time consuming and tedious task, and due to the volume of emails and contact requests I receive, I simply cannot do it.
  3. Be respectful of the space. I put a lot of my own personal time into creating these free weekly tutorials. On average, each tutorial takes me 15-20 hours to put together. I love offering these guides to you and I take pride in the content I create. Therefore, I will not approve comments that include large code blocks/terminal output as it destroys the formatting of the page. Kindly be respectful of this space.
  4. Be patient. I receive 200+ comments and emails per day. Due to spam, and my desire to personally answer as many questions as I can, I hand moderate all new comments (typically once per week). I try to answer as many questions as I can, but I'm only one person. Please don't be offended if I cannot get to your question
  5. Do you need priority support? Consider purchasing one of my books and courses. I place customer questions and emails in a separate, special priority queue and answer them first. If you are a customer of mine you will receive a guaranteed response from me. If there's any time left over, I focus on the community at large and attempt to answer as many of those questions as I possibly can.

Thank you for keeping these guidelines in mind before submitting your comment.

Leave a Reply

[email]
[email]