Neural Style Transfer with OpenCV

Click here to download the source code to this post.

In this tutorial, you will learn how to apply neural style transfer to both images and real-time video using OpenCV, Python, and deep learning. By the end of this guide, you’ll be able to generate beautiful works of art with neural style transfer.

The original neural style transfer algorithm was introduced by Gatys et al. in their 2015 paper, A Neural Algorithm of Artistic Style (in fact, this is the exact algorithm that I teach you how to implement and train from scratch inside Deep Learning for Computer Vision with Python).

In 2016, Johnson et al. published Perceptual Losses for Real-Time Style Transfer and Super- Resolution, which frames neural style transfer as a super-resolution-like problem using perceptual loss. The end result is a neural style transfer algorithm which is up to three orders of magnitude faster than the Gatys et al. method (there are a few downsides though and I’ll be discussing them later in the guide).

In the rest of this post you will learn how to apply the neural style transfer algorithm to your own images and video streams.

To learn how to apply neural style transfer using OpenCV and Python, just keep reading!

Looking for the source code to this post?
Jump right to the downloads section.

Neural Style Transfer with OpenCV

In the remainder of today’s guide I will be demonstrating how you can apply the neural style transfer algorithm using OpenCV and Python to generate your own works of art.

The method I’m discussing here today is capable of running in near real-time on a CPU and is fully capable of obtaining super real-time performance on your GPU.

We’ll start with a brief discussion of neural style transfer, including what it is and how it works.

From there we’ll utilize OpenCV and Python to actually apply neural style transfer.

What is neural style transfer?

Figure 1: Neural style transfer with OpenCV example. Our content image (left). Style image (middle). Stylized output (right).

Neural style transfer is the process of:

  1. Taking the style of one image
  2. And then applying it to the content of another image

An example of the neural style transfer process can be seen in Figure 1. On the left we have our content image — a serene view of myself enjoying a beer on top of a mountain in the Black Forest of Germany, overlooking the town of Baden.

In the middle is our style image, Vincent van Gogh’s famous The Starry Night.

And on the right is the output of applying the style of van Gogh’s Starry Night to the content of my photo of Germany’s Black Forest. Notice how we have retained the content of the rolling hills, forest, myself, and even the beers, but have applied the style of Starry Night — it’s as if Van Gogh had applied his masterful paint strokes to our scenic view!

The question is, how do we define a neural network to perform neural style transfer?

Is that even possible?

You bet it is — and we’ll be discussing how neural style transfer is made possible in the next section.

How does neural style transfer work?

Figure 2: Neural Style Transfer with OpenCV possible (Figure 1 of Gatys et. al. 2015).

At this point you’re probably scratching your head and thinking something along the lines of: “How do we define a neural network to perform style transfer?”

Interestingly, the original 2015 paper by Gatys et al. proposed a neural style transfer algorithm that does not require a new architecture at all. Instead, we can take a pre-trained network (typically on ImageNet) and define a loss function that will enable us to achieve our end goal of style transfer and then optimize over that loss function.

Therefore, the question isn’t “What neural network do we use?” but rather “What loss function do we use?”

The answer is a three-component loss function, including:

  1. Content loss
  2. Style loss
  3. Total-variation loss

Each component is individually computed and then combined in a single meta-loss function. By minimizing the meta-loss function we will be in turn jointly optimizing the content, style, and total-variation loss as well.

While the Gatys et al. method produced beautiful neural style transfer results, the problem was that it was quite slow.

Johnson et al. (2016) built on the work of Gatys et al., proposing a neural style transfer algorithm that is up to three orders of magnitude faster. The Johnson et al. method frames neural style transfer as a super-resolution-like problem based on perceptual loss functions.

While the Johnson et al. method is certainly fast, the biggest downside is that you cannot arbitrarily select your style images like you could in the Gatys et al. method.

Instead, you first need to explicitly train a network to reproduce the style of your desired image. Once the network is trained, you can then apply it to any content image you wish. You should see the Johnson et al. method as a more of an “investment” in your style image — you better like your style image as you’ll be training your own network to reproduce its style on content images.

Johnson et al. provide documentation on how to train your own neural style transfer models on their official GitHub page.

Finally, it’s also worth noting that that in Ulyanov et al.’s 2017 publication, Instance Normalization: The Missing Ingredient for Fast Stylization, it was found that swapping batch normalization for instance normalization (and applying instance normalization at both training and testing), leads to even faster real-time performance and arguably more aesthetically pleasing results as well.

I have included both the models used by Johnson et al. in their ECCV paper along with the Ulyanov et al. models in the “Downloads” section of this post — be sure to download them so you can follow along with the remainder of this guide.

And if you’re interested in learning more about how neural style transfer works, be sure to refer to my book, Deep Learning for Computer Vision with Python.

Project structure

Today’s project includes a number of files which you can grab from the “Downloads” section.

Once you’ve grabbed the scripts + models + images, you can inspect the project structure with the tree  command:

Once you use the “Downloads” section of the blog post to grab the .zip, you won’t need to go hunting for anything else online. I’ve provided a handful of test images/  as well as a number of models/  that have already been trained by Johnson et. al. You’ll also find three Python scripts to work with and we’ll be reviewing two of them today.

Implementing neural style transfer

Let’s get started implementing neural style transfer with OpenCV and Python.

Open up your neural_style_transfer.py  file and insert the following code:

First, we import our required packages and parse command line arguments.

Our notable imports are:

  • imutils: This package is pip-installable via pip install --upgrade imutils . I recently released imutils==0.5.1 , so don’t forget to upgrade!
  • OpenCV: You need OpenCV 3.4 or better in order to use today’s code. You can install OpenCV 4 using my tutorials for Ubuntu and macOS.

We have two required command line arguments for this script:

  • --model : The neural style transfer model path. I’ve included 11 pre-trained models for you to use in the “Downloads”.
  • --image : Our input image which we’ll apply the neural style to. I’ve included 4 sample images. Feel free to experiment with your own as well!

You do not have to change the command line argument code — the arguments are passed and processed at runtime. If you aren’t familiar with how this works, be sure to read my command line arguments + argparse blog post.

Now comes the fun part — we’re going to load our image + model and then compute neural style transfer:

In this code block we proceed to:

  • Load a pre-trained neural style transfer model into memory as net  (Line 17).
  • Load the input image  and resize it (Lines 21 and 22).
  • Construct a blob  by performing mean subtraction (Lines 27 and 28). Read about cv2.dnn.blobFromImage  and how it works in my previous blog post.
  • Perform a  forward  pass to obtain an  output  image (i.e., the result of the neural style transfer process) on Line 31. I’ve also surrounded this line with timestamps for benchmarking purposes.

Next, it is critical that we post-process the output  image:

For the particular image I’m using for this example, the output  NumPy array will have the shape (1, 3, 452, 600) :

  • The 1  indicates that we passed a batch size of one (i.e., just our single image) through the network.
  • OpenCV is using channels-first ordering here, indicating there are 3  channels in the output image.
  • The final two values in the output shape are the number of rows (height) and number of columns (width).

We reshape the matrix to simply be (3, H, W)  (Line 36) and then “de-process” the image by:

  1. Adding back in the mean values we previously subtracted (Lines 37-39).
  2. Scaling (Line 40).
  3. Transposing the matrix to be channels-last ordering (Line 41).

The final step is to show the output of the neural style transfer process to our screen:

Neural style transfer results

In order to replicate my results, you will need to grab the “Downloads” for this blog post.

Once you’ve grabbed the files, open up terminal and execute the following command:

Figure 3: Neural Style Transfer with OpenCV applied to a picture of me feeding a giraffe. The output is stylized from The Great Wave Off Kanagawa which you can see in the top-left corner.

Now, simply change the command line arguments to use a screen capture from my favorite movie, Jurassic Park, as the content image, and then The Scream style model:

Figure 4: A Jurassic Park movie frame undergoes Neural Style Transfer using OpenCV. The Scream style is applied to the image producing an artistic effect.

And changing the command line arguments in your terminal once more:

Figure 5: Using the artist Francis Picabia’s famous Udnie, Young American Girl oil painting, an image of Lionel Messi, Argentinian soccer player, undergoes Neural Style Transfer with OpenCV.

Figure 5 is arguably my favorite — it just feels like it could be printed and hung on a wall in a sports bar.

In these three examples, we’ve created deep learning art! In the terminal output, the time elapsed to compute the output image is shown — each CNN model is a little bit different and you should expect different timings for each of the models.

Challenge! Can you create fancy deep learning artwork with neural style transfer? I’d love to see you tweet your artwork results — just use the hashtag, #neuralstyletransfer and mention me in the tweet (@PyImageSearch). Also, be sure to give credit to the artists and photographers — tag them if they are on Twitter as well.

Real-time neural style transfer

Now that we’ve learned how to apply neural style transfer to single images, let’s learn how to apply the process to (near) real-time video as well.

The process is quite similar to performing neural style transfer on a static image. In this script, we’ll:

  • Utilize a special Python iterator which will allow us to cycle over all available neural style transfer models in our models  path.
  • Start our webcam video stream — our webcam frames will be processed in (near) real-time. Slower systems may lag quite a bit for certain larger models.
  • Loop over incoming frames.
  • Perform neural style transfer on the frame, post-process the output, and display the result to the screen (you’ll recognize this from above as it is nearly identical).
  • If the user presses the “n” key on their keyboard, we’ll utilize the iterator to cycle to the next neural style transfer model without having to stop/restart the script.

Without further ado, let’s get to it.

Open up your neural_style_transfer_video.py  file and insert the following code:

We begin by importing required packages/modules.

From there, we just need the path to our models/  directory (a selection of models is included with today’s “Downloads”). The command line argument, --models , coupled with argparse , allows us to pass the path at runtime.

Next, let’s create our model path iterator:

Once we begin processing frames in a while  loop (to be covered in a few code blocks), a “n” keypress will load the “next” model in the iterator. This allows you to see the effect of each neural style model in your video stream without having to stop your script, change your model path, and then restart.

To construct our model iterator, we:

  • Grab and sort paths to all neural style transfer models (Lines 18 and 19).
  • Assign a unique ID to each (Line 23).
  • Use itertools  and cycle  to create an iterator (Line 27). Essentially, cycle  allows us to create a circular list which when you reach the end of it, starts back at the beginning.

Calling the  next  method of the modelIter  grabs our first modelID  and modelPath  (Line 28).

If you are new to Python iterators or iterators in general (most programming languages implement them), then be sure to give this article by RealPython a read.

Let’s load the first neural style transfer model and initialize our video stream:

On Line 32, we read the first neural style transfer model using its path.

Then on Lines 36 and 37, we initialize our video stream so we can grab frames from our webcam.

Let’s begin looping over frames:

Our while  loop begins on Line 41.

Lines 43-57 are nearly identical to the previous script we reviewed with the only exception being that we load a frame from the video stream rather than an image file on disk.

Essentially we grab the frame , preprocess it into a blob , and send it through the CNN. Be sure to refer to scroll up to my previous explanation if you didn’t read it already.

There’s a lot of computation going on behind the scenes here in the CNN. If you’re curious how to train your own neural style transfer model with Keras, be sure to refer to my book, Deep Learning for Computer Vision with Python.

Next, we’ll post-process and display the output  image:

Again, Lines 61-66 are identical to the static image neural style script above where I explained these lines in detail. These lines are critical to you seeing the correct result. Our output image is “de-processed” by reshaping, mean addition (since we subtracted the mean earlier), rescaling, and transposing.

The output of our neural style transfer is shown on Lines 70 and 71, where both the original and processed frames are displayed on the screen.

We also capture keypresses on Line 72. The keys are processed in the next block:

There are two keys that will cause different behaviors while the script is running:

  • “n”: Grabs the “next” neural style transfer model path + ID and loads it (Lines 76-80). If we’ve reached the last model, the iterator will cycle back to the beginning.
  • “q”: Pressing the “q” key will “quit” the while  loop (Lines 83 and 84).

Cleanup is then performed on the remaining lines.

Real-time neural style transfer results

Once you’ve used the “Downloads” section of this tutorial to download the source code and neural style transfer models, you can execute the following command to apply style transfer to your own video streams:

As you can see, it’s easy to cycle through the neural style transfer models using a single keypress.

I have included a short demo video of myself applying neural transfer below:

 

Where can I learn more about neural style transfer?

If you’re interested in learning more about neural style transfer, including the history, theory, and implementing your own custom neural style transfer pipeline with Keras, I would suggest you take a look at my book, Deep Learning for Computer Vision with Python:

Inside the book I discuss the Gatys et al. method in detail, including fully-documented Python + Keras code.

Using the methods discussed in the book, you’ll be able to apply neural style transfer to your own style and content images.

Inside the book you’ll also find:

  • Super practical walkthroughs that present solutions to actual, real-world image classification problems, challenges, and competitions.
  • Hands-on tutorials (with lots of code) that not only show you the algorithms behind deep learning for computer vision but their implementations as well.
  • A no-nonsense teaching style that is guaranteed to help you master deep learning for image understanding and visual recognition.

To learn more about the book (and grab your set of free sample chapters + table of contents), just use this link!

Summary

In today’s blog post you learned how to apply neural style transfer to both images and video using OpenCV and Python.

Specifically, we utilized the models trained by Johnson et al. in their 2016 publication on neural style transfer — for your convenience, I have included the models in the “Downloads” section of this blog post.

I hope you enjoyed today’s tutorial on neural style transfer!

Be sure to use Twitter and the comments section to post links to your own beautiful works of art — I can’t wait to see them!

To be notified when future blog posts are published here on PyImageSearch, just enter your email address in the form below!

Downloads:

If you would like to download the code and images used in this post, please enter your email address in the form below. Not only will you get a .zip of the code, I’ll also send you a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL! Sound good? If so, enter your email address and I’ll send you the code immediately!

, , ,

56 Responses to Neural Style Transfer with OpenCV

  1. Huy Ngo August 27, 2018 at 10:20 am #

    Absolutely amazing tutorial, Adrian! Thank you for doing this!

    • Adrian Rosebrock August 27, 2018 at 4:07 pm #

      Thanks Huy Ngo, I’m so happy you liked the guide! Have fun with it 🙂

  2. Ritika Agarwal August 27, 2018 at 11:21 am #

    Awesome tutorial!!

    • Adrian Rosebrock August 27, 2018 at 4:07 pm #

      Thanks Ritika! 🙂

  3. Mark West August 27, 2018 at 11:46 am #

    Great post! Even though I know this will run *very* slowly on a Raspberry Pi 3 B+ I’m stil tempted to give it a try!

    • Adrian Rosebrock August 27, 2018 at 4:07 pm #

      It will run slower for sure, but it will still work. You could at the very least still apply it to single images.

  4. nisar August 27, 2018 at 12:31 pm #

    can you provide source guide for vehicle automation?

    • Adrian Rosebrock August 27, 2018 at 4:07 pm #

      Vehicle automation is a pretty big area, far more than what could be covered in a single blog post. Perhaps if you could be a bit more specific?

  5. holger August 27, 2018 at 12:59 pm #

    Wow – i really really liked the jurassic park picture 🙂
    “…fully capable of obtaining super real-time performance on your GPU…”

    I want to have this – this would mean running the model directly on torch(with gpu support enabled) – right? Open cv has no cuda support and open cl / halide also ist no big help imho.

    • Adrian Rosebrock August 27, 2018 at 4:06 pm #

      The Jurassic Park one is my favorite as well! You could run the model directly via Torch but I also know that OpenCV is working on Python + CUDA bindings. Once those are available I will be sharing how to use them.

  6. Walid August 27, 2018 at 4:01 pm #

    Worked great as usual
    My question is , how can I show the style image too besides the original and the output

    Thanks a lot
    Walid

    • Adrian Rosebrock August 27, 2018 at 4:05 pm #

      The networks used in this example were actually pre-trained using the style image so you can’t actually “show” the style image unless you wanted to explicitly save it to disk and then show it when executing. I have included links to each of the respective works of art in the post though 🙂

  7. Aldo August 27, 2018 at 4:38 pm #

    Great post Adrian!
    I have a question, how can I get more models in ‘.t7’ extension?

  8. Juan August 27, 2018 at 6:52 pm #

    Hi Adrian! very good post! I have a question regarding the paintings that are used as a base in the neural network, is it possible to change them? Is the retraining of the model necessary with each one? thank you very much!

    • Adrian Rosebrock August 28, 2018 at 9:39 am #

      If you wanted to use a different style image you would need to train the network from scratch. This behavior is one of the downsides of the Johnson et al. method to neural style transfer. The original Gatys et al. method allowed you to supply your style and content images on the fly (but would take longer to perform the transfer).

  9. qiang92 August 27, 2018 at 9:14 pm #

    Why don’t you use enumerate(models) instead of zip(range(0, len(modelPaths)), (modelPaths)))

  10. secondaria August 27, 2018 at 11:20 pm #

    Adrian Rosebrock, thank you for your blog post.Really thank you! Awesome.

    • Adrian Rosebrock August 28, 2018 at 9:39 am #

      Thanks so much, I’m so happy to hear you enjoyed the tutorial! 🙂

  11. yueyu August 28, 2018 at 2:06 am #

    How do I know which models of OpenCV DNN support? I can’t see that it supports style migration model again

    • Adrian Rosebrock August 28, 2018 at 9:40 am #

      The OpenCV docs actually mention which layer types are supported, tips on migrating networks, etc.

  12. Mark C August 28, 2018 at 3:01 am #

    This looks great i did some generative neural network and it is really cool.

    • Adrian Rosebrock August 28, 2018 at 9:40 am #

      Nice job, Mark!

  13. atom August 28, 2018 at 5:19 am #

    Great post again, Mr Incredible!

    One odd wonder around reason you trained networks using torch instead others

    • Adrian Rosebrock August 28, 2018 at 9:41 am #

      I actually didn’t train the networks. The networks were trained by Johnson et al.

  14. Karla August 28, 2018 at 8:09 am #

    Excellent tutorial, congratulations!!

    One question, how can we save the output image?

    I’ve been trying to do so, but as float32 ndarray, imwrite function does not display the same result as imshow. The conversion to uint8 (in order to save the image to disk) makes the colors from the saved image completely different to the ones seen in the image of the output window.

    Thanks in advance!

    Karla

    • Adrian Rosebrock August 28, 2018 at 9:42 am #

      Interesting — I just replicated the same error as well. I’m traveling for PyImageConf right now so I can’t provide a solution but in the meantime I would suggest posting on the official OpenCV GitHub page to see if this is a known bug or if there is a workaround.

      • Karla August 29, 2018 at 4:35 am #

        Thank you for attempting a quick try Adrian, I’ll try to come back with a solution. Have fun at PyImageConf! 🙂

        • Adrian Rosebrock August 30, 2018 at 9:01 am #

          Thanks Karla!

  15. Toni August 28, 2018 at 9:52 am #

    Nice Tutorial Adrian.

    Can we save output results as a png or jpg file ? cv2.imwrite function produces empty files.

    Thanks a lot,

    Toni

    • Adrian Rosebrock August 28, 2018 at 2:04 pm #

      See my reply to Karla — I would appreciate if other PyImageSearch readers could look into it as well 🙂

  16. Big Adam August 28, 2018 at 11:43 am #

    Thank you for the great tutorial.
    I am confused about line 40, why do we scale the values?

    • Karla August 29, 2018 at 3:47 am #

      The output is a Numpy array with values within the [0-1] range, and so we convert the color scale to [0-255] range for RGB values, since this is the range for each individual color (8-bit Red, Green, Blue). By dividing by 255, the [0-255] range can be described with a [0.0-1.0] range where 0.0 means 0 (0x00) and 1.0 means 255 (0xFF).

      • BMSuser September 17, 2018 at 9:37 pm #

        output *= 255.0
        cv2.imwrite(“saved.png”, output)

  17. Z Abrams August 28, 2018 at 1:39 pm #

    Cool post!
    Any chance you can explain/link-to how we can create a model using Keras (TF backend, like in your book), and then use that instead of a .t7 Torch file? I see that OpenCV has a readNetFromTensorflow , so is there a way to save the model and then reload it?
    [I’ve heard Tensorflow is “slower” than Caffe/Torch, but let’s forget that for now.]
    Also, I assume the speed is Network dependent, so basing it off a VGG16/19 seems a bit overkill considering newer, leaner models (mobilenet/googlenet/etc).There’s no reason these things shouldn’t be <5MB, like ENet.

    • Adrian Rosebrock August 28, 2018 at 2:03 pm #

      You can’t use this code directly for a Keras-trained model. You would need to swap out the OpenCV model loading code for the Keras code and then apply it directly to the images/frames. The OpenCV + TensorFlow model loading functionalities are getting better, but they’re not quite there (in my opinion). You’re better off using strict TensorFlow or strict Keras to load a model and classify an image/frame than trying to make it work with OpenCV.

  18. Abhishek Singh August 28, 2018 at 3:08 pm #

    Hey Adrian, Any help here?

    python neural_style_transfer.py –image images/giraffe.jpg \
    > –model models/eccv16/the_wave.t7

    [INFO] Importing Neural Style Model
    Traceback (most recent call last):
    File “neural_style_transfer.py”, line 15, in
    net = cv2.dnn.readNetFromTorch(args[“model”])

    AttributeError: ‘module’ object has no attribute ‘dnn’

    • Adrian Rosebrock August 30, 2018 at 9:11 am #

      It sounds like you’re using an older version of OpenCV. What version of OpenCV are you using? I recommend OpenCV 3.4.1 or newer for this post.

  19. jorge August 31, 2018 at 1:55 pm #

    Hi Adrian great post!

    I would like to save the modified image using cv2.imwrite(‘famMod.jpg’,output) at the end of the script but the file saved didn’t show anything.

    You know why.

    • Adrian Rosebrock September 5, 2018 at 9:18 am #

      Make sure you read the comments section. In particular, my reply to Karla.

  20. Tracy August 31, 2018 at 5:40 pm #

    Is there a way for me to try this out without buying the book?

    • Adrian Rosebrock September 5, 2018 at 9:15 am #

      Hey Tracy — I’m not sure what you mean regarding buying my book. You don’t have to buy anything if you don’t want. Just use the “Downloads” section of the blog post to download my source code and example images.

  21. J Serena September 1, 2018 at 6:06 am #

    Hello Adrian,

    I starter with Machine Learning few months ago and your articles have been the lighthouse that guided me.

    I hope you will be writing so interesting articles for a long time.

    • Adrian Rosebrock September 5, 2018 at 9:12 am #

      Thank you for the kind words, I really appreciate it 🙂

  22. Raj September 8, 2018 at 3:36 pm #

    Thanks Adrian. All the articles here are brilliant. I really helps me understand the concepts and applications since I am a newbie in this field.

    • Adrian Rosebrock September 11, 2018 at 8:22 am #

      Thanks Raj, I’m happy you are enjoying the tutorials 🙂 And congratulations on getting your start in computer vision and deep learning.

  23. nettoyer September 12, 2018 at 5:01 pm #

    Hello Adrian, there is my image saving workaround via ‘imageio’:
    import imageio

    output = cv2.cvtColor(output, cv2.COLOR_BGR2RGB)
    imageio.imwrite(‘output.jpg’, output)

    P.S.
    many thanks for your great example that push to learn this area

    • Adrian Rosebrock September 14, 2018 at 9:46 am #

      Thanks so much for sharing!

      • Alex September 15, 2018 at 3:00 am #

        Thank you all very much. Also trying to save – but gives an error

        WARNING:root:Lossy conversion from float32 to uint8.

  24. Deepak September 28, 2018 at 1:06 pm #

    Can u write a blog on self driving car
    Basically Lane following raspberry pi car
    Stop detector,red light etc

  25. Marco Heimeshoff October 17, 2018 at 10:56 am #

    This is wonderful. How do I activate the GPU though, right now it is just running on the processor.

    I have a Windows 10 64bit machine with a GTX1060 and Cuda is installed, if that helps

  26. Jim October 19, 2018 at 6:52 pm #

    How to use GPU acceleration with this Python code? I’ve install CUDA and numba, as this video https://www.youtube.com/watch?v=vMZ7tK-RYYc has told. While, when I running the code, I found my GPU still not occupied.

    Thanks a lot!

    • Adrian Rosebrock October 20, 2018 at 7:25 am #

      Unfortunately at this time you cannot use CUDA acceleration via Python. I suspect support is coming soon but it’s not there yet.

  27. Eon Kim October 29, 2018 at 12:41 pm #

    Thanks Adrian. Your articles are very helpful to me. I have a question. What is the license for this code? Can I correct this code and push it on the github?

    • Adrian Rosebrock October 29, 2018 at 1:00 pm #

      Yes, but I would appreciate an attribution with a link back to the blog post.

Leave a Reply