Generating art with guided deep dreaming.

van_gogh_animated

One of the main benefits of the bat-country Python package for deep dreaming and visualization is its ease of use, extensibility, and customization.

And let me tell you, that customization really came in handy last Friday when the Google Research team released an update to their deep dream work, demonstrating a method to “guide” your input images to visualize the features of a target image.

The results were quite impressive — so I decided to port the functionality to bat-country.

Truth be told, it only took ~20 minutes from start-to-finish to get the code together. I honestly spent more time running the Python scripts to gather example images and updating the documentation than I did updating the codebase.

The secret to this quick turnaround is the extensibility of the BatCountry  class where nearly every function and every method can be overridden and extended.

Want to change how each image is pre-processed or post-processed? No problem. Define your own custom processor and pass it in. Want to change the objective function? Again, just define your own objective and you’re good to go.

In fact, defining your own custom objective function is the the exact route I took when extending  bat-country . I simply defined a new objective function, allowing the step function to be further customized, and we’re done!

In the remainder of this blog post we’ll play around with the new bat-country update to perform guided dreaming — and even use it to generate our own art using guided deep dreaming!

Guided deep dreaming

Last Friday the Google Research team posted an update to their deep dream working demonstrating it was possible to guide your dreaming process by supplying a seed image. This method passes your input image through the network in a similar manner, but this time using your seed image to guide and influence the output.

Using bat-country , it’s just as easy to perform guided dreaming as deep dreaming. Here’s some quick sample code:

If you don’t already have bat-country  installed, either pull down the code from the GitHub repo or use pip to install it on your system:  pip install bat-country or  pip install --upgrade bat-country . You’ll also need a working installation of Caffe.

What’s nice about this approach is that we can “guide” what the output image looks like. Here I use a seed image of Vincent van Gogh’s Starry Night and apply to an image of clouds:

Figure 1: An example of applying guided dreaming using Starry Night and a cloud image.

Figure 1: An example of applying guided dreaming using Starry Night and a cloud image.

As you can see, the output cloud image after applying guided dreaming appears to mimic many of the brush strokes of Van Gogh’s painting.

Which got me thinking — what would happen if I took some well-known paintings from extremely famous artists such as Andy Warhol, MC Escher, Pablo Picasso, Jackson Pollock, and Vincent van Gogh and used them as inputs and guides to each other?

What would the results look like? Would the artistic style of each painting be transferred to the other?

In order to test this out, I collected images of the following pieces of work:

And then I passed each of them through the demo_guided.py  script detailed above.

Generating art with deep dreaming

Overall, the results look really fantastic. I’m especially pleased with how Wheat Field with Cypresses and Guernica turned out when used as input images and the others paintings as guided images.

Below you can find some of my favorite images:

Vincent van Gogh – Wheat Field with Cypresses

Figure 3: Vincent van Gogh's Wheat Field with Cypresses guided using Warhol's Marilyn Monroe.

Figure 2: Vincent van Gogh’s Wheat Field with Cypresses guided using Warhol’s Marilyn Monroe. Notice how eyes are now present in the sky.

Vincent van Gogh's Wheat Field with Cypresses guided using Jackson Pollock's Energy Made Visible.

Figure 3: Vincent van Gogh’s Wheat Field with Cypresses guided using Jackson Pollock’s Energy Made Visible.

Figure 5: Vincent van Gogh's Wheat Field with Cypresses guided using Picasso's Guernica.

Figure 4: Vincent van Gogh’s Wheat Field with Cypresses guided using Picasso’s Guernica.

Pablo Picasso – Guernica

Figure X: Pablo Picasso's  Guernica guided with MC Escher's Sky and Water I.

Figure 5: Pablo Picasso’s Guernica guided with MC Escher’s Mosaic II.

Figure X: Pablo Picasso – Guernica's guided with Jackson Pollock's Energy Made Visible.

Figure 6: Pablo Picasso’s Guernica guided with Jackson Pollock’s Energy Made Visible.

Figure X: Pablo Picasso Guernica's guided with Andy Warhol's Marilyn Monroe.

Figure 7: Pablo Picasso’s Guernica guided with Andy Warhol’s Marilyn Monroe.

MC Escher – Mosaic II

Figure X: MC Esher's Mosaic II guided by Jackson Pollock's Energy Made Visible.

Figure 8: MC Esher’s Mosaic II guided by Jackson Pollock’s Energy Made Visible.

Figure X: MC Esher's Mosaic II guided by Vincent van Gogh's Wheat Field with Cypresses.

Figure 9: MC Esher’s Mosaic II guided by Vincent van Gogh’s Wheat Field with Cypresses.

Andy Warhol – Marilyn Monroe

Figure 2: Andy Warhol's Marilyn Monroe guided using MC Escher's Mosaic II.

Figure 10: Andy Warhol’s Marilyn Monroe guided using MC Escher’s Sky and Water I.

Figure 3: Andy Warhol's Marilyn Monroe guided using Picasso's Guernica.

Figure 11: Andy Warhol’s Marilyn Monroe guided using Picasso’s Guernica.

Summary

In this blog post I reviewed the updates to the bat-country package will allow for guided deep dreaming by supplying two images: an input image that will be passed through the network, and a seed image that the network will use to “guide” the output.

I then took the updated code and used it to generate art from famous works, such as Jackson Pollock’s Energy Made Visible, Pablo Picasso’s Guernica, and Vincent van Gogh’s Wheat Field with Cypresses.

Definitely consider installing the bat-country  package on your system and giving deep dreaming a try! It’s strangely addictive (and not to mention, a lot of fun) to play around and generate your own images.

Finally, if you’re interested in deep learning, deep dreaming, and computer vision, think about signing up for the PyImageSearch Newsletter by entering your email address in the form below. I send out regular updates on the PyImageSearch blog, each filled with actionable, real-world computer vision projects.

See you next time!

, , , , , , , ,

9 Responses to Generating art with guided deep dreaming.

  1. gwern July 13, 2015 at 3:14 pm #

    Interesting that no matter what you seem to combine, eyes will pop up.

    • Sterling Hulett January 28, 2017 at 4:39 am #

      I believe that they are using neurons as if they’re dopped with LSD, seriously. I’m pretty sure it’s a deep learning project with neurons as if they where on LSD. I could be wrong; though, I thought I had read something about it.

      • Adrian Rosebrock January 28, 2017 at 6:44 am #

        You typically see “eyes” and “dog-like” objects in the images because many of the ImageNet classes contain animals (specifically dogs) which of course have eyes.

  2. Jeremy M July 13, 2015 at 10:19 pm #

    Hi Adrian,

    Did you install Caffe on your Mac?

    Jeremy

    • Adrian Rosebrock July 14, 2015 at 6:20 am #

      Hi Jeremy, I actually to all my deep learning work on my Amazon EC2 instance so I can leverage a more powerful GPU.

      • Jeremy M July 14, 2015 at 10:07 pm #

        Thanks Adrian. I’m struggling with installing Caffe on my Mac. There are lot of steps to it. I’ll try using Docker.

  3. Bassel July 14, 2015 at 11:24 am #

    Many thanks for the useful example although I can’t get caffe to install on my windows 7 – 64 bit using the standalone package from:
    https://github.com/niuzhiheng/caffe

    any advice!!

Leave a Reply