So you’re telling me that we are going to build a Pokedex in Python that can visually recognize Pokemon? Just like in the TV show? – Ash Ketchum
Yes, Ash. That’s exactly what I’m telling you. The next series of blog posts will be dedicated to building a Pokedex that can visually recognize Pokemon from the original Red, Blue, and Green (JP) games for the Game Boy.
Don’t believe me? Check out this video:
Motivation: Visually Recognizing Pokemon
I’ve been playing Pokemon (the original Red and Blue) since I was a little kid. I love these games and I know them front to back. In fact, every year my friend and I hold the Pokethon. We drink some beer and do a speed run of Pokemon Blue as fast as we can. This past year I managed to beat the entire game in 2 hours, 46 minutes. Was I pretty happy with that achievement? You bet. But there’s always room for improvement. Next year’s goal is sub 2 hours, 30 minutes.
I’m getting a little off topic here. A few years back I decided that I wanted to beat Pokemon Green. The Green version was only available in Japan, and as you might guess, all the text is in Japanese. I can’t read Japanese, but I still managed to get through the game.
And I thought to myself, this is what Ash Ketchum must have felt like.
Here I am, playing this game. I can’t read the text, I can’t follow the actual story of the game. But through random encounters and battles with rivals, I’m able to see and experience a lot of different Pokemon.
Now, I obviously know what each Pokemon is based on their sprite images. But let’s pretend that I didn’t.
How might I identify these Pokemon?
And how might I easily and conveniently lookup any stats related to the Pokemon?
Perhaps the most obvious strategy would be to go to Google and type in the name of the Pokemon in Japanese (character by character, according to what is on my screen), translate it, and then lookup the stats in english.
But we study computer vision on this blog. So instead we’re going to build a computer vision solution.
Your Phone is Your Pokedex
If you’re anything like me, you have your phone with you wherever you go. It’s probably sitting next to you when you are playing Pokemon, just in case you get a txt or a call.
And in all likelihood, your phone has a camera.
So here’s what’s going to happen:
- We are playing Pokemon on our Game Boy (either, Red, Blue, or Green version) when we encounter a Pokemon that we do not know.
- We would like to identify this Pokemon, so we take out our phone and snap a photo of our Game Boy screen.
- We then take this picture, apply our computer vision and image search engine techniques, and we are able to automatically identify the Pokemon.
Again, if you don’t believe me, just take a look at the video above.
How are we going to do this?
In order to even “find” the Game Boy screen in our query image we are first going to make use of many image processing and computer vision techniques. For example, we’ll be using techniques such as:
- Edge detection
- Perspective warping
After we have our Game Boy screen, we’ll crop out the Pokemon and then describe the Pokemon. We’ll characterize the shape of the Pokemon by using shape descriptors. In this way we can abstractly represent the Pokemon using only a list of numbers (i.e. an image feature vector).
Don’t worry. I’ll go through every and every detail as we work through this series. By the time we are finished, you’ll have a firm understanding of how to use shape descriptors to describe objects in images. And in this case, the “objects” will be Pokemon.
Building our Pokedex will be a six step process:
- An introduction to building a Pokedex (that’s this blog post)
- Scraping the web and building our Pokemon sprite database
- Indexing our database using shape descriptors
- Finding the Game Boy screen in our image (Part 1)
- Finding the Game Boy screen (Part 2)
- Searching and identifying the Pokemon
Libraries You Will Need
I will assume that you have the following libraries installed:
Don’t want spend a bunch of time installing these packages?
I have a pre-configured Ubuntu virtual machine with all the necessary computer vision and image processing pre-installed!
Click here and check out the eBook + virtual machine package to learn more.
Gotta Identify ‘Em All
Are you ready to embark on your very own Pokemon identification journey?
I know I am.