An interview with Gary Song, deep learning practitioner at Unity Technologies

In this blog post, I interview Gary Song, a deep learning practitioner at Unity Technologies.

We’re now at the one-year anniversary of COVID-19. It’s been a particularly rough year for all of us. For Gary, it was really bad.

But as his story shows, there are always ways to turn lemons into lemonade … if you’re willing to put in the hard work.

In 2020, Gary was managing a family emergency, and right as he returned home, COVID-19 struck. The economy went into a tailspin. The pandemic was especially hard on his employer, resulting in tremendous pay cuts.

Anxious about his future, both financially and professionally, Gary worked hard studying computer vision and deep learning. He created projects that demonstrated his knowledge of the field. And he put his resume out there, even though hiring conditions were tough.

In short, Gary invested in himself — and despite a worldwide pandemic going on, he landed a deep learning practitioner position at one of the world’s most famous video game companies.

I love sharing stories like Gary’s. I’m a firm believer that every person on this earth is the master of their own destiny … but in order to achieve your full potential, you need to put in the hard work.

A little luck doesn’t help either, but as one of my favorite sayings goes:

Fortune favors hard workers.

It’s amazing to see how far the world has fallen in one year — but it’s equally incredible to see how fast we’re recovering. It’s anyone’s guess when things will return to “normal” (or whatever the new version of normal is), but given that we’re now at the one-year anniversary of COVID-19, I couldn’t think of a better, more inspirational story to share.

Join me in learning how Gary Song landed a deep learning job at one of the world’s most famous video game companies, despite a worldwide pandemic going on.

An interview with Gary Song, deep learning practitioner at Unity Technologies

Adrian: Hi Gary! Thank you for taking the time to do this interview. I know you’re busy with your new job. It’s a pleasure to have you on the PyImageSearch blog.

Gary: Hey Adrian! The pleasure’s all mine.

---

Adrian: COVID has made the past year really hard on people and businesses. How did COVID impact you and your job?

Gary: I definitely went through a rough stretch at the beginning of the year. I had just returned from a family emergency overseas, and COVID pretty much hit right after I came back. My then employer was affected pretty much immediately, and pay cuts came swiftly. I was thankful to be still employed, but with only uncertainty ahead, the stress and anxiety quickly became overbearing.

---

Adrian: After the pay cuts, you ended up leaving your old job, and during a worldwide pandemic, you landed a new job as a deep learning practitioner at Unity Technologies, a video game software development company. That’s amazing. Congratulations! Can you walk us through that process? How did you have the courage to leave your job during COVID and then land this amazing position?

Gary: Thanks! Absolutely. So the first thing I remember was the hiring bar suddenly going up everywhere. Many places stopped hiring for junior and intermediate positions, and existing offers were being rescinded. Even in that climate, I knew that settling for a job just to be employed could easily be a death sentence for my career, so I had to be somewhat selective.

The way I tried to make my resume stand out was to highlight the fact that I had quickly built a prototype leveraging computer vision and deep learning to solve an existing business problem. That probably helped a lot in getting interviews.

I knew I was well prepared for those interviews because I had hands-on experience and knowledge of the nuances of working with deep learning models from working through the PyImageSearch courses and reading the blog. That, I think, was the differentiator.

I mean, anyone with a semester of multivariate calculus can answer some general questions about deep learning, ya know? But it takes the experience of preparing the data, training the models, debugging and improving bad results, etc., all of which your materials cover, to really know it well.

Now, I did always have to make it clear that my experience up to that point was solely with the computer vision applications of deep learning, but the knowledge was definitely transferrable, being applicable even in my current day-to-day work.

---

Adrian: What are your day-to-day responsibilities at Unity? What types of deep learning models are you working with?

Gary: It’s essentially a cycle of meeting with stakeholders and structuring the project around the business requirements, then understanding the data and iterating on the model.

For example, I’m currently working on a churn prediction project. Although it’s fairly well known that gradient boosting algorithms tend to beat simple deep learning models on tabular data, I still get to use deep learning to understand the data and complement the results of the gradient boosting models.

As an example, latent space features from a very deep multi-task model can be used to look for objective-aware clusters to help us better understand our customers, which is important beyond the scope of this project.

Because my role is on the business side of things, I don’t often encounter unstructured data like images, so unless there’s a business case, I won’t get to do much computer vision in this particular role. However, I know that we have been very successful with our perception toolkit for generating synthetic data for object detection and image segmentation. It’s very cool, and I recommend everyone to check it out!

---

Adrian: What was your background in computer vision and deep learning before you joined Unity?

Gary: My knowledge of computer vision was just everything I learned in the PyImageSearch Gurus class, plus a few things I picked up on my own doing projects.

On the other hand, I knew deep learning fundamentals and major developments well, could implement models from papers from scratch, knew how to customize existing model architectures, and had a good sense of the effectiveness of models for given business use cases.

The deep learning fundamentals were from taking some online classes, reading books, watching lecture videos, and reading papers. Most of the implementation experience, however, came from following the Deep Learning for Computer Vision with Python ImageNet bundle.

Once I understood each model’s components, I started reading the papers cited in the books to understand the implementation details. Another thing I like about the PyImageSearch books is that they cite the original research papers, whereas a lot of online courses don’t do that.

---

Adrian: How did you first become interested in computer vision and deep learning?

Gary: I think the first time I encountered computer vision and deep learning together at the nuts and bolts level was in 2018 when my good friend and then colleague Jing Wang experimented with it.

Since deep learning has been trending as one of the most disruptive technologies over the last few years, I naturally wanted to learn it, but with new technologies, it’s never clear if it’s worth the time investment to learn if you’ve only heard about it. So, seeing someone at work use it made it clear that adding this to my toolkit was something I should prioritize.

I chose computer vision as my entry point to deep learning because computer vision seemed to be very intuitive, so it’s much easier to come up with hypotheses and applications for the technology.

---

Adrian: Do you have any recommendations for readers who want to follow in your footsteps?

Gary: I can definitely speak to this. Deep learning wasn’t offered in any courses back when I was in school, so I don’t have a very deep academic background in deep learning. As such, most of this advice is for aspiring practitioners in the industry, i.e., this is about eventually getting a role where deep learning is a staple part of your job. In no particular order:

• Keep an eye on developments in the hardware space. Certain things may be done now due to computational constraints but may become less relevant once the hardware is powerful enough. Certain models may also just be out of reach for the average practitioner with access to only consumer-grade hardware. Due to the experiment-heavy nature of deep learning in practice, this can severely limit what you can afford to study, even if you use cloud resources. This was one reason I chose computer vision rather than NLP, as cutting-edge NLP models have grown incredibly big and hence, cost-prohibitive to experiment with.

• Don’t hesitate to invest in computing power, whether it’s hardware or cloud. Computing power will let you experiment faster so that you learn faster.

• Don’t hesitate to invest in courses that walk you through the end-to-end deep learning pipeline. Good courses that walk you through the end-to-end deep learning pipeline will save you time so that you can focus on the part you want to study. Also, it’s generally better to have a curriculum and all the information in one place rather than you having to dig through various sources on the web.

• Build something and think about how to scale it. The modeling process is usually only part of the job. The point is that you want to demonstrate a capacity to handle all aspects of a project. This isn’t necessarily something specific to deep learning, but something employers will care about.

• Read source code. There are lots to be learned from good implementations, including how to think about a problem and its solution.

• Read at least one paper and implement the model from scratch. The devil is in the details. You’ll get a much deeper understanding of the principles behind model architectures in general.

• In general, try to dedicate time to deep dives into your topic of interest. Out-of-the-box solutions will only take you so far and can be used as quick and dirty baselines, but it is only through these deep dives that you’ll understand how to improve your results.

Finally, here is some advice from Andrej Karpathy that I’ve taken to heart regarding becoming an expert:

---

Adrian: You’ve been a PyImageSearch reader and customer since June 2019! Thank you for supporting PyImageSearch and me. What PyImageSearch books and courses do you own? And how did they help prepare for your new job at Unity?

Gary: I actually own all of them, except for the OCR one, which I’m going to get for a side project. The books and courses served as guided labs where I could get hands-on experience and become comfortable with all parts of the deep learning pipeline, which is important in the industry.

These materials were really my introduction to the “real” world of deep learning outside of a theory and derivations-heavy class. As such, they served as a survey of the landscape of deep learning by not only covering the models but also the datasets that are used to benchmark against.

Some of you might’ve heard of the notion that when learning a new subject, it’s important to build a “zoo,” as in, to be aware of, and to understand the interesting and illustrative cases. This is essentially what the PyImageSearch books and courses do for you.

---

Adrian: Would you recommend these books and courses to other budding developers, students, and researchers trying to learn computer vision, deep learning, and OpenCV?

Gary: 100%! I think these books and courses are tremendously valuable, especially for the demographic I’m in.

When I was in school, there weren’t any classes for deep learning, so I never got an opportunity to build a mental model of how everything fits together. With these books and courses, I was able to do that.

I recommend getting started with the Deep Learning for Computer Vision with Python ImageNet bundle. It can be used as guided deep learning labs or as a reference. The code is also very helpful because official documentation can be incomplete or, even worse, can have a scope that exceeds your needs, miring you with information overload.

Lastly, but perhaps most importantly, it is dense in practical knowledge not often mentioned elsewhere.

---

Adrian: If a PyImageSearch reader wants to connect with you, what is the best place to connect with you?

Gary: I’m always available on LinkedIn or through PyImageSearch direct messaging. I look forward to connecting with everyone!

Summary

In this blog post, we interviewed Gary Song, a deep learning practitioner at the famous video game development company, Unity Technologies.

Gary landed his deep learning position at Unity in the middle of the COVID-19 pandemic. He used the pay cuts at his previous job as motivation to study, improve himself, and land a position that he was not only proud of, but more stable as well.

I’m so incredibly proud of Gary. He’s put in the hard work, and he’s now enjoying the fruits of his labor.

Remember, fortune favors hard workers — are you working hard? Or hardly working?