Rectified Adam (RAdam) optimizer with Keras

In this tutorial, you will learn how to use Keras and the Rectified Adam optimizer as a drop-in replacement for the standard Adam optimizer, potentially leading to a higher accuracy model (and in fewer epochs).

Today we’re kicking off a two-part series on the Rectified Adam optimizer:

  1. Rectified Adam (RAdam) optimizer with Keras (today’s post)
  2. Is Rectified Adam actually *better* than Adam? (next week’s tutorial)

Rectified Adam is a brand new deep learning model optimizer introduced by a collaboration between members of the University of Illinois, Georgia Tech, and Microsoft Research.

The goal of the Rectified Adam optimizer is two-fold:

  1. Obtain a more accurate/more generalizable deep neural network
  2. Complete training in fewer epochs

Sound too good to be true?

Well, it might just be.

You’ll need to read the rest of this tutorial to find out.

To learn how to use the Rectified Adam optimizer with Keras, just keep reading!

Looking for the source code to this post?
Jump right to the downloads section.

Rectified Adam (RAdam) optimizer with Keras

In the first part of this tutorial, we’ll discuss the Rectified Adam optimizer, including how it’s different than the standard Adam optimizer (and why we should care).

From there I’ll show you how to use the Rectified Adam optimizer with the Keras deep learning library.

We’ll then run some experiments and compare Adam to Rectified Adam.

What is the Rectified Adam optimizer?

Figure 1: Using the Rectified Adam (RAdam) deep learning optimizer with Keras. (image source: Figure 6 from Liu et al.)

A few weeks ago the deep learning community was all abuzz after Liu et al. published a brand new paper entitled On the Variance of the Adaptive Learning Rate and Beyond.

This paper introduced a new deep learning optimizer called Rectified Adam (or RAdam for short).

Rectified Adam is meant to be a drop-in replacement for the standard Adam optimizer.

So, why is Liu et al.’s contribution so important? And why is the deep learning community so excited about it?

Here’s a quick rundown on why you should care about it:

  • Learning rate warmup heuristics work well to stabilize training.
  • These heuristics also work well to improve generalization.
  • Liu et al. decided to study the theory behind learning rate warmup…
  • …but they found a problem with adaptive learning rates — during the first few batches the model did not generalize well and had very high variance.
  • The authors studied the problem in detail and concluded that the issue can be resolved/mitigated by:
    • 1. Applying warm up with a low initial learning rate.
    • 2. Or, simply turning off the momentum term for the first few sets of input batches.
  • As training continues, the variance will stabilize, and from there, the learning rate can be increased and the momentum term can be added back in.

The authors call this optimizer Rectified Adam (RAdam), a variant of the Adam optimizer, as it “rectifies” (i.e., corrects) the variance/generalization issues apparent in other adaptive learning rate optimizers.

But the question remains — is Rectified Adam actually better than standard Adam?

To answer that, you’ll need to finish reading this tutorial and read next week’s post which includes a full comparison.

For more information about Rectified Adam, including details on both the theoretical and empirical results, be sure to refer to Liu et al.’s paper.

Project structure

Let’s inspect our project layout:

Our ResNet CNN is contained within the pyimagesearch  module. The resnet.py  file contains the exact ResNet model class included with Deep Learning for Computer Vision with Python.

We will train ResNet on the CIFAR-10 dataset with both the Adam or RAdam optimizers inside of train.py , which we’ll review later in this tutorial. The training script will generate an accuracy/loss plot each time it is run — two .png files for each of the Adam and Rectified Adam experiments are included in the “Downloads”.

Installing Rectified Adam for Keras

This tutorial requires the following software to be installed in your environment:

  • TensorFlow
  • Keras
  • Rectified Adam for Keras
  • scikit-learn
  • matplotlib

Luckily, all of the software is pip installable. If you’ve ever followed one of my installation tutorials, then you know I’m a fan of virtualenv and virtualenvwrapper for managing Python virtual environments. The first command below, workon , assumes that you have these packages installed, but it is optional.

Let’s install the software now:

The original implementation of RAdam by Liu et al. was in PyTorch; however, a Keras implementation was created by Zhao HG.

You can install the Keras implementation of Rectified Adam via the following command:

To verify that the Keras + RAdam package has been successfully installed, open up a Python shell and attempt to import keras_radam:

Provided there are no errors during the import, you can assume Rectified Adam is successfully installed on your deep learning box!

Implementing Rectified Adam with Keras

Let’s now learn how we can use Rectified Adam with Keras.

If you are unfamiliar with Keras and/or deep learning, please refer to my Keras Tutorial. For a full review of deep learning optimizers, refer to the following chapters of Deep Learning for Computer Vision with Python:

  • Starter Bundle – Chapter 9: “Optimization Methods and Regularization Techniques”
  • Practitioner Bundle –  Chapter 7: “Advanced Optimization Methods”

Otherwise, if you’re ready to go, let’s dive in.

Open up a new file, name it train.py, and insert the following code:

Lines 2-15 import our packages and modules. Most notably, Lines 10 and 11 import Adam  and RAdam  optimizers. We will use the "Agg"  backend of matplotlib so that we can save our training plots to disk (Line 3).

Lines 18-24 then parse two command line arguments:

  • --plot : The path to our output training plot.
  • --optimizer : The type of optimizer that we’ll use for training (either  adam  or radam).

From here, let’s go ahead and perform a handful of initializations:

Lines 27 and 28 initialize the number of epochs to train for as well as our batch size. Feel free to tune these hyperparameters, just keep in mind that they will affect results.

Lines 33-35 load and preprocess our CIFAR-10 data including scaling data to the range [0, 1].

Lines 38-40 then binarize our class labels from integers to vectors.

Lines 43-45 construct our data augmentation object. Be sure to refer to my data augmentation tutorial if you are new to data augmentation, how it works, or why we use it.

Our CIFAR-10 class labelNames  are listed on Lines 48 and 49.

Now we’re to the meat of this tutorial — initializing either the Adam or RAdam optimizer:

Depending on the --optimizer  command line argument, we’ll either initialize:

With our optimizer ready to go, now we’ll compile and train our model:

We compile ResNet  with our specified optimizer (either Adam or RAdam) via Lines 64-67.

Lines 70-75 launch the training process. Be sure to refer to my tutorial on Keras’ fit_generator method if you are new to using this function to train a deep neural network with Keras.

To wrap up, we print our classification report and plot our loss/accuracy curves over the duration of the training epochs:

Standard Adam Optimizer Results

To train ResNet on the CIFAR-10 dataset using the Adam optimizer, make sure you use the “Downloads” section of this blog post to download the source guide to this guide.

From there, open up a terminal and execute the following command:

Figure 2: To achieve a baseline, we first train ResNet using the Adam optimizer on the CIFAR-10 dataset. We will compare the results to the Rectified Adam (RAdam) optimizer using Keras.

Looking at our output you can see that we obtained 90% accuracy on our testing set.

Examining Figure 2 shows that there is little overfitting going on as well — our training progress is quite stable.

Rectified Adam Optimizer Results

Now, let’s train ResNet on CIFAR-10 using the Rectified Adam optimizer:

Figure 3: The Rectified Adam (RAdam) optimizer is used in conjunction with ResNet using Keras on the CIFAR-10 dataset. But how to the results compare to the standard Adam optimizer?

Notice how the --optimizer  switch is set to radam for this second run of our training script.

But wait a second — why are we only obtaining 85% accuracy here?

Isn’t the Rectified Adam optimizer supposed to outperform standard Adam?

Why is our accuracy somehow worse?

Let’s discuss that in the next section.

Is Rectified Adam actually better than Adam?

If you look at our results you’ll see that the standard Adam optimizer outperformed the new Rectified Adam optimizer.

What’s going on here?

Isn’t Rectified Adam supposed to obtain higher accuracy and in fewer epochs?

Why is Rectified Adam performing worse than standard Adam?

Well, to start, keep in mind that we’re looking at the results from only a single dataset here — a true evaluation would look at the results across multiple datasets.

…and that’s exactly what I’ll be doing next week!

To see a full-blown comparison between Adam and Rectified Adam, and determine which optimizer is better, you’ll need to tune in for next week’s blog post!

What’s next?

Figure 4: My deep learning book, Deep Learning for Computer Vision with Python, is trusted by employees and students of top institutions.

If you’re interested in diving head-first into the world of computer vision/deep learning and discovering how to:

  • Select the best optimizer for the job
  • Train Convolutional Neural Networks on your own custom datasets
  • Replicate the results of state-of-the-art papers, including ResNet, SqueezeNet, VGGNet, and others
  • Train your own custom Faster R-CNN, Single Shot Detectors (SSDs), and RetinaNet object detectors
  • Use Mask R-CNN to train your own instance segmentation networks
  • Train Generative Adversarial Networks (GANs)

…then be sure to take a look at my book, Deep Learning for Computer Vision with Python!

My complete, self-study deep learning book is trusted by members of top machine learning schools, companies, and organizations, including Microsoft, Google, Stanford, MIT, CMU, and more!

Readers of my book have gone on to win Kaggle competitions, secure academic grants, and start careers in CV and DL using the knowledge they gained through study and practice.

My book not only teaches the fundamentals, but also teaches advanced techniques, best practices, and tools to ensure that you are armed with practical knowledge and proven coding recipes to tackle nearly any computer vision and deep learning problem presented to you in school, in your research, or in the modern workforce.

Be sure to take a look  — and while you’re at it, don’t forget to grab your (free) table of contents + sample chapters.

Summary

In this tutorial, you learned how to use the Rectified Adam optimizer as a drop-in replacement for the standard Adam optimizer using the Keras deep learning library.

We then ran a set of experiments comparing Adam performance to Rectified Adam performance. Our results show that standard Adam actually outperformed the RAdam optimizer.

So what gives?

Liu et al. reported higher accuracy with fewer epochs in their paper — are we doing anything wrong?

Is something broken with our Rectified Adam optimizer?

To answer those questions you’ll need to tune in next week where I’ll be providing a full set of benchmark experiments comparing Adam to Rectified Adam. You won’t want to miss next week’s post, it’s going to be a good one!

To download the source code to this post (and be notified when next week’s tutorial goes live), be sure to enter your email address in the form below!

Downloads:

If you would like to download the code and images used in this post, please enter your email address in the form below. Not only will you get a .zip of the code, I’ll also send you a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL! Sound good? If so, enter your email address and I’ll send you the code immediately!

, , , , , ,

15 Responses to Rectified Adam (RAdam) optimizer with Keras

  1. Zubair Ahmed September 30, 2019 at 10:44 am #

    Wow, a post on RAdam, so quick!

    • Adrian Rosebrock September 30, 2019 at 2:48 pm #

      Thanks Zubair! Next week’s post is going to be even better 🙂

  2. David Bonn September 30, 2019 at 2:29 pm #

    This is cool stuff.

    Even if “rectified adam” doesn’t work I suspect that there is some kind of fancy learning rate scheduling along these lines that will work. Having gone through much of the deep learning learning curve/hazing ritual anything that promises to make that process a little less painful is worth discovering.

    • Adrian Rosebrock September 30, 2019 at 2:50 pm #

      I’ll discuss it in more detail in part two of this series, but my general advice to DL practitioners is that model performance doesn’t always have to do with the “best” optimizer. Instead, it’s much more aligned with the practitioner’s skill level using a particular optimizer. That’s the exact reason why SGD is used to train the majority of the state-of-the-art networks. There are “better” optimizers than SGD, but so many practitioners are comfortable using SGD that they can obtain high accuracy model despite “better” optimizers existing.

  3. Xu Zhang September 30, 2019 at 7:41 pm #

    Thank you for your post. Surely, many state-of-the-art networks are trained by SGD. But how to use SGD successfully is a question because we can’t automatically schedule the learning rate. For other advanced optimizers, such as Adam, RAdam, and so on, the optimizers themselves have the learning rate scheduler. For lower-skilled practitioners, they are easy to use. Would you like to show some examples which achieved state-of-art performance using SGD in details? Many thanks

    • Adrian Rosebrock October 1, 2019 at 1:44 pm #

      You can’t automatically schedule the learning rate and I argue that you shouldn’t until you’ve ran a few trial experiments using SGD and ctrl + c training to establish a baseline first.

      Once you have that baseline try to improve upon it using automatic learning rate schedulers, CLR, and dynamic methods like Adam, Rectified Adam, etc.

      Stay tuned for next week’s post where I break down how nearly every popular state-of-the-art model you encounter was trained using SGD.

  4. Anh Tuấn Hoàng October 1, 2019 at 12:04 am #

    I hope to read the next article sooner

    • Adrian Rosebrock October 1, 2019 at 1:43 pm #

      It will be online this coming Monday, stay tuned!

  5. Aniket Saxena October 1, 2019 at 12:20 pm #

    Yes, I also used Adam after getting worse performance of my model with SGD. But, with Adam, I also used cyclical learning rate from Leslie Smith’s paper. And it turns out that I got 1.85 as total mean absolute error with all 5 auxillary losses ended up with values less than 1, for a regression problem I am currently working on, which is a very good result without having been rely very much on snapshot ensembles with cosine annealing learning rate. So, I totally agree with what Adrian said that its totally up to practitioner’s skill to make a particular optimizer better to get it to work to come up with very much promising results. Anyway, thank you so much Adrian to always come out with these awesome tutorials. Only you and Jason are two guys who helped me a lot to come up with multi-input multi-output deep neural networks for this regression problem. One question is that is IEEE Access a good journal to submit my deep learning paper??

    • Adrian Rosebrock October 1, 2019 at 1:47 pm #

      Thanks Aniket — I think you’ll enjoy next week’s tutorial where I go into more detail on why it’s an important for a deep learning practitioner to learn how to operate a given optimizer and ideally obtain mastery level experience with it. I often recommend SGD as the very first optimizer for a practitioner to learn how to master. From there, typically Adam and RMSprop. Given my current experiments with Adam and Rectified Adam, I would suggest readers to use Adam, note the results, and if Adam is working well, sub in Rectified Adam to see if generalization improves.

  6. Zeev October 3, 2019 at 3:18 pm #

    Thanks for the post.
    I started using tensorflow-gpu 2.0.0

    The line below in train.py results in KeyError: ‘acc’ exception
    plt.plot(N, H.history[“acc”], label=”train_acc”)

    To correct the problem I printed print(H.history.keys()) and changed
    plt.plot(N, H.history[“accuracy”], label=”train_acc”) # was plt.plot(N, H.history[“acc”]…
    plt.plot(N, H.history[“val_accuracy”], label=”val_acc”) # was plt.plot(N, H.history[“val_acc”]…

    • Adrian Rosebrock October 4, 2019 at 4:22 am #

      Correct, in TensorFlow 2.0 they changed “acc to “accuracy”.

  7. Dmitriy Serebryanskiy October 8, 2019 at 1:41 am #

    Great post! Also for TF 2.0 use
    from keras_radam.training import RAdamOptimizer

  8. Alex October 15, 2019 at 8:37 pm #

    Hi Adrian,

    It might be a versioning issue or something, but as it stands I believe there is a typo in the train script, such that it crashes right at the end. Amending lines 97 and 98 to:

    plt.plot(N, H.history[“accuracy”], label=”train_acc”)
    plt.plot(N, H.history[“val_accuracy”], label=”val_acc”)

    Seemed to work for me.

    Thanks,
    Alex

    • Adrian Rosebrock October 17, 2019 at 7:04 am #

      Hey Alex — you’re using TensorFlow 2.0. In TF 2 they changed “acc” to “accuracy”.

Before you leave a comment...

Hey, Adrian here, author of the PyImageSearch blog. I'd love to hear from you, but before you submit a comment, please follow these guidelines:

  1. If you have a question, read the comments first. You should also search this page (i.e., ctrl + f) for keywords related to your question. It's likely that I have already addressed your question in the comments.
  2. If you are copying and pasting code/terminal output, please don't. Reviewing another programmers’ code is a very time consuming and tedious task, and due to the volume of emails and contact requests I receive, I simply cannot do it.
  3. Be respectful of the space. I put a lot of my own personal time into creating these free weekly tutorials. On average, each tutorial takes me 15-20 hours to put together. I love offering these guides to you and I take pride in the content I create. Therefore, I will not approve comments that include large code blocks/terminal output as it destroys the formatting of the page. Kindly be respectful of this space.
  4. Be patient. I receive 200+ comments and emails per day. Due to spam, and my desire to personally answer as many questions as I can, I hand moderate all new comments (typically once per week). I try to answer as many questions as I can, but I'm only one person. Please don't be offended if I cannot get to your question
  5. Do you need priority support? Consider purchasing one of my books and courses. I place customer questions and emails in a separate, special priority queue and answer them first. If you are a customer of mine you will receive a guaranteed response from me. If there's any time left over, I focus on the community at large and attempt to answer as many of those questions as I possibly can.

Thank you for keeping these guidelines in mind before submitting your comment.

Leave a Reply

[email]
[email]