Auto-Keras and AutoML: A Getting Started Guide

In this tutorial, you will learn how to use Auto-Keras, an open source alternative to Google’s AutoML, for automated machine learning and deep learning.

When training a neural network on a dataset there are two primary objectives a deep learning practitioner is trying to optimize and balance:

  1. Defining a neural network architecture that lends itself to the nature of the dataset
  2. Tuning a set of hyperparameters over many experiments that will lead to a model with high accuracy and ability to generalize to data outside the training and testing sets. Typical hyperparameters that need to be tuned include the optimizer algorithm (SGD, Adam, etc.), learning rate and learning rate scheduling, and regularization, to name a few

Depending on the dataset and problem, it can take a deep learning expert upwards of tens to hundreds of experiments to find a balance between neural network architecture and hyperparameters.

These experiments can add up to hundreds to thousands of hours in GPU compute time.

And that’s just for experts — what about non-deep learning experts?

Enter Auto-Keras and AutoML:

The end goal of both Auto-Keras and AutoML is to reduce the barrier to entry to performing machine learning and deep learning through the use of automated Neural Architecture Search (NAS) algorithms.

Auto-Keras and AutoML enable non-deep learning experts to train their own models with minimal domain knowledge of either deep learning or their actual data.

Using AutoML and Auto-Keras, a programmer with minimal machine learning expertise can apply these algorithms to achieve state-of-the-art performance with very little effort.

Sound too good to be true?

Well, maybe — but you’ll need to read the rest of the post first to find out why.

To learn more about AutoML (and how to automatically train and tune a neural network with Auto-Keras), just keep reading!

Looking for the source code to this post?
Jump right to the downloads section.

Auto-Keras and AutoML: A Getting Started Guide

In the first part of this blog post, we’ll discuss Automated Machine Learning (AutoML) and Neural Architecture Search (NAS), the algorithm that makes AutoML possible when applied to neural networks and deep learning.

We’ll also briefly discuss Google’s AutoML, a suite of tools and libraries allowing programmers with limited machine learning expertise to train high accuracy models on their own data.

Of course, Google’s AutoML is a proprietary algorithm (it’s a bit on the expensive side of as well).

An alternative to AutoML is the open source Auto-Keras, built around Keras and PyTorch.

I’ll then show you how to automatically train a network using Auto-Keras as well as to evaluate it.

What is Automated Machine Learning (AutoML)?

Figure 1: Auto-Keras is an alternative to Google’s AutoML. These software projects can help you train models automatically with little intervention. They are great options for novice deep learning practitioners or to obtain a baseline to beat later on.

Outside of unsupervised learning (automatically learning patterns from unlabeled data), automated machine learning for non-experts is considered the “holy grail” of machine learning.

Imagine the ability to automatically create a machine learning model via:

  1. Installing a library/using a web interface
  2. Pointing the library/interface to your data
  3. Automatically training a model on the data without having to tune the parameters/requiring an intimate understanding of the algorithms powering it

Some companies are trying to create such solutions — a big one is Google’s AutoML.

Google AutoML enables developers and engineers with very limited machine learning experience to automatically train neural networks on their own datasets.

Under the hood Google’s AutoML algorithms are iterative:

  1. Training a network on a training set
  2. Evaluating the network on a testing set
  3. Modifying the neural network architecture
  4. Tuning hyperparameters
  5. Repeating the process

The programmer or engineer using AutoML doesn’t need to define their own neural network architecture or tune the hyperparameters — AutoML is doing that for them automatically.

Neural Architecture Search (NAS) makes AutoML possible

Figure 2: Neural Architecture Search (NAS) produced a model summarized by these graphs when searching for the best CNN architecture for CIFAR-10 (source: Figure 4 of Zoph et al.)

Both Google’s AutoML and Auto-Keras are powered by an algorithm called Neural Architecture Search (NAS).

Given your input dataset, a Neural Architecture Search algorithm will automatically search for the most optimal architecture and corresponding parameters.

Neural Architecture Search essentially replaces the deep learning engineer/practitioner with a set of algorithms that automatically tunes the model!

In the context of computer vision and image recognition, a Neural Architecture Search algorithm will:

  1. Accept an input training dataset
  2. Optimize and find architectural building blocks called “cells” — these cells are automatically learned and may look similar to inception, residual, or squeeze/fire micro-architectures
  3. Continually train and search the “NAS search space” for more optimized cells

If the user of the AutoML system is an experienced deep learning practitioner then they may decide to:

  1. Run the NAS on a significantly smaller subset of the training dataset
  2. Find an optimal set of architectural building blocks/cells
  3. Take these cells and manually define a deeper version of the network found during the architecture search
  4. Train the network on the full training set using their own expertise and best practices

Such an approach is a hybrid between a fully automated machine learning solution and one that requires an expert deep learning practitioner — often this approach will lead to better accuracy than what the NAS finds on its own.

I would recommend reading Neural Architecture Search with Reinforcement Learning (Zoph and Le, 2016) along with Learning Transferable Architectures for Scalable Image Recognition (Zoph et al., 2017) for more details on how these algorithms work.

Auto-Keras: An open source alternative to Google’s AutoML

Figure 3: The Auto-Keras package was developed by the DATA Lab team at Texas A&M University. Auto-Keras is an open source alternative to Google’s AutoML.

The Auto-Keras package, developed by the DATA Lab team at Texas A&M University, is an alternative to Google’s AutoML.

Auto-Keras also utilizes the Neural Architecture Search but applies “network morphism” (keeping network functionality while changing the architecture) along with Bayesian optimization to guide the network morphism for more efficient neural network search.

You can find the full details of the Auto-Keras framework in Jin et al.’s 2018 publication, Auto-Keras: Efficient Neural Architecture Search with Network Morphism.

Project structure

Go ahead and grab the zip from the “Downloads” section of today’s blog post.

From there you should unzip the file and navigate into it using your terminal.

Let’s inspect today’s project with the tree  command:

Today we’re going to be reviewing a single Python script: train_auto_keras.py .

Since there will be a lot of output printed to the screen, I’ve opted to save our classification reports (generated with the help of scikit-learn’s classification_report  tool) as text files to disk. Inspecting the output/  folder above you can see a handful of the reports that have been generated. Go ahead and print one to your terminal ( cat output/14400.txt ) to see what it looks like.

Installing Auto-Keras

Figure 4: The Auto-Keras package depends upon Python 3.6, TensorFlow, and Keras.

As the Auto-Keras GitHub repository states, Auto-Keras is in a “pre-release” state — it is not an official release.

Secondly, Auto-Keras requires Python 3.6 and is only compatible with Python 3.6.

If you are using any other version of Python other than 3.6 you will not be able to utilize the Auto-Keras package.

To check your Python version just use the following command:

Provided you have Python 3.6 you can install Auto-Keras using pip:

If you have any issues installing or utilizing Auto-Keras make sure you post on the official Auto-Keras GitHub Issues page where the authors will be able to help you.

Implementing our training script with Auto-Keras

Let’s go ahead and implement our training script using Auto-Keras. Open up the train_auto_keras.py  file and insert the following code:

To begin, we import necessary packages on Lines 2-5:

  • As previously mentioned, we’ll be using scikit-learn’s classification_report  to calculate statistics which we’ll save in our output files.
  • We’re going to use the CIFAR-10 Dataset, conveniently built into keras.datasets .
  • Then comes our most notable import, autokeras , which I’ve imported as  ak  for shorthand.
  • The os  module is required as we’ll accommodate path separators on various operating systems when building output file paths.

Let’s define the main  function for our script on Line 7. We’re required to wrap our code in a main  function due to how Auto-Keras and TensorFlow handle threading. See this GitHub issue thread for more details.

Our base OUTPUT_PATH  is defined on Line 9.

Now let’s initialize a list of training times for Auto-Keras:

Lines 13-20 define a set of TRAINING_TIMES , including  [1, 2, 4, 8, 12, 24]  hours. We’ll be exploring the effect of longer training times on accuracy using Auto-Keras today.

Let’s load the CIFAR-10 dataset and initialize class names:

Our CIFAR-10 data is loaded and stored into training/testing splits on Line 25.

Subsequently, we’ll scale this data to the range of [0, 1] (Lines 26 and 27).

Our class labelNames  are initialized on Lines 30 and 31. These 10 classes are included in CIFAR-10. Take note that order is important here.

And now let’s begin looping over our TRAINING_TIMES  , each time putting Auto-Keras to use:

The code block above is the heart of today’s script. On Line 35 we’ve defined a loop over each of our TRAINING_TIMES  , where we:

  • Initialize our model  ( ak.ImageClassifier ) and allow training to start (Lines 39 and 40). Notice that we didn’t instantiate an object for a particular CNN class as we have in previous tutorials such as this one. Nor do we need to tune hyperparameters as we typically do. Auto-Keras handles all of this for us and provides a report of its findings.
  • Once the time limit has been reached, take the best model  and parameters Auto-Keras has found + re-train the model (Line 41).
  • Evaluate and construct the classification report  (Lines 44-47).
  • Write the classification report  along with the accuracy score  to disk so we can evaluate the effect of longer training times (Lines 50-54).

We’ll repeat this process for each of our TRAINING_TIMES .

Finally, we’ll check for and start the main  thread of execution:

Here we’ve checked to ensure that this is the main  thread of execution and then the main  function.

Just 60 lines later, we’re done writing our Auto-Keras with CIFAR-10 example script. But we’re not done yet…

Training a neural network with Auto-Keras

Let’s go ahead and train our neural network using Auto-Keras.

Make sure you use the “Downloads” section of this tutorial to download the source code.

From there, open up a terminal, navigate to where you downloaded the source code, and execute the following command:

Here you can see that our script is instructing Auto-Keras to perform six sets of experiments.

The total training time, including the time limits + the time to re-fit the model, was a little over 3 days on an NVIDIA K80 GPU.

Auto-Keras results

Figure 5: Using Auto-Keras usually is a very time-consuming process. Training with Auto-Keras produces the best models for CIFAR-10 in the 8-12 hour range. Past that, Auto-Keras is not able to optimize further.

In Figure 5 above you can see the effect of the amount of training time (x-axis) on overall accuracy (y-axis) using Auto-Keras.

Lower training times, namely 1 and 2 hours, lead to ~73% accuracy. Once we train for 4 hours we are able to achieve up to 93% accuracy.

The best accuracy we obtain is in the 8-12 range where we achieve 95% accuracy.

Training for longer than 8-12 hours does not increase our accuracy, implying that we have reached a saturation point and Auto-Keras is not able to optimize further.

Are Auto-Keras and AutoML worth it?

Figure 6: Is Auto-Keras (or AutoML) worth it? It is certainly a great step forward in the industry and is especially helpful for those without deep learning domain knowledge. That said, seasoned deep learning experts can craft architectures + train them in significantly less time + achieve equal or greater accuracy.

Outside of unsupervised learning (automatically learning patterns from unlabeled data), automated machine learning for non-experts is considered the “holy grail” of machine learning.

Both Google’s AutoML and the open source Auto-Keras package attempt to bring machine learning to the masses, even without significant technical experience.

While Auto-Keras worked reasonably well for CIFAR-10, I ran a second set of experiments using my previous post on deep learning, medical imagery, and malaria detection.

In that previous post, I obtained 97.1% accuracy using a simplified ResNet architecture which took under one hour to train.

I then let Auto-Keras run for 24 hours on the same dataset — the result was barely 96% accuracy, less than my hand-defined architecture.

Both Google’s AutoML and Auto-Keras are great steps forward; however, automated machine learning is nowhere near solved.

Automatic machine learning (currently) does not beat having expertise in deep learning — domain expertise, specifically in the data you are working with, is absolutely critical to obtain a higher accuracy model.

My suggestion is to invest in your own knowledge, don’t rely on automated algorithms.

To be a successful deep learning practitioner and engineer you need to bring the right tool to the job. Use AutoML and Auto-Keras for what they are, tools, and then continue to fill your own toolbox with additional knowledge.

Summary

In today’s blog post, we discussed Auto-Keras and AutoML, a set of tools and libraries to perform automated machine learning and deep learning.

The end goal of both Auto-Keras and AutoML is to reduce the barrier to entry to performing machine learning and deep learning through the use of Neural Architecture Search (NAS) algorithms.

NAS algorithms, the backbone of Auto-Keras and AutoML, will automatically:

  1. Define and optimize a neural network architecture
  2. Tune the hyperparameters to the model

The primary benefits include:

  • Being able to perform machine learning and deep learning with little expertise
  • Obtaining a high accuracy model with the ability to generalize to data outside the training and testing set
  • Getting up and running quickly with either a GUI interface or a simple API
  • A potentially state-of-the-art performance with little effort

Of course, there is a price to be paid — two prices in fact.

First, Google’s AutoML is expensive, approximately $20/hour.

To save funds you could go with Auto-Keras, an open source alternative to Google’s AutoML, but you still need to pay for GPU compute time.

Replacing an actual deep learning expert with a NAS algorithm will require many hours of computing to search for optimal parameters.

While we achieved a high accuracy model for CIFAR-10 (~96% accuracy), when I applied Auto-Keras to my previous post on medical deep learning and malaria prediction, Auto-Keras only achieved 96.1% accuracy, a full percentage point lower than my 97% accuracy (and Auto-Keras required 2,300% more compute time!)

While Auto-Keras and AutoML may be a step in the right direction in terms of automated machine learning and deep learning, there is still quite a bit of work to be done in this area.

There is no silver bullet for solving machine learning/deep learning with off-the-shelf algorithms. Instead, I recommend you invest in yourself as a deep learning practitioner and engineer.

The skills you learn today and tomorrow will pay off tremendously in the future.

I hope you enjoyed today’s tutorial!

To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), just enter your email address in the form below!

Downloads:

If you would like to download the code and images used in this post, please enter your email address in the form below. Not only will you get a .zip of the code, I’ll also send you a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL! Sound good? If so, enter your email address and I’ll send you the code immediately!

, , , , , ,

33 Responses to Auto-Keras and AutoML: A Getting Started Guide

  1. golemu January 7, 2019 at 11:50 am #

    Great tutorial. This post is a great way to start my year! I look forward to augmenting auto-keras to my research and see if it helps improves my models. I look forward to gathering great insight from auto-keras/autoML.

    As always, you provide valuable insight to machine learning and deep learning. Looking forward to reading more! Enjoy your 2019!!

    –golemu

    • Adrian Rosebrock January 7, 2019 at 12:23 pm #

      Thanks so much, Golemu!

      • golemu January 7, 2019 at 2:12 pm #

        No problem. But I did have a question.

        Will Auto-keras work for multi-task learning?

        • Adrian Rosebrock January 7, 2019 at 2:22 pm #

          As far as I understand, no. Most applications with Auto-Keras are used for image and text.

          • golemu January 11, 2019 at 2:24 am #

            Is there a fit_generator in auto-keras? My dataset is too large to fit into memory all at once. Any suggestions?

          • Adrian Rosebrock January 11, 2019 at 9:28 am #

            Good question. As far as I know there is not a fit_generator function in Auto-Keras. I would suggest reaching out to the Auto-Keras developers on GitHub.

  2. Brian S. Penn January 7, 2019 at 11:55 am #

    Hi Adrian,

    Thanks for another great tutorial!!

    How do AutoML and Auto-Keras compare to AutoWeka? I’ve used AutoWeka for a couple of projects and it works reasonably well.

    • Adrian Rosebrock January 7, 2019 at 12:23 pm #

      I’ve never used AutoWeka before, but the general idea behind AutoWeka is that it’s not just neural networks and deep learning — it’s all the algorithms implemented inside Weka along with their associated hyperparameters. AutoWeka seeks to not only search among algorithms but their hyperparameters as well.

      • brian penn January 8, 2019 at 6:08 pm #

        Yes, I had one dataset that AutoWeka recommended an SVM solution. I actually like this idea b/c neural networks don’t always provide optimal solutions.

  3. Cenk January 7, 2019 at 2:48 pm #

    Excellent tutorial with tangible figures! Many thanks for letting us see what is under the hood indeed.
    It makes me think that AutoML/AutoKeras is like a car with automatic transmission. It consumes more and is not currently good enough for performance challenges but good for general purpose soft-driving mostly. In formula 1 race (challenge in a short time), you would not use a car with an automatic transmission but a manual transmission car (hyperparameter tuning) with a F1 Pilot (Deep Learning Expert).

    Having said that I believe that NAS will evolve in time and will be pervasively used across all domains in parallel to ongoing computing power increases. Just like cars with automatic transmissions override those with manual transmissions today …

    • Adrian Rosebrock January 8, 2019 at 6:47 am #

      Great analogy, Cenk!

  4. David Bonn January 7, 2019 at 4:02 pm #

    Adrian — this is a very interesting tutorial. I agree with your evaluation that an experienced practitioner in the arts of Deep Learning can easily produce better results in less time, but systems such as this could still be valuable for someone starting down the very steep deep learning curve.

    Thanks again for the blog!

    • Adrian Rosebrock January 8, 2019 at 6:45 am #

      Thanks David!

  5. Mercy Ranjit January 7, 2019 at 9:06 pm #

    Hey Adrian,

    I greatly benefit from your posts. I would also like to thank you for the books you had released on deep learning(I purchased all the bundles), I used them for my research course work preparation and got up-to speed in no-time. Thank you so much for writing these amazing books.

    I wanted to reference the below blog which gives a comparison on all the options available including the one from Microsoft.

    https://medium.com/@santiagof/auto-is-the-new-black-google-automl-microsoft-automated-ml-autokeras-and-auto-sklearn-80d1d3c3005c

    Thanks,
    Mercy

    • Adrian Rosebrock January 8, 2019 at 6:42 am #

      Thanks Mercy, I really appreciate the kind words. Thank you for making PyImageSearch possible by going through my Deep Learning book. I’m so happy to hear that it helped you with your research 🙂

      Also, thanks so much for sharing the AutoML comparisons!

  6. Xu Zhang January 8, 2019 at 7:27 pm #

    A great article again. Thank you very much.

    For each iteration, is it true that the algorithm decides how many epochs it will run?

    • Adrian Rosebrock January 11, 2019 at 9:58 am #

      Correct. The algorithm trains until either accuracy/loss is not improving or the training time has ran out.

  7. Mohanad Abdulsattar January 8, 2019 at 10:15 pm #

    I dont know how thank you Adrian Rosebrock for your endless effort and support. I wish to meet you in person to talk more in details about my ideas and imagination in this field. I was really blessed and lucky to find such a smart and generous person like you.

    • Adrian Rosebrock January 11, 2019 at 9:55 am #

      Thank you for the kind words, Mohanad 🙂

  8. kalyanramu January 9, 2019 at 11:11 am #

    Is it possible to run AutoML in google colab? Google Colab is free

  9. Basit Aslam January 14, 2019 at 12:57 am #

    Dear Adrian Rosebrock,
    Really glad to find your blogs which are totally totally gem and upscale for the OpenCv learner. I never worked on such projects in my life and this is my first time that i am going to work on it. I want to start it from the scratch till my final project but have no mentor. Well, if you are able to help me then please read the below context and answer me according to that. That will be really appreciate to you. Thank you in advance
    ***Project Discription***
    Be a PHD student my supervisor gave me a task to run a camera and put some chart in front of it. mark some point on the image and according to that with the equal distance from the camera align it parallel with the help of motor to align it.
    if you can provide me the email address of yours then i can share some of the image scenarios on that behalf i need to align the camera. That will be much helpful for me. Thank you so much dear once again.

    • Adrian Rosebrock January 16, 2019 at 10:02 am #

      The best place to reach me is via the PyImageSearch contact form but if you’re having a problem with your final project I would suggest chatting with your supervisor instead — I don’t know the particulars of your project and your supervisor will be better equipped to give you targeted advice for your project. Best of luck with it!

    • rahman January 23, 2019 at 11:30 pm #

      Dear Basit Aslam,
      I can understand how do you feel and how hard to do a project you don’t have any experience yet. I was struggling to finish my project on the topic of deep learning which was something not only new but strange for me. Google helps a “bit” but not sufficient to help me out. Thank God I found this blog and invest my future by buying the most amazing book to start deep learning and python. So, I really recommend you to have a look the book and invest in your future:
      https://www.pyimagesearch.com/deep-learning-computer-vision-python-book/

      Trust me, you will thank me later 🙂

      note: sorry to promote your book Adrian, I only want to share the wonderful knowledge you’ve done in your amazing book

      • Adrian Rosebrock January 25, 2019 at 7:13 am #

        Thank you Rahman, I really appreciate your kind words — I’m also so happy the book was able to help you with your project, congrats!

  10. rahman January 23, 2019 at 1:30 am #

    As always, a great, nice and clean tutorial from the expert one.
    Adrian, would you please provide in your next post all about pre-trained network in CNN including what is that, why we need it, and the most important how to use it etc.
    I truly believe that your ‘beginner’ readers will be very happy to learn about the pre-trained CNN from your nice tutorial.
    wish you all the best Adrian

    • Adrian Rosebrock January 25, 2019 at 7:31 am #

      Hey Rahman — have you taken a look at Deep Learning for Computer Vision with Python? The book is designed to help you understand CNNs, pre-trained CNNs, and train your CNNs from scratch on custom datasets. Even if you are beginner the book will absolutely help you get up to speed. Give it a look, I believe it will really help you.

  11. dergham March 20, 2019 at 10:06 am #

    thank you for your very nice tutorial
    I have a question
    since I am a trader I want to know if auto keras can be used for stock price prediction
    most (or all) examples I see Initialize the model ( ak.ImageClassifier )
    is there other models to deal with numeric data or time series ?

    • Adrian Rosebrock March 22, 2019 at 8:57 am #

      Sorry, I have only used AutoKeras for work in image classification. For non-image questions you should ask the AutoKeras creators.

  12. Van April 16, 2019 at 6:45 pm #

    Nice tutorial. Do you have tutorials where you use your own data?

  13. At May 2, 2019 at 7:57 am #

    Do we have to just split raw data and feed it to the AutoML or we also need to do feature engineering?

    • Adrian Rosebrock May 8, 2019 at 1:44 pm #

      That really depends on your exact project and dataset. I would run your data as-is through AutoML/Auto-Keras to obtain a baseline. From there try some feature engineering before passing it into AutoML.

  14. Joseph July 20, 2019 at 8:54 pm #

    Hey Adrian! Wonderful article thanks for sharing. I really want to get into deep learning I was wondering if there was any software a non technical person could use to easily train and learn from data? I don’t know coding and as a single father of two working for an investment bank, it’s impossible for me to find the time.

Leave a Reply

[email]
[email]