Running Keras models on iOS with CoreML

In last week’s blog post, you learned how to train a Convolutional Neural Network (CNN) with Keras.

Today, we’re going to take this trained Keras model and deploy it to an iPhone and iOS app using what Apple has dubbed “CoreML”, an easy-to-use machine learning framework for Apple applications:

To recap, thus far in this three-part series, we have learned how to:

  1. (Quickly) create a deep learning image dataset
  2. Train a Keras + Convolutional Neural Network on our custom dataset
  3. Deploy our Keras model to an iPhone app with CoreML (this post)

My goal today is to show you how simple it is to deploy your Keras model to your iPhone and iOS using CoreML.

How simple you say?

To be clear, I’m not a mobile developer by any stretch of the imagination, and if I can do it, I’m confident you can do it as well.

Feel free to use the code in today’s post as a starting point for your own application.

But personally, I’m going to continue the theme of this series and build a Pokedex. A Pokedex is a device that exists in the world of Pokemon, a popular TV show, video game, and trading card series (I was/still am a huge Pokemon nerd).

Using a Pokedex you can take a picture of a Pokemon (animal-like creatures that exist in the world of Pokemon) and the Pokedex will automatically identify the creature for for you, providing useful information and statistics, such as the Pokemon’s height, weight, and any special abilities it may have.

You can see an example of a Pokedex in action at the top of this blog post, but again, feel free to swap out my Keras model for your own — the process is quite simple and straightforward as you’ll see later in this guide.

To learn how you can deploy a trained Keras model to iOS and build a deep learning iPhone app, just keep reading.

Looking for the source code to this post?
Jump right to the downloads section.

Running Keras models on iOS with CoreML

Today’s blog post is broken down into four parts.

First, I’ll give some background on CoreML, including what it is and why we should use it when creating iPhone and iOS apps that utilize deep learning.

From there, we’ll write a script to convert our trained Keras model from a HDF5 file to a serialized CoreML model — it’s an extremely easy process.

Next, we’ll create a Swift project in Xcode. This step is painless for those that know their way around Xcode, but for me I had to learn as I went along using resources online (I’m not a mobile expert and it’s been a long time since I’ve needed to use Xcode).

My hope is that I’ve provided you enough detail that you don’t need to pull up a search engine unless you’re modifying the code.

At some point you’ll likely want to register for the Apple Developer Program — I’ll squeeze this in just before we test the app on our iPhone.

Finally, we’ll compile the app and deploy the Keras model to our iPhone and iOS.

What is CoreML and who is it for?

Figure 1: To make a CoreML deep learning computer vision app on your iPhone, follow these steps: (1) Gather images, (2) Train and save your model with Keras, (3) Convert your model file coremltools, (4) Import the model into your Xcode Swift app, (5) Write Swift code to run inferences on frames from your camera, (6) Deploy to your iPhone and have fun!

CoreML is a machine learning framework created by Apple with the goal of making machine learning app integration easy for anyone that wants to build a machine learning mobile app for iOS/iPhone.

CoreML supports Caffe, Keras, scikit-learn, and more.

For today, you just need a trained, serialized Keras model file to convert into a CoreML (Xcode compatible) file. This could be:

If you choose to use your own custom model you’ll want to check the CoreML documentation to ensure the layers utilized inside your network are supported.

From there all you need is a few short lines of code to load the model and run inferences.

Apple’s CoreML development team really couldn’t have made it any easier and they deserve some well-earned praise. Great job to you all.

I’m a computer vision + deep learning expert, not an app developer

I’ll be totally upfront and candid:

I’m not a mobile app developer (and I don’t claim to be).

Sure, I’ve built previous apps like ID My Pill and Chic Engine, but mobile development isn’t my strong suit or interest. In fact, those apps were created with PhoneGap/Cordova using HTML, JavaScript, and CSS without any Objective-C or Swift knowledge.

Instead, I’m a computer vision guy through and through. And when it comes to mobile apps, I lean heavily on easy-to-use frameworks such as PhoneGap/Cordova and (now) CoreML.

To learn the CoreML basics for this blog post, I gleaned this project from the knowledge of other expert developers on the web. Without them, I would be lost.

One app developer in particular, Mark Mansur, shared an excellent article on how to put together a deep learning + iOS app.

Much of today’s code is based on Mark’s post and project, with only a small modification or two. Thanks to Mark, this project is possible. Thanks Mark!

Making a Keras model compatible with iOS with CoreML and Python

In this section, we’re going to make use of the pip-installable coremltools package.

To install coremltools , ensure you’re in a Python virtual environment with relevant libraries (we’re using Keras) and enter the following command:

From there, grab my converter script and associated files by scrolling down to the “Downloads” section of this blog post and downloading the code.

Once you’re ready, open coremlconverter.py  and follow along:

Lines 2-5 import our required packages.

If you don’t have coremltools  installed, be sure to refer above this code block for installation instructions.

Then we parse our command line arguments. We have two arguments:

  • --model : The path to the pre-trained, serialized Keras model residing on disk.
  • --labelbin : The path to our class label binarizer. This file is a scikit-learn LabelBinarizer  object from our previous post where we trained the CNN. If you do not have a LabelBinarizer  object you will need to modify the code to hardcode the set of class_labels .

From there, let’s load the class labels and our Keras model:

On Lines 17-19, we load our class label pickle file, and store the class_labels  as a list.

Next, we load the trained Keras model on a single line (Line 23).

From there, let’s call the converter from coremltools  and save the resulting model to disk:

Beginning on Line 27, we call the coremltools.converters.keras.convert  function. Be sure to refer to the docs for the keyword parameter explanations. We are utilizing the following parameters today:

  • model : The Keras model we are converting. You can actually just put a string path + filename here, but I elected to enter the model object — the API supports both methods.
  • input_names="image" : Quoted from the docs: “Optional name(s) that can be given to the inputs of the Keras model. These names will be used in the interface of the Core ML models to refer to the inputs of the Keras model. If not provided, the Keras inputs are named to [input1, input2, …, inputN] in the Core ML model. When multiple inputs are present, the input feature names are in the same order as the Keras inputs.”
  • image_input_names="image" : Quoted from the docs: “Input names to the Keras model (a subset of the input_names parameter) that can be treated as images by Core ML. All other inputs are treated as MultiArrays (N-D Arrays).”
  • image_scale=1/255.0 : This parameter is very important. It’s common practice to scale the pixel intensities of your images to [0, 1] prior to training your network. If you performed this type of scaling, be sure to set the image_scale  parameter to the scale factor. Double and triple-check and scaling and preprocessing you may have done during training and ensure you reflect these preprocessing steps during the conversion process.
  • class_labels=class_labels : Here we supply the set of class labels our model was trained on. We obtained our class_labels  from our LabelBinarizer  object. You can also hardcode the class_labels  if you wish.
  • is_bgr=True : This parameter is easy to overlook (I found out the hard way). If your model was trained with BGR color channel ordering, then it is important to set this value to True  so that CoreML operates as intended. If your model was trained with RGB images, you can safely ignore this parameter. If your images are not BGR or RGB, refer to the docs for further instruction.

I’d also like to point out that you can add red/green/blue/gray biases via the parameters if you’re performing mean subtraction on your query image from within your iPhone app. This is required for many ImageNet models, for example. Be sure to refer to the docs if you need to perform this step. Mean subtraction is a common pre-processing step covered in Deep Learning for Computer Vision with Python.

The last step on our script is to save the output CoreML protobuf model:

Xcode expects this file to have the extension .mlmodel . Therefore, I elected to handle this with code rather than a command line argument to avoid possible problems down the road.

Line 35 drops the .model  extension from the input path/filename and replaces it with .mlmodel  storing the result as output .

From there, Line 37 saves the file to disk using the correct filename.

That’s all there is to this script. Thanks Apple CoreML developers!

Running the Keras to CoreML conversion script

Our script can be executed by passing two command line arguments:

  1. The path to the model
  2. The path to the label binarizer.

Each of those files was created in last week’s blog post but are included in this week’s download as well.

Once you’re ready, enter the following command in your terminal and review the output as needed:

Then, list the contents of your directory:

…and you’ll see pokedex.mlmodel  which can be imported right into Xcode (we’ll proceed to do this in the next section in Step 4). Interestingly, you can see that the file is smaller than the original Keras model which likely means CoreML stripped any optimizer state status during the conversion process.

Note: In an effort to allow for my Pokedex app to recognize when the camera is aimed at an “everyday object” and not a Pokemon (with a goal of eliminating false positives of our Pokemon friends), I added a class called “background”. I then retrained the model using the exact code from last week. The background class consisted of 250 images randomly sampled from the UKBench dataset residing on my system.

Creating a Swift + CoreML deep learning project in Xcode

Figure 2: A Swift iPhone app paired with a CoreML model makes for the perfect deep learning computer vision mobile app.

Step 0: Prepare your development environment

The zeroth step for this section is to download and install Xcode on your Macintosh computer. If your version of Xcode is not at least version 9.0, then you’ll need to upgrade. At some point my Xcode insisted that I upgrade to version 9.3 to support my iPhone iOS 11.3.

Warning: Upgrading Xcode can break other development software + environments on your machine (such as a Python virtual environment with OpenCV installed). Proceed with caution or use a service such as MacInCloud so you do not break your local development environment.

Once you’ve installed/checked for the proper version of XCode, you’ll be ready to continue on.

Step 1: Create the project

For organization purposed, I opted to create a folder called xcode  in my home directory to house all of my Xcode projects. I created the following directory: ~/adrian/xcode .

From there, launch Xcode and create a “Single View App” as shown in Figure 3.

Figure 3: Creating a “Single View App” in Xcode is the first step to creating a deep learning computer vision smartphone app.

Next, you can name your project whatever you’d like — I named mine “pokedex” as shown below.

Figure 4: In Xcode, you can name your project or app whatever you’d like. I’m making a “pokedex” CoreML deep learning app.

Step 2: Kill off the storyboard

A storyboard is a view controller (think Model/View/Controller architecture). We’re going to get rid of the view controller for today’s simple app. Our view will be created programmatically instead.

Go ahead and delete Main.storyboard  from the file manager on the left as in Figure 5.

Figure 5: Delete Main.storyboard in Xcode — we don’t need it for this deep learning computer vision iOS app.

Then, click the high level app name in the tree (“pokedex” in my case) and scroll to “Deployment info”. Erase the contents of the text box labeled “Main Interface”

Figure 6: Delete the Main Interface. We’ll be making an interface programmatically in our iOS deep learning app.

Step 3: Add an element to info.plist

Our app accesses the camera, so we need to prepare an authorization message. This can easily be done in info.plist.

Click the “+” button as is shown in Figure 7 and add the Key + Value.  The Key must match exactly to “Privacy – Camera Usage Description”, but the value can be any message you’d like.

Figure 7: Add a “Privacy – Camera Usage Description” to our info.plist because our deep learning CoreML app will utilize the iPhone camera. For a higher resolution version of this image, click here.

Step 4: Create the app window and root view controller

We still need a view even though we’ve gotten rid of the storyboard. For this step, you’ll want to copy and paste the following code into AppDelegate.swift . The function is already defined — you just need to paste the body from below:

Step 5: Drag the CoreML model file into Xcode

Using Finder on your Mac, navigate to the CoreML .mlmodel  file that you created above (or just grab my pokedex.mlmodel  from the “Downloads” section of this blog post).

Then, drag and drop it into the project tree. It will import automatically and create the relative Swift classes:

Figure 8: Adding the Keras deep learning model to the iOS project.

Step 6: Build the ViewController

Open ViewController.swift  and import our required packages/frameworks:

Lines 10-12 import three required packages for this project.

The UIKit  package is a common framework for developing the view of an iOS application and allows for text, buttons, table views, and navigation.

The AVFoundation  framework is for audiovisual media on iOS — we’ll employ it to capture from the camera.

We’re going to use the Vision  framework for our custom CoreML model classification, but the framework allows for much more than that. With the Vision framework, you can perform face detection, facial landmark detection, barcode recognition, feature tracking, and more.

Now that we’ve imported the relevant frameworks, let’s create the ViewController  class and begin with a text label:

On Line 14 the ViewController  class is defined while inheriting UIViewController  and AVCaptureVideoDataOutputSampleBufferDelegate . That’s a mouthful and really brings be back to my days of coding in Java!

In this class, we’re first going to define a UILabel  which will hold our class label and associated probability percentage text. Lines 16-23 handle this step.

Next, we’re going to override the viewDidLoad  function:

The viewDidLoad  function is called after the view has been loaded. For view controllers created via code, this is after loadView .

On Line 25 we use the override  keyword so that the compiler knows that we’re overriding the inherited class function.

Since we’re overriding the function, we need to call the super/parent function as is shown on Line 27.

From there, we establish the capture session (Line 30) followed by adding the label  as a subview (Lines 31 and 32).

I’m including this next function as a matter of completeness; however, we’re not actually going to make any changes to it:

If you are experiencing memory warnings when testing your app, you can override  the didReceiveMemoryWarning  function with additional actions. We’re going to leave this as-is and move on.

Let’s try to set up camera capture access using iOS and Swift:

Now remember — I’m not an expert at iOS development, but the codeblock above is relatively easy to follow.

First, we need to create a capture session (Line 44) and query for a camera + check for any errors (Lines 47-57).

From there, we output the feed to the screen in a previewLayer  (Lines 60-64) and start the session (Lines 67 and 68).

Let’s classify the frame and draw the label text on the screen:

Nothing magic is happening in this block — it may just be a little unfamiliar to Python developers.

We load the CoreML model on Line 73.

Then, we classify a given frame and grab the results on Lines 76-79. We can then grab the first predicted result from the CoreML model, storing it as an object named Observation  (Line 82).

The predicted class label can be extracted via Observation.identifier  (Line 85). We also format the confidence to show only two places past the decimal (Line 86). We set the label  text with these two components (Lines 89-91).

And finally, we establish a video pixel buffer and execute the request (Lines 97-100).

We’ve reached the final function where we’ll establish a location on the screen for the label:

The two settings in setupLabel  speak for themselves — essentially we constrain the label to the bottom center.

Don’t forget the final bracket to mark the end of the ViewController class!

It’s easy to make minor syntax errors if you aren’t used to Swift, so be sure to use the “Downloads” section of this blog post to grab the entire project.

Registering for the Apple Developer Program

In order to deploy the project to your iPhone, first enroll in the Apple Developer Program.

After you’re enrolled, accept the certificates on your iPhone. I remember this being very easy somewhere in settings but I don’t recall where.

It is a nearly instant process to enroll, wait for Xcode and your iPhone to sync, and then accept certificates. I ended up paying the $100 but I later found out that it’s possible to create a free developer account via this blog post. Don’t make my mistake if you want to save some funds (and potentially use those extra funds to purchase a copy of Deep Learning for Computer Vision with Python).

Testing our Keras + CoreML deep learning app on the iPhone

Now we’re ready to compile and test our Keras + iOS/iPhone deep learning app!

I recommend first deploying your app via USB. From there if you want to share it with others, you could take advantage of TestFlight before publishing in the App Store.

We’re going to use USB today.

First, plug in your iPhone to your Mac via USB. You likely have to unlock your iPhone with your pin and when iTunes prompts you to trust the device, you should.

From there, in the Xcode menubar, select Product > Destination > Adrian's iPhone .

Then to build and run all in one swoop, select Product > Run . If you have any build errors, you’ll need to resolve them and try again.

If you are successful, the app will be installed and opened automatically on your iPhone.

At this point, you can go find Pokemon in the wild (playing cards, stuffed Pokemon, or action figures). A big shoutout to GameStop, Target, and Wal-mart where I caught some Pokemon critters. You might get lucky and find a whole box for a few bucks on CraigsList or eBay. If you don’t find any, then load some photos/illustrations on your computer and aim your iPhone at your screen.

Here’s my CoreML app in action:

Figure 9: Our Keras deep neural network deployed to iOS and iPhone is able to run in real-time.

To watch the full video on YouTube, just press play on the video here:

It’s definitely a simple app, but I’m quite proud that I have this on my phone to show my friends, fellow Pokemon nerds, and PyImageSearch readers.

I’d like to thank Mark Mansur for his inspiring, detailed this blog post which made today’s tutorial possible.

If I had had more time, I may have placed a button on the UI so that I could take snapshots of the Pokemon I encounter in the wild. I’ll leave this to the Swift + iOS experts — with practice and determination, it could be you!

Note: Screenshots are easy on iPhone/iOS. I assume you already know how to take them. If not, Google it. Screencast videos are relatively easy too. Just add the feature via Settings > Control Center > Customize Controls  and then go back to the app and swipe up from the bottom (further details here).

Compatibility Note: This app was tested with iOS 11.3 on an iPhone 6s, iPhone 7, and iPhone X. I used xCode 9.3 to build the app.

What about Android? Does CoreML work on Android?

CoreML is an Apple toolkit and is meant only for iPhone, iOS, and other Apple applications. CoreML does not work on Android.

I do not have any experience developing Android apps and I’ve never used Android Studio.

I also do not have an Android phone.

But if there is enough interest in the comments section I will consider borrowing an Android phone from a friend and trying to deploy a Keras model to it.

Summary

In today’s blog post we saw that it’s incredibly easy to leverage the CoreML framework to take (trained) Keras models and deploy them to the iPhone and iOS.

I hope you see the value in Apple’s CoreML framework — it is quite impressive and a huge testament to the Apple developers and machine learning engineers for creating a tool that can ingest a deep neural network (that could be trained via a variety of popular deep learning libraries) and output a model that is nearly drag-and-drop compatible with the iPhone and iOS.

We used Swift for today’s iPhone app. While Swift isn’t as straightforward as Python (take that statement with a grain of salt because I’m a bit biased), given how easy CoreML is, you’ll be able to take this project and build your own, polished apps in no time.

Now, I have a challenge for you.

I used Pokemon for this series of tutorials as it was one of my favorite childhood pastimes and brought back some wonderful nostalgic memories.

Now it’s your turn to let your imagination run wild.

What deep learning vision app would you like to build? Do you want to recognize the make and model of cars, build an animal species classifier, detect fruits and vegetables? Something else?

If so, you’ll want to start by taking a look at my book, Deep Learning for Computer Vision with Python.

Inside my book you will:

  • Learn the foundations of machine learning and deep learning in an accessible manner that balances both theory and implementation
  • Study advanced deep learning techniques, including object detection, multi-GPU training, transfer learning, and Generative Adversarial Networks (GANs)
  • Replicate the results of state-of-the-art papers, including ResNet, SqueezeNet, VGGNet, and others on the 1.2 million ImageNet dataset

I have no doubt in my mind that you’ll be able to train your own custom deep neural networks using my book.

Be sure to take a look (and while you’re at it, don’t forget to grab your free table of contents + sample chapters PDF of the book).

From there, using both (1) your newly trained deep learning model and (2) today’s lesson, you can undoubtedly create a deep learning + Keras iPhone app and deploy it to the app store (and perhaps even make some money off the app as well, if you’re so inclined).

I’ll be back next week with a special bonus “part four” to this tutorial. I’m so excited about it that I might even drop some hints on Twitter leading up to Monday. Stay tuned!

Be sure that you don’t miss out on my next blog post (and more to come) by entering your email in the form below.

Downloads:

If you would like to download the code and images used in this post, please enter your email address in the form below. Not only will you get a .zip of the code, I’ll also send you a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL! Sound good? If so, enter your email address and I’ll send you the code immediately!

, , , , , , , , , ,

65 Responses to Running Keras models on iOS with CoreML

  1. Manuel April 23, 2018 at 10:52 am #

    Hi Adrian, I’d be very interested in learning how to deploy a Keras model in Android.
    I’m also very curious about tensorflow.js!

    This is a great series, very practical and surely I’ll use it as a starting point in the near future.

    • Adrian Rosebrock April 23, 2018 at 11:23 am #

      Thanks Manuel!

      • Gilad April 23, 2018 at 1:11 pm #

        Hi Adrian – learned a lot from this series.
        I also would like to learn how to run keras on android.
        Thx,
        Gilad

        • Adrian Rosebrock April 23, 2018 at 4:57 pm #

          Thanks Gilad!

  2. Alexander April 23, 2018 at 10:58 am #

    Thanks for the Post. But I have a question. How can we handle the conversion of relatively complex models (like RetinaNet, for example) that contains custom objects and custom layers?

    • Adrian Rosebrock April 23, 2018 at 11:21 am #

      Unfortunately you cannot, at least not easily. Keep in mind that these models need to run using Apple’s CoreML framework. If CoreML does not have an equivalent layer implementation it will not be able to handle the conversion. The solution would ultimately be implementing a custom layer type for CoreML which is not something I have done and cannot really comment on the process.

  3. Tony Holdroyd April 23, 2018 at 11:08 am #

    I only have an Android phone, I would love to be able to deploy your app to it.
    I hope enough people agree!

    • Adrian Rosebrock April 23, 2018 at 11:20 am #

      Thanks for the comment, Tony 🙂

    • Shorav Suriyal April 23, 2018 at 12:15 pm #

      I have done this on Andorid

  4. Kejitan April 23, 2018 at 11:27 am #

    Hello Adrian, Great tutorial as is a custom 🙂 . Would love to have the application run on Android.

    • Adrian Rosebrock April 23, 2018 at 11:31 am #

      Thanks Kejitan, I will keep that in mind. If I can get a friend to lend me an Android device I might be able to do it.

  5. Eswar Sai Krishna G April 23, 2018 at 11:30 am #

    Please do for Android also!

    • Adrian Rosebrock April 23, 2018 at 11:56 am #

      If I can do it, I will!

  6. Nadav April 23, 2018 at 11:45 am #

    Hello Adrian, thanks for the great tutorial. It would be great to have a tutorial about keras for android!

    • Adrian Rosebrock April 23, 2018 at 11:57 am #

      Thanks Nadav, I’ll try to do Android as well. It might take a few weeks though. I have quite literally zero experience in Android and I do not own an Android device.

      • Lúcio Corrêa April 23, 2018 at 12:53 pm #

        Hi Adrian, thank you for the excellent material, as always. It’s not a replacement for CoreML, but for Android you can use Tensorflow Lite.

        • Adrian Rosebrock April 23, 2018 at 4:59 pm #

          Thanks, Lúcio. I’ve heard of TensorFlow Lite but don’t know much about it. I’ll have to look into it. Thanks again!

  7. AMAS April 23, 2018 at 11:51 am #

    Hi Adrian, thanks for sharing the great work, really interesting.

    By the way, can I use .pb model to make such IOS App?

    • Adrian Rosebrock April 23, 2018 at 11:58 am #

      I haven’t tried it but I heard that Google has a tool for CoreML and TensorFlow Lite that you may want to look into.

  8. Charles Freitas April 23, 2018 at 12:03 pm #

    Very good Adrian Rosebrock. Yours posts are very practical and didaticals, I learn much with all of them. Actually I’m building an app for recognizer from foods, it’s my thesis from master’s degree. Man, I would like much of see this implemetation in android, or at least how I can do this. Thanks so much! Hugs my friend.

    • Adrian Rosebrock April 23, 2018 at 12:39 pm #

      Thanks Charles 🙂 If I can get my hands on an Android I’ll try to get the model to run on it.

  9. Akshay April 23, 2018 at 12:39 pm #

    Really great series. Your blog is the go-to place for practical computer vision. Please make a tutorial for Android devices.

    • Adrian Rosebrock April 23, 2018 at 4:58 pm #

      Thank you for the kind words, Akshay. If I can get my hands on an Android device I will figure it out 🙂

  10. NEBYU ZEKARIYAS April 23, 2018 at 1:09 pm #

    HI Ardrian Thanks for This Great Post but please try to do the android version of this post please please..

    • Adrian Rosebrock April 23, 2018 at 4:58 pm #

      Hey Nebyu — do you have an Android device that you like to use? I’m looking for recommendations.

  11. Laszlo April 23, 2018 at 1:53 pm #

    An Android version as well, please! 🙂

    • Adrian Rosebrock April 23, 2018 at 4:57 pm #

      Thanks Laszlo. Fingers crossed that I can get a friend to lend me their Android 😉

  12. Navendu Sinha April 23, 2018 at 1:54 pm #

    Great project Adrian.

    • Adrian Rosebrock April 23, 2018 at 4:57 pm #

      Thanks Navendu, I really appreciate it!

  13. Hamza Rashid April 23, 2018 at 7:09 pm #

    Android too please!

    • Adrian Rosebrock April 24, 2018 at 5:26 am #

      Thanks Hamza 🙂

  14. Michael April 23, 2018 at 8:22 pm #

    +1 for Android version! I’m currently playing around with tensorflow-lite on Android so a guide on how you made your app run on Android would be great 🙂

    • Adrian Rosebrock April 24, 2018 at 5:26 am #

      Thanks Michael. I’ll admit that I am not an Android user so I don’t have much experience there. But I hope that the very likely future guide will help in some way. It may not be for another month or so until I can do it though.

  15. Abkul April 24, 2018 at 3:23 am #

    Thanks a lot for your always EXCELLENT and CLEAR tutorial.

    I am joining others to request you to do Android version.

    Please comment on Android application development using TensorFLow Lite.

    • Adrian Rosebrock April 24, 2018 at 5:26 am #

      Thanks Abkul.

  16. Jose Luis V. April 24, 2018 at 5:06 am #

    Hello Adrian, thank you for this series of tutorials. I would like to have this app in my Android and if it were not possible, maybe it is in a raspberry

    • Adrian Rosebrock April 24, 2018 at 5:25 am #

      Thanks Jose. Let me see if I can get a hold of an Android. As for the Raspberry Pi… 😉

  17. Akash April 24, 2018 at 9:00 am #

    Hi Adrian! Thanks again for another superb tutorial!
    I would love to see this Android as well

    • Adrian Rosebrock April 24, 2018 at 5:32 pm #

      Thanks, Akash!

  18. Alex April 24, 2018 at 11:15 am #

    Hi Adrian, how can I use this model for object detection?

    • Adrian Rosebrock April 24, 2018 at 5:34 pm #

      You cannot use networks trained for classification to perform object detection, the only exception being if you want to use an image pyramid + sliding window. Instead, you would want to train a dedicated deep learning object detection framework such as Faster R-CNN, SSD, or YOLO. I cover all of these methods, including the image pyramid + sliding window technique inside my book, Deep Learning for Computer Vision with Python.

  19. Ryan April 24, 2018 at 6:02 pm #

    Hey Adrian!

    I’m in the process of packaging an image recognition model as part of my Master’s Thesis and am considering of deploying to mobile. Therefore I have am evaluating the best tech stack for my purposes.

    In the Medium Blog Post written by the developer of the famous Not Hotdog App, the author used React Native so that his model (using pre-trained Keras implementation of Google’ sMobileNet) could be deployed on both Android and iOS: https://medium.com/@timanglade/how-hbos-silicon-valley-built-not-hotdog-with-mobile-tensorflow-keras-react-native-ef03260747f3.

    In the post the author describes his reasons for choosing the React Native approach:

    “Instead of using TensorFlow on iOS, we looked at using Apple’s built-in deep learning libraries instead (BNNS, MPSCNN and later on, CoreML). We would have designed the network in Keras, trained it with TensorFlow, exported all the weight values, re-implemented the network with BNNS or MPSCNN (or imported it via CoreML), and loaded the parameters into that new implementation. However, the biggest obstacle was that these new Apple libraries are only available on iOS 10+, and we wanted to support older versions of iOS. As iOS 10+ adoption and these frameworks continue to improve, there may not be a case for using TensorFlow on device in the near future.”

    What would you say main tradeoffs are with choosing between the two options?

    Thanks!

    See you at PyImageConf!

    -Ryan

    • Adrian Rosebrock April 25, 2018 at 5:28 am #

      Hey Ryan — as I mention in the blog post, I’m not an expert in mobile development by any means. I cannot really comment on the tradeoffs between using React Native vs. the standard Apple libraries. I’ve heard really great things about React Native but don’t have a ton of experience in it. My general understanding is that if you want pure speed, implementing with the native libraries will likely give you that. If you want portability, Reactive Native. Other than that, I can’t really comment. Sorry I couldn’t be of more help here, but as I said in the post, I’m not a mobile developer nor do I claim to be.

  20. Cesar Flores April 24, 2018 at 10:54 pm #

    Very good post….as always…..

    Hoping to see this in Android…..

    Thanks…

    • Adrian Rosebrock April 25, 2018 at 5:24 am #

      Thank you Cesar, if I can get my hands on an Android I will consider it.

  21. moshel April 25, 2018 at 12:31 am #

    Thanks Adrian for the really interesting series. Would also be interested in Android, specifically if you can get it to work on tensorflow lite, which is Google “answer” to coreML.
    I have managed (with lots of problems) to get tensorflow lite to work on rpi but apparently MobileNetSSD is still unsupported or something (there is a java wrapper that is supposed to handle the unsupported layers?). As my main goal is object detection, not sure this will be helpful for me but still, its a step forward!

    • Adrian Rosebrock April 25, 2018 at 5:24 am #

      Thanks Moshel. I haven’t done much work with TensorFlow Lite so unfortunately I do not have any suggestions for the troubles. If I end up going the Android route I may give the MobileNet SSD a try.

  22. Elizabeth April 25, 2018 at 3:07 am #

    Thanks for the tutorial, I will try certainly it soon, but I have the opposite problem – I don’t have iPhone, but maybe I will be able to borrow one from my friend… I would love to see the variant for Android phone, too. Maybe using the object detector instead of the classification one, for example, the drowsiness detection. What do you think about it? 😉

    • Adrian Rosebrock April 25, 2018 at 5:19 am #

      If I can get my hands on an Android device I will consider an Android version of this tutorial.

  23. Caleb April 25, 2018 at 7:39 pm #

    +1 for Android

  24. Chris April 27, 2018 at 10:00 am #

    Love this series of posts. Really enjoy trying things like this that can be easily applied to real-world uses/or easily demo something useful.

    Glad you did this for iOS, as I’m a hardcore Apple user like you. If you’d done it the other way around, I’d be as eager as all the Android guys are now!

    Seriously tempted by your book hardcopy pack, but may wait until you’ve written your updated one. Really looking forward to it!

    Love your work – I’m definitely wanting to find more reasons to do CV!

  25. Frank Doepke April 28, 2018 at 1:16 am #

    Hi Adrian,
    A small advise on how to use Vision and CoreML a bit more effective in your example: You want to create the VNCoreMLModel and VNCoreMLModelRequest only once outside the captureOutput and then reuse the request for each frame. Both are not super expansive operations but you want to keep the work on the captureOutput to a minimum to maintain frame rate.

    • Adrian Rosebrock April 28, 2018 at 6:01 am #

      Thanks Frank! That is helpful, thank you for sharing.

  26. MImranKhan May 1, 2018 at 1:35 am #

    please please make a tutorial on android and keras like you made on ios please

    • Adrian Rosebrock May 1, 2018 at 1:23 pm #

      It will be in the works soon if I get my hands on an Android device!

  27. Dave Xanatos May 6, 2018 at 7:53 pm #

    PS., I tried the system’s advice and changed the line to read:

    model.compile(loss=”sparse_categorical_crossentropy”, optimizer=opt, metrics=[“accuracy”])

    AND… it’s running through the epochs now, but all loss AND all acc are reporting 0.0000e+00, so I suspect I screwed something up in royal fashion 🙂

    Thanks,

    Dave

  28. Cho May 10, 2018 at 2:50 pm #

    Thanks Adrian for your great post. I am trying to build the post using pre-trained VGG16-Imagenet model. But I failed converting ‘mlmodel.’ Can I ask how to do this?

    • Adrian Rosebrock May 11, 2018 at 10:25 am #

      Hi Cho — I would suggest posting on the official coremltools GitHub repo if you ran into any issues converting the model.

  29. Daniel Jordan May 17, 2018 at 2:26 am #

    Hi Adrian,

    I have may be a stupid questions but I could not be figured it out why. I follow your document and my own keras model I test with very good results. My model should use cars to predict what kind of car it is and on the server the results are great.

    When I convert it to a XCode mlmodel it works sometimes but let say more than not so good.

    I tried it again with your model or others model I found on the internet. Always the mlmodel gives not a good result. like I was hoping.

    Any Idea why the mlmodel is so much difference?

    • Adrian Rosebrock May 17, 2018 at 6:38 am #

      Hey Daniel, thanks for the comment. To be honest I’m not sure why your mlmodel wouldn’t be performing as well as the original Keras model. My guess is that you are doing some additional image preprocessing prior to classification in your Python code but this same preprocessing is not done for the mlmodel.

  30. Heng Sokvisal June 11, 2018 at 3:56 am #

    Hi Adrian . I wonder how can the model know that, the image is not the pokemon but a background? I have follow your series but my results is , even if there is no pokemon in the photo , it still predict and gussing the pokemon name. I don’t know why.

    • Adrian Rosebrock June 13, 2018 at 5:55 am #

      For this post we trained the model on a “background” class consisting of random images that do not contain Pokemon. In the previous post in the series there was not a background class but I added one in this post since we were using video and it would be confusing if the model was predicting a Pokemon (even with very low probability) for every frame in the video. Instead, the model was retrained.

  31. Nishant Asthana August 16, 2018 at 4:14 am #

    HI Adrian – thanks for the tutorial. When I run the coremlconverter.py command

    I am getting the folllowing error and I have searched Stackoverflow but not sure how to resolve it

    ValueError: unsupported pickle protocol: 3

    I am usign the downloaded version of the code. Can you help?

    • Adrian Rosebrock August 17, 2018 at 7:25 am #

      Re-train the model using your version of Python, then export it.

Quick Note on Comments

Please note that all comments on the PyImageSearch blog are hand-moderated by me. By moderating each comment on the blog I can ensure (1) I interact with and respond to as many readers as possible and (2) the PyImageSearch blog is kept free of spam.

Typically, I only moderate comments every 48-72 hours; however, I just got married and am currently on my honeymoon with my wife until early October. Please feel free to submit comments of course! Just keep in mind that I will be unavailable to respond until then. For faster interaction and response times, you should join the PyImageSearch Gurus course which includes private community forums.

I appreciate your patience and thank you being a PyImageSearch reader! I will see you when I get back.

Leave a Reply