Machine Learning in Python

Struggling to get started with machine learning using Python? In this step-by-step, hands-on tutorial you will learn how to perform machine learning using Python on numerical data and image data.

By the time you are finished reading this post, you will be able to get your start in machine learning.

To launch your machine learning in Python education, just keep reading!

Looking for the source code to this post?
Jump right to the downloads section.

Machine Learning in Python

Inside this tutorial, you will learn how to perform machine learning in Python on numerical data and image data.

You will learn how to operate popular Python machine learning and deep learning libraries, including two of my favorites:

  • scikit-learn
  • Keras

Specifically, you will learn how to:

  1. Examine your problem
  2. Prepare your data (raw data, feature extraction, feature engineering, etc.)
  3. Spot-check a set of algorithms
  4. Examine your results
  5. Double-down on the algorithms that worked best

Using this technique you will be able to get your start with machine learning and Python!

Along the way, you’ll discover popular machine learning algorithms that you can use in your own projects as well, including:

  1. k-Nearest Neighbors (k-NN)
  2. Naïve Bayes
  3. Logistic Regression
  4. Support Vector Machines (SVMs)
  5. Decision Trees
  6. Random Forests
  7. Perceptrons
  8. Multi-layer, feedforward neural networks
  9. Convolutional Neural Networks (CNNs)

This hands-on experience will give you the knowledge (and confidence) you need to apply machine learning in Python to your own projects.

Install the required Python machine learning libraries

Before we can get started with this tutorial you first need to make sure your system is configured for machine learning. Today’s code requires the following libraries:

  • NumPy: For numerical processing with Python.
  • PIL: A simple image processing library.
  • scikit-learn: Contains the machine learning algorithms we’ll cover today (we’ll need version 0.20+ which is why you see the --upgrade  flag below).
  • Keras and TensorFlow: For deep learning. The CPU version of TensorFlow is fine for today’s example.
  • OpenCV: While we aren’t using OpenCV for this blog post, imutils depends upon it (next bullet). Because of this, you can simply use pip to install OpenCV, just bear in mind that you won’t have the full install of OpenCV and you can’t customize it.
  • imutils: My personal package of image processing/computer vision convenience functions

Each of these can be installed in your environment (virtual environments recommended) with pip:

Datasets

In order to help you gain experience performing machine learning in Python, we’ll be working with two separate datasets.

The first one, the Iris dataset, is the machine learning practitioner’s equivalent of “Hello, World!” (likely one of the first pieces of software you wrote when learning how to program).

The second dataset, 3-scenes, is an example image dataset I put together — this dataset will help you gain experience working with image data, and most importantly, learn what techniques work best for numerical/categorical datasets vs. image datasets.

Let’s go ahead and get a more intimate look at these datasets.

The Iris dataset

Figure 1: The Iris dataset is a numerical dataset describing Iris flowers. It captures measurements of their sepal and petal length/width. Using these measurements we can attempt to predict flower species with Python and machine learning. (source)

The Iris dataset is arguably one of the most simplistic machine learning datasets — it is often used to help teach programmers and engineers the fundamentals of machine learning and pattern recognition.

We call this dataset the “Iris dataset” because it captures attributes of three Iris flower species:

  1. Iris Setosa
  2. Iris Versicolor
  3. Iris Virginica

Each species of flower is quantified via four numerical attributes, all measured in centimeters:

  1. Sepal length
  2. Sepal width
  3. Petal length
  4. Petal width

Our goal is to train a machine learning model to correctly predict the flower species from the measured attributes.

It’s important to note that one of the classes is linearly separable from the other two — the latter are not linearly separable from each other.

In order to correctly classify these the flower species, we will need a non-linear model.

It’s extremely common to need a non-linear model when performing machine learning with Python in the real world — the rest of this tutorial will help you gain this experience and be more prepared to conduct machine learning on your own datasets.

The 3-scenes image dataset

Figure 2: The 3-scenes dataset consists of pictures of coastlines, forests, and highways. We’ll use Python to train machine learning and deep learning models.

The second dataset we’ll be using to train machine learning models is called the 3-scenes dataset and includes 948 total images of 3 scenes:

  • Coast (360 of images)
  • Forest (328 of images)
  • Highway (260 of images)

The 3-scenes dataset was created by sampling the 8-scenes dataset from Oliva and Torralba’s 2001 paper, Modeling the shape of the scene: a holistic representation of the spatial envelope.

Our goal will be to train machine learning and deep learning models with Python to correctly recognize each of these scenes.

I have included the 3-scenes dataset in the “Downloads” section of this tutorial. Make sure you download the dataset + code to this blog post before continuing.

Steps to perform machine learning in Python

Figure 3: Creating a machine learning model with Python is a process that should be approached systematically with an engineering mindset. These five steps are repeatable and will yield quality machine learning and deep learning models.

Whenever you perform machine learning in Python I recommend starting with a simple 5-step process:

  1. Examine your problem
  2. Prepare your data (raw data, feature extraction, feature engineering, etc.)
  3. Spot-check a set of algorithms
  4. Examine your results
  5. Double-down on the algorithms that worked best

This pipeline will evolve as your machine learning experience grows, but for beginners, this is the machine learning process I recommend for getting started.

To start, we must examine the problem.

Ask yourself:

  • What type of data am I working with? Numerical? Categorical? Images?
  • What is the end goal of my model?
  • How will I define and measure “accuracy”?
  • Given my current knowledge of machine learning, do I know any algorithms that work well on these types of problems?

The last question, in particular, is critical — the more you apply machine learning in Python, the more experience you will gain.

Based on your previous experience you may already know an algorithm that works well.

From there, you need to prepare your data.

Typically this step involves loading your data from disk, examining it, and deciding if you need to perform feature extraction or feature engineering.

Feature extraction is the process of applying an algorithm to quantify your data in some manner.

For example, when working with images we may wish to compute histograms to summarize the distribution of pixel intensities in the image — in this manner, we can characterize the color of the image.

Feature engineering, on the other hand, is the process of transforming your raw input data into a representation that better represents the underlying problem.

Feature engineering is a more advanced technique and one I recommend you explore once you already have some experience with machine learning and Python.

Next, you’ll want to spot-check a set of algorithms.

What do I mean by spot-checking?

Simply take a set of machine learning algorithms and apply them to the dataset!

You’ll likely want to stuff the following machine learning algorithms in your toolbox:

  1. A linear model (ex. Logistic Regression, Linear SVM),
  2. A few non-linear models (ex. RBF SVMs, SGD classifiers),
  3. Some tree and ensemble-based models (ex. Decision Trees, Random Forests).
  4. A few neural networks, if applicable (Multi-layer Perceptrons, Convolutional Neural Networks)

Try to bring a robust set of machine learning models to the problem — your goal here is to gain experience on your problem/project by identifying which machine learning algorithms performed well on the problem and which ones did not.

Once you’ve defined your set of models, train them and evaluate the results.

Which machine learning models worked well? Which models performed poorly?

Take your results and use them to double-down your efforts on the machine learning models that performed while discarding the ones that didn’t.

Over time you will start to see patterns emerge across multiple experiments and projects.

You’ll start to develop a “sixth sense” of what machine learning algorithms perform well and in what situation.

For example, you may discover that Random Forests work very well when applied to projects that have many real-valued features.

On the other hand, you might note that Logistic Regression can handle sparse, high-dimensional spaces well.

You may even find that Convolutional Neural Networks work great for image classification (which they do).

Use your knowledge here to supplement traditional machine learning education — the best way to learn machine learning with Python is to simply roll up your sleeves and get your hands dirty!

A machine learning education based on practical experience (supplemented with some super basic theory) will take you a long way on your machine learning journey!

Let’s get our hands dirty!

Now that we have discussed the fundamentals of machine learning, including the steps required to perform machine learning in Python, let’s get our hands dirty.

In the next section, we’ll briefly review our directory and project structure for this tutorial.

Note: I recommend you use the “Downloads” section of the tutorial to download the source code and example data so you can easily follow along.

Once we’ve reviewed the directory structure for the machine learning project we will implement two Python scripts:

  1. The first script will be used to train machine learning algorithms on numerical data (i.e., the Iris dataset)
  2. The second Python script will be utilized to train machine learning on image data (i.e., the 3-scenes dataset)

As a bonus we’ll implement two more Python scripts, each of these dedicated to neural networks and deep learning:

  1. We’ll start by implementing a Python script that will train a neural network on the Iris dataset
  2. Secondly, you’ll learn how to train your first Convolutional Neural Network on the 3-scenes dataset

Let’s get started by first reviewing our project structure.

Our machine learning project structure

Be sure to grab the “Downloads” associated with this blog post.

From there you can unzip the archive and inspect the contents:

The Iris dataset is built into scikit-learn. The 3-scenes dataset, however, is not. I’ve included it in the 3scenes/  directory and as you can see there are three subdirectories (classes) of images.

We’ll be reviewing four Python machine learning scripts today:

  • classify_iris.py : Loads the Iris dataset and can apply any one of seven machine learning algorithms with a simple command line argument switch.
  • classify_images.py : Gathers our image dataset (3-scenes) and applies any one of seven Python machine learning algorithms
  • nn_iris.py : Applies a simple multi-layer neural network to the Iris dataset
  • basic_cnn.py : Builds a Convolutional Neural Network (CNN) and trains a model using the 3-scenes dataset

Implementing Python machine learning for numerical data

Figure 4: Over time, many statistical machine learning approaches have been developed. You can use this map from the scikit-learn team as a guide for the most popular methods. Expand.

The first script we are going to implement is classify_iris.py  — this script will be used to spot-check machine learning algorithms on the Iris dataset.

Once implemented, we’ll be able to use classify_iris.py  to run a suite of machine learning algorithms on the Iris dataset, look at the results, and decide on which algorithm works best for the project.

Let’s get started — open up the classify_iris.py  file and insert the following code:

Lines 2-12 import our required packages, specifically:

  • Our Python machine learning methods from scikit-learn (Lines 2-8)
  • A dataset splitting method used to separate our data into training and testing subsets (Line 9)
  • The classification report utility from scikit-learn which will print a summarization of our machine learning results (Line 10)
  • Our Iris dataset, built into scikit-learn (Line 11)
  • A tool for command line argument parsing called argparse  (Line 12)

Using argparse , let’s parse a single command line argument flag, --model  on Lines 15-18. The --model  switch allows us to choose from any of the following models:

The models  dictionary on Lines 23-31 defines the suite of models we will be spot-checking (we’ll review the results of each of these algorithms later in the post):

  • k-Nearest Neighbor (k-NN)
  • Naïve Bayes
  • Logistic Regression
  • Support Vector Machines (SVMs)
  • Decision Trees
  • Random Forests
  • Perceptrons

The keys can be entered directly in the terminal following the --model  switch. Here’s an example:

From there the KNeighborClassifier  will be loaded automatically. This conveniently allows us to call any one of 7 machine learning models one-at-a-time and on demand in a single Python script (no editing the code required)!

Moving on, let’s load and split our data:

Our dataset is easily loaded with the dedicated load_iris  method on Line 36. Once the data is in memory, we go ahead and call train_test_split  to separate the data into 75% for training and 25% for testing (Lines 37 and 38).

The final step is to train and evaluate our model:

Lines 42 and 43 train the Python machine learning model  (also known as “fitting a model”, hence the call to .fit ).

From there, we evaluate the model  on the testing set (Line 47) and then print  a classification_report  to our terminal (Lines 48 and 49).

Implementing Python machine learning for images

Figure 5: A linear classifier example for implementing Python machine learning for image classification (Inspired by Karpathy’s example in the CS231n course).

The following script, classify_images.py , is used to train the same suite of machine learning algorithms above, only on the 3-scenes image dataset.

It is very similar to our previous Iris dataset classification script, so be sure to compare the two as you follow along.

Let’s implement this script now:

First, we import our necessary packages on Lines 2-16. It looks like a lot, but you’ll recognize most of them from the previous script. The additional imports for this script include:

  • The LabelEncoder  will be used to transform textual labels into numbers (Line 9).
  • A basic image processing tool called PIL/Pillow (Line 12).
  • My handy module, paths , for easily grabbing image paths from disk (Line 13). This is included in my personal imutils package which I’ve released to GitHub and PyPi.
  • NumPy will be used for numerical computations (Line 14).
  • Python’s built-in os  module (Line 16). We’ll use it for accommodating path separators among different operating systems.

You’ll see how each of the imports is used in the coming lines of code.

Next let’s define a function called extract_color_stats :

Most machine learning algorithms perform very poorly on raw pixel data. Instead, we perform feature extraction to characterize the contents of the images.

Here we seek to quantify the color of the image by extracting the mean and standard deviation for each color channel in the image.

Given three channels of the image (Red, Green, and Blue), along with two features for each (mean and standard deviation), we have 3 x 2 = 6 total features to quantify the image. We form a feature vector by concatenating the values.

In fact, that’s exactly what the  extract_color_stats  function is doing:

  • We split the three color channels from the image  on Line 22.
  • And then the feature vector is built on Lines 23 and 24 where you can see we’re using NumPy to calculate the mean and standard deviation for each channel

We’ll be using this function to calculate a feature vector for each image in the dataset.

Let’s go ahead and parse two command line arguments:

Where the previous script had one argument, this script has two command line arguments:

  • --dataset : The path to the 3-scenes dataset residing on disk.
  • --model : The Python machine learning model to employ.

Again, we have seven machine learning models to choose from with the --model  argument:

After defining the models  dictionary, we’ll need to go ahead and load our images into memory:

Our imagePaths  are extracted on Line 53. This is just a list of the paths themselves, we’ll load each actual image shortly.

I’ve defined two lists, data  and labels  (Lines 54 and 55). The data  list will hold our image feature vectors and the class labels  corresponding to them. Knowing the label for each image allows us to train our machine learning model to automatically predict class labels for our test images.

Lines 58-68 consist of a loop over the imagePaths  in order to:

  1. Load each image  (Line 61).
  2. Extract a color stats feature vector (mean and standard deviation of each channel) from the image  using the function previously defined (Line 62).
  3. Then on Line 63 the feature vector is added to our data  list.
  4. Finally, the class label  is extracted from the path and appended to the corresponding labels  list (Lines 67 and 68).

Now, let’s encode our labels  and construct our data splits:

Our textual  labels  are transformed into an integer representing the label using the LabelEncoder  (Lines 71 and 72):

Just as in our Iris classification script, we split our data into 75% for training and 25% for testing (Lines 76 and 77).

Finally, we can train and evaluate our model:

These lines are nearly identical to the Iris classification script. We’re fitting (training) our model  and evaluating it (Lines 81-86). A classification_report  is printed in the terminal so that we can analyze the results (Lines 87 and 88).

Speaking of results, now that we’re finished implementing both classify_irs.py  and classify_images.py , let’s put them to the test using each of our 7 Python machine learning algorithms.

k-Nearest Neighbor (k-NN)

Figure 6: The k-Nearest Neighbor (k-NN) method is one of the simplest machine learning algorithms.

The k-Nearest Neighbors classifier is by far the most simple image classification algorithm.

In fact, it’s so simple that it doesn’t actually “learn” anything. Instead, this algorithm relies on the distance between feature vectors. Simply put, the k-NN algorithm classifies unknown data points by finding the most common class among the k closest examples.

Each data point in the k closest data points casts a vote and the category with the highest number of votes wins!

Or, in plain English: “Tell me who your neighbors are, and I’ll tell you who you are.”

For example, in Figure 6 above we see three sets of our flowers:

  • Daises
  • Pansies
  • Sunflowers

We have plotted each of the flower images according to their lightness of the petals (color) and the size of the petals (this is an arbitrary example so excuse the non-formality).

We can clearly see that the image is a sunflower, but what does k-NN think given our new image is equal distance to one pansy and two sunflowers?

Well, k-NN would examine the three closest neighbors (k=3) and since there are two votes for sunflowers versus one vote for pansies, the sunflower class would be selected.

To put k-NN in action, make sure you’ve used the “Downloads” section of the tutorial to download the source code and example datasets.

From there, open up a terminal and execute the following command:

Here you can see that k-NN is obtaining 95% accuracy on the Iris dataset, not a bad start!

Let’s look at our 3-scenes dataset:

On the 3-scenes dataset, the k-NN algorithm is obtaining 75% accuracy.

In particular, k-NN is struggling to recognize the “highway” class (~56% accuracy).

We’ll be exploring methods to improve our image classification accuracy in the rest of this tutorial.

For more information on how the k-Nearest Neighbors algorithm works, be sure to refer to this post.

Naïve Bayes

Figure 7: The Naïve Bayes machine learning algorithm is based upon Bayes’ theorem (source).

After k-NN, Naïve Bayes is often the first true machine learning algorithm a practitioner will study.

The algorithm itself has been around since the 1950s and is often used to obtain baselines for future experiments (especially in domains related to text retrieval).

The Naïve Bayes algorithm is made possible due to Bayes’ theorem (Figure 7).

Essentially, Naïve Bayes formulates classification as an expected probability.

Given our input data, D, we seek to compute the probability of a given class, C.

Formally, this becomes P(C | D).

To actually compute the probability we compute the numerator of Figure 7 (ignoring the denominator).

The expression can be interpreted as:

  1. Computing the probability of our input data given the class (ex., the probability of a given flower being Iris Setosa having a sepal length of 4.9cm)
  2. Then multiplying by the probability of us encountering that class throughout the population of the data (ex. the probability of even encountering the Iris Setosa class in the first place)

Let’s go ahead and apply the Naïve Bayes algorithm to the Iris dataset:

We are now up to 98% accuracy, a marked increase from the k-NN algorithm!

Now let’s apply Naïve Bayes to the 3-scenes dataset for image classification:

Uh oh!

It looks like we only obtained 66% accuracy here.

Does that mean that k-NN is better than Naïve Bayes and that we should always use k-NN for image classification?

Not so fast.

All we can say here is that for this particular project and for this particular set of extracted features the k-NN machine learning algorithm outperformed Naive Bayes.

We cannot say that k-NN is better than Naïve Bayes and that we should always use k-NN instead.

Thinking that one machine learning algorithm is always better than the other is a trap I see many new machine learning practitioners fall into — don’t make that mistake.

For more information on the Naïve Bayes machine learning algorithm, be sure to refer to this excellent article.

Logistic Regression

Figure 8: Logistic Regression is a machine learning algorithm based on a logistic function always in the range [0, 1]. Similar to linear regression, but based on a different function, every machine learning and Python enthusiast needs to know Logistic Regression (source).

The next machine learning algorithm we are going to explore is Logistic Regression.

Logistic Regression is a supervised classification algorithm often used to predict the probability of a class label (the output of a Logistic Regression algorithm is always in the range [0, 1]).

Logistic Regression is heavily used in machine learning and is an algorithm any machine learning practitioner needs Logistic Regression in their Python toolbox.

Let’s apply Logistic Regression to the Iris dataset:

Here we are able to obtain 98% classification accuracy!

And furthermore, note that both the Setosa and Versicolor classes are classified 100% correctly!

Now let’s apply Logistic Regression to the task of image classification:

Logistic Regression performs slightly better than Naive Bayes here, obtaining 69% accuracy but in order to beat k-NN we’ll need a more powerful Python machine learning algorithm.

Support Vector Machines (SVMs)

Figure 9: Python machine learning practitioners will often apply Support Vector Machines (SVMs) to their problems. SVMs are based on the concept of a hyperplane and the perpendicular distance to it as shown in 2-dimensions (the hyperplane concept applies to higher dimensions as well).

Support Vector Machines (SVMs) are extremely powerful machine learning algorithms capable of learning separating hyperplanes on non-linear datasets through the kernel trick.

If a set of data points are not linearly separable in an N-dimensional space we can project them to a higher dimension — and perhaps in this higher dimensional space the data points are linearly separable.

The problem with SVMs is that it can be a pain to tune the knobs on an SVM to get it to work properly, especially for a new Python machine learning practitioner.

When using SVMs it often takes many experiments with your dataset to determine:

  1. The appropriate kernel type (linear, polynomial, radial basis function, etc.)
  2. Any parameters to the kernel function (ex. degree of the polynomial)

If, at first, your SVM is not obtaining reasonable accuracy you’ll want to go back and tune the kernel and associated parameters — tuning those knobs of the SVM is critical to obtaining a good machine learning model. With that said, let’s apply an SVM to our Iris dataset:

Just like Logistic Regression, our SVM obtains 98% accuracy — in order to obtain 100% accuracy on the Iris dataset with an SVM, we would need to further tune the parameters to the kernel.

Let’s apply our SVM to the 3-scenes dataset:

Wow, 83% accuracy!

That’s the best accuracy we’ve seen thus far!

Clearly, when tuned properly, SVMs lend themselves well to non-linearly separable datasets.

Decision Trees

Figure 10: The concept of Decision Trees for machine learning classification can easily be explained with this figure. Given a feature vector and “set of questions” the bottom leaf represents the class. As you can see we’ll either “Go to the movies” or “Go to the beach”. There are two leaves for “Go to the movies” (nearly all complex decision trees will have multiple paths to arrive at the same conclusion with some shortcutting others).

The basic idea behind a decision tree is to break classification down into a set of choices about each entry in our feature vector.

We start at the root of the tree and then progress down to the leaves where the actual classification is made.

Unlike many machine learning algorithms such which may appear as a “black box” learning algorithm (where the route to the decision can be hard to interpret and understand), decision trees can be quite intuitive — we can actually visualize and interpret the choice the tree is making and then follow the appropriate path to classification.

For example, let’s pretend we are going to the beach for our vacation. We wake up the first morning of our vacation and check the weather report — sunny and 90 degrees Fahrenheit.

That leaves us with a decision to make: “What should we do today? Go to the beach? Or see a movie?”

Subconsciously, we may solve the problem by constructing a decision tree of our own (Figure 10).

First, we need to know if it’s sunny outside.

A quick check of the weather app on our smartphone confirms that it is indeed sunny.

We then follow the Sunny=Yes branch and arrive at the next decision — is it warmer than 70 degrees out?

Again, after checking the weather app we can confirm that it will be > 70 degrees outside today.

Following the >70=Yes branch leads us to a leaf of the tree and the final decision — it looks like we are going to the beach!

Internally, decision trees examine our input data and look for the best possible nodes/values to split on using algorithms such as CART or ID3. The tree is then automatically built for us and we are able to make predictions.

Let’s go ahead and apply the decision tree algorithm to the Iris dataset:

Our decision tree is able to obtain 95% accuracy.

What about our image classification project?

Here we obtain 74% accuracy — not the best but certainly not the worst either.

Random Forests

Figure 11: A Random Forest is a collection of decision trees. This machine learning method injects a level of “randomness” into the algorithm via bootstrapping and random node splits. The final classification result is calculated by tabulation/voting. Random Forests tend to be more accurate than decision trees. (source)

Since a forest is a collection of trees, a Random Forest is a collection of decision trees.

However, as the name suggestions, Random Forests inject a level of “randomness” that is not present in decision trees — this randomness is applied at two points in the algorithm.

  • Bootstrapping — Random Forest classifiers train each individual decision tree on a bootstrapped sample from the original training data. Essentially, bootstrapping is sampling with replacement a total of D times. Bootstrapping is used to improve the accuracy of our machine learning algorithms while reducing the risk of overfitting.
  • Randomness in node splits — For each decision tree a Random Forest trains, the Random Forest will only give the decision tree a portion of the possible features.

In practice, injecting randomness into the Random Forest classifier by bootstrapping training samples for each tree, followed by only allowing a subset of the features to be used for each tree, typically leads to a more accurate classifier.

At prediction time, each decision tree is queried and then the meta-Random Forest algorithm tabulates the final results.

Let’s try our Random Forest on the Iris dataset:

As we can see, our Random Forest obtains 96% accuracy, slightly better than using just a single decision tree.

But what about for image classification?

Do Random Forests work well for our 3-scenes dataset?

Using a Random Forest we’re able to obtain 84% accuracy, a full 10% better than using just a decision tree.

In general, if you find that decision trees work well for your machine learning and Python project, you may want to try Random Forests as well!

Neural Networks

Figure 12: Neural Networks are machine learning algorithms which are inspired by how the brains work. The Perceptron, a linear model, accepts a set of weights, computes the weighted sum, and then applies a step function to determine the class label.

One of the most common neural network models is the Perceptron, a linear model used for classification.

A Perceptron accepts a set of inputs, takes the dot product between the inputs and the weights, computes a weighted sum, and then applies a step function to determine the output class label.

We typically don’t use the original formulation of Perceptrons as we now have more advanced machine learning and deep learning models. Furthermore, since the advent of the backpropagation algorithm, we can train multi-layer Perceptrons (MLP).

Combined with non-linear activation functions, MLPs can solve non-linearly separable datasets as well.

Let’s apply a Multi-layer Perceptron machine learning algorithm to our Iris dataset using Python and scikit-learn:

Our MLP performs well here, obtaining 98% classification accuracy.

Let’s move on to image classification with an MLP:

The MLP reaches 81% accuracy here — quite respectable given the simplicity of the model!

Deep Learning and Deep Neural Networks

Figure 13: Python is arguably the most popular language for Deep Learning, a subfield of machine learning. Deep Learning consists of neural networks with many hidden layers. The process of backpropagation tunes the weights iteratively as data is passed through the network. (source)

If you’re interested in machine learning and Python then you’ve likely encountered the term deep learning as well.

What exactly is deep learning?

And what makes it different than standard machine learning?

Well, to start, it’s first important to understand that deep learning is a subfield of machine learning, which is, in turn, a subfield of the larger Artificial Intelligence (AI) field.

The term “deep learning” comes from training neural networks with many hidden layers.

In fact, in the 1990s it was extremely challenging to train neural networks with more than two hidden layers due to (paraphrasing Geoff Hinton):

  1. Our labeled datasets being too small
  2. Our computers being far too slow
  3. Not being able to properly initialize our neural network weights prior to training
  4. Using the wrong type of nonlinearity function

It’s a different story now. We now have:

  1. Faster computers
  2. Highly optimized hardware (i.e., GPUs)
  3. Large, labeled datasets
  4. A better understanding of weight initialization
  5. Superior activation functions

All of this has culminated at exactly the right time to give rise to the latest incarnation of deep learning.

And chances are, if you’re reading this tutorial on machine learning then you’re most likely interested in deep learning as well!

To gain some experience with neural networks, let’s implement one using Python and Keras.

Open up the nn_iris.py  and insert the following code:

Let’s import our packages.

Our Keras imports are for creating and training our simple neural network (Lines 2-4). You should recognize the scikit-learn imports by this point (Lines 5-8).

We’ll go ahead and load + split our data and one-hot encode our labels on Lines 13-20. A one-hot encoded vector consists of binary elements where one of them is “hot” such as [0, 0, 1]  or [1, 0, 0]  in the case of our three flower classes.

Now let’s build our neural network:

Our neural network consists of two fully connected layers using sigmoid activation.

The final layer has a “softmax classifier” which essentially means that it has an output for each of our classes and the outputs are probability percentages.

Let’s go ahead and train and evaluate our model :

Our model  is compiled on Lines 30-32 and then the training is initiated on Lines 33 and 34.

Just as with our previous two scripts, we’ll want to check on the performance by evaluating our network. This is accomplished by making predictions on our testing data and then printing a classification report (Lines 38-40).

There’s a lot going on under the hood in these short 40 lines of code. For an in-depth walkthrough of neural network fundamentals, please refer to the Starter Bundle of Deep Learning for Computer Vision with Python or the PyImageSearch Gurus course.

We’re down to the moment of truth — how will our neural network perform on the Iris dataset?

Wow, perfect! We hit 100% accuracy!

This neural network is the first Python machine learning algorithm we’ve applied that’s been able to hit 100% accuracy on the Iris dataset.

The reason our neural network performed well here is because we leveraged:

  1. Multiple hidden layers
  2. Non-linear activation functions (i.e., the sigmoid activation function)

Given that our neural network performed so well on the Iris dataset we should assume similar accuracy on the image dataset as well, right? Well, we actually have a trick up our sleeve — to obtain even higher accuracy on image datasets we can use a special type of neural network called a Convolutional Neural Network.

Convolutional Neural Networks

Figure 14: Deep learning Convolutional Neural Networks (CNNs) operate directly on the pixel intensities of an input image alleviating the need to perform feature extraction. Layers of the CNN are stacked and patterns are learned automatically. (source)

Convolutional Neural Networks, or CNNs for short, are special types of neural networks that lend themselves well to image understanding tasks. Unlike most machine learning algorithms, CNNs operate directly on the pixel intensities of our input image — no need to perform feature extraction!

Internally, each convolution layer in a CNN is learning a set of filters. These filters are convolved with our input images and patterns are automatically learned. We can also stack these convolution operates just like any other layer in a neural network.

Let’s go ahead and learn how to implement a simple CNN and apply it to basic image classification.

Open up the basic_cnn.py  script and insert the following code:

In order to build a Convolutional Neural Network for machine learning with Python and Keras, we’ll need five additional Keras imports on Lines 2-8.

This time, we’re importing convolutional layer types, max pooling operations, different activation functions, and the ability to flatten. Additionally, we’re using the Adam  optimizer rather than SGD as we did in the previous simple neural network script.

You should be acquainted with the names of the scikit-learn and other imports by this point.

This script has a single command line argument, --dataset . It represents the path to the 3-scenes directory on disk again.

Let’s load the data now:

Similar to our classify_images.py  script, we’ll go ahead and grab our imagePaths  and build our data and labels lists.

There’s one caveat this time which you should not overlook:

We’re operating on the raw pixels themselves rather than a color statistics feature vector. Take the time to review classify_images.py  once more and compare it to the lines of basic_cnn.py .

In order to operate on the raw pixel intensities, we go ahead and resize each image to 32×32 and scale to the range [0, 1] by dividing by 255.0  (the max value of a pixel) on Lines 36 and 37. Then we add the resized and scaled image  to the data  list (Line 38).

Let’s one-hot encode our labels and split our training/testing data:

And then build our image classification CNN with Keras:

On Lines 55-67, demonstrate an elementary CNN architecture. The specifics aren’t important right now, but if you’re curious, you should:

Let’s go ahead and train + evaluate our CNN model:

Our model is trained and evaluated similarly to our previous script.

Let’s give our CNN a try, shall we?

Using machine learning and our CNN we are able to obtain 92% accuracy, far better than any of the previous machine learning algorithms we’ve tried in this tutorial!

Clearly, CNNs lend themselves very well to image understanding problems.

What do our Python + Machine Learning results mean?

On the surface, you may be tempted to look at the results of this post and draw conclusions such as:

  • “Logistic Regression performed poorly on image classification, I should never use Logistic Regression.”
  • “k-NN did fairly well at image classification, I’ll always use k-NN!”

Be careful with those types of conclusions and keep in mind the 5-step machine learning process I detailed earlier in this post:

  1. Examine your problem
  2. Prepare your data (raw data, feature extraction, feature engineering, etc.)
  3. Spot-check a set of algorithms
  4. Examine your results
  5. Double-down on the algorithms that worked best

Each and every problem you encounter is going to be different in some manner.

Over time, and through lots of hands-on practice and experience, you will gain a “sixth sense” as to what machine learning algorithms will work well in a given situation.

However, until you reach that point you need to start by applying various machine learning algorithms, examining what works, and re-doubling your efforts on the algorithms that showed potential.

No two problems will be the same and, in some situations, a machine learning algorithm you once thought was “poor” will actually end up performing quite well!

Here’s how you can learn Machine Learning in Python

If you’ve made it this far in the tutorial, congratulate yourself!

It’s okay if you didn’t understand everything. That’s totally normal.

The goal of today’s post is to expose you to the world of machine learning and Python.

It’s also okay if you don’t have an intimate understanding of the machine learning algorithms covered today.

I’m a huge champion of “learning by doing” — rolling up your sleeves and doing hard work.

One of the best possible ways you can be successful in machine learning with Python is just to simply get started.

You don’t need a college degree in computer science or mathematics.

Sure, a degree like that can help at times but once you get deep into the machine learning field you’ll realize just how many people aren’t computer science/mathematics graduates.

They are ordinary people just like yourself who got their start in machine learning by installing a few Python packages, opening a text editor, and writing a few lines of code.

Ready to continue your education in machine learning, deep learning, and computer vision?

If so, click here to join the PyImageSearch Newsletter.

As a bonus, I’ll send you my FREE 17-page Computer Vision and OpenCV Resource Guide PDF.

Inside the guide, you’ll find my hand-picked tutorials, books, and courses to help you continue your machine learning education.

Sound good?

Just click the button below to get started!

Grab your free  Computer Vision and Machine Learning Resource Guide
 

Summary

In this tutorial, you learned how to get started with machine learning and Python.

Specifically, you learned how to train a total of nine different machine learning algorithms:

  1. k-Nearest Neighbors (k-NN)
  2. Naive Bayes
  3. Logistic Regression
  4. Support Vector Machines (SVMs)
  5. Decision Trees
  6. Random Forests
  7. Perceptrons
  8. Multi-layer, feedforward neural networks
  9. Convolutional Neural Networks

We then applied our set of machine learning algorithms to two different domains:

  1. Numerical data classification via the Iris dataset
  2. Image classification via the 3-scenes dataset

I would recommend you use the Python code and associated machine learning algorithms in this tutorial as a starting point for your own projects.

Finally, keep in mind our five-step process of approaching a machine learning problem with Python (you may even want to print out these steps and keep them next to you):

  1. Examine your problem
  2. Prepare your data (raw data, feature extraction, feature engineering, etc.)
  3. Spot-check a set of algorithms
  4. Examine your results
  5. Double-down on the algorithms that worked best

By using the code in today’s post you will be able to get your start in machine learning with Python — enjoy it and if you want to continue your machine learning journey, be sure to check out the PyImageSearch Gurus course, as well as my book, Deep Learning for Computer Vision with Python, where I cover machine learning, deep learning, and computer vision in detail.

To download the source code this post, and be notified when future tutorials are published here on PyImageSearch, just enter your email address in the form below.

Downloads:

If you would like to download the code and images used in this post, please enter your email address in the form below. Not only will you get a .zip of the code, I’ll also send you a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL! Sound good? If so, enter your email address and I’ll send you the code immediately!

, , , , , , ,

101 Responses to Machine Learning in Python

  1. sudonto January 14, 2019 at 10:45 am #

    Another great tutorial from you AGAIN! You always know what readers want (except the price of your magic book -ImageNet Bundle).

    • Adrian Rosebrock January 14, 2019 at 3:02 pm #

      Hi Sudonto — I’m so glad you enjoyed the tutorial. The ImageNet Bundle book is priced according to the value it provides the reader. I’m biased (since I wrote the book), but I think it’s the best deep learning book on the market (I’ve read quite a few of them). Other PyImageSearch readers who have picked up a copy will tell you the same. Keep in mind that you’re not only getting 900+ pages of the most comprehensive CV + DL education online, but you’re also getting super highly quality (and not to mention, well documented) code, video tutorials, and even pre-configured development environments as well.

      I’m happy to discuss with you over email if you have any questions.

      • Hilman January 14, 2019 at 6:28 pm #

        I won’t lie sudonto.
        As one of the buyer of the ImageNet Bundle book, I would say it worth every penny you paid.

        I still haven’t finished reading it yet but I can see how the book can provide you with the knowledge that will be needed before you go further into the ML/DL field (you know, before you read the ML/DL technical paper that full of weird terms).

      • Hendrik August 29, 2019 at 1:20 am #

        Hi Adrian. I am a blind guy and struggling to decide what way is the best to develop a banknote recognition app. It do not have to authenticate but only tell what denomination is found in the image. Which of the machine learning options will be the best. knn or cnn or one of the others. Could you please give advise on which approach to take. Also product recognition. Thank you

        • Adrian Rosebrock September 5, 2019 at 10:40 am #

          Using a CNN would likely be overkill. First find the banknote in the image. Extract the corners (i.e., where the domination value is). Run OCR on those regions. Or, train a simple ML classifier to recognize the digits. You may need a CNN for the digit recognition component but again, it may be overkill.

          I hope that helps!

  2. Al January 14, 2019 at 10:47 am #

    Amazing post !

    • Adrian Rosebrock January 14, 2019 at 2:57 pm #

      Thanks Al!

  3. Becky January 14, 2019 at 10:52 am #

    Hi Adrian,

    This is Becky from your Linkedin and I just wanna thank you. You made my progress with python way easier.

    • Adrian Rosebrock January 14, 2019 at 2:57 pm #

      Hey Becky! Thanks so much I’m glad you enjoyed the tutorial.

  4. Gogul January 14, 2019 at 10:57 am #

    Just amazing tutorial ever! Thank you so much Adrian.

    • Adrian Rosebrock January 14, 2019 at 2:58 pm #

      You’re welcome, Gogul!

  5. Huang-Yi Li January 14, 2019 at 11:08 am #

    Awesome tutorial ! whether you can recommend some online courses about machine learning ( or some books ?

    • Adrian Rosebrock January 14, 2019 at 3:08 pm #

      Hi Huang-Yi, I’m glad you enjoyed the tutorial! The PyImageSearch Gurus course contains an entire module on “Image Classification and Machine Learning” and another module on “Deep Learning”. If your interest is applying machine learning and deep learning to computer vision and image processing, that would be where to start. For more general machine learning content, I recommend Jason’s blog and books at MachineLearningMastery.com.

  6. Tom January 14, 2019 at 2:18 pm #

    Great post! I did have some trouble with the Logistic Regression section, had to change the paremeter from “auto” to “multinomial” but after that everything worked.

    Maybe I’ve got a newer version of Keras I’m not sure.

    Thanks!
    Tom

    • Adrian Rosebrock January 14, 2019 at 5:17 pm #

      Which version of scikit-learn are you using? This post assumes scikit-learn 0.20+ (the Logistic Regression code uses scikit-learn, not Keras, so the Keras version wouldn’t be a problem).

  7. Latha H N January 14, 2019 at 3:17 pm #

    I would like to try all these algorithms in Matlab.
    Can you please help me.
    Is it possible to do with Matlab on Windows OS?

    • Adrian Rosebrock January 14, 2019 at 5:07 pm #

      It’s been a good many years since I’ve used MATLAB. By the time I was in undergrad and starting grad school MATLAB had largely fallen out of favor with Python being the suggested programming language for machine learning. Unfortunately I don’t have any suggested resources for MATLAB. I’m sure scikit-learn and similar package still exist for MATLAB though, you’ll have do more digging. Sorry I don’t have a direct answer to your question!

  8. RAHMOUN ABDELLATIF January 14, 2019 at 3:59 pm #

    OUTSTANDING!!!

    Thanks

    • Adrian Rosebrock January 14, 2019 at 5:06 pm #

      Thanks Rahmoun!

  9. Alex January 14, 2019 at 7:20 pm #

    Thanks for the great and understandable tutorial serials

    • Adrian Rosebrock January 16, 2019 at 9:47 am #

      Thanks so much Alex 🙂

  10. Lucas Luk January 14, 2019 at 8:27 pm #

    Thanks for your GREAT work!!! I learn a lot!

    • Adrian Rosebrock January 16, 2019 at 9:47 am #

      Thanks Lucas!

  11. SJ Lee January 14, 2019 at 8:30 pm #

    This is GREAT WORK!!

    • Adrian Rosebrock January 16, 2019 at 9:46 am #

      Thank you, I’m glad you enjoyed it!

  12. Akshay Chaturvedi January 15, 2019 at 12:30 am #

    A great Article Adrain, thanks for the valuable information. The thing i am still searching is the tuning of the parameters. As you said in SVM “it can perform better if tuned”. Just a question from a beginner is how to tune the SVM or any other working model ? Is hit and trial is just the option ??

    Your answer will definitely help many new beginners

    • Adrian Rosebrock January 16, 2019 at 9:45 am #

      Take a look at this tutorial where I teach you how to tune the hyperparameters. You can then apply the same methods to an SVM or any other model in scikit-learn.

  13. Muhammad Awais January 15, 2019 at 12:59 am #

    Always inspiring, love this tutorial. Best for absolute beginners 🙂

    • Adrian Rosebrock January 16, 2019 at 9:43 am #

      Thanks Muhammad!

  14. Jay January 15, 2019 at 1:00 am #

    Thanks for the great tutorial.
    Running classify_images.py on mac in a python 3.7 virtual environment fails as imutils has a dependency on cv2 in convenience.py.

    Thanks

    • Pat Ryan January 16, 2019 at 10:54 am #

      Hi Jay
      I found this also to be true if you are starting out from a ‘clean’ virtual environment. What I did was harvest the paths.py file from imutils and put that into the project.

      Also – I found that TensorFlow won’t install for python 3.7, I had to use python 3.6.x.

      • Adrian Rosebrock January 19, 2019 at 8:20 am #

        Thanks Pat. You are also correct that TensorFlow does not yet support Python 3.7. You would need to either (1) utilize Python 3.6 or (2) compile TensorFlow from scratch for Python 3.7.

  15. Abkul January 15, 2019 at 1:13 am #

    Adrian, thank you for another beyond Excellence tutorial!!!

    AND Your books are first class!!!!

    Will appreciate if you could do a tutorial on combination of CNN/RNN + LSTM in future

    • Adrian Rosebrock January 16, 2019 at 9:43 am #

      I have some ideas for a LSTM tutorial but haven’t quite gotten there. I hope to cover it in the future.

  16. Ron January 15, 2019 at 6:50 am #

    Thank you!
    It is all well explained and very useful for all levels.

    • Adrian Rosebrock January 16, 2019 at 9:41 am #

      Thanks Ron, I’m glad you found it helpful!

  17. Antonio January 15, 2019 at 4:36 pm #

    Adrián, I just want to say thank you. You do a great job and effort to share this information, I have learned a lot by reading your posts, you always have something interesting to share and I have something new to learn.

    • Adrian Rosebrock January 16, 2019 at 9:36 am #

      Thank you for the kind words, Antonio 🙂

  18. Daniel January 15, 2019 at 7:41 pm #

    Epic article again Adrian. Inspiring me to improve my work and communication as always. Have shared with the team and will be sharing with others in the future as a ‘start here’ when I get asked.

    • Adrian Rosebrock January 16, 2019 at 9:35 am #

      Thanks so much, Daniel!

  19. Dayne January 16, 2019 at 7:57 am #

    Appreciate your great work Adrian!!

    I think I found my very own Computer Vision University…

    • Adrian Rosebrock January 16, 2019 at 9:28 am #

      Thanks Dayne, I really appreciate that 🙂

  20. Sahil Makandar January 16, 2019 at 8:36 am #

    Hey Adrian, please anything share about night vision such as the step by step approach to resolving such problem. Thank you.

    • Adrian Rosebrock January 16, 2019 at 9:26 am #

      I’ll be covering “night vision” + computer vision in my upcoming Raspberry Pi book, stay tuned!

  21. Pat Ryan January 16, 2019 at 9:13 am #

    Thank you Adrian. I purchased the ‘Practical Python and OpenCV’ and thought it was just awesome. That is how I received notification of this tutorial that I am working through now.

    I ran into a slight issue. For me the classify_images script failed on ‘from imutils import paths’ saying cv2 was not found.

    Not a big deal, I just harvested the paths.py and have the script alongside the classify_images.py and changed the import to,’import paths’.

    Since you mentioned you specifically did not want a dependency on cv2 I thought I would point this out.

    Thanks again for your always great content.

    • Adrian Rosebrock January 16, 2019 at 9:25 am #

      Thanks for picking up a copy of Practical Python and OpenCV, Pat! I’m glad you’re enjoying.

      It regards to your import errors it sounds like you’re missing the “workon” command to access your Python virtual environment:

      $ workon py3cv4

      From there all imports should work.

  22. Pat Ryan January 16, 2019 at 10:52 am #

    Hi Adrian

    I have noticed in all but the classify_iris.py you do not supply a random_state parameter to the ‘train_test_split’ function.

    I believe this means it is possible as we run through the examples we will get different results from yours because we are using different training and testing data.

    I just want to point this out incase others are wondering why the results would be different.

    Thanks again for the great tutorials.

    • Adrian Rosebrock January 19, 2019 at 8:21 am #

      I simply didn’t include the pseudo-random number for the train_test_split function. There is no other reasoning behind it.

  23. dadi January 16, 2019 at 11:24 am #

    Very awesome !

    • Adrian Rosebrock January 19, 2019 at 8:20 am #

      Thanks, I’m glad you liked it!

  24. befama January 16, 2019 at 1:59 pm #

    Great job!

    • Adrian Rosebrock January 19, 2019 at 8:20 am #

      Thanks so much!

  25. Andrew January 16, 2019 at 2:14 pm #

    Adrian,

    Thank you so much for your blog, it is extremely helpful! I may have missed it in a previous post but is it possible to get a list of all the images along with the label that the model assigned to them, or just a list of images that the model misclassified? My specific research is turning acoustic data into images via time frequency analysis (spectrogram) and then attempting to determine if a scuba diver is present. I have been successful using a model that is based on your SimpleNet model, getting F1 scores of 85 or greater, but it would be useful for me to see which images are misclassified. Please do not feel obligated to respond. Thank you for your time!

    -Andrew

    • Adrian Rosebrock January 19, 2019 at 8:20 am #

      Yes, just loop over all images in the testing and/or training sets, classify them, and check to see if the predicted label matches the ground-truth label.

  26. Suraj January 17, 2019 at 2:44 am #

    Thank you Adrian. I have been following your tutorials and learning various techniques since long and thank for being my invisible teacher ever. I am from Pune, India where I get very fewer resources and few helps, but thanks to you and your blog which always helped me.

    I am working on various self interest projects and I am facing two problems where I think only you can help.
    1. I want to measure time and display for how long a person was detected in webcam feed. I have followed your face recognition tutorial and now am able to recognize person from live webcam feed but not able to get time period for each.
    2. I read another tutorial from another blog of which if you want i can share link. In that problem I want to perform CNN on satellite images and have to find if there are roads, electricity, agriculture land, water resources in that area from satellite image.

    You can made tutorial of it also or we can communicate over email if you wish to help me.
    Thanks in advance..

    • Adrian Rosebrock January 19, 2019 at 8:19 am #

      Hey Suraj, thanks for the kind words — it’s my pleasure to help you on your journey. To address your questions:

      1. It sounds like the computer vision comment is solved it’s just a bit of “programming/engineering” that’s tripping you up. All you need to do is use the “time” or “datetime” modules of Python. Grab a timestamp when a person is first detected (I would use a Python dictionary to store the name as the key and the timestamp as the value) and then grab a timestamp when they leave the frame. Compute the difference between the timestamps and you’re done!

      2. Take a look into semantic segmentation algorithms.

      • Suraj January 20, 2019 at 4:17 am #

        Thank You very much Adrian. I will keep you updated with my progress

  27. Akash Chakraborty January 17, 2019 at 3:37 am #

    I am using Python 2.7 and I think tensorflow isn’t supported in this version and I have to use this version only for another package (pyavrophonetic) so what are my options?

    • Adrian Rosebrock January 19, 2019 at 8:17 am #

      TensorFlow does still support Python 2.7 so that’s likely not the issue. What is the problem you are encountering with Python 2.7 and TensorFlow?

      To be honest I would really recommend you switch to Python 3. Python 2.7’s end of life is quickly approaching.

  28. Laveena January 17, 2019 at 5:56 pm #

    Omg! This is awesome

    • Adrian Rosebrock January 19, 2019 at 8:16 am #

      Thanks Laveena!

  29. Bob O'Donnell January 17, 2019 at 6:20 pm #

    Adrian,p

    Still wading through all the information in here, excellent job!

    I know you typically use other virtual environments, but for those who use conda, to get scikit-learn to 0.20 first make sure you are not in an environment, then “conda update conda” to update all packages, then enter your environment and “conda update scikit-learn”. With that upgrade, all the scripts work so far, but running classify images with mlp I only get 77-80.

    Thanks again for all the great work you do in offering these blog/lessons!

    Bob O

  30. Jackko January 18, 2019 at 1:44 am #

    Thank you very much. I have learned a little about deep learning, and I want to look back at the traditional machine learning algorithm. Your tutorial has taught me a lot. Thanks again. –A college student.

    • Adrian Rosebrock January 19, 2019 at 8:16 am #

      That’s great, I’m happy you found the tutorial so helpful!

  31. Ashar Khan January 18, 2019 at 9:47 am #

    Great work! Can you do some computer vision with GANs?

  32. Tomy January 22, 2019 at 1:22 am #

    Nice work. Great effort. Thanks

    • Adrian Rosebrock January 22, 2019 at 9:06 am #

      Thanks Tomy!

  33. Andrew January 22, 2019 at 4:21 am #

    Adrian Rosebrock Amazing tutorial!

    Please tell me Adrian. I want to try to use ANN (Annoy or PyNNDescent) to get similar faces (neighbors). do you have a tutorial on ANN? I want to understand how to store data for them so that each time not to index. I must store data in pickle?
    Thank you advanced

    • Adrian Rosebrock January 22, 2019 at 9:05 am #

      I don’t have a tutorial on approximate nearest neighbors (yet). I’ve considered doing what but haven’t had a chance to. I’ll see if I can create one sooner rather than later, thanks for the suggestion!

  34. Alex January 22, 2019 at 7:38 am #

    Nice tutorial.

    I am confused by one thing – you use “Accuracy” as the metric for evaluation but this is not included in the classification_report output. When I use accuracy_score from sklearn.metrics on KNN and Classify Images I get 74% not 75%.

    From a bit more investigation I can see you are using weighted average precision as accuracy? would you consider these synonymous.

    • Adrian Rosebrock January 22, 2019 at 9:00 am #

      Typically yes, accuracy and precision go hand in hand. You’ll even see publications refer to accuracy as precision and vice versa. That said, there are subtle differences between the two depending on what you’re doing and what you’re reporting on. The following article does a nice job summarizing the differences.

      • Alex January 22, 2019 at 10:09 am #

        Thanks –
        That is fair – I would agree that there are similarities and some differences between the two. FYI – The link you posted is broken, could you repost?

        • Adrian Rosebrock January 25, 2019 at 7:49 am #

          I have updated the link, it should work now.

  35. kevin January 22, 2019 at 5:48 pm #

    Excellent work Adrian (as usual)
    But how do I classify MY iris ie now I have these methods you have given us, how do I take one line of CSV data of my petal and sepal sizes and ask the model what my iris is ?

    Thanks
    Kevin

    • Adrian Rosebrock January 25, 2019 at 7:34 am #

      You would create a list of your iris measurement values and then pass them through the model.predict function. From there the model will classify the iris.

  36. Matthias Heng January 23, 2019 at 8:08 am #

    Very nice tutorial! Thanks!
    Do you know why multiple run of knn model of image_classify provide different result? For example, when i run “py -3.6 classify_images.py –model knn”, i got scene vs precision as {coast:0.79,forest:0.82,highway:0.70} for first time, then {coast:0.73,forest:0.82,highway:0.67} for second time, i keep trying and confirm that every time it provide different result.
    However, if i run knn model of iris_classify multiple time, i got the same result every time, species vs precision as {setosa:1.00,versicolor:0.92,virginica:0.91}.
    Is is something to be expected?

    • Adrian Rosebrock January 25, 2019 at 7:23 am #

      Each time a new random training and testing split is created. If you want to reproduce the results each and every time add a random_state to the train_test_split function.

  37. Anthony The Koala January 26, 2019 at 6:19 am #

    This is about CNNs, where the code starts above “And then build our image classification CNN with Keras” and above “On Lines 55-67”.

    As far as I understand:
    * as you keep addding filters, you start with 8, double it each time until you reach the number of pixel sizes (32×32) – as specified by first layer’s input shape of 32x32x3
    * you also have a kernel size of 3×3

    Throughout the tutorial, you emphasize that you have to test all model types. one could say the same for CNNs as there is variability between layers and kernel size.

    My questions are:
    * How many layers do you need? In your example, it’s 3. Could you have 5 layers starting with

    * Kernel size, (1,1),(3,3),(5,5), (2n+1, 2n+1)

    Or put it another way – which do you adjust first, layers then kernel size or kernel size then layers?

    Also, what if the aspect ratio of the image is not 1:1 as in the above example Say aspect ratio is 16:9, do you have to trim it to have an aspect ratio of 1:1?

    Thank you again for an interesting tutorial.

    Anthony of Sydney.

    • Adrian Rosebrock January 29, 2019 at 6:56 am #

      The number of layers you use for a CNN is a hyperparameter you tune when training your networks. I address all of your questions, including my best practices, tips, and suggestions on hyperparameter tuning, kernel sizes, and handling aspect ratios inside Deep Learning for Computer Vision with Python. If you’re interested in training your own networks, playing with different architectures, and understanding how to increase the accuracy of your models I would really encourage you to pick up a copy.

  38. Miguel January 28, 2019 at 12:44 pm #

    Awesome tutorial.
    But how do i load the iris data from a iris.csv instead of the sklearn built in dataset?

    • Adrian Rosebrock January 28, 2019 at 5:46 pm #

      There are a few options but my primary two recommendations would be:

      1. Use NumPy’s np.genfromtxt function
      2. Use Panda’s pd.read_csv function

  39. Peter January 28, 2019 at 1:04 pm #

    Very good stuff.
    Is there a way to save the trained model to disk and then load it later?
    Thanks

    • Adrian Rosebrock January 28, 2019 at 5:45 pm #

      Yes. For Keras you would use model.save and load_model. For scikit-learn you would use “pickle”:

  40. Arif January 31, 2019 at 2:08 am #

    Thank you again Adrian!
    One question; are all these methods non-deterministic? Especially for the image classifying, each time I run the SAME METHOD, on the SAME DATA, I will get DIFFERENT RESULT.
    I understand neural networks are non-deterministic, but so seems knn, decision trees, svm and the others.

    • Adrian Rosebrock February 1, 2019 at 6:55 am #

      Take a look at the “random_state” attribute to the “train_test_split” function. We are computing different training and testing splits each time. You can ensure the same split is made each time by explicitly supplying a random state.

  41. Dimitrios February 12, 2019 at 7:50 am #

    You are OpenCV rockstar buddy.
    Awesome effort for speading the knowledge.

    I have a question , it would be possible to train the model and then run it with images taken from a webcam?

  42. slava March 5, 2019 at 2:21 pm #

    Hey, Adrian. Thanks for the great tutorial.

    I’m geting an error while trying to install tesnorflow

    ” Could not find a version that satisfies the requirement tensorflow (from versions: )
    No matching distribution found for tensorflow”

    i think tensorflow doesn’t support python 3.7…
    Do you know any solutions of this problem?

    • Adrian Rosebrock March 8, 2019 at 5:57 am #

      TensorFlow does not yet support Python 3.7 which is why you cannot pip install it. You can either:

      1. Downgrade to Python 3.6 and install
      2. Compile TensorFlow from source for Python 3.7

  43. Matheus March 27, 2019 at 8:19 am #

    Thanks a lot, Adrian, for being so dedicated and didactic. Time is precious and you clearly put your time to the PyImageSearch. Yesss!!!

    Started following you today in Twitter and I am glad to have subscribed to your newsletter too (since December).

    When reading your posts and tutorials, you make us enthusiasts of Deep Learning and confident it is possible to start from scratch.

    I am not computer scientist nor mathematic but got encouraged.

    Please, keep it up. The world needs more people like you.

    Cheers,
    Matheus

    • Adrian Rosebrock March 27, 2019 at 8:23 am #

      Thanks for the kind words, Matheus! I really appreciate that 🙂

  44. jessica May 15, 2019 at 2:13 pm #

    Hello, thank you very much for your information, at this moment I have a problem when executing the code
    I hope you can solve my doubt soon

    Excuse me my english, I talk spanish

    ValueError: With n_samples=0, test_size=0.25 and train_size=None, the resulting train set will be empty. Adjust any of the aforementioned parameters.

    • Adrian Rosebrock May 15, 2019 at 2:25 pm #

      Which script are you executing when that error occurs?

  45. WW May 31, 2019 at 10:20 am #

    Cool stuff!

    • Adrian Rosebrock June 6, 2019 at 8:39 am #

      Thanks, I’m glad you liked it!

  46. krishnan chittur August 10, 2019 at 7:06 pm #

    Good examples (thanks!)

    I do have a question on how the model (after training, testing) could be used to make predictions on a totally unknown set (of images) – assume A, B, C (labels) and create
    a model (e.g. SVM) – and you get predictions/etc – How can I use it to say predict “D” which was NOT part of the training/test set used to develop/refine the model? I am missing something … i.e. how can you store/dump the model values that could be used to make predictions on a totally different data set of images? something like
    joblib.dump(model,nameoffile)?

  47. krishnan chittur August 10, 2019 at 8:41 pm #

    I was able to find a solution! – joblib can save the results – I was curious as to why the labels had to be converted from characters to numbers -? it is necessary -? Does not seem so …

    • Adrian Rosebrock August 16, 2019 at 5:52 am #

      You can use “joblib” or “pickle” to save the models. I prefer “pickle” but either will work.

  48. Adi August 14, 2019 at 12:42 pm #

    Great post, Adrian. Quick question, I understand how to train the data, but how do you introduce a new image that the ML algorithm will classify? Thanks

Leave a Reply

[email]
[email]