Adding a web interface to our image search engine with Flask

This is a guest post by Michael Herman from Real Python – learn Python programming and web development through hands-on, interesting examples that are useful and fun!

In this tutorial, we’ll take the command line image search engine from the previous tutorial and turn it into a full-blown web application using Python and Flask. More specifically, we’ll be creating a Single Page Application (SPA) that consumes data via AJAX (on the front-end) from an internal, REST-like API via Python/Flask (on the back-end).

The end product will look like this:

7_sixth-iteration

New to Flask? Start with the official Quickstart guide or the “Flask: QuickStart” chapter in the second Real Python course.

Setup

You can setup you development either with or without Docker.

With Docker:

If you don’t have Docker installed, follow the official Docker documentation to install both Docker and boot2docker. Then with boot2docker up and running, run docker version  to test the Docker installation.

Create a directory to house your project “flask-image-search”.

Grab the _setup.zip from the repository, unzip the files, and add them to your project directory.

Now build the Docker image:

Once built, run the Docker container:

Open your web browser and navigate to the IP address associated with the DOCKER_HOST variable – which should be http://192.168.59.103/; if not, run boot2docker ip  to get the correct address – you should see the text “Welcome!” in your browser.

Without Docker:

Create a directory to house your project “flask-image-search”.

Grab the _setup.zip from the repository, unzip the files, and add them to your project directory. Create and activate a virtualenv, then install the requirements via Pip:

Basics

Since you’ve already built the search engine, we just need to transfer the relevant code to Flask. Essentially, we’re just going to wrap the image search engine in Flask.

Right now your project directory should look like this:

The Dockerfile and the files with the “config” directory are used specifically to get our app up an running in the Docker container (if you used Docker, of course). Don’t worry too much about how these work, but if you are curious you can reference the inline comments in the Dockerfile. Within the “app” directory the index.csv file as well as the files within the “pyimagesearch” directory are specific to, well, the image search engine. Reference the previous tutorial for more information.

Now, let’s take a closer look at the files and folders within the “app” directory that are specific to Flask:

  1. The app.py file is our Flask application. Be sure to reference the inline comments within the file to fully understand what’s happening. It’s important to note that in most cases you should break your app into smaller pieces. However, since this app (as well as our finished app) are small, we can get away with storing all the functionality in a single file.
  2. The “static” directory houses static files, like – stylesheets, JavaScript files, images, etc.
  3. The “templates” directory houses our app’s templates. Take a look at the relationship between the _base.html and index.html templates. This is called template inheritance. For more on this, check out this blog post.

That’s it for our current project structure. With that, let’s start building!

Workflow

Put simply, we’ll focus on two routes/endpoints:

  1. The main route (‘/’): This route handles the main user interaction. Users can select an image (which sends a POST request to the search route) and then displays the similar images.
  2. The search route (‘/search’): This route handles POST requests. It will take an image (name) and then using the majority of the search engine code return similar images (URLs).

Back-end

Main Route

The back-end code is already set up. That’s right – We just need to render a template when a user requests / . We do however, need to update the template, index.html, as well as add HTML, CSS, and Javascript/jQuery code. This will be handled in the Front-end section.

Search Route

Again, this route is meant to:

  • Handle POST requests,
  • Take an image and search for similar images (using the already completed search engine code), and
  • Return the similar images (in the form of URLs) in JSON format

Add the following code to app.py, just below the main route.

What’s happening?

  1. We define the endpoint, /search' , along with the allowed HTTP request methods, methods=['POST'] . Jump back to the /  main endpoint real quick. Notice how we did not specify the allowed request methods. Why? That’s because by default all endpoints respond to GET requests.
  2. We grab the image, image_url = request.form.get('img')  and then using a try/except we search for similar images.
  3. Compare the loop in the above code, to the loop in search.py from the previous tutorial. Here, instead of outputting the results, we’re simply grabbing them and adding them to a list. This list is then passed into a special Flask function called jsonify  which returns a JSON response.

Be sure to update the imports-

-and add the following variable, just below the creation of the Flask instance, which specifies the path to the index.csv file used in the image search-

We’ll look at the exact output of this in the next section.

Front-end

So, with our back-end code done, we just need to update the structure and feel (via HTML and CSS) as well as add user interaction (via JavaScript/jQuery). To help with this, we’ll use the Bootstrap front-end framework.

Open the _base.html template.

We’ve already included the Bootstrap stylesheet (via a CDN) along with the jQuery and Bootstrap JavaScript libraries and a custom stylesheet and JavaScript file (both of which reside in the “static” folder):

Template

First, let’s update the index.html template:

Now let’s test…

With Docker: 

Rebuild the Docker image and then run the new container:

Without Docker: 

Navigate to your app in the browser, and you should see:

1_first-iteration

As you can tell, we added four images on the left side and a results table on the right side. Pay attention to the CSS selectors ( id s and class es) in the above HTML code. The row  and col-md-x  classes are associated with the Bootstrap grid system. The remaining id s and class es are used for adding styles with CSS and/or interaction via JavaScript/jQuery.

JavaScript/jQuery

Note: If you’re unfamiliar with JavaScript and jQuery basics, please check out the Madlibs tutorial.

Let’s break down the user interaction by each individual piece of interaction.

Image Click

The interaction begins with an image click. In other words, the end user clicks one of the four images on the left side of the page with the end goal of finding similar images.

Update the main.js file:

Run your app. Either:

  • Rebuild the Docker image, and run the new container.
  • Run  python app/app.py 

Then navigate to your app in the browser. Open your JavaScript console and then click one of the images. You should see:

2_second-iterationSo, the jQuery code handles the click event by grabbing the URL of the specific image clicked and adding a CSS class (which we still need to add to the CSS file). The link between the jQuery code and the HTML is the img  class – $(".img").click(function()  and class="img" , respectively. This should be clear.

AJAX Request

With the image URL in hand, we can now send it to the back-end via an AJAX request, which is a client-side technology for making asynchronous requests that don’t cause an entire page refresh. Most SPAs use some sort of asynchronous technology to prevent a page refresh when requesting data since this enhances the overall user experience.

Update main.js like so:

You know the drill: Run the app, and then refresh your browser. Click an image again, and then after a few seconds you should see:

3_third-iterationNote: This request is quite slow since we are searching a CSV rather than an actual database – i.e., SQLite, Postgres, MySQL. It’s a fairly trivial job to convert the data to a database. Try this on your own. Feel free to comment below if you have questions and/or a solution that you’d like us to look at. Cheers!

This time after the user click, we send a POST request to the /search  endpoint, which includes the image URL. The back-end does it’s magic (grabbing the image, running the search code) and then returns the results in JSON format. The AJAX request has two handlers – one for a success and one for a failure. Jumping back to the back-end, the /search  route either returns a 200 response (a success) or a 500 response (a failure) along with the data or an error message:

Back to the front-end… since the result was successful, you can see the data in the JavaScript console:

This is just an array of JSON objects. Go ahead and expand the array and open an individual object:

4_json-dataSo, each object has an image and a score that represents the “similarity” between the query image and the result image. The smaller the score is, the more “similar” the query is to the result. A similarity of zero would indicate “perfect similarity”. This is the exact data we want to present to the end user.

Update the DOM

We’re in the home stretch! Let’s update the success and error handlers so that once one of them receives data from the back-end, we append that data to the DOM:

In the success handler, we loop through the results, add in some HTML (for the table), then append the data to an id of results  (which is already part of the HTML template). The error handler does not actually update the DOM with the exact error returned. Instead, we log the exact error it to the console, for us to see, and then “unhide” an HTML element with an id of error  (which, again, we need to add to the HTML template).

We also need to add some global variables to the top of the file JavaScript file:

Look through the file and see if you can find how these variables are put to use.

Finally, before we test, let’s update the index.html template…

Add:

Right above:

Okay. Think about what’s going to happen now when we test. if all went well, you should see:

5_fourth-iteration

Boom! You can even click on the image URLs to see the actual results (e.g., the similar images). Notice how you can see the error though. We still have some cleaning up to do.

DOM Cleanup

With the main functionality done, we just need to do a bit of housekeeping. Update main.js like so:

Take a look at the added code…

and

and

and

and

We’re just hiding and showing different HTML elements based on the user interaction and whether the results of the AJAX request is a success or failure. If you’re really paying attention you probably saw that there is a new CSS selector that you have not seen before – #searching . What does this mean? Well, first off we need to update the template…

Add:

Right above:

Now, let’s test! What’s different? Well, when the end user clicks an image, the text Searching...  appears, which disappears when results are added. Then if the user clicks another image, the previous results disappear, the Searching...  text reappears, and finally the new results are added to the DOM.

Take a breath. Or two. We’re now done with the JavaScript portion. It’s a good idea to review this before moving on.

CSS

There’s a lot we could do, style-wise, but let’s keep it simple. Add the following code to main.css:

Run the app, which should now look like this:

6_fifth-iterationThe big change is now when a user clicks an image, a red border appears around it, just reminding the end user which image s/he clicked. Try clicking another image. The red border should now appear around that image. Return to the JavaScript file and review the code to find out how this works.

Refactor

We could stop here, but let’s refactor the code slightly to show thumbnails of the top three results. This is an image search engine, after all – We should display some actual images!

Starting with the back-end, update the search()  view function so that it returns only the top three results:

Next update the for loop within the success handler main.js:

Finally, add the following CSS style:

You should now have:

7_sixth-iteration

Boom!

Conclusion and Next steps

To recap, we took the search engine code from the first tutorial and wrapped in in Flask to create a full-featured web application. If you’d like to continue working with Flask and web development in general, try:

  • Replacing the static CSV file with a relational database;
  • Updating the overall user experience by allowing a user to upload an image, rather than limiting the user to search only by the four images;
  • Adding unit and integration tests;
  • Deploying to Heroku.

Be sure to check out the Real Python course to learn how to do all of these and more.

Cheers!

Interested in learning more?

Interested in learning more about the algorithms behind building image search engines? Or utilizing Python web frameworks to upload images and make them searchable? We are considering creating an online course covering image search engines in more depth, from both a technical computer vision side, and a practical web development side.

If this sounds interesting to you, enter your email address in the form below and (if we get enough interest) we will keep you up to date with how the course progresses.

, , , , , , ,

24 Responses to Adding a web interface to our image search engine with Flask

  1. Guri Holmes December 8, 2014 at 2:22 pm #

    Hi Adrian,
    I wanted to ask how do you add code snippets in your code. Which script do you use?

    • Adrian Rosebrock December 8, 2014 at 6:18 pm #

      Hi Guri, I’m not sure I understand your question, but I use a WordPress plugin to embed the code snippets into each post.

  2. Raghu December 9, 2014 at 1:46 am #

    Hello Adrian,
    When are you planning to release the paperback version of your books in India?

    • Adrian Rosebrock December 9, 2014 at 7:28 am #

      Hi Raghu, great question. Shoot me an email and we can chat about the hardcopy version of the book.

  3. Lukas Essien December 15, 2014 at 5:33 am #

    Hi Adrian,
    I already own your other books and they are really good books for beginners. I was looking for a good book on CBIR, but could not find one. Can you recommend one? Moreover, it would be great if you could assort your blog post on CBIR in the form of a book with a bit of high level concepts like BoG etc.
    Thank You

    • Adrian Rosebrock December 15, 2014 at 7:34 am #

      Hi Lukas, great question. Send me an email and let’s chat about this some more. I have some things in the works that I think you might be really interested in :-)

  4. Kiheum May 5, 2015 at 4:25 pm #

    Hi Adrian, Thanks for your great blog!
    I have a question about this blog because I’ve got some problem.
    When I follow your blog, I stuck at “Run the app, which should now look like this:”.
    The image ,under that point, show 10 results but they doesn’t match with query image(127502.png). It’s different with local test results(last blog’s results).
    And also I can’t drive the last result image when I follow this blog’s code.
    My results are queer.
    I guess “return jsonify(results=(RESULTS_ARRAY[::-1][:3]))”point makes problem, but not sure.
    If you know difference between local and web app’s result,please, let me know

    Always thank for your book and blog

    • Adrian Rosebrock May 5, 2015 at 4:38 pm #

      Hey Kiheum, thanks for the comment. That’s really strange — I’m getting the same local results as on the simple web server. The results should be the same.

    • Navneet March 28, 2016 at 7:56 pm #

      The cause of above problem(posted by @Kiheum) is due to the fact that opencv stores images as BGR(reversed RGB) as opposed to that in case of skimage package(RGB).I got it fixed by adding an extra line “img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)” in colordescriptor.py
      Credits to Adrian for this http://www.pyimagesearch.com/2015/03/02/convert-url-to-image-with-python-and-opencv/ blog. Method#2 is helpful in this case.

      • Adrian Rosebrock March 29, 2016 at 7:00 am #

        Thanks for sharing Navneet!

  5. crankdaworld May 7, 2015 at 11:22 am #

    Hi Adrian,
    I tried using the docker and it works fine. As mentioned by Kiheum, the results locally and at web server are different.
    Is there a way, in which i can run the web server locally, i mean the backend in my mac and test it?

    • Adrian Rosebrock May 7, 2015 at 12:00 pm #

      You should be able to launch the Flask webserver locally. Here’s a link to the Flask docs with explanation on how to do that.

  6. panovr May 8, 2015 at 7:04 pm #

    How can I use the data set in my local directory? For example, if I put the “vacation-photos” data set in “static” directory of the project, how to modify the necessary files?
    Thanks!

    • Adrian Rosebrock May 9, 2015 at 7:53 am #

      I would suggest first going back to the previous tutorial on building an image search engine to use your own custom dataset of images. All of the heavy lifting is done via the command line arguments where you supply the paths to your datasets. After you have your image search engine working from the command line, you can then move it to the web server without much of an issue.

  7. Zhaoyun July 23, 2015 at 6:30 am #

    Hi Adrian,
    I’ve tried to run the code as posted in this page ,but been kept getting the error message as follow: “ValueError: View function did not return a response”.
    It seems to work OK if I remove the cv2 part,but when I put them back on as posted,the error I mentioned may occur.
    Do you have any idea how this might happen?
    Thanks.

    • Adrian Rosebrock July 23, 2015 at 6:43 am #

      It sounds like OpenCV has not been installed on your system if the cv2 code is throwing the error.

      • Zhaoyun July 23, 2015 at 6:53 am #

        I did install OpenCV and I tested it worked fine.I think the traceback says Flask throws the error.But I just don’t know why,it seems strange since you or others don’t have the error.

        • cs January 12, 2016 at 12:48 pm #

          I having the same view function error , is it because i running a cv3 ?

          • Andreu February 19, 2016 at 12:56 pm #

            Hi.
            You can use the dataset image folder locally. First, copy it into the static folder.

            The view function error is related to the Flask endpoints. I have solved changing to:

            app = Flask(__name__, static_url_path = “”, static_folder = “static”)
            in app.py

            The dataset image folder is into the static folder. You need to change the main.js to:
            // global
            var url = ‘dataset/’;

  8. Bran July 23, 2016 at 5:20 pm #

    Hi Adrian,
    If I create and activate a vertualenv, where should I place the project?

    • Adrian Rosebrock July 27, 2016 at 2:44 pm #

      You can place the project in whatever directory you would like. I normally like to organize my code in: /home/adrian/projects, but that’s entirely up to you.

  9. Lee August 24, 2016 at 11:22 am #

    It would really be nice if there was more information here at PyImage on making the image functions more usable – specifically a web UI.

    For example, what good is a motion detection camera if you can’t watch the live video during the event?

    For a real-time video or image application, remote management of the system is required, .. there would seldom be access to the X system of the machine.

    • Adrian Rosebrock August 24, 2016 at 12:11 pm #

      In general, PyImageSearch is dedicated to teaching computer vision and image processing techniques. Once you understand these techniques, you can apply them to any type of application you want — whether GUI based or not. I am also not a GUI developer so I don’t have plans to do more GUI related tutorials in the future. That said, the real-time video tutorials on the PyImageSearch video tutorials always include a method to view the live video stream using the cv2.imshow function. This blog post even demonstrates how to use TKinter with a video stream.

  10. Martin November 12, 2016 at 4:24 pm #

    I have a problem, when I’m clicking the image it showing me “TypeError: data is undefined” in file main.js and results are not showing up. Someone can help me?

Leave a Reply