BDCC - project 1

Big Data & Cloud Computing

Eduardo R. B. Marques, DCC/FCUP

Updates:

• Apr. 20 - deadline revised to Apr. 29 + submission instructions.
• Apr. 4 - Revised group conditions.
• Apr. 3 - initial version.

SUBMISSION INSTRUCTIONS AVAILABLE HERE

Summary

In this project you will have to program an AppEngine app that provides information about images taken from the Open Images dataset. Additionally, the app employs a TensorFlow model for image classification derived using AutoML Vision.

A partially completed application is provided as a starting point, including a functional TensorFlow model, and a demo application with all the features can be tested at https://bdcc22project.appspot.com.

The primary work will consist in:

• Defining a BigQuery data set from given CSV files using a Python script or Colab notebook.
• Programming the app endpoints that are missing. Each endpoint needs to query data stored in BigQuery, and reference the corresponding image files stored in a public storage bucket;
• Deriving your own TensorFlow model with AutoML, replacing the one that is provided with the application. AutoML requires a dataset that you should define using a Python script or a Colab notebook.

Some additional challenges are also proposed (15 % of the grade).

Project work and delivery

• You may work on your own or with a colleague. If the group includes working students, groups may be formed by three students.
• The deadline for the project delivery is April 25 29.
• You will have to deliver a report of your work in PDF format, and your code in a ZIP file. The details regarding project delivery will be announced later.
• In the report you should provide:
  • Basic info
    • The author(s) identification;
    • The identifier of the Google Cloud project;
    • The URL for the AppEngine app
  • An explanation of how your app works in terms of BigQuery queries (you may include code snippets) per each implemented endpoint.
  • An explanation of how the TensorFlow data set preparation script/notebook works.
• The ZIP file should contain the application code plus the additional Python scripts or Colab notebooks.

Getting started

Create a new project

Recommended: in the Google cloud console, create a new project for all developments to avoid confusion with other work you may have done previously.

Get the code

Download the ZIP file for the initial application code - app.zip.

You may then upload it to the Cloud Shell environment, and then unzip it.

$ unzip app.zip

Setup Cloud shell

Make sure you have the Pillow package installed.

$ pip3 install Pillow

Run the app for the first time

$ cd app
$ python3 main.py

Use the Web preview for development, and deploy only from time to time to test more stable versions.

Deploy it for the first time

$ gcloud app deploy

Note:

• An initial deployment is necessary for the image classification to work in the cloud shell environment (the application bucket for files must be created).

• If you get an error like the one shown below, click on the link displayed to enable the Cloud Build API, and wait 5-10 minutes before deploying again. Clicking on the link is sufficient to enable the Build API (you do NOT need to “create credentials” in the Cloud Build API).

  Error Response: [7] Access Not Configured. Cloud Build has not been used in project bdcc22project before or it is disabled. 
  Enable it by visiting https://console.developers.google.com/apis/api/cloudbuild.googleapis.com/overview?project=bdcc22project 
  then retry. 
  If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.

Data model

To get access to the data in the BigQuery console use Add Data > Pin a project > Enter project name and specify bdcc22project.

You can use the bdcc22project.openimages BigQuery data set for initial development of the app, but at some point define your own BigQuery data set PROJECT_ID.openimages and use it in your app. For that purpose, write a Python Script or Colab notebook that defines the necessary BigQuery data set. Check the following links to obtain the data in CSV format:

• Image classes (classes.csv)
• Labels per image (image-labels.csv)
• Relations (relations.csv)

References:

• BigQuery Python API
• BigQuery - table creation (Lab 3 notebook)

Application endpoints

The following application endpoints are already implemented:

• classes
• image_search
• image_classify_classes
• image_classify

You must implement the remaining ones:

• image_search_multiple
• relations
• relations_search
• image_info

The visual aspect of generated HTML pages may differ, but the displayed data and the functionality provided through HTML links should be similar.

References:

• BigQuery Python API
• Flask
• Jinja templates
• HTML reference

TensorFlow model using AutoML

Using a Python script or Colab notebook you must derive an AutoML dataset for a minimum of 10 image classes such that:

• For each class there should be 100 image files split into 3 groups: 80 for training, 10 for validation and 10 for testing.
• The image classes must differ from those already supported in the provided AutoML model. Consider for instance a set of different animals, kinds of food or objects, do not worry otherwise in picking “bad images”. Just avoid generic classes (e.g., “Mammal” “Auto part”, “Human”) that may have images that are too heterogenous.

An example CSV format for an AutoML data set is provided in static/tflite/automl.csv (and also here: https://bdcc22project/appspot.com/static/tflite/automl.csv)

You may use the score_image.py program to test your TensorFlow model from the Cloud Shell command line.

$ ./score_image.py file1.jpg file2.jpg ... 

• Reference:
  • AutoML Vision - Edge device model quick-start
  • If you are curious about the machine learning techniques involved have a look at Section 10.4 (“Deep Learning - A Shallow Introduction”) of the “Cloud Computing for Science and Engineering” textbook.

Additional challenges

Use of the Cloud Vision API

Develop an alternative app endpoint for image classification that makes use of label detection through the Google Cloud Vision API using the corresponding Python client API.

Explain what you did in the project report.

Define a Docker image for the app

AppEngine is enabled by containers internally. Why not define your own container explicitly for the app? All it takes is a DockerFile :)

The app should run in the Cloud Shell environment as a container instance, and then also through Cloud Run.

Explain what you did in the project report, in particular the meaning of commands in your DockerFile.