mlkit/tutorials/mlkit_image_labeling_model_maker.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "mlkit_image_labeling_model_maker.ipynb",
      "provenance": [],
      "private_outputs": true,
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.8"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "h2q27gKz1H20"
      },
      "source": [
        "##### Copyright 2019 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "colab_type": "code",
        "id": "TUfAcER1oUS6",
        "colab": {}
      },
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Gb7qyhNL1yWt"
      },
      "source": [
        "# Create ML Kit Image labeling model with Tensorflow Lite Model Maker"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "nDABAblytltI"
      },
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://www.tensorflow.org/lite/tutorials/model_maker_image_classification\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />View on TensorFlow.org</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/tutorials/model_maker_image_classification.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/tutorials/model_maker_image_classification.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://storage.googleapis.com/tensorflow_docs/tensorflow/tensorflow/lite/g3doc/tutorials/model_maker_image_classification.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "m86-Nh4pMHqY"
      },
      "source": [
        "Model Maker library simplifies the process of adapting and converting a TensorFlow neural-network model to particular input data when deploying this model for on-device ML applications.\n",
        "\n",
        "This notebook shows an end-to-end example that utilizes this Model Maker library to create an image labeling model for ML Kit custom Image Labeling and Object Detection and Tracking features."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "bcLF2PKkSbV3"
      },
      "source": [
        "## Prerequisites\n",
        "\n",
        "To run this example, we first need to install serveral required packages, including Model Maker package that in github [repo](https://github.com/tensorflow/examples/tree/master/tensorflow_examples/lite/model_maker)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "6cv3K3oaksJv",
        "colab": {}
      },
      "source": [
        "!pip install git+https://github.com/tensorflow/examples.git#egg=tensorflow-examples[model_maker]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Gx1HGRoFQ54j"
      },
      "source": [
        "Import the required packages."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "XtxiUeZEiXpt",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "\n",
        "import tensorflow as tf\n",
        "\n",
        "from tensorflow_examples.lite.model_maker.core.data_util.image_dataloader import ImageClassifierDataLoader\n",
        "from tensorflow_examples.lite.model_maker.core.task import image_classifier\n",
        "from tensorflow_examples.lite.model_maker.core.task.model_spec import ImageModelSpec\n",
        "from tensorflow_examples.lite.model_maker.core.task import configs\n",
        "from tensorflow_examples.lite.model_maker.core import compat\n",
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "brWCSWSn4ASn",
        "colab_type": "text"
      },
      "source": [
        "Make sure to set tf_version as 1 to produce models with uint8 input and output types to be compatible with ML Kit."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WfwMNeYY4L98",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "compat.setup_tf_behavior(tf_version=1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "SiZZ5DHXotaW"
      },
      "source": [
        "## Get the data path\n",
        "\n",
        "Let's get some images to play with this simple end-to-end example. Hundreds of images is a good start for Model Maker while more data could achieve better accuracy."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "colab_type": "code",
        "id": "3jz5x0JoskPv",
        "colab": {}
      },
      "source": [
        "image_path = tf.keras.utils.get_file(\n",
        "      'flower_photos',\n",
        "      'https://storage.googleapis.com/download.tensorflow.org/example_images/flower_photos.tgz',\n",
        "      untar=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "a55MR6i6nuDm"
      },
      "source": [
        "You could replace `image_path` with your own image folders. As for uploading data to colab, you could find the upload button in the left sidebar shown in the image below with the red rectangle. Just have a try to upload a zip file and unzip it. The root file path is the current path.\n",
        "\n",
        "<img src=\"https://storage.googleapis.com/download.tensorflow.org/models/tflite/screenshots/model_maker_image_classification.png\" alt=\"Upload File\" width=\"800\" hspace=\"100\">\n",
        "\n",
        "Make sure the file structure is correct. For example, the flower dataset contains 3670 images belonging to 5 classes. \n",
        "\n",
        "The dataset has the following directory structure:\n",
        "\n",
        "<pre>\n",
        "<b>flower_photos</b>\n",
        "|__ <b>daisy</b>\n",
        "    |______ 100080576_f52e8ee070_n.jpg\n",
        "    |______ 14167534527_781ceb1b7a_n.jpg\n",
        "    |______ ...\n",
        "|__ <b>dandelion</b>\n",
        "    |______ 10043234166_e6dd915111_n.jpg\n",
        "    |______ 1426682852_e62169221f_m.jpg\n",
        "    |______ ...\n",
        "|__ <b>roses</b>\n",
        "    |______ 102501987_3cdb8e5394_n.jpg\n",
        "    |______ 14982802401_a3dfb22afb.jpg\n",
        "    |______ ...\n",
        "|__ <b>sunflowers</b>\n",
        "    |______ 12471791574_bb1be83df4.jpg\n",
        "    |______ 15122112402_cafa41934f.jpg\n",
        "    |______ ...\n",
        "|__ <b>tulips</b>\n",
        "    |______ 13976522214_ccec508fe7.jpg\n",
        "    |______ 14487943607_651e8062a1_m.jpg\n",
        "    |______ ...\n",
        "</pre>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "NNRNv_mloS89"
      },
      "source": [
        "If you prefer not to upload your images to the cloud, you could try to run the library locally following the [guide](https://github.com/tensorflow/examples/tree/master/tensorflow_examples/lite/model_maker) in github.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "w-VDriAdsowu"
      },
      "source": [
        "## Run the example\n",
        "The example just consists of 4 lines of code as shown below, each of which representing one step of the overall process.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "6ahtcO86tZBL"
      },
      "source": [
        "Step 1.   Load input data specific to an on-device ML app. Split it to training data and testing data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "lANoNS_gtdH1",
        "colab": {}
      },
      "source": [
        "train_data, test_data = ImageClassifierDataLoader.from_folder(image_path).split(0.9)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Y_9IWyIztuRF"
      },
      "source": [
        "Step 2. Customize the TensorFlow model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "yRXMZbrwtyRD",
        "colab": {}
      },
      "source": [
        "model = image_classifier.create(train_data)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "oxU2fDr-t2Ya"
      },
      "source": [
        "Step 3. Evaluate the model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "wQr02VxJt6Cs",
        "colab": {}
      },
      "source": [
        "loss, accuracy = model.evaluate(test_data)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NtCLU30l4ZSV",
        "colab_type": "text"
      },
      "source": [
        "Step 4. Setup config for quantized model with uint8 input and output type"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "W4QMeF974hgD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "config = configs.QuantizationConfig.create_full_integer_quantization(\n",
        "    representative_data=test_data, is_integer_only=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "eVZw9zU8t84y"
      },
      "source": [
        "Step 4.  Export to TensorFlow Lite model.\n",
        "\n",
        "Here, we export TensorFlow Lite model with [metadata](https://www.tensorflow.org/lite/convert/metadata) which provides a standard for model descriptions.\n",
        "You could download it in the left sidebar same as the uploading part for your own use."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Zb-eIzfluCoa",
        "colab": {}
      },
      "source": [
        "model.export(export_dir='.', quantization_config=config)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gwIhJ6-93qrw",
        "colab_type": "text"
      },
      "source": [
        "After this simple 4 steps, we could further use TensorFlow Lite model file in ML Kit Image Labeling and Object Detection and Tracking features."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Lpppfth3kgJA",
        "colab_type": "text"
      },
      "source": [
        "Tensorflow Lite Model Maker allows changing model architecture to suit different needs. Here is the instructions of how to change model architecture:\n",
        "https://www.tensorflow.org/lite/tutorials/model_maker_image_classification#change_the_model"
      ]
    }
  ]
}