Video_Classification_With_Pre_Trained_Models_Project.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Video Classification With Pre-Trained Models Project",
      "private_outputs": true,
      "provenance": [],
      "collapsed_sections": [
        "copyright",
        "s5UQCAU90N81"
      ]
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "copyright"
      },
      "source": [
        "#### Copyright 2020 Google LLC."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "khlO4Bu21oZ4"
      },
      "source": [
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AlzIlBsScJJ_"
      },
      "source": [
        "# Video Classification with Pre-Trained Models Project"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nTirVS4FWaPx"
      },
      "source": [
        "In this project we will import a pre-existing model that recognizes objects and use the model to identify those objects in a video. We'll edit the video to draw boxes around the identified object, and then we'll reassemble the video so the boxes are shown around objects in the video."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YTVUYxPwcHhp"
      },
      "source": [
        "# Exercises"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LdIOgOHP1ces"
      },
      "source": [
        "## Exercise 1: Coding"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jhTEOK1ZmqN8"
      },
      "source": [
        "You will process a video frame by frame, identify objects in each frame, and draw a bounding box with a label around each car in the video.\n",
        " \n",
        "Use the [SSD MobileNet V1 Coco](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf1_detection_zoo.md) (*ssd_mobilenet_v1_coco*) model. The video you'll process can be found [on Pixabay](https://pixabay.com/videos/cars-motorway-speed-motion-traffic-1900/). The 640x360 version of the video is smallest and easiest to handle, though any size should work since you must scale down the images for processing.\n",
        " \n",
        "Your program should:\n",
        " \n",
        "* Read in a video file (use the one in this colab if you want)\n",
        "* Load the TensorFlow model linked above\n",
        "* Loop over each frame of the video\n",
        "* Scale the frame down to a size the model expects\n",
        "* Feed the frame to the model\n",
        "* Loop over detections made by the model\n",
        "* If the detection score is above some threshold, draw a bounding box onto the frame and put a label in or near the box\n",
        "* Write the frame back to a new video\n",
        " \n",
        "Some tips:\n",
        " \n",
        "* Processing an entire video is slow, so consider truncating the video or skipping over frames during development. Skipping frames will make the video choppy. But you'll be able to see a wider variety of images than you would with a truncated video with all of the original frames in the clip.\n",
        "* The model expects a 300x300 image. You'll likely have to scale your frames to fit the model. When you get a bounding box, that box is relative to the scaled image. You'll need to scale the bounding box out to the original image size.\n",
        "* Don't start by trying to process the video. Instead, capture one frame and work with it until you are happy with your object detection, bounding boxes, and labels. Once you get those done, use the same logic on the other frames of the video.\n",
        "* The [Coco labels file](https://github.com/nightrome/cocostuff/blob/master/labels.txt) can be used to identify classified objects.\n",
        " "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7XM35vYWSbim"
      },
      "source": [
        "### **Student Solution**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ivTzfzQN5jDk"
      },
      "source": [
        "# Your code goes here\n",
        "import cv2 as cv\n",
        "from PIL import Image as im\n",
        "import tensorflow as tf\n",
        "import urllib.request\n",
        "import os\n",
        "import tarfile\n",
        "import shutil\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "cars_video = cv.VideoCapture('cars.mp4')\n",
        "\n",
        "height = int(cars_video.get(cv.CAP_PROP_FRAME_HEIGHT))\n",
        "width = int(cars_video.get(cv.CAP_PROP_FRAME_WIDTH))\n",
        "fps = cars_video.get(cv.CAP_PROP_FPS)\n",
        "total_frames = int(cars_video.get(cv.CAP_PROP_FRAME_COUNT))\n",
        "\n",
        "print(f'height: {height}')\n",
        "print(f'width: {width}')\n",
        "print(f'frames per second: {fps}')\n",
        "print(f'total frames: {total_frames}')\n",
        "print(f'video length (seconds): {total_frames / fps}')\n",
        "\n",
        "cars_video.release()\n",
        "\n",
        "cars_video = cv.VideoCapture('cars.mp4')\n",
        "total_frames = int(cars_video.get(cv.CAP_PROP_FRAME_COUNT))\n",
        "\n",
        "frames_read = 0\n",
        "\n",
        "for current_frame in range(0, total_frames):\n",
        "  cars_video.set(cv.CAP_PROP_POS_FRAMES, current_frame)\n",
        "  ret, _ = cars_video.read()\n",
        "  if not ret:\n",
        "    raise Exception(f'Problem reading frame {current_frame} from video')\n",
        "  if (current_frame+1) % 50 == 0:\n",
        "    print(f'Read {current_frame+1} frames so far')\n",
        "\n",
        "cars_video.release()\n",
        "\n",
        "print(f'Read {total_frames} frames')\n",
        "\n",
        "base_url = 'http://download.tensorflow.org/models/object_detection/'\n",
        "file_name = 'ssd_mobilenet_v1_coco_2018_01_28.tar.gz'\n",
        "\n",
        "url = base_url + file_name\n",
        "\n",
        "urllib.request.urlretrieve(url, file_name)\n",
        "\n",
        "#os.listdir()\n",
        "\n",
        "dir_name = file_name[0:-len('.tar.gz')]\n",
        "\n",
        "if os.path.exists(dir_name):\n",
        "  shutil.rmtree(dir_name) \n",
        "\n",
        "tarfile.open(file_name, 'r:gz').extractall('./')\n",
        "\n",
        "# os.listdir(dir_name)\n",
        "\n",
        "frozen_graph = os.path.join(dir_name, 'frozen_inference_graph.pb')\n",
        "\n",
        "with tf.io.gfile.GFile(frozen_graph, \"rb\") as f:\n",
        "  graph_def = tf.compat.v1.GraphDef()\n",
        "  loaded = graph_def.ParseFromString(f.read())\n",
        "\n",
        "\n",
        "outputs = (\n",
        "  'num_detections:0',\n",
        "  'detection_classes:0',\n",
        "  'detection_scores:0',\n",
        "  'detection_boxes:0',\n",
        ")\n",
        "\n",
        "def wrap_graph(graph_def, inputs, outputs, print_graph=False):\n",
        "  wrapped = tf.compat.v1.wrap_function(\n",
        "    lambda: tf.compat.v1.import_graph_def(graph_def, name=\"\"), [])\n",
        "  return wrapped.prune(\n",
        "    tf.nest.map_structure(wrapped.graph.as_graph_element, inputs),\n",
        "    tf.nest.map_structure(wrapped.graph.as_graph_element, outputs)\n",
        "  )\n",
        "    \n",
        "model = wrap_graph(graph_def=graph_def, inputs=[\"image_tensor:0\"], outputs=outputs)\n",
        "\n",
        "#input video\n",
        "input_video = cv.VideoCapture('cars.mp4')\n",
        "\n",
        "class_dict = {\n",
        "    3.0: 'car',\n",
        "    6.0: 'bus',\n",
        "    7.0: 'train',\n",
        "    8.0: 'truck',\n",
        "    77.0: 'cell phone'\n",
        "}\n",
        "\n",
        "#getting height width fps\n",
        "height = int(input_video.get(cv.CAP_PROP_FRAME_HEIGHT))\n",
        "width = int(input_video.get(cv.CAP_PROP_FRAME_WIDTH))\n",
        "fps = input_video.get(cv.CAP_PROP_FPS)\n",
        "\n",
        "#generate video file to be written\n",
        "fourcc = cv.VideoWriter_fourcc(*'mp4v')\n",
        "output_video = cv.VideoWriter('processed.mp4', fourcc, fps, (300, 300))\n",
        "\n",
        "\n",
        "for i in range(0, int(input_video.get(cv.CAP_PROP_FRAME_COUNT)), 5):\n",
        "  input_video.set(cv.CAP_PROP_POS_FRAMES, i)\n",
        "  ret, frame = input_video.read()\n",
        "  if ret == True:\n",
        "    processed = cv.resize(frame,(width, height),fx=0,fy=0, interpolation = cv.INTER_CUBIC)\n",
        "    output_video.write(processed)\n",
        "    output_video.write(frame)\n",
        "\n",
        "  tensor = tf.convert_to_tensor([processed], dtype=tf.uint8)\n",
        "  detections = model(tensor)\n",
        "\n",
        "  num_pet = int(detections[0].numpy()[0])\n",
        "  classes = detections[1].numpy()[0, :num_pet]\n",
        "  scores = detections[2].numpy()[0, :num_pet]\n",
        "  boxes = detections[3].numpy()[0, :num_pet]\n",
        "  color = None\n",
        "  H, W, _ = processed.shape\n",
        "  for x in range(num_pet):\n",
        "    if scores[x] > 0.5:\n",
        "      box = boxes[x]\n",
        "      y1, x1, y2, x2 = box\n",
        "      x1 *= W \n",
        "      x2 *= W \n",
        "      y1 *= H\n",
        "      y2 *= H\n",
        "      cv.rectangle(processed, (int(x1), int(y1)), (int(x2), int(y2)), (255, 0, 0), 2)\n",
        "      cv.putText(processed, class_dict[classes[x]], (int(x1), int(y2+30)), cv.FONT_HERSHEY_SIMPLEX, 1, [255, 0, 0], 2) \n",
        "    output = cv.resize(processed, (300, 300))\n",
        "\n",
        "  output_video.write(output)\n",
        "input_video.release()\n",
        "output_video.release()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VEGDiC-IhcrM"
      },
      "source": [
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HniKdSXg0YHR"
      },
      "source": [
        "## Exercise 2: Ethical Implications"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W4FvC1Aa0ZT5"
      },
      "source": [
        "Even the most basic models have the potential to affect segments of the population in different ways. It is important to consider how your model might positively and negatively affect different types of users.\n",
        "\n",
        "In this section of the project, you will reflect on the positive and negative implications of your model. Frame the context of your model creation using this narrative:\n",
        "\n",
        "> The city of Seattle is attempting to reduce traffic congestion in its downtown area. As part of this project, they plan to allow each local driver one free trip to downtown Seattle per week. After that, the driver will have to pay a $50 toll for each extra day per week driven. As an early proof of concept for this project, your team is tasked with using machine learning to correctly identify automobiles on the road. The next phase of the project will involve detecting license plate numbers and then cross-referencing that data with RFID chips that should be mounted in all local drivers' cars."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lkyzwVQr0brd"
      },
      "source": [
        "### **Student Solution**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gy4I2vG60ebd"
      },
      "source": [
        "**Positive Impact**\n",
        "\n",
        "Your model is trying to solve a problem. Think about who will benefit from that problem being solved and write a brief narrative about how the model will help."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k59MK1Ah0fWy"
      },
      "source": [
        "> *Hypothetical entities will benefit because...*\n",
        "the model could benefitthe traffic association to count the traffic flow improve the traffic flow at the heavy-traffic locations, throughout different times of the day. As well as the police in terms of speedint regulation and tickets.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gzqkrLnk0hMU"
      },
      "source": [
        "**Negative Impact**\n",
        "\n",
        "Models rarely benefit everyone equally. Think about who might be negatively impacted by the predictions your model is making. This person(s) might not be directly using the model, but they might be impacted indirectly."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Hefa1JdP0kj3"
      },
      "source": [
        "> *Hypothetical entities will be negatively impacted because...*\n",
        "This model will have a negative  impact on the drivers who consitenly drive over the speed limit. Or even one-off situations where someone temporarily goes over the speed limit.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Uax2HAzd0mHX"
      },
      "source": [
        "**Bias**\n",
        "\n",
        "Models can be biased for many reasons. The bias can come from the data used to build the model (e.g., sampling, data collection methods, available sources) and/or from the interpretation of the predictions generated by the model.\n",
        "\n",
        "Think of at least two ways bias might have been introduced to your model and explain both below."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6bJGm-qs0oQV"
      },
      "source": [
        "> *One source of bias in the model could be...One bias that we notices is that not every vehicle is detected in our output, which could lead to a lower accuracy.*\n",
        "\n",
        "> *Another source of bias in the model could be...A second bias that we noticied is incorrect classifying, as our model misidentified some of our cars.*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ybb1zAkC0p2e"
      },
      "source": [
        "**Changing the Dataset to Mitigate Bias**\n",
        "\n",
        "Having bias in your dataset is one of the primary ways in which bias is introduced to a machine learning model. Look back at the input data you fed to your model. Think about how you might change something about the data to reduce bias in your model.\n",
        "\n",
        "What change or changes could you make to reduce the bias in your dataset? Consider the data you have, how and where it was collected, and what other sources of data might be used to reduce bias.\n",
        "\n",
        "Write a summary of changes that could be made to your input data."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UFsnF4_h08DD"
      },
      "source": [
        "> *Since the data has potential bias A we can adjust... I think we could just add more training data or have a smaller, less traffic video to allow our program to be more accurate and precise.*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ChEJbhXA02pW"
      },
      "source": [
        "**Changing the Model to Mitigate Bias**\n",
        "\n",
        "Is there any way to reduce bias by changing the model itself? This could include modifying algorithmic choices, tweaking hyperparameters, etc.\n",
        "\n",
        "Write a brief summary of changes you could make to help reduce bias in your model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kEAhgO_U0p8Y"
      },
      "source": [
        "> *Since the model has potential bias A, we can feed in training videos that contain individual colors so that the trained model can better identify and classify(box every moving objects correctly) cars.*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rShB5BQv0wix"
      },
      "source": [
        "**Mitigating Bias Downstream**\n",
        "\n",
        "Models make predictions. Downstream processes make decisions. What processes and/or rules should be in place for people and systems interpreting and acting on the results of your model to reduce bias? Describe these rules and/or processes below."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C__BwBP-00HN"
      },
      "source": [
        "> *Since the predictions have potential bias A, we can adjust...*\n",
        "Since the predictions have potential bias in favored to specific cars with specific colors, then we can come up a way to preprocess the videoes so that different color cars can be correctly exposed in the videoes for training. one potential way to preprocess the videoes is to adjust the threshold to expose different colors."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1L_4RNXphYtI"
      },
      "source": [
        "---"
      ]
    }
  ]
}