Functional Data Programming Project.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Functional Data Programming Project - PrestaCop\n",
    "\n",
    "## Subject\n",
    "\n",
    "### Client\n",
    "\n",
    "PrestaCop, a company specializing in service delivery for police forces, wants to create a drone service to help police systems make parking tickets.\n",
    "\n",
    "A camera with a pattern recognition software identifies license plates and characterizes infractions.\n",
    "\n",
    "PrestaCop has teams to develop the drone and does work. The first prototype is finished it can qualify some infraction. However Prestacop service comes with a software as service offer that PrestaCop struggle to create.\n",
    "\n",
    "### Drone description\n",
    "\n",
    "Each drone send message regularly, each standard message contains : drone location, time, drone id.\n",
    "\n",
    "In the event of a violation, the drone sends separately:\n",
    "\n",
    "- 1 image\n",
    "- a standard message with additional field : a violation code describing the nature of the offense, image id\n",
    "\n",
    "A human operator can take control of any drone at any time. Taking control of the drone is done through an already existing software using the drone id.\n",
    "\n",
    "In doing so the employee is free to observe vehicles from any angle while driving the drone.\n",
    "\n",
    "### Alert\n",
    "\n",
    "When the drone can not qualify a possible offense with accuracy, its message’s violation code indicate it requires human intervention.\n",
    "\n",
    "If this happen a operator (probably a NYPD officer) using the remote control feature described below  understand the situation and may send a new message with the right violation code.  (Example a police officer will take control of the drone in order to read a car’s plate that the drone can’t read).\n",
    "\n",
    "After a few tests performed prestacop estimates that this represents 1% of observed violations.\n",
    "\n",
    "\n",
    "### Statistics\n",
    "\n",
    "PrestaCop is convinced that we need to keep every drone’s messages in order to make statistics and improve their services. But they still don’t know what kindof question/statistic they will want to address.\n",
    "\n",
    "### Initial feed\n",
    "\n",
    "PrestaCop has forged a partnership with the New York police to retrieve the history of their tickets data. They want to use NYPD historical data to improvethose statistics thus they want to transform it to its equivalent drone data to feed their information system. This historical data is existing data on tickets previously written by NYPD (without Prestacop’s system).\n",
    "\n",
    "NYPD poses 2 constraints: its computers are old and not very powerful, [This history is stored in a large CSV that you can find here.](https://www.kaggle.com/new-york-city/nyc-parking-tickets#Parking_Violations_Issued_-_Fiscal_Year_2015.csv)\n",
    "\n",
    "### Failed attempt\n",
    "\n",
    "To create a POC PrestaCop hired a team of data-scientists and Prestacop manager expect this team of data-scientists to provide a program that uses few memory resources and can send to PrestaCop computers (or cloud) NYPD historical data.\n",
    "\n",
    "Despite all their efforts, PrestaCop's data teams have not been able to set up a program that is light enough to send them data from New York police computers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preliminary questions\n",
    "\n",
    "1. What technical/business constraints should the architecture meet to fulfill the requirement described by the customer in paragraph «Statistics» ? (In other words the customer has express some needs, some existing solutions, it comes with limitations).\n",
    "\n",
    "2. Same question with the paragraph «Alert»\n",
    "\n",
    "3. What mistake(s) from Prestacop can explains the failed attempt?\n",
    "\n",
    "4. Prestacop has likely forgot some technical information in the regular message sent by the drone. In the future this information could help Prestacop make its product much more profitable. Which information?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Project\n",
    "\n",
    "PrestaCop understand this is beyond their team limits, it can not put in place an information system to deal with the drone’s data. PrestaCop asks you for advice to design an architecture allowing them to create a product they could sell to different police forces.\n",
    "\n",
    "Ideally you should also tell PrestaCop how to power this product with NYPD data.\n",
    "\n",
    "\n",
    "It's up to you to report and recommend what to do."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preliminary questions - Our answers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Technical / Business constraints of the architecture (Statistics)\n",
    "\n",
    "Because the drones will be flying and collecting data about parking in New York, so it is reasonable to suppose the amount of data the drones will collect will be rapidly huge.\n",
    "\n",
    "Since the custommer wants to keep track of the data, we must store it somewhere, for that, and considering the volume of data that we will have to process, a NoSQL database seems to be the best solution because it allows to provide better response times with huge volumes of data.\n",
    "\n",
    "For this project, a document-oriented NoSQL database should be sufficient and powerful enough, so MongoDB will be the database of choice, but once again, the volume of data that should be stored without any transformation will quickly become problematic without a distributed storage solution like HDFS or AWS S3 to back it up.\n",
    "\n",
    "Now, we have another problem: we will have dozens of drones sending data at regular intervals, probably at the same time, so we will have multiple producers and we will also have multiple consumers: an analysis program which should dispatch incoming requests for human assistance and another one that will simply write new records from the drones to the database. For the drone navigation data, another program will consume data to gather all data from drones at any time and as soon as it is emitted.\n",
    "\n",
    "This producer/consumer problem will be easy to solve with Kafka streams: a stream into which valid signals from the drones will be sent by the drones and gathered by analysis programs and database writing program, another one into which navigation data will be sent by the drones and gathered by a central control program and another one for the human assistance demands from the drones that will be listened to by multiple instances of the same program which enables its users to take control of drones when needed.\n",
    "\n",
    "Processing such volumes of data is a good indicator that we should use functional programming hence Scala will be the language of choice because of its integration with Spark for data processing and analysis as well as Kafka for streaming.\n",
    "\n",
    "In addition to that, this solution enables us to setup an easy backup database/storage system because oonly the raw data will be circulating in the streams, so it won't be complicated to plug another system to write data to another database to the concerned streams: this is a huge benefit because such data should really be backed up."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Technical / Business constraints of the architecture (Alert)\n",
    "\n",
    "As explained in the previous answer, we need multiple streams and multiple programs to recognize and handle events of different natures. Considering that a drone \"knows\" when it needs human assistance, sending an event into a Kafka stream with the data required to connect to it and awaiting for an operator to take control might be sufficient: the architecture described above seems to be quite good."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Mistakes PrestaCop made\n",
    "\n",
    "Handling huge amount of data is not really problematic, but it is always better to know what you want to do with the data so we can only store the data we want to keep and use. Taking advantage of existing data is good, but wanting to use too much data on a regular basis will always have a (negative) drawback on performances. Since performances are a concern for PrestaCop and the NYPD, this is what the first mistake is. In addition to that performance issue, it is a no brainer to guess that NYPD computers are not brand new and are not optimized for data analysis because they are not suited to perform heavy computations and make extensive use of bandwidth, this also might explain why PrestaCop failed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Technical Information that the drone should send\n",
    "\n",
    "The drone does not send its battery data when reporting but it might be a good idea to do so and hence provide better control over the drones fleet."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------\n",
    "\n",
    "# Project\n",
    "\n",
    "For this project, we have existing data from the NYPD that should be sent into a Kafka stream as if it was drone data.\n",
    "Because we do not want the program to use a lot of resources, we must paralellize the data, hence the use of a DataFrame, a structure which is alreeady optimized for paralellization.\n",
    "\n",
    "## Spark initialization and loading data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "spark = org.apache.spark.sql.SparkSession@41548c26\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "org.apache.spark.sql.SparkSession@41548c26"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import org.apache.spark.sql.SparkSession\n",
    "\n",
    "val spark = SparkSession.builder.appName(\"SparkTest\").config(\"spark.master\", \"local\").getOrCreate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "raw2014 = [Summons Number: string, Plate ID: string ... 49 more fields]\n",
       "raw2015 = [Summons Number: string, Plate ID: string ... 49 more fields]\n",
       "raw2016 = [Summons Number: string, Plate ID: string ... 49 more fields]\n",
       "raw2017 = [Summons Number: string, Plate ID: string ... 41 more fields]\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "[Summons Number: string, Plate ID: string ... 41 more fields]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val raw2014 = spark.read.format(\"csv\").option(\"header\", \"true\").load(\"./Parking_Violations_Issued_-_Fiscal_Year_2014.csv\")\n",
    "val raw2015 = spark.read.format(\"csv\").option(\"header\", \"true\").load(\"./Parking_Violations_Issued_-_Fiscal_Year_2015.csv\")\n",
    "val raw2016 = spark.read.format(\"csv\").option(\"header\", \"true\").load(\"./Parking_Violations_Issued_-_Fiscal_Year_2016.csv\")\n",
    "val raw2017 = spark.read.format(\"csv\").option(\"header\", \"true\").load(\"./Parking_Violations_Issued_-_Fiscal_Year_2017.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "violationCodes2014 = [Violation Code: string]\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "100"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val violationCodes2014 = raw2014.select(raw2014(\"Violation Code\")).distinct\n",
    "violationCodes2014.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "raw2014rdd = MapPartitionsRDD[183] at rdd at <console>:30\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "MapPartitionsRDD[183] at rdd at <console>:30"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val raw2014rdd = violationCodes2014.rdd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Array(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "raw2014rdd.map(_(0).toString.toInt).collect().sorted"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Statistics\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Summons Number: string, Plate ID: string ... 41 more fields]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Apache Toree - Scala",
   "language": "scala",
   "name": "apache_toree_scala"
  },
  "language_info": {
   "codemirror_mode": "text/x-scala",
   "file_extension": ".scala",
   "mimetype": "text/x-scala",
   "name": "scala",
   "pygments_lexer": "scala",
   "version": "2.11.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}