1_OpenCV_Jupyter_webcam.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lesson 1. Displaying webcam video in IPython notebook at (relatively) high framerate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When working on my project I realized, I can use OpenCV in Python to grap image from webcam as Numpy array, modify it and then display it using OpenCV's **cv2.imshow()**. OpenCV will create a window and push my frame there. However, this will not work in a IPython notebook. I found few solutions to implement the same functionality, but they all were slow (about 250 ms per frame).\n",
    "\n",
    "Here, I combine and modify these two examples to get achieve about 5 times higher framerate:\n",
    "1. __[Showing webcame image using OpenCV and matplotlib](https://gist.github.com/tibaes/35b9dbd7cbf81a98955067aa318290e7#file-video)__. Important moment here is that previous frame is cleared from screen using **IPython.display.clear_output()**\n",
    "2. __[Minimal code for rendering a numpy array as an image in a Jupyter notebook in memory](https://gist.github.com/kylemcdonald/2f1b9a255993bf9b2629)__. It uses PIL to convert NumPy array to .PNG format in order to display it with **IPython.display.display()**\n",
    "\n",
    "Both are relatively slow. The slowest step in the first one is the __[matplotlib.pyplot.imshow()](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.imshow.html)__ and the second one spend most of the time converting array data to PNG in __[PIL.Image.save()](https://pillow.readthedocs.io/en/3.1.x/reference/Image.html#PIL.Image.Image.save)__.\n",
    "\n",
    "But converting to PNG is not the fastest and only give me 2-3 FPS. If I use JPEG instead, framerate goes up to 36 FPS, which is not bad."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the required modules\n",
    "import cv2\n",
    "from IPython.display import clear_output\n",
    "import time\n",
    "import PIL.Image\n",
    "from io import BytesIO\n",
    "import IPython.display\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Use 'jpeg' instead of 'png' (~5 times faster)\n",
    "def array_to_image(a, fmt='jpeg'):\n",
    "    #Create binary stream object\n",
    "    f = BytesIO()\n",
    "    \n",
    "    #Convert array to binary stream object\n",
    "    PIL.Image.fromarray(a).save(f, fmt)\n",
    "    \n",
    "    return IPython.display.Image(data=f.getvalue())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_frame(cam):\n",
    "    # Capture frame-by-frame\n",
    "    ret, frame = cam.read()\n",
    "    \n",
    "    #flip image for natural viewing\n",
    "    frame = cv2.flip(frame, 1)\n",
    "    \n",
    "    return frame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cam = cv2.VideoCapture(0)\n",
    "\n",
    "d = IPython.display.display(\"\", display_id=1)\n",
    "d2 = IPython.display.display(\"\", display_id=2)\n",
    "while True:\n",
    "    try:\n",
    "        t1 = time.time()\n",
    "        frame = get_frame(cam)\n",
    "        frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n",
    "        \n",
    "        im = array_to_image(frame)\n",
    "        \n",
    "        d.update(im)\n",
    "        \n",
    "        t2 = time.time()\n",
    "        \n",
    "        s = f\"\"\"{int(1/(t2-t1))} FPS\"\"\"\n",
    "        d2.update( IPython.display.HTML(s) )\n",
    "    except KeyboardInterrupt:\n",
    "        print()\n",
    "        cam.release()\n",
    "        IPython.display.clear_output()\n",
    "        print (\"Stream stopped\")\n",
    "        break"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}