Realtime_video_ipython.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Displaying webcam video in IPython notebook at (relatively) high framerate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When working on my project I realized, I can use OpenCV in Python to grap image from webcam as Numpy array, modify it and then display it using OpenCV's **cv2.imshow()**. OpenCV will create a window and push my frame there. However, this will not work in a IPython notebook. I found few solutions to implement the same functionality, but they all were slow (about 250 ms per frame).\n",
    "\n",
    "Here, I combine and modify these two examples to get achieve about 5 times higher framerate:\n",
    "1. __[Showing webcame image using OpenCV and matplotlib](https://gist.github.com/tibaes/35b9dbd7cbf81a98955067aa318290e7#file-video)__. Important moment here is that previous frame is cleared from screen using **IPython.display.clear_output()**\n",
    "2. __[Minimal code for rendering a numpy array as an image in a Jupyter notebook in memory](https://gist.github.com/kylemcdonald/2f1b9a255993bf9b2629)__. It uses PIL to convert NumPy array to .PNG format in order to display it with **IPython.display.display()**\n",
    "\n",
    "Both are relatively slow. The slowest step in the first one is the __[matplotlib.pyplot.imshow()](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.imshow.html)__ and the second one spend most of the time converting array data to PNG in __[PIL.Image.save()](https://pillow.readthedocs.io/en/3.1.x/reference/Image.html#PIL.Image.Image.save)__.\n",
    "\n",
    "But converting to PNG is not the fastest and only give me 2-3 FPS. If I use JPEG instead, framerate goes up to 36 FPS, which is not bad."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the required modules\n",
    "import cv2\n",
    "from IPython.display import clear_output\n",
    "import time\n",
    "import PIL.Image\n",
    "from cStringIO import StringIO\n",
    "import IPython.display\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Use 'jpeg' instead of 'png' (~5 times faster)\n",
    "def showarray(a, fmt='jpeg'):\n",
    "    f = StringIO()\n",
    "    PIL.Image.fromarray(a).save(f, fmt)\n",
    "    IPython.display.display(IPython.display.Image(data=f.getvalue()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_frame(cam):\n",
    "    # Capture frame-by-frame\n",
    "    ret, frame = cam.read()\n",
    "    \n",
    "    #flip image for natural viewing\n",
    "    frame = cv2.flip(frame, 1)\n",
    "    \n",
    "    return frame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Stream stopped\n"
     ]
    }
   ],
   "source": [
    "cam = cv2.VideoCapture(0)\n",
    "try:\n",
    "    while(True):\n",
    "        t1 = time.time()\n",
    "        \n",
    "        # Capture frame-by-frame\n",
    "        frame = get_frame(cam)\n",
    "\n",
    "        # Convert the image from OpenCV BGR format to matplotlib RGB format\n",
    "        # to display the image\n",
    "        frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n",
    "        showarray(frame)\n",
    "        t2 = time.time()\n",
    "\n",
    "        print(\"%f FPS\" % (1/(t2-t1)))\n",
    "        # Display the frame until new frame is available\n",
    "        clear_output(wait=True)\n",
    "except KeyboardInterrupt:\n",
    "    cam.release()\n",
    "    print \"Stream stopped\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}