pynvml-bokeh-demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visualizing GPU Resource Utilization with PyNVML and Bokeh\n",
    "\n",
    "- **Author:** Rick Zamora (email: rzamora@nvidia.com)\n",
    "- **Last Update:** 5/15/2019\n",
    "\n",
    "**Note:** *In the 5/15/2019 live demonstration, the \"GPU-Utilization\" and \"GPU-Resources\" applications were used as JupyterLab extensions. The integration of the PyNVML-Bokeh server was finished by [Ben Zaitlen](https://github.com/quasiben) just before the live demonsration (thanks Ben!). In the future, this notebook will be updated to reflect the JupyterLab integration.* \n",
    "\n",
    "### Introduction\n",
    "\n",
    "This notebook provides a brief demonstration of how to visualize GPU metrics using PyNVML and Bokeh. The demonstration has three general goals:\n",
    "\n",
    "1. Introduce/discuss the PyNVML python bindings for the [NVIDIA Management Library (NVML)](https://developer.nvidia.com/nvidia-management-library-nvml)\n",
    "2. Discuss a specific example of NVML-Bokeh integration for GPU-metric visualization\n",
    "3. Provide a simple benchmark (using Dask) to visualize multi-GPU resource utilization\n",
    "\n",
    "### Base Environment Setup\n",
    "\n",
    "In order to visualize GPU utilization for this demo, we start by creating a base conda environment with [RAPIDS](https://rapids.ai/) and [Jupyter](https://jupyter.org/) packages:\n",
    "```\n",
    "conda create --name bokeh-pynvml \\\n",
    "    -c defaults -c nvidia -c rapidsai \\\n",
    "    -c pytorch -c numba -c conda-forge \\\n",
    "    cudf=0.7 cuml=0.7 python=3.7 cudatoolkit=9.2 \\\n",
    "    nodejs jupyterlab dask dask-cudf dask-cuda bokeh -y\n",
    "conda activate bokeh-pynvml\n",
    "```\n",
    "\n",
    "Note that I am personally using a DGX machine with eight V100 NVIDIA GPUs to write this notebook (`Ubuntu 16.04.5 LTS (GNU/Linux 4.4.0-135-generic x86_64`).\n",
    "\n",
    "Before or after activating our base conda environment, we should also choose a specific root-directory location for this demo:\n",
    "```\n",
    "export demo_home='/home/nfs/rzamora/workspace/pynvml-bokeh-demo'\n",
    "mkdir $demo_home; cd $demo_home\n",
    "```\n",
    "\n",
    "### Python Bindings for the NVIDIA Management Library (PyNVML)\n",
    "\n",
    "PyNVML is a python wrapper for the [NVIDIA Management Library (NVML)](https://developer.nvidia.com/nvidia-management-library-nvml), which is a C-based API for monitoring and managing various states of NVIDIA GPU devices. NVML is directly used by the better-known [NVIDIA System Management Interface](https://developer.nvidia.com/nvidia-system-management-interface) (`nvidia-smi`). According to the NVIDA developer site, NVML provides access to the following query-able states (in additional to modifiable states not discussed here):\n",
    "\n",
    "- **ECC error counts**: Both correctable single bit and detectable double bit errors are reported. Error counts are provided for both the current boot cycle and for the lifetime of the GPU.\n",
    "- **GPU utilization**: Current utilization rates are reported for both the compute resources of the GPU and the memory interface.\n",
    "- **Active compute process**: The list of active processes running on the GPU is reported, along with the corresponding process name/id and allocated GPU memory.\n",
    "- **Clocks and PState**: Max and current clock rates are reported for several important clock domains, as well as the current GPU performance state.\n",
    "- **Temperature and fan speed**: The current core GPU temperature is reported, along with fan speeds for non-passive products.\n",
    "- **Power management**: For supported products, the current board power draw and power limits are reported.\n",
    "- **Identification**: Various dynamic and static information is reported, including board serial numbers, PCI device ids, VBIOS/Inforom version numbers and product names.\n",
    "\n",
    "Although several different python wrappers for NVML currently exist, we will use the [PyNVML](https://github.com/gpuopenanalytics/pynvml) package hosted by GoAi on GitHub. This version of PyNVML uses `ctypes` to wrap most of the NVML C API.  For this demo, we will focus on a small subset of the API needed to query real-time GPU-resource utilization:\n",
    "\n",
    "- `nvmlInit()`: Initialize an NVML profiling session\n",
    "- `nvmlShutdown()`: Finalize an NVML profiling session\n",
    "- `nvmlDeviceGetCount()`: Get the number of available GPU devices\n",
    "- `nvmlDeviceGetHandleByIndex()`: Get a handle for a device (given an integer index)\n",
    "- `nvmlDeviceGetMemoryInfo()`: Get a memory-info object (given a device handle)\n",
    "- `nvmlDeviceGetUtilizationRates()`: Get a utlization-rate object (given a device handle)\n",
    "- `nvmlDeviceGetPcieThroughput()`: Get a PCIe-trhoughput object (given a device handle)\n",
    "\n",
    "In the current version of PyNVML, the python function names are usually chosen to exactly match the C API. For example, to query the current GPU-utilization rate on every available device, the code would look something like this:\n",
    "\n",
    "```\n",
    "In [1]: from pynvml import *\n",
    "In [2]: nvmlInit()\n",
    "In [3]: ngpus = nvmlDeviceGetCount()\n",
    "In [4]: for i in range(ngpus):\n",
    "   ...:     handle = nvmlDeviceGetHandleByIndex(i)\n",
    "   ...:     gpu_util = nvmlDeviceGetUtilizationRates(handle).gpu\n",
    "   ...:     print('GPU %d Utilization = %d%%' % (i, gpu_util))\n",
    "   ...:\n",
    "GPU 0 Utilization = 43%\n",
    "GPU 1 Utilization = 0%\n",
    "GPU 2 Utilization = 15%\n",
    "GPU 3 Utilization = 0%\n",
    "GPU 4 Utilization = 36%\n",
    "GPU 5 Utilization = 0%\n",
    "GPU 6 Utilization = 0%\n",
    "GPU 7 Utilization = 11%\n",
    "```\n",
    "\n",
    "Of course, if there is nothing currently running on any of the GPUs, all devices will show 0% utilization. In this demo, we will use simple python code (like in the above example) to query GPU metrics in real time.  To install [PyNVML](https://github.com/gpuopenanalytics/pynvml) from source:\n",
    "```\n",
    "git clone https://github.com/gpuopenanalytics/pynvml.git\n",
    "cd pynvml\n",
    "pip install -e .\n",
    "```\n",
    "\n",
    "Note that this version of PyNVML is also hosted on [PyPI](https://pypi.org/project/pynvml/) and [Conda Forge](https://anaconda.org/conda-forge/pynvml), so you can alternatively use `pip install pynvml` or `conda install -c conda-forge pynvml` without cloning the repository.  For example, here is a screenshot of the PyPI page for the PyNVML package I am using:\n",
    "\n",
    "![alt text](https://raw.githubusercontent.com/rjzamora/notebooks/master/pynvml-bokeh-files/pypi-ss.png)\n",
    "\n",
    "\n",
    "### A PyNVML Bokeh-Server Example\n",
    "\n",
    "Although it is pretty cool that we can use python to query the current state of our NVIDIA GPUs, what we really want in practice is an intuitive visualization of the most important metrics.  In order for the visualization to *paint* a complete/useful picture of the system, the NVML data will need to be automatically updated in real time. \n",
    "\n",
    "The good news is that the `server` module within the [Bokeh](https://bokeh.pydata.org/en/latest/) python library provides the perfect solution for this task!  In fact, the process of building programmatic bokeh servers is already nicely outlined in a [great blog post by Matt Rocklin](http://matthewrocklin.com/blog/work/2017/06/28/simple-bokeh-server) (thanks Matt!). \n",
    "\n",
    "For this demo, we will use a fork of the [`jupyterlab-bokeh-server`](https://github.com/ian-r-rose/jupyterlab-bokeh-server) repository, developed by [Ian Rose](https://github.com/ian-r-rose) and [Matt Rocklin](https://github.com/mrocklin).  Within this fork, the `pyunvml` branch is based on the `system-resources` branch of the upstream repository.  The `system-resources` branch is a great reference, because it already includes code for visualizing CPU resource utilization (see the *Code Details* section for further implementation details).\n",
    "\n",
    "#### Downloading the Bokeh-Server Code\n",
    "\n",
    "To access the code for NVML-metric visualization, clone the `pynvml` branch of [`rjzamora/jupyterlab-bokeh-server`](https://github.com/rjzamora/jupyterlab-bokeh-server):\n",
    "\n",
    "```\n",
    "cd $demo_home\n",
    "git clone https://github.com/rjzamora/jupyterlab-bokeh-server.git -b pynvml\n",
    "```\n",
    "\n",
    "#### Running the PyNVML Bokeh Server\n",
    "\n",
    "Despite the existence of `jupyterlab` within the name of the repository used for this demo, the server is not yet integrated as a jupyterlab extension.  For now, we need to run the `jupyterlab_bokeh_server/server.py` script directly. For example:\n",
    "\n",
    "```\n",
    "python $demo_home/jupyterlab-bokeh-server/jupyterlab_bokeh_server/server.py 5000 > server.out 2>&1 &\n",
    "```\n",
    "\n",
    "After the bokeh server is launched, you can navigate to `http://<IP>:5000` in your web browser. If everything is working correctly, you will see the following menu page:\n",
    "\n",
    "![alt text](https://raw.githubusercontent.com/rjzamora/notebooks/master/pynvml-bokeh-files/bokeh-app-ss.png)\n",
    "\n",
    "##### GPU-Utilization Bar Plot\n",
    "\n",
    "If you click on the **GPU-Utilization** link listed in the main menu, you will see a bar-chart of the current GPU compute utilization (y-scale being 1-100%).  For the dask benchmark (discussed below), I saw the following output for a single snapshot (your snapshot might show more or less utilization):\n",
    "\n",
    "![alt text](https://raw.githubusercontent.com/rjzamora/notebooks/master/pynvml-bokeh-files/gpu-utilization-ss.png)\n",
    "\n",
    "##### GPU-Resources Stacked Line Plot\n",
    "\n",
    "If you click on the **GPU-Resources** link listed in the main menu, you will see a figure with four stacked line plots: \n",
    "\n",
    "- **GPU Utilization (per Device) [%]**: Plot of the GPU-**compute** utilization for each device. Each GPU is plotted with a different color, and the units are percentage.\n",
    "- **Memory Utilization (per Device)**: Plot of the GPU-**memory** utilization for each device. Each GPU is plotted with a different color, and the units are  GiB.\n",
    "- **Total Utilization [%]**: Plot of the **total** GPU **memory** and **compute** utilization. Units are percentage.\n",
    "- **Total PCI Throughput [MB/s]**: Plot of the **total** PCIe **TX** and **RX** data throughput. Units are MB/s.\n",
    "\n",
    "For example, when running the dask benchmark (discussed below), I see the following output:\n",
    "\n",
    "![alt text](https://raw.githubusercontent.com/rjzamora/notebooks/master/pynvml-bokeh-files/gpu-resources-ss.png)\n",
    "\n",
    "\n",
    "#### Code Details\n",
    "\n",
    "The pyNVML-specific code needed for this demo can be found in the `jupyterlab_bokeh_server/server.py` and `jupyterlab_bokeh_server/nvml_apps.py` files of my `jupyterlab-bokeh-server` fork. In `server.py`, the only significant change to the upstream repository is the addition of new `gpu`, `gpu_resource_timeline`, and `pci` bokeh applications (which are all defined in `nvml_apps.py`):\n",
    "\n",
    "```\n",
    "try:\n",
    "    import nvml_apps\n",
    "    routes = {\n",
    "        \"/CPU-Utilization\": cpu,\n",
    "        \"/Machine-Resources\": resource_timeline,\n",
    "        \"/GPU-Utilization\": nvml_apps.gpu,\n",
    "        \"/GPU-Resources\": nvml_apps.gpu_resource_timeline,\n",
    "        \"/PCI-Throughput\": nvml_apps.pci,\n",
    "    }\n",
    "```\n",
    "\n",
    "In order for the server to constantly refresh the pyNVML data used by the bokeh applications, we use bokeh's `ColumnDataSource` class to define the *source* of data in each of our plots. The `ColumnDataSource` class allows you to pass an `update` function for each type of data, which can be called within a dedicated callback function (`cb`) for each application.  For example, the `gpu` application is defined like this:\n",
    "\n",
    "```\n",
    "def gpu(doc):\n",
    "    fig = figure(title=\"GPU Usage\", sizing_mode=\"stretch_both\", y_range=[0, 100])\n",
    "\n",
    "    gpu = [ pynvml.nvmlDeviceGetUtilizationRates( gpu_handles[i] ).gpu for i in range(ngpus) ]\n",
    "    left = list(range(len(gpu)))\n",
    "    right = [l + 0.8 for l in left]\n",
    "    source = ColumnDataSource({\"left\": left, \"right\": right, \"gpu\": gpu})\n",
    "    mapper = LinearColorMapper(palette=all_palettes['RdYlBu'][4], low=0, high=100)\n",
    "\n",
    "    fig.quad(\n",
    "        source=source, left=\"left\", right=\"right\", bottom=0, top=\"gpu\", color={\"field\": \"gpu\", \"transform\": mapper}\n",
    "    )\n",
    "\n",
    "    doc.title = \"GPU Utilization [%]\"\n",
    "    doc.add_root(fig)\n",
    "\n",
    "    def cb():\n",
    "        source.data.update({\"gpu\": [ pynvml.nvmlDeviceGetUtilizationRates( gpu_handles[i] ).gpu for i in range(ngpus) ]})\n",
    "\n",
    "    doc.add_periodic_callback(cb, 200)\n",
    "```\n",
    "\n",
    "Note that the real-time update of PyNVML GPU-utilization data is performed within the `source.data.update()` call."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sample GPU Benchmark\n",
    "\n",
    "If you have followed this notebook so far, there is a decent chance that you saw some pretty boring plots for the GPU activity on your own system (unless you happened to be running a GPU-intensive application at the time). In case you don't have a decent GPU benchmark on hand, I am including a code snippent from the [join-indexed](https://github.com/mrocklin/dask-gpu-benchmarks/blob/master/join-indexed.ipynb) example from the [dask-gpu-benchmarks](https://github.com/mrocklin/dask-gpu-benchmarks) repository:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/nfs/rzamora/miniconda3/envs/bokeh-pynvml/lib/python3.7/site-packages/distributed/deploy/local.py:138: UserWarning: diagnostics_port has been deprecated. Please use `dashboard_address=` instead\n",
      "  \"diagnostics_port has been deprecated. \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 13.9 s, sys: 1.76 s, total: 15.6 s\n",
      "Wall time: 2min 14s\n"
     ]
    }
   ],
   "source": [
    "from dask.distributed import Client, wait\n",
    "from dask_cuda import LocalCUDACluster\n",
    "import dask\n",
    "import cudf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster = LocalCUDACluster(diagnostics_port=9000)\n",
    "client = Client(cluster)\n",
    "client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "left = dask.datasets.timeseries(\n",
    "    '2000', '2001', \n",
    "    dtypes={'id': int, 'x': float, 'y': float},\n",
    "    freq='10ms',\n",
    "    partition_freq='2d',\n",
    ")\n",
    "left.index = left.index.astype(int)\n",
    "left = left.persist()\n",
    "\n",
    "right = dask.datasets.timeseries(\n",
    "    '2000', '2001', \n",
    "    dtypes={'z': float},\n",
    "    freq='100ms',\n",
    "    partition_freq='5d',\n",
    ")\n",
    "right.index = right.index.astype(int)\n",
    "right = right.persist()\n",
    "\n",
    "gleft = left.map_partitions(cudf.from_pandas)\n",
    "gright = right.map_partitions(cudf.from_pandas)\n",
    "gleft, gright = dask.persist(gleft, gright)  # persist data in device memory\n",
    "\n",
    "out = gleft.merge(gright, left_index=True, right_index=True, how='inner')  # this is lazy\n",
    "out = out.persist()\n",
    "%time _ = wait(out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since this example is designed to use every available GPU device, it is a great fit for this demonstration.  If you happen to have other GPU benchmarks that also produce interesting PyNVML visualizations, please do share :)\n",
    "\n",
    "Thanks for reading!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}