pytorch-101.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pytorch 101 - Tensors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pytorch is a python package targeted for mainly two uses\n",
    "\n",
    "- A replacement for NumPy to use your GPU thus accelerating computing\n",
    "- Dee learning development platform"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first concept we need to make sure we understand is the concept of a tensor.  \n",
    "\n",
    "If you are familiar with NumPy then you are a lucky cookie.  \n",
    "Tensors are similar to NumPy's ndarrays with the brilliant characteristic of being capable to use GPU to accelerate computing. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Lets import\n",
    "import torch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can create: \n",
    "- Empty tensors\n",
    "- Random\n",
    "- Filled with 0's\n",
    "- Filled with 1's\n",
    "- Directly from data\n",
    "- From another tensor\n",
    "\n",
    "And some other tricks that we'll cover later"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 5.9294e+00,  4.5808e-41,  5.9294e+00],\n",
      "        [ 4.5808e-41,  4.4842e-44,  0.0000e+00],\n",
      "        [ 8.9683e-44,  0.0000e+00, -1.1312e-36],\n",
      "        [ 3.0915e-41,  2.0319e-43,  0.0000e+00],\n",
      "        [-1.1344e-36,  3.0915e-41,  5.9293e+00]])\n"
     ]
    }
   ],
   "source": [
    "# Empty\n",
    "x = torch.empty(5,3)\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0.0549, 0.3294, 0.1466],\n",
      "        [0.4971, 0.2633, 0.1515],\n",
      "        [0.4355, 0.9915, 0.0374],\n",
      "        [0.8308, 0.4187, 0.0894],\n",
      "        [0.2701, 0.6461, 0.8348]])\n"
     ]
    }
   ],
   "source": [
    "# Random\n",
    "x = torch.rand(5,3)\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0, 0, 0],\n",
      "        [0, 0, 0],\n",
      "        [0, 0, 0],\n",
      "        [0, 0, 0],\n",
      "        [0, 0, 0]])\n"
     ]
    }
   ],
   "source": [
    "# 0's but type long\n",
    "x = torch.zeros(5, 3, dtype=torch.long)\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1, 1, 1],\n",
      "        [1, 1, 1],\n",
      "        [1, 1, 1],\n",
      "        [1, 1, 1],\n",
      "        [1, 1, 1]], dtype=torch.int32)\n"
     ]
    }
   ],
   "source": [
    "# 1's but type int\n",
    "x = torch.ones(5, 3, dtype=torch.int)\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([5.5000, 3.0000])\n"
     ]
    }
   ],
   "source": [
    "# Directly from data\n",
    "x = torch.tensor([5.5, 3])\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1., 1., 1.],\n",
      "        [1., 1., 1.],\n",
      "        [1., 1., 1.],\n",
      "        [1., 1., 1.],\n",
      "        [1., 1., 1.]], dtype=torch.float64)\n",
      "tensor([[-0.8837, -1.9384,  0.0824],\n",
      "        [ 0.5409, -1.2753,  1.2485],\n",
      "        [ 1.0305, -2.1184,  0.5200],\n",
      "        [-0.9521,  0.8252,  1.6442],\n",
      "        [ 0.2127,  1.5367, -1.3651]])\n",
      "torch.Size([5, 3])\n"
     ]
    }
   ],
   "source": [
    "# We can also create tensors from other tensors and change the dtype on the new ones\n",
    "x = x.new_ones(5, 3, dtype=torch.double)\n",
    "print(x)\n",
    "x = torch.randn_like(x, dtype=torch.float)\n",
    "print(x)\n",
    "\n",
    "# And of course we can get the size of a sensor\n",
    "print(x.size())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Operations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's obviously imposible to cover all operations but for the sake of giving some sort of example let keep going ;)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1., 1., 1.],\n",
      "        [1., 1., 1.],\n",
      "        [1., 1., 1.],\n",
      "        [1., 1., 1.],\n",
      "        [1., 1., 1.]])\n",
      "tensor([[0.4207, 0.9760, 0.3910],\n",
      "        [0.2187, 0.1812, 0.1678],\n",
      "        [0.1113, 0.9096, 0.2253],\n",
      "        [0.8655, 0.1390, 0.5084],\n",
      "        [0.2376, 0.6938, 0.3795]])\n",
      "tensor([[1.4207, 1.9760, 1.3910],\n",
      "        [1.2187, 1.1812, 1.1678],\n",
      "        [1.1113, 1.9096, 1.2253],\n",
      "        [1.8655, 1.1390, 1.5084],\n",
      "        [1.2376, 1.6938, 1.3795]])\n"
     ]
    }
   ],
   "source": [
    "# Addition using \"+\" operator\n",
    "x = torch.ones(5, 3)\n",
    "y = torch.rand(5, 3)\n",
    "\n",
    "print(x)\n",
    "print(y)\n",
    "print(x + y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1.3145, 1.4350, 1.0026],\n",
      "        [1.3657, 1.7708, 1.2255],\n",
      "        [1.5436, 1.6044, 1.2447],\n",
      "        [1.1253, 1.5685, 1.0296],\n",
      "        [1.9169, 1.5857, 1.6694]])\n"
     ]
    }
   ],
   "source": [
    "# Or use a function\n",
    "print(torch.add(x, y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1.3145, 1.4350, 1.0026],\n",
      "        [1.3657, 1.7708, 1.2255],\n",
      "        [1.5436, 1.6044, 1.2447],\n",
      "        [1.1253, 1.5685, 1.0296],\n",
      "        [1.9169, 1.5857, 1.6694]])\n"
     ]
    }
   ],
   "source": [
    "# Or output to a tensor\n",
    "result = torch.empty(5, 3)\n",
    "torch.add(x, y, out=result)\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1.4207, 1.9760, 1.3910],\n",
      "        [1.2187, 1.1812, 1.1678],\n",
      "        [1.1113, 1.9096, 1.2253],\n",
      "        [1.8655, 1.1390, 1.5084],\n",
      "        [1.2376, 1.6938, 1.3795]])\n"
     ]
    }
   ],
   "source": [
    "# Or in-place\n",
    "y.add_(x)\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Indexing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All indexing in a tensor works like NumPy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([1., 1., 1., 1., 1.])\n"
     ]
    }
   ],
   "source": [
    "print(x[:, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Resizing/Reshape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To reshape/resize tensors the function to use is `.view()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([4, 4]) torch.Size([16]) torch.Size([2, 8])\n"
     ]
    }
   ],
   "source": [
    "x = torch.randn(4, 4)\n",
    "y = x.view(16)\n",
    "z = x.view(-1, 8)  # the size -1 is inferred from other dimensions\n",
    "print(x.size(), y.size(), z.size())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For one element tensors the best way to obtain such is by the `.item()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([-1.2545])\n",
      "-1.2545496225357056\n"
     ]
    }
   ],
   "source": [
    "x = torch.randn(1)\n",
    "print(x)\n",
    "print(x.item())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Numpy bridge"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember when I said tensors are NumPy ndarrays were very similar?  \n",
    "Well, converting between these two is super easy here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tensor to NumPy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([1., 1., 1., 1., 1.])\n"
     ]
    }
   ],
   "source": [
    "a = torch.ones(5)\n",
    "print(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1. 1. 1. 1. 1.]\n"
     ]
    }
   ],
   "source": [
    "b = a.numpy()\n",
    "print(b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NumPy and Tensors will share the same space in memory as long as the tensors are located on 'CPU' (more on this later)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([3., 3., 3., 3., 3.])\n",
      "[3. 3. 3. 3. 3.]\n"
     ]
    }
   ],
   "source": [
    "# Addition in place\n",
    "a.add_(1)\n",
    "print(a)\n",
    "print(b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NumPy to Tensor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Same way, when converting an `ndarray` to `Tensor it will use CPU memory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2. 2. 2. 2. 2.]\n",
      "tensor([2., 2., 2., 2., 2.], dtype=torch.float64)\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "a = np.ones(5)\n",
    "b = torch.from_numpy(a)\n",
    "np.add(a, 1, out=a)\n",
    "print(a)\n",
    "print(b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CUDA Tensors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, the moment you've all been waiting for *drum rolls**  \n",
    "  \n",
    "  We've talked so far on how ndaarrays and Tensors are similar and interchangable and the question why do we need tensors should arise.  \n",
    "  \n",
    "  Well, because we can move Tensors to GPU memory, how do we do this? With the `.to()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let us run this cell only if CUDA is available\n",
    "# We will use ``torch.device`` objects to move tensors in and out of GPU\n",
    "if torch.cuda.is_available():\n",
    "    device = torch.device(\"cuda\")          # a CUDA device object\n",
    "    y = torch.ones_like(x, device=device)  # directly create a tensor on GPU\n",
    "    x = x.to(device)                       # or just use strings ``.to(\"cuda\")``\n",
    "    z = x + y\n",
    "    print(z)\n",
    "    print(z.to(\"cpu\", torch.double))       # ``.to`` can also change dtype together!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTE:** The previous code will not run unless you have properly configurated your GPU and CUDA.  \n",
    "\n",
    "This is a whole different beast and it's interesting in its own but it will not be covered in here for now."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}