carla-simulator.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "carla-simulator.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gpY-9ehVv70N"
      },
      "source": [
        "# Running CARLA simulator\n",
        "This tutorial shows how to run and visualize CARLA on colab.\n",
        "\n",
        "\n",
        "\n",
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/MichaelBosello/carla-colab/blob/master/carla-simulator.ipynb\">\n",
        "    <img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />\n",
        "    Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/MichaelBosello/carla-colab\">\n",
        "    <img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />\n",
        "    View source on GitHub</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_HATkoTPmgg5"
      },
      "source": [
        "<img src=\"https://raw.githubusercontent.com/MichaelBosello/carla-colab/master/img/screen.png\" width=\"750\">"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tsln_9Oc4rUR"
      },
      "source": [
        "---\n",
        "---\n",
        "---\n",
        "\n",
        "## Introduction\n",
        "[Carla](https://github.com/carla-simulator/carla) is an open-source simulator for autonomous driving research.\n",
        "\n",
        "Since it is hardware demanding, it could be useful for students to run it on colab.\n",
        "\n",
        "---\n",
        "\n",
        "## Acknowledgements\n",
        "\n",
        "Thanks to *Tomohiro*  for developing the code that allows remote access to colab.\n",
        "Check out his [page](https://internet-of-tomohiro.netlify.app/google_colab/vnc.en.html) and [GitHub](https://github.com/demotomohiro/remocolab).\n",
        "\n",
        "---\n",
        "\n",
        "## Update\n",
        "\n",
        "Ngrok doesn't work anymore with Google Colab (see [this issue](https://github.com/demotomohiro/remocolab/issues/79)). The tutorial has been updated to use Argo tunnel instead.\n",
        "\n",
        "---\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5jsE5t9G6nY4"
      },
      "source": [
        "## Requirements\n",
        "\n",
        "\n",
        "\n",
        "*   ~~[Ngrok](https://ngrok.com/) account (you don't need to buy premium plans).~~\n",
        "*   [Install cloudflared](https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/install-and-setup/installation) on your PC.\n",
        "*   TurboVNC Viewer. [Download here](https://sourceforge.net/projects/turbovnc/). (RealVNC and Microsoft Remote Desktop __don't__ work).\n",
        "*   Ssh client (already present in all recent os).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "__kiSsQE-R7v"
      },
      "source": [
        "## Change the runtime\n",
        "In colab: click \"Runtime\" in the top bar -> \"Change runtime type\" -> change hardware accelerator to GPU."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-YdPX4a3_XFg"
      },
      "source": [
        "## Installation & setup of remote access \n",
        "\n",
        "\n",
        "\n",
        "1.   Run the snippet below\n",
        "2.   ~~It will ask for your ngrok authtoken.~~\n",
        "\n",
        "  *   ~~login to ngrok -> from the left menu: Authentication>Your Authroken -> copy~~\n",
        "  *   ~~Enter it~~\n",
        "\n",
        "3. ~~Select ngrok region (the region closest to you should be the fastest)~~\n",
        "4. Wait the end of the process\n",
        "  * It sets up OpenSSH server, VirtualGL and TurboVNC.\n",
        "  * Then, access passwords are shown\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w4Ywwr43AGR9"
      },
      "source": [
        "!pip install git+https://github.com/demotomohiro/remocolab.git\n",
        "import remocolab\n",
        "remocolab.setupVNC()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zbMpTEeFG_lO"
      },
      "source": [
        "## Connect TurboVNC\n",
        "\n",
        "1. Open a ssh connection to colab (It is necessary to secure VNC connection)\n",
        "  * Execute the ssh command showed above\n",
        "  * Use the \"colab password\" displayed above\n",
        "2. Run TurboVNC Viewer\n",
        "  * The connection address is `localhost:1`\n",
        "  * Use the \"VNC password\" displayed above\n",
        "\n",
        "\n",
        " if everything worked properly, you will see a screen."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d3OWsY6jNQqJ"
      },
      "source": [
        "## Download CARLA\n",
        "\n",
        "If you need another [version](https://github.com/carla-simulator/carla/blob/master/Docs/download.md) of CARLA, change the link below. You must choose a Linux version.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rI7R3brAJgyQ"
      },
      "source": [
        "! sudo -u colab wget -O /home/colab/CARLA.tar.gz https://carla-releases.s3.eu-west-3.amazonaws.com/Linux/CARLA_0.9.10.tar.gz"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AozeZpibPB6u"
      },
      "source": [
        "## Extract the zip"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rfRiKysEL3eR"
      },
      "source": [
        "! sudo -u colab mkdir /home/colab/carla\n",
        "! sudo -u colab tar -xf /home/colab/CARLA.tar.gz -C /home/colab/carla"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BFrH9p95pjg2"
      },
      "source": [
        "## Setup PythonAPI\n",
        "\n",
        "The PythonAPI in the zip doesn't work with the current Colab Python version. If you change the Carla version, if you install another Python version, or if Google updates Colab, then you may not need to execute the following commands (or you may need to find another egg file accordingly)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "25tiIe4WsXpt"
      },
      "source": [
        "! sudo -u colab rm -rf /home/colab/carla/PythonAPI/carla/dist/*.egg"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FA5rNhxpp0zw"
      },
      "source": [
        "! sudo -u colab wget https://github.com/MichaelBosello/carla-colab/raw/master/carla-0.9.10-py3.6-linux-x86_64.egg -P /home/colab/carla/PythonAPI/carla/dist/"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m7kFVqEfT59y"
      },
      "source": [
        "## Run CARLA"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wok54OiA7E91"
      },
      "source": [
        "! sudo -u colab DISPLAY=:1 vglrun /home/colab/carla/CarlaUE4.sh"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "61zYsUrmQ5T7"
      },
      "source": [
        "# Run client\n",
        "\n",
        "**You can run the client on colab or on your remote pc**\n",
        "\n",
        "---\n",
        "\n",
        "## Client on colab\n",
        "Run the following commands in a terminal on the remote desktop\n",
        "\n",
        "### Install requirement\n",
        "\n",
        "1. `sudo pip3 install pygame`\n",
        "\n",
        "  * The password for colab user is shown above\n",
        "\n",
        "### Run the 'manual control' example\n",
        "\n",
        "2. `cd /home/colab/carla/PythonAPI/examples`\n",
        "\n",
        "3. `python3 manual_control.py`\n",
        "\n",
        "---\n",
        "\n",
        "## Client on your pc\n",
        "\n",
        "Perform the following steps in your pc\n",
        "\n",
        "### Install requirements\n",
        "\n",
        "1. [install Carla](https://carla.readthedocs.io/en/latest/start_quickstart/#a-debian-carla-installation) (use the same version of the server)\n",
        "\n",
        "2. `pip3 install pygame`\n",
        "\n",
        "### Port forwarding\n",
        "\n",
        "You have to forward two ports in addition to the one used for VNC\n",
        "\n",
        "Copy the ssh command used for the VNC connection and run it with the port changed (change 5901 which is the VNC port). You have to run it two times (in two additional terminals) with those two ports: 2000 and 2001.\n",
        "\n",
        "**For example (note that you have to change YOUR_ISTANCE_PORT to the port specified in your command):**\n",
        "\n",
        "`ssh -o UserKnownHostsFile=/dev/null -o VisualHostKey=yes -p YOUR_ISTANCE_PORT -L 2000:localhost:2000 colab@2.tcp.eu.ngrok.io\n",
        "`\n",
        "\n",
        "`ssh -o UserKnownHostsFile=/dev/null -o VisualHostKey=yes -p YOUR_ISTANCE_PORT -L 2001:localhost:2001 colab@2.tcp.eu.ngrok.io\n",
        "`\n",
        "\n",
        "### Run the 'manual control' example\n",
        "\n",
        "1. `cd /opt/carla-simulator/PythonAPI/examples`\n",
        "\n",
        "2. `python3 manual_control.py`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qmpPO81jUEh1"
      },
      "source": [
        "---\n",
        "---\n",
        "---\n",
        "\n",
        "# Extra\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "## Display smooth animation\n",
        "Open TurboVNC Viewer options and change \"Encoding method\" to \"Tight + Medium-Quality JPEG\" or \"Tight + Low-Quality JPEG(WAN)\". Some noises could appear but it will look smoother.\n",
        "\n",
        "---\n",
        "## Tips\n",
        "\n",
        "### Keeping the virtual machine alive\n",
        "You may want to use [this Chrome extension](https://chrome.google.com/webstore/detail/colab-alive/eookkckfbbgnhdgcbfbicoahejkdoele).\n",
        "\n",
        "### How to transfer code\n",
        "Options:\n",
        "* Develop on your pc and use git to trasfer the code\n",
        "* Develop on your pc and mount your Google Drive from colab (see snippet below)\n",
        "* Install a code editor in the remote desktop (remember to save your code often on drive)\n",
        "\n",
        "### Training\n",
        "\n",
        "Save the checkpoints of your model on your Google Drive (see snippet below)\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qz4MypzAvltW"
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9EoV3sC1vr37"
      },
      "source": [
        "\n",
        "\n",
        "---\n",
        "\n",
        "## Notes\n",
        "\n",
        "\n",
        "\n",
        "### CARLA command\n",
        "\n",
        "The command to run the carla server is:\n",
        "\n",
        "`DISPLAY=:1 vglrun /home/colab/carla/CarlaUE4.sh`\n",
        "\n",
        "If you want to run the CARLA server in Python, you must use `subprocess.Popen` with the above command\n",
        "\n",
        "### Users and terminals\n",
        "\n",
        "The colab web-page runs commands as root. To execute them as a normal user I used `sudo -u colab`. This is necessary beacause __Unreal Engine can't be run as root__.\n",
        "\n",
        "If you prefer, you can use the same commands in the remote Desktop or ssh terminal without `sudo -u colab`.\n",
        "\n",
        "Python clients works only from the remote desktop or ssh client, and not from colab web-page. \n",
        "\n",
        "### Check OpenGL\n",
        "\n",
        "if you want to check the openGL implementation you can run in the remote desktop `vglrun /opt/VirtualGL/bin/glxinfo | grep NVIDIA`\n",
        "\n",
        "You should see something like:\n",
        "\n",
        "```\n",
        "OpenGL vendor string: NVIDIA Corporation\n",
        "OpenGL core profile version string: 4.6.0 NVIDIA 418.67\n",
        "OpenGL core profile shading language version string: 4.60 NVIDIA\n",
        "OpenGL version string: 4.6.0 NVIDIA 418.67\n",
        "OpenGL shading language version string: 4.60 NVIDIA\n",
        "```\n",
        "\n",
        "If you get errors, it could be the VNC viewer (again, use TurboVNC).\n",
        "\n",
        "### Closing Carla\n",
        "Close CARLA always from the remote desktop, if you interrupt the execution from the colab web-page you will interrupt also the VNC server."
      ]
    }
  ]
}