2-LAB_2_Taxi.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "LAB-2-Taxi.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/anadiedrichs/rl-ai/blob/master/2-LAB_2_Taxi.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HnL9hNPERI6_",
        "colab_type": "text"
      },
      "source": [
        "# Laboratorio #2 de Aprendizaje por refuerzo - Inteligencia Artificial - DISI 2019\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zuIiY5q7RWOG",
        "colab_type": "text"
      },
      "source": [
        "**Realizado por: Ing. Ana Laura Diedrichs**\n",
        "\n",
        "Consultas: lunes 19 hs en sala consulta de sistemas.\n",
        "\n",
        "Contacto por otros horarios de consulta o dudas: \n",
        "* Email ana.diedrichs@frm.utn.edu.ar\n",
        "* Telegram  @anadiedrichs\n",
        "* en el grupo de la materia por dudas generales\n",
        "\n",
        "## Objetivos del laboratorio:\n",
        "\n",
        "* Mediante un problema de juguete evaluar las diferencias entre el comportamiento aleatorio de un agente y su comportamiento con aprendizaje por refuerzo\n",
        "* Comprensión del funcionamiento de Q-learning\n",
        "* Evaluación de hiperparámetros alfa, gamma y epsilon en Q-learning.\n",
        "* Ser una entrada en calor o training del uso del entorno colab.research.google.com\n",
        "\n",
        "## Pre-requisitos o pre-condiciones\n",
        "* Tener una cuenta google (gmail)\n",
        "* Tener instalado el navegador google chrome \n",
        "* Contar con conectividad a internet \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KUYmBoiufH2c",
        "colab_type": "text"
      },
      "source": [
        "# Entrega y uso de este laboratorio"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nP0mcOHhfL3o",
        "colab_type": "text"
      },
      "source": [
        "**USO**\n",
        "\n",
        "* Antes que cualquier cosa,  cree una copia de este notebook: Click en *File*, luego *Save a Copy in Drive*\n",
        "* Renombre el archivo con el siguiente formato: APELLIDO_LEGAJO_titulonotebook.ipynb \n",
        "Ejemplo: DIEDRICHS_99999_Cab-taxi-problem.ipynb\n",
        "* Use el notebook, complete las actividades del final. \n",
        "* Este laboratorio es una actividad individual.\n",
        "\n",
        "\n",
        "**ENTREGA**\n",
        "\n",
        "* Una vez finalizado el laboratorio, complete [el formulario de entrega](https://forms.gle/AqdeVPA38chsJqR99)  indicando:\n",
        " * Apellido\n",
        " * Nombre\n",
        " * Nro Legajo\n",
        " * Turno (tarde o noche)\n",
        " * link de su notebook. El mismo se obtiene si realiza click en *Share* (esquina superior derecha) y luego en *Get shareable link* \n",
        " \n",
        " \n",
        " No se aceptarán otras formas de entrega distintas a la mencionada.\n",
        " \n",
        " \n",
        " Fecha límite de entrega: se indicará en clase o por la lista de correos. Última semana de clases es la entrega."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tznv9desR3Va",
        "colab_type": "text"
      },
      "source": [
        "# Introducción\n",
        "\n",
        "Vamos a trabajar utilizando este notebook y una librería llamada OpenAI (https://gym.openai.com/) y realizaremos simulaciones de aprendizaje por refuerzo.\n",
        "\n",
        "Ejecute los siguientes dos bloques de código (click en la flechita a la izquierda del bloque de código).\n",
        "\n",
        "Primero instalamos las librerías necesarias necesarias."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0sALMHPwGESj",
        "colab_type": "code",
        "outputId": "49c26566-a029-4c26-d863-a92bce8b0af6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 241
        }
      },
      "source": [
        "!pip install cmake 'gym' scipy"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: cmake in /usr/local/lib/python3.6/dist-packages (3.12.0)\n",
            "Requirement already satisfied: gym in /usr/local/lib/python3.6/dist-packages (0.10.11)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (1.3.0)\n",
            "Requirement already satisfied: requests>=2.0 in /usr/local/lib/python3.6/dist-packages (from gym) (2.21.0)\n",
            "Requirement already satisfied: numpy>=1.10.4 in /usr/local/lib/python3.6/dist-packages (from gym) (1.16.4)\n",
            "Requirement already satisfied: pyglet>=1.2.0 in /usr/local/lib/python3.6/dist-packages (from gym) (1.3.2)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from gym) (1.12.0)\n",
            "Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests>=2.0->gym) (1.24.3)\n",
            "Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests>=2.0->gym) (2.8)\n",
            "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests>=2.0->gym) (3.0.4)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests>=2.0->gym) (2019.3.9)\n",
            "Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from pyglet>=1.2.0->gym) (0.16.0)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lkTmxV76OQw6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# este es un comentario en el código\n",
        "import gym # importa librería gym\n",
        "import pandas as pd # carga la librería pandas y la llamaremos como pd\n",
        "import matplotlib.pyplot as plt # carga la librería pyplot y la llamaremos de forma abreviada como plt\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Lj5OREewGX-g",
        "colab_type": "text"
      },
      "source": [
        "## Descripción del ambiente\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v5H-bEnqSUNT",
        "colab_type": "text"
      },
      "source": [
        "El siguiente bloque de código crea el ambiente **Taxi**, del inglés *environment* y su abreviatura *env*.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Fz1iwEzCGFBw",
        "colab_type": "code",
        "outputId": "6f1915a3-f997-402a-da1d-91542701cc8e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 153
        }
      },
      "source": [
        "env = gym.make(\"Taxi-v2\").env # crea / instancia ambiente llamado Taxi-v2\n",
        "\n",
        "env.render() # nos muestra una imagen de como luce el ambiente."
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "+---------+\n",
            "|R: | : :\u001b[35mG\u001b[0m|\n",
            "| : : : : |\n",
            "| : : : : |\n",
            "| | : | : |\n",
            "|\u001b[34;1mY\u001b[0m|\u001b[43m \u001b[0m: |B: |\n",
            "+---------+\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8nR8P6uXDkcI",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "* El trabajo del agente es recoger un pasajero de una ubicación y llevarlo a un destino. \n",
        "* Se reciben +20 puntos por llevarlo al destino indicado y se pierte un punto por cada paso realizado (*time-step*)\n",
        "* Se penaliza con 10 puntos (-10) por intentar recoger un pasajero o dejarlo en un destino que no era el indicado.\n",
        "* El cuadradito relleno de amarillo representa el taxi cuando no lleva pasajeros y se pone verde cuando lleva un pasajero.\n",
        "* El *pipe* (símbolo | ) representa una pared o muro, la misma no puede ser atravesada por el taxi.\n",
        "* Hay cuatro (4) ubicaciones idenficadas con distintas letras. \n",
        "\n",
        "R, G, Y, B  son posibles destino de recogida y llegada/destino de pasajeros. \n",
        "\n",
        "La letra en azul representa una ubicación actual para recoger un pasajero, y la letra lila o púrpura representa el destino del pasajero.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IB0DZn3xGd6p",
        "colab_type": "text"
      },
      "source": [
        "Veamos que otras cosas podemos hacer con *environment*"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VEwfMWJiGPHd",
        "colab_type": "code",
        "outputId": "61b448f7-6eb6-4e3b-c54a-5d3cc4062870",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 187
        }
      },
      "source": [
        "env.reset() # me \"resetea\" el ambiente a uno nuevo, uno aleatorio\n",
        "env.render() # muestra la imagen del ambiente\n",
        "\n",
        "print(\"Action Space {}\".format(env.action_space)) # cuántas acciones puede tomar el agente\n",
        "print(\"State Space {}\".format(env.observation_space)) # cuántos espacios o estados s puede tomar el agente\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "+---------+\n",
            "|R: | : :\u001b[35mG\u001b[0m|\n",
            "| : : : : |\n",
            "| : : : : |\n",
            "| | : | : |\n",
            "|Y|\u001b[43m \u001b[0m: |\u001b[34;1mB\u001b[0m: |\n",
            "+---------+\n",
            "\n",
            "Action Space Discrete(6)\n",
            "State Space Discrete(500)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tH17yrOAGzJt",
        "colab_type": "text"
      },
      "source": [
        "Como nos muestra la salida del *print* tenemos un espacio de acción de tamaño 6 (seis) y un espacio de estados de tamaño 500. \n",
        "\n",
        "Los 500 estados posibles corresponden a la codificación que incluye: ubicación del taxi, ubicación del pasajero y ubicación destino del pasajero. (ver actividad nro 2))\n",
        "\n",
        "Nuestro algoritmo de aprendizaje por refuerzo (RL) no necesita más que estas dos cosas: el conjunto de acciones posibles y el conjunto de estados posibles. \n",
        "\n",
        "Necesitamos identificar cada estado de forma unívoca, asignándole un número único al estado (un ID) y que el algortimo RL elija algunas de las acciones posibles a tomar, que están etiquetadas desde el 0 cero al 5 cinco, donde:\n",
        "\n",
        "\n",
        "* 0 = sur\n",
        "* 1 = nore\n",
        "* 2 = este\n",
        "* 3 = oeste\n",
        "* 4 = *pickup* o recoger pasajero\n",
        "* 5 = *dropoff* o dejar en su destino al pasajero\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FhkcKjSdAPM-",
        "colab_type": "text"
      },
      "source": [
        "En cada unidad de tiempo discreta de nuestro simulador, según una acción ejecutada sobre el ambiente, el mismo regresa los siguientes elementos:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "b6JrgiRbAKvQ",
        "colab_type": "code",
        "outputId": "6b74c3d9-8f28-4410-b2fd-57ae24a5a5e1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "env.step(3)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(433, -1, False, {'prob': 1.0})"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GjKeUoTZAdyZ",
        "colab_type": "text"
      },
      "source": [
        "* **Observación**  (objecto): el estado del ambiente o un objeto específico del ambiente que representa la observación del mismo.\n",
        "* **Recompensa** o Reward (float): es la recompensa alcanzada por la acción previa.\n",
        "\n",
        "* **Done** (boolean): si es momento de \"resetear el ambiente\". La mayoría de las tareas en varios ambientes si el valor done = True indica que el episodio ha terminado (perdiste una vida en un video juego, por ejemplo) o se alcanzó un objetivo (dejamos pasajero en su destino, para este ejemplo).\n",
        "* **Info** (dict): podemos ignorar este atributo, brinda información de diagnóstico para debugging. No hay que usar nada de esto para aprendizaje/entrenamiento.\n",
        "\n",
        "#### [ACTIVIDAD] Detalle cuáles son los valores de recompensa para este problema\n",
        "\n",
        "Vea la [info del entorno](https://gym.openai.com/envs/Taxi-v2/)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VCia6KknK-6p",
        "colab_type": "text"
      },
      "source": [
        "RESPUESTA\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UZq-Lh8OLDJG",
        "colab_type": "text"
      },
      "source": [
        "Veamos como está codificado el estado y el ambiente del Taxi."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tNFzJtaxHfS9",
        "colab_type": "code",
        "outputId": "a86cda7d-2874-4062-8e88-82328d67b28a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 170
        }
      },
      "source": [
        "state = env.encode(3, 1, 1, 0) # (fila de posición del taxi, columna de posición del taxi, índice del pasajero, índice del destino)\n",
        "print(\"State:\", state)\n",
        "\n",
        "env.s = state\n",
        "env.render()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "State: 324\n",
            "+---------+\n",
            "|\u001b[35mR\u001b[0m: | : :\u001b[34;1mG\u001b[0m|\n",
            "| : : : : |\n",
            "| : : : : |\n",
            "| |\u001b[43m \u001b[0m: | : |\n",
            "|Y| : |B: |\n",
            "+---------+\n",
            "  (West)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tbwlNWkAHulC",
        "colab_type": "text"
      },
      "source": [
        "#### **[ACTIVIDAD]** Modifique la primer línea del siguiente script para ver dónde es la posición (3,4) del mapa y la (0,0)\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oxVAWT4BH4VH",
        "colab_type": "code",
        "outputId": "ca3ce9bb-366a-4306-c6f8-08120555ad73",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 170
        }
      },
      "source": [
        "state = env.encode(0, 0, 2, 0) # (taxi row, taxi column, passenger index, destination index)\n",
        "print(\"State:\", state) # muestra el número de estado, una codificación.\n",
        "\n",
        "env.s = state\n",
        "env.render() # nos muestra una representación gráfica del estado."
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "State: 8\n",
            "+---------+\n",
            "|\u001b[35m\u001b[43mR\u001b[0m\u001b[0m: | : :G|\n",
            "| : : : : |\n",
            "| : : : : |\n",
            "| | : | : |\n",
            "|\u001b[34;1mY\u001b[0m| : |B: |\n",
            "+---------+\n",
            "  (West)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "30iqoY9mCC54",
        "colab_type": "text"
      },
      "source": [
        "**[ACTIVIDAD]** Modifique el siguiente script, verifique cuáles son los valores posibles de los índices de pasajero y destino. ¿Qué representan.?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WgnnB4XTB_-x",
        "colab_type": "code",
        "outputId": "91d94f9b-73f3-4468-e2ff-16944b4c081f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 170
        }
      },
      "source": [
        "state = env.encode(0, 0, 2, 0)# (fila de posición del taxi, columna de posición del taxi, índice del pasajero, índice del destino)\n",
        "print(\"State:\", state)\n",
        "env.s = state\n",
        "env.render()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "State: 8\n",
            "+---------+\n",
            "|\u001b[35m\u001b[43mR\u001b[0m\u001b[0m: | : :G|\n",
            "| : : : : |\n",
            "| : : : : |\n",
            "| | : | : |\n",
            "|\u001b[34;1mY\u001b[0m| : |B: |\n",
            "+---------+\n",
            "  (West)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bZOiJLl6JNkI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vevUF5vNKUEl",
        "colab_type": "text"
      },
      "source": [
        "## Comportamiento del agente aleatorio (random search)\n",
        "\n",
        "Ejecute el siguiente bloque de código"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EA-l1zcTCaLr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "env.s = 329  # estado desde el cual arranca. "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rxSfpsoTCci-",
        "colab_type": "text"
      },
      "source": [
        "#### **[ACTIVIDAD]**  Visualice el estado. Reutilice código anterior."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ybjTfwxvCn2h",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# coloque aquí el código y ejecute"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_I_6jFrrCx6p",
        "colab_type": "text"
      },
      "source": [
        "Inicializamos variables\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WB28nmFaCz_1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "epochs = 0 # contador de iteraciones\n",
        "penalties, reward = 0, 0 # penalidades y recompensas\n",
        "frames = [] # para la animación \n",
        "rewards = [] # guardamos la recompensa obtenida por cada acción\n",
        "done = False"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u_EDLG-KJD5T",
        "colab_type": "text"
      },
      "source": [
        "El siguiente código muestra la ejecución de un agente de búsqueda aleatoria.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iFNhosdWKUOx",
        "colab_type": "code",
        "outputId": "5836279b-cd27-4109-a4ae-fb529796fe60",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "# mientras no llegue al objetivo\n",
        "while not done:\n",
        "    action = env.action_space.sample() # elige la próxima acción de forma aleatoria\n",
        "    state, reward, done, info = env.step(action) # se ejecuta la acción sobre el ambiente\n",
        "\n",
        "    if reward == -10:  # si lo penalizan, suma una penalización.\n",
        "        penalties += 1\n",
        "    \n",
        "    rewards.append({'epoch':epochs,'reward':reward}) #guarda el nro de iteración y valor de recompensa obtenida.\n",
        "    \n",
        "    # Guarda información de cada frame en un diccionario, para posterior animación\n",
        "    frames.append({\n",
        "        'frame': env.render(mode='ansi'),\n",
        "        'state': state,\n",
        "        'action': action,\n",
        "        'reward': reward\n",
        "        }\n",
        "    )\n",
        "\n",
        "    epochs += 1 # incrementa número de iteración\n",
        "    \n",
        "    \n",
        "print(\"Cuantos pasos o epochs o time-steps corrió: {}\".format(epochs))\n",
        "print(\"Penalidades (cuántas veces fue penalizado): {}\".format(penalties))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Cuantos pasos o epochs o time-steps corrió: 1826\n",
            "Penalidades (cuántas veces fue penalizado): 575\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RK19YF_dYPLl",
        "colab_type": "text"
      },
      "source": [
        "A continuación visualizamos esas recompensas otorgadas por cada episodio de ejecución del agente."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "40tuakahWrmD",
        "colab_type": "code",
        "outputId": "dfd8cd6e-e9e2-4808-af09-c687dcaf6add",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 286
        }
      },
      "source": [
        "df = pd.DataFrame(rewards)\n",
        "\n",
        "plt.plot(df['reward'])\n",
        "# agregue título al gráfico\n",
        "# agregue etiqueta al eje x y al eje y\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x7f92055b0cc0>]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 30
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD8CAYAAAB0IB+mAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAF1dJREFUeJzt3XmQHOV5x/HfM8euVlodK7QSQrfQ\nYQvHSLAIMJgjYCyIjQy+UBwsbGzhCiqbSlwuHFIOSZUrthMfOXzJARunwFfZFCR2jIEQk1R8LUQG\nYQwSBJdRBFqDy8JGx+7Mkz+md5hdViu009f2+/1UTe1Mz0z30+/0/LbnnXe6zd0FACi+UtYFAADS\nQeADQCAIfAAIBIEPAIEg8AEgEAQ+AASCwAeAQBD4ABAIAh8AAlHJuoBWc+bM8aVLl2ZdBgBMKvfd\nd9+v3L33SI/LVeAvXbpU/f39WZcBAJOKmf3ipTyOLh0ACASBDwCBIPABIBAEPgAEgsAHgEC0Hfhm\ntsjM7jGzn5nZQ2b2vmj6bDO708x2Rn972i8XADBRcezhD0n6U3dfI+k0SVeb2RpJ10q6291XSro7\nug0AyEjbge/ue9z9/uj6c5IelrRA0kZJN0UPu0nSG9pdFgAU0afuelT/uXMg8eXE2odvZkslrZP0\nI0nz3H1PdNdTkuYd5jlbzKzfzPoHBpJfYQDIm8/c85j++7FnEl9ObIFvZt2SvinpGnff13qfN86U\nPubZ0t19m7v3uXtfb+8RfxkMAJigWALfzKpqhP3N7v6taPLTZjY/un++pL1xLAsAisbH3h+OXRyj\ndEzSDZIedvdPtNx1u6TN0fXNkm5rd1kAUFSWwjLiOHjaGZIul/SgmW2Ppv2ZpI9I+rqZXSnpF5Le\nEsOyAAAT1Hbgu/t/6fD/nM5rd/4AUHSeTo8Ov7QFgDywFPp0CHwACASBDwCBIPABIGMpdeET+ACQ\nB5bCwEwCHwACQeADQMY8pXGZBD4A5ADDMgEAsSHwASBjjNIBgICkcfA0Ah8AAkHgA0DGOHgaAIQk\nhWE6BD4ABILAB4BAEPgAEAgCHwBygGGZAIDYEPgAkKG0DpwmEfgAkAscPA0AEBsCHwAylGKPDoEP\nAHkwaU5xaGY3mtleM9vRMu16M9ttZtujy0VxLAsAMDFx7eF/SdKGMaZ/0t3XRpfvxLQsACiMFHt0\n4gl8d79X0rNxzAsAQlSEUTpbzeyBqMunZ6wHmNkWM+s3s/6BgYGEywGAcCUZ+J+VdLyktZL2SPr4\nWA9y923u3ufufb29vQmWAwBhSyzw3f1pd6+5e13SFyStT2pZADBZFeKXtmY2v+XmJZJ2HO6xABC6\nNA6eVoljJmb2FUnnSJpjZk9K+gtJ55jZWjW+hH5C0lVxLAsAMDGxBL67bxpj8g1xzBsAimzSDcsE\nALSnCMMyAQA5QeADQIY4eBoABMZS6NMh8AEgEAQ+AGTIUxynQ+ADQCAIfAAIBIEPAIEg8AEgQwzL\nBIDA8EtbAEBsCHwACASBDwA5YCkcEZ/AB4BAEPgAkCFG6QBAYBilAwCIDYEPABni4GkAEJgUenQI\nfAAIBYEPABlilA4ABGbSjNIxsxvNbK+Z7WiZNtvM7jSzndHfnjiWBQCYmLj28L8kacOoaddKutvd\nV0q6O7oNAMhILIHv7vdKenbU5I2Sboqu3yTpDXEsCwCKJMUu/ET78Oe5+57o+lOS5iW4LACY1Apz\n8DR3dx3mH5mZbTGzfjPrHxgYSKMcAAhSkoH/tJnNl6To796xHuTu29y9z937ent7EywHAPLHUxyX\nmWTg3y5pc3R9s6TbElwWAExqk2lY5lck/UDSajN70syulPQRSa8xs52Szo9uAwAyUoljJu6+6TB3\nnRfH/AGgqIoySgcAkCMEPgAEgsAHgAxx8DQACIylMEyHwAeAQBD4ABAIAh8AskQfPgCEhXPaAgBi\nQ+ADQIY8xT4dAh8AcmDSHDwNAJB/BD4AZIhf2gJAYBilAwCIDYEPABniePgAEBgOngYAiA2BDwCB\nIPABIEOe4rhMAh8AcoBf2gIAYkPgA0CGGJYJAIFJ45e2laQXYGZPSHpOUk3SkLv3Jb1MAMCLJR74\nkXPd/VcpLQsAJg0OngYAoSnIL21d0vfM7D4z25LC8gAAY0ijS+dMd99tZnMl3WlmP3f3e4fvjP4J\nbJGkxYsXp1AOAORHoU5x6O67o797Jd0qaf2o+7e5e5+79/X29iZdDgDk0qQ/Hr6ZTTOz6cPXJV0g\naUeSywQAjC3pLp15km6NDvtZkXSLu3834WUCAMaQaOC7++OSTkxyGQAwqTEsEwDCwsHTAACxIfAB\nIEMcPA0AAmMpDMwk8AEgEAQ+AGSIg6cBQGAYpQMAiA2BDwAZKtTB0wAARzbpD54GAMgPAh8AMsQo\nHQAIDKN0AACxIfABIBAEPgBkiIOnAUBgOHgaACA2BD4AZMhTHJdJ4ANAHjAsEwAQFwIfADLEL20B\nIDAcPA0AEJvEA9/MNpjZI2a2y8yuTXp5AICxJRr4ZlaW9GlJF0paI2mTma1JcpkAMBlZCkdPS3oP\nf72kXe7+uLsfkvRVSRsTXiYAYAxJB/4CSb9suf1kNA0AkLLMv7Q1sy1m1m9m/QMDA1mXAwCpKtKw\nzN2SFrXcXhhNa3L3be7e5+59vb29CZcDAPlUhGGZP5G00syWmVmHpMsk3Z7wMgEAY6gkOXN3HzKz\nrZLukFSWdKO7P5TkMgFgMvEUj4ifaOBLkrt/R9J3kl4OAExmaZzTNvHAT4u768lf71fdXZVySUO1\nusolU63ucm80ZkelJJOpFF3/zf5BuUtTO8t6/mBNZmo+V5I6K2WVStL+QzVJUle1rLpLg7W6zKRa\nvbGswaG6qpWSqiXTgcG6pk+paLBe1/5DNXV1lJvPl6Qp1bLMpHpdOjhUU6VcUr3u6upo1DC7u0O/\n/t0h1aNvckpmqpZL6qyUVK2U9MxvD6qroyy5NFh3DdXq6qiU5C7V3TVUc1UrL6zD1I6K3F37B2ty\nl6qVUmP9yyU9d2BoxDoPt1fJTOWSyUw6NNSYf7lkzfoPDNYa61Yta2pnRb87OCTphccO1V2VkunA\nYE3VcklTO8qN10FqPnbka9f4W600ehirpcY67zswqJKZOiolHRqqq2dah5757cHm81pf36mdZR0c\nrKtabjy2s9qoIypbh4bqzWUNr8/w62Im9UzrkHvjcc8fGmqub63uMjVqcHlzPl3VsgbrjcJb265S\nLqmrWta+/YMyazxuKGrTA4M1VcqmjnKp+XrM7Kpq34HB5vp0lBt1D9W8sZ24N2uplEqqlEfWPfza\nHxisaUq1rENDdZVK1nz9W7fn0bqqZXVWyqqUTc8fqul3B4dUssbrXi6ZXGpu24NDjW2+FKVS3Rvr\nVClbs006KiUN1Vyd1ZIGa41tc8aUxvoNvw/KJWu+3nV3dVRKmtXVob3PHWjOu1Z39Uzt0L4Dg833\nQeu2N3p7rZRLmlJpLPPA4Att07p9DG9fw23RVS03X4PhoO2qlnWoVpdZ41QktbqrVDJVy6bBmmtw\nqLHMcsk0WKs330/1uqtSNplMQ/XG81szZLBWV63uzXUvl03dnRXtP1TTgcGantp3YMzXJwmFCfx/\n+Pdd+sSdj2ZdBgBMSKWc/KDJzIdlxuX2n/5f1iUc1h+eujjrEgBdeeayrEvI3ImLZmVdwmGdszr5\nUYqFCfw8W790tqZ1lLMuoxBKaYxdK6gzVhyTdQmZWzx7atYlHNaUSvIZUZjAT/M0YUcrjS9jQpHG\n8UaKKo2TZCPfihP4WReAVBBZKKo09mUKE/gIQ4k9/Imj6XLdJViEX9qmJ8e7+GZGV0RMaMaJo+lo\ng8IEfo7zXqZ8f8cwmRD4E8dOR74V4Xj4qSFQw8AXjygqunQKgh2r+OS5DzbvaLp8fwfEl7ZHIc/7\n9+yVxoduiYmj6VCcwM9x4vNGiw9NiaKiDx8YhX+eE8cnTT4hFibw0zym9NEysaHFpUQn/oSxCdIG\nxQn8/Oa9zBhFFJfA369AWwj8VBBTceGTEjBxhQl8hIEeHbQj9O2HwE8BO6VxojGBiSpM4Oe5j5yI\nig//PIGJK07gZ10AUhH6R3K0J/ShqYUJ/Dzji8b45Pmn8ci/UuCJV5jVz3GPTuD7FPGiLYGJK07g\n57hTh53S+PBpCZi4xALfzK43s91mtj26XJTUshAO8h7tCH2HoZLw/D/p7n+b8DIk5bxLJ+xtLFa0\nJTBxBerSya/QRwbEibYEJi7pwN9qZg+Y2Y1m1pPkgvK8h09GxYdhmWhH6JtPW4FvZneZ2Y4xLhsl\nfVbS8ZLWStoj6eOHmccWM+s3s/6BgYF2ykEAGJaJdoS+/bTVh+/u57+Ux5nZFyT962HmsU3SNknq\n6+trYz89v7v4YW9iMaMxgQlLcpTO/Jabl0jakdSypHx36YQ+MgBAPiQ5SudjZrZWjV3vJyRdleCy\ncrx/z05pnGhLtCP0fa/EAt/dL09q3ggXn5bQjtC3nuIMy8xxnw4ZBSAPihP4WRcwDsaOx4eWRDtC\n/4RYmMAHgCMJPO+LE/g57tEJfiMDkA8FCvz8Jj55H5/8vspA/hUn8LMuYDwkfmxoSrQj9O/TChP4\nCEOu/7Ej90LvXi1O4Oc4CULfqwCQD4UJ/BznffB7FXGiKdGO0Lef4gQ+X9oCOILQd74KE/gIQ37/\nrWMy4IdXBZHnIAh9IwOQD8UJ/BwnPnkfH5oS7Qh9+ylM4OdZ6BsZkBehf9ouTOB7rjt1ACB7xQn8\nHOd94DsVAHKiOIGfdQHjIvGBPAh956swgQ8ARxJ43hco8HO8ix/6XgWAfChM4Of5S1vyHsiH0He+\nChP4eRb6UDAgL0qBvxcLE/h5HqUDAHlQnMDPuoBxhL1PASAvihP4Od7FD/xTJNCWON8/ob8V2wp8\nM3uzmT1kZnUz6xt13wfNbJeZPWJmr22vzMmNE6AAExfruyfwva9Km8/fIelSSZ9vnWhmayRdJukE\nScdJusvMVrl7rc3lHVZ+9+8BIB/a2sN394fd/ZEx7too6avuftDd/1fSLknr21nWkWtJcu7tCXyn\nAsiN0N+KSfXhL5D0y5bbT0bTEvH9RweSmnVsujra/TAFSaqUC/O1U+om645HV7Uc27wqpUnaCDE5\n4rvHzO4ysx1jXDbGUYCZbTGzfjPrHxiYWHB3d1b02hPmSZJesWCGNp++RO85+3idvapX6xbP0lmr\nekc8vlpuvOjnru5VZ+WFJti49jjNmFLRp966VvNnThnxnPeet1KSdMJxMzSnu1OS9OqVc/Te31+h\ny05ZNGZdPVOr2rR+kVbM7dbXrzpNUztGbriXrFugU5b26II1jdrndHc079t67gp95m0nacXc7ma9\nkjR3eqeWHDNVkrRu8Szd/K5Tm/etXTRL65fNliSVR23Y06e88A9neKMvl0y3vPtUbVq/WK+Jarjn\n/edo7vTOEeswb0an5k7vbK73yrnd+sCG1S9a3+lTKjpp8azm7W+853R1d1a04YRj1Tu9Uwt7unTO\n6l7NmlrVbVefoTedvFCb1i/W5y8/WWeumKNV87pfNM/lc6Y117u7s6K/2niCzl3dq0vXNfYfpo1q\n045yqVnDpesWaEa03ledvVznvWzuiHb4u8vW6p/e3qfzXz53xDzuuOas5vV3nLFUi2Z3SZIWzOrS\nlrOW6/0XrGref9aq3ua2csu7Tm2G6syuqr57zatHvA7dnS8se9W8bq1d1KizUjJNb7lv8eypesWC\nGSNqag3r1fOma2ZXdcT9f3zO8brwFcdq8+lLdMWrlupftp6pVx1/jCSpb0mPrj73eJ227Bh9+71n\nqlIydVZKeuXCmSPm8aHXrRkRiItmd2nT+kXN+W89d4VmdlX10Tf+nq698GV65xnL1FkpaWpHudmu\nm09footPPE7L50zT2at6deyMKXrTyQv1yoUzdWK0vr3TO/XOM5bpz//g5XrzyQslNd5XkrThhGOb\ny++qlnXsjCm6besZ+uIVp0iSplRLWjZnmt7St7D5uJOX9GjtollacsxUbT13ha5//RpJ0nEzp2je\njE795cUn6Kqzl+uPTlusd5+1vLmsL15xij751hObyxqubUp1ZCwOZ8R1F728Oe2GzX3advnJmjej\nU+e9bOT2c8y0jma7DTtuVJ5IjW11eP3fF+VL0iyO0S1m9h+S3u/u/dHtD0qSu/91dPsOSde7+w/G\nm09fX5/39/e3XQ8AhMTM7nP3viM9LqnPx7dLuszMOs1smaSVkn6c0LIAAC9Bu8MyLzGzJyWdLunb\n0Z683P0hSV+X9DNJ35V0dZIjdAAAR9bWN4nufqukWw9z34clfbid+QMA4sOQBwAIBIEPAIEg8AEg\nEAQ+AASCwAeAQMTyw6u4mNmApF9M8OlzJP0qxnKSQp3xmQw1StQZt8lQZ9o1LnH33iM9KFeB3w4z\n638pvzTLGnXGZzLUKFFn3CZDnXmtkS4dAAgEgQ8AgShS4G/LuoCXiDrjMxlqlKgzbpOhzlzWWJg+\nfADA+Iq0hw8AGEchAt/MNkQnS99lZtdmWMciM7vHzH4Wndz9fdH0681st5ltjy4XtTwnk5O9m9kT\nZvZgVM/weQxmm9mdZrYz+tsTTTcz+/uozgfM7KSUalzd0mbbzWyfmV2Th/Y0sxvNbK+Z7WiZdtTt\nZ2abo8fvNLPNKdT4N2b286iOW81sVjR9qZntb2nTz7U85+RoW9kVrUfM5xUfs86jfo2TzoHD1Pm1\nlhqfMLPt0fTM2nNc7j6pL5LKkh6TtFxSh6SfSlqTUS3zJZ0UXZ8u6VFJayRdr8YJYkY/fk1Ub6ek\nZdF6lFOq9QlJc0ZN+5ika6Pr10r6aHT9Ikn/psYpQU+T9KOMXuenJC3JQ3tKOkvSSZJ2TLT9JM2W\n9Hj0tye63pNwjRdIqkTXP9pS49LWx42az4+jui1ajwtTaMujeo3TyIGx6hx1/8clfSjr9hzvUoQ9\n/PWSdrn74+5+SNJX1TiJeurcfY+73x9df07Swxr/XL6pn+z9CDZKuim6fpOkN7RM/7I3/FDSLDOb\nn3Jt50l6zN3H+2Feau3p7vdKenaM5R9N+71W0p3u/qy7/1rSnZI2JFmju3/P3Yeimz+UtPBFT2wR\n1TnD3X/ojbT6cst6JVbnOA73GieeA+PVGe2lv0XSV8abRxrtOZ4iBH6qJ0x/qcxsqaR1kn4UTdoa\nfYy+cfijvrKt3SV9z8zuM7Mt0bR57r4nuv6UpHnR9Ty08WUa+WbKW3tKR99+Wdf7TjX2MIctM7P/\nMbPvm9mro2kLorqGpVnj0bzGWbflqyU97e47W6blrT0LEfi5Y2bdkr4p6Rp33yfps5KOl7RW0h41\nPvpl7Ux3P0nShZKuNrOzWu+M9j5yMYTLzDokXSzpG9GkPLbnCHlqv7GY2XWShiTdHE3aI2mxu6+T\n9CeSbjGzGYd7fgpy/xqPskkjd0jy1p6SihH4uyW1niJ+YTQtE2ZWVSPsb3b3b0mSuz/t7jV3r0v6\ngl7oZsisdnffHf3dq8ZZy9ZLenq4qyb6uzfrOiMXSrrf3Z+W8tmekaNtv0zqNbMrJL1O0tuif0yK\nukieia7fp0Z/+KqontZun1RqnMBrnNlrb2YVSZdK+trwtLy157AiBP5PJK00s2XRnuBlapxEPXVR\nP94Nkh5290+0TG/t775E0vC3/Jmc7N3MppnZ9OHranyRtyOqZ3ikyGZJt7XU+fZotMlpkn7T0nWR\nhhF7T3lrzxZH2353SLrAzHqiLosLommJMbMNkj4g6WJ3f75leq+ZlaPry9Vou8ejOveZ2WnR9v32\nlvVKss6jfY2zzIHzJf3c3ZtdNXlrz6a0vh1O8qLGKIhH1fgvel2GdZypxsf4ByRtjy4XSfpnSQ9G\n02+XNL/lOddFdT+ilL6tV2Mkw0+jy0PDbSbpGEl3S9op6S5Js6PpJunTUZ0PSupLsU2nSXpG0syW\naZm3pxr/gPZIGlSjH/bKibSfGv3ou6LLO1KocZcafd3D2+fnose+MdoWtku6X9LrW+bTp0bgPibp\nHxX9YDPhOo/6NU46B8aqM5r+JUnvGfXYzNpzvAu/tAWAQBShSwcA8BIQ+AAQCAIfAAJB4ANAIAh8\nAAgEgQ8AgSDwASAQBD4ABOL/AahVGMq4txy4AAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K1oXrx7ILvTC",
        "colab_type": "text"
      },
      "source": [
        "#### **[ACTIVIDAD]** \n",
        "\n",
        "Agregue título al gráfico anterior y etiqueta a los ejes x e y usando las funciones de matplotlib usando las funciones \n",
        "\n",
        "* xlabel(\"Nombre del eje x\")\n",
        "* ylabel(\"Nombre del eje y\")\n",
        "* title(\"Título del gráfico\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jMdFtWWiX1q-",
        "colab_type": "text"
      },
      "source": [
        "El siguiente gráfico muestra la suma acumulada de las recompensas a través del tiempo. Observamos claramente una tendencia negativa, pues el agente ha sido penalizado (recompensa negativa) la mayor parte del tiempo."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HlHSkXH3XzT0",
        "colab_type": "code",
        "outputId": "f0f96769-b5f6-4ec4-91fd-a269c367c0c3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 286
        }
      },
      "source": [
        "plt.plot(df['reward'].cumsum())"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x7f920539a7f0>]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 31
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAD8CAYAAACCRVh7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xd8VfX9x/HXJ4tVNmGGPaSAiBDZ\nQ/ZwBK1aFBUKFPesRSxt1VrrwAWK4ELBBYgD6kKQPQTCHjLCkJCyIQyZCd/fHzm2V36EADfJuTd5\nPx+P8+Dezznn3vc9CfeTs805h4iIyNlE+B1ARERCl5qEiIhkSk1CREQypSYhIiKZUpMQEZFMqUmI\niEim1CRERCRTahIiIpIpNQkREclUlN8BglWmTBlXrVo1v2OIiISVJUuW7HXOxWY1Xdg3iWrVqpGY\nmOh3DBGRsGJmP53PdNrcJCIimVKTEBGRTKlJiIhIptQkREQkUyHXJMysm5mtN7MkMxvsdx4Rkfws\npJqEmUUCI4DuQD3gZjOr528qEZH8K6SaBNAUSHLObXbOnQTGAQk+ZxIRybdCrUlUApIDnm/3atnq\n9GnHhMXJfLt6Z3a/tIhInhKWJ9OZ2UBgIECVKlUueH4HvP/DT6SkHqNhXHEqliiUzQlFRPKGUFuT\nSAEqBzyP82q/4px70zkX75yLj43N8qzy/ycywngyoT6pR0/S8tnpPPXlWk6kpV98ahGRPCrUmsRi\noLaZVTezGKAXMDkn3qhxlZJ8cU8rGlcpwTtzt9DsX9/zzDc/krT7SE68nYhIWDLnnN8ZfsXMegCv\nAJHAaOfc0+eaPj4+3gV77aaJS7Yzeu4W1u44BMDNTaswsG0NqpcpEtTrioiEKjNb4pyLz3K6UGsS\nFyo7msQvknYf5m9frGHB5n0A9GlRlb9eXY/oyFBb4RIRCc75Ngl9+wWoVbYoHw9szlf3t6ZN7TKM\nWfAT178+n59PpPkdTUTEF2oSZ1G/YnHG9mvKAx1rsyrlIC2e+Z7lyal+xxIRyXVqEpkwMx7qXIdR\ntzbGATe9sYCl2w74HUtEJFepSWShW4MKfDSgOQWjIvjDu4v5eNE2jp/S4bIikj+oSZyHS+OK816/\npgA89tkqWj47nXU7D/mcSkQk56lJnKfGVUqyaEhHnr+hIUdPppHw2jw+SUzm9OnwPjpMRORc1CQu\nQIGoSG6Kr8z4gS0oWjCaP09cyRVPT2P2hj1+RxMRyRFqEhfhssolmPtoe/7ZswFppx23j15Ewoh5\nfLpkO+F+3omISCA1iYtUMDqSW5tX5esH2tC/dXWS9x/lT5+soP+YRNK1CUpE8gg1iSBVKlGIv11d\nj3mPdqB3sypMX7ebgWMTOXZSR0CJSPhTk8gmhWIi+WfPBtzctArfr9vNlS/M4P0FWzmis7VFJIyp\nSWQjM+OZ6y/lxRsv40Taaf42aQ09hs1h467DfkcTEbkoahI54HdN4lj61848fV0Dtu0/yjWvzWXO\nRh0BJSLhR00ih0REGL2bVeXL+1oTYcZt7yzi7g+XkHr0pN/RRETOm5pEDmtQqTgLBnfkmssq8vWq\nnTT6x1QGjEkkJfWY39FERLKk+0nkosVb9zNm/la+XLkDgL4tq/FwlzoUKxjtczIRyW90P4kQdEW1\nUrx2S2Mm39uKptVL8d78rdwwcj6b9+iWqSISmtQkfNAwrgQT7mjBP3s2YMOuI3QbNocVul+FiIQg\nNQkf3do8Y8d2TGQEN45awIJN+/yOJCLyK2oSPmtQqTgT72pBoZhIbn7rBx6ftFr3qxCRkBFUkzCz\nG81sjZmdNrP4M8Y9ZmZJZrbezLoG1Lt5tSQzGxxQr25mC736eDOLCSZbOKlbvhhfP9CG5jVKMWbB\nTwwYk8jrM5OYuX63LhgoIr4Kdk1iNXA9MDuwaGb1gF5AfaAb8LqZRZpZJDAC6A7UA272pgV4DnjZ\nOVcLOAD0DzJbWKlUohDjBrbg/g612LTnCM9/u56+7y6m3dCZJO8/6nc8EcmngmoSzrkfnXPrzzIq\nARjnnDvhnNsCJAFNvSHJObfZOXcSGAckmJkBHYCJ3vxjgJ7BZAtXD3e5hAWPdWTF4114tFtdkg8c\npc3zM/h40Ta/o4lIPpRT+yQqAckBz7d7tczqpYFU51zaGfWzMrOBZpZoZol79uTNy10ULxTNXVfW\n5Iu7W1EztgiPfbaKETOS/I4lIvlMlk3CzKaZ2eqzDAm5EfBsnHNvOufinXPxsbGxfsXIFZdVLsFn\nd7eiYVxxhk5ZT7uhM/h29Q6/Y4lIPhGV1QTOuU4X8bopQOWA53FejUzq+4ASZhblrU0ETp/vFS8U\nzSd3tuDtOVt4a85m7vxgKVdeEsuoW5tQMDrS73gikofl1OamyUAvMytgZtWB2sAiYDFQ2zuSKYaM\nnduTXcYhPDOAG7z5+wCTcihbWCoQFck97Wsxe1B7+reuzsz1e+j99kKSdutsbRHJOcEeAnudmW0H\nWgBfmdkUAOfcGmACsBb4FrjHOZfurSXcC0wBfgQmeNMCPAo8bGZJZOyjeCeYbHlVsYLR/O3qejzQ\nsTZLfjpA11dmk7h1v9+xRCSP0gX+wtjqlIP0Gb2I46fS+XhgcxrGlfA7koiECV3gLx9oUKk4o/te\nwanTjmtfm0f3YXMYNWsTe4+c8DuaiOQRahJh7rLKJZj+p3bc2a4mh46d4tlv1tHu+RnM3bjX72gi\nkgeoSeQBcSULM7h7XeYN7sDYfk0BuPWdhUxZs9PnZCIS7tQk8pi2dWKZ8ciVVCtdmPs+XsbqlIN+\nRxKRMKYmkQeVLVaQ9/7QlKgIo997i1m/87DfkUQkTKlJ5FHVyhTh9d6N2X34BF1fmc2waRv9jiQi\nYUhNIg+78pKyzBnUniuqleTlaRt4+qu1pKWf9juWiIQRNYk8rnKpwozt14xm1Uvx1pwtJIyYx+QV\n/9GNjUTkvKhJ5AOFYiL5YEAz7u9Ymw27DnP/x8vo/PIsdh067nc0EQlxahL5RHRkBA93rsPqJ7vy\nVM8GJO8/RvsXZjIhMZnDx0/5HU9EQpSaRD5TICqS25pX5cMBzSgQFcGgiStp+ex0Zm/Im/flEJHg\nqEnkU61qlWH+4I68dsvlGHD76EX85fNVHDyqtQoR+R81iXysUEwkVzesyHcPtaN1rTJ8tHAb7V6Y\nwYJN+/yOJiIhQk1CKF+8IB8MaMYbtzXhZNpp7ng/kZ0HtVNbRNQkJEDX+uWZeGdLjp5Mp/kz3/Po\nxJU6VFYkn1OTkF+pV7EYE+5sQatapRmfmEyLZ77XhQJF8jE1Cfl/GlcpyYcDmjOsVyPS0h13vL+E\nIZ+vItxvUCUiF05NQjKV0KgScx/tQLs6sXy4cBvPT1mvRiGSz6hJyDkVLxzN6L5X0LxGKUbO3ES3\nV+YwYkaSTsATySeCahJmNtTM1pnZSjP73MxKBIx7zMySzGy9mXUNqHfzaklmNjigXt3MFnr18WYW\nE0w2yT6REcaHA5pzf4daHDuVztAp67ly6Exe+m49J9K0Y1skL7NgNh+YWRdgunMuzcyeA3DOPWpm\n9YCPgaZARWAaUMebbQPQGdgOLAZuds6tNbMJwGfOuXFmNgpY4ZwbmVWG+Ph4l5iYeNGfQS7clyv/\nw2vTk1i38zCVShTi6oYVuDE+jlpli/odTUTOk5ktcc7FZzVdUGsSzrnvnHNp3tMfgDjvcQIwzjl3\nwjm3BUgio2E0BZKcc5udcyeBcUCCmRnQAZjozT8G6BlMNsk5VzesyDcPtOGphPoULRjFG7M3033Y\nHL5YluJ3NBHJZtm5T6If8I33uBKQHDBuu1fLrF4aSA1oOL/UJUSZGbe1qMa3D7ZlyoNtKV4omgfH\nL+elqRu0c1skD8mySZjZNDNbfZYhIWCaIUAa8GFOhg14v4FmlmhmiXv26MJ0frukfFHmPtqBehWK\nMfz7jXR8aZburS2SR2TZJJxznZxzDc4yTAIws77A1UBv978/IVOAygEvE+fVMqvvA0qYWdQZ9cwy\nvemci3fOxcfGxp7XB5WcVTA6ki/va82gbpewbd9Rrh85n/mb9vodS0SCFOzRTd2AQcC1zrmjAaMm\nA73MrICZVQdqA4vI2FFd2zuSKQboBUz2mssM4AZv/j7ApGCySe6LiDDuvrIWk+9tTYGoCG5/ZxHf\n6WxtkbAW7D6J14CiwFQzW+4dlYRzbg0wAVgLfAvc45xL9/Y53AtMAX4EJnjTAjwKPGxmSWTso3gn\nyGzik3oVi/Hlfa0pXiiage8v0b0qRMJYUIfAhgIdAhu6dh86ztWvzmXfzycZdWsTOtcr53ckEfHk\nyiGwIudStlhBPvpjM8oXK8h9Hy9lwJjFugS5SJhRk5AcVatsUcYNbM5Vl1Zk2o+76TliHj/t+9nv\nWCJyntQkJMdVLlWYF2+6jDdva8Luw8e584OlpKWf9juWiJwHNQnJNV3ql+f5Gy7jxx2H6DZsDgeP\n6SKBIqFOTUJy1e8aV2Jw97ok7T5CxxdnMnP9br8jicg5qElIrjIz7mxXk9d7N+bQ8TT6vruY12cm\nafOTSIhSkxBf9Li0Agsf60iN2CI8/+16rn51LqlHT/odS0TOoCYhvilZJIZpD7XjkS51WLfzMO1f\nmMljn61kw67DfkcTEY+ahPgqIsK4t0NtRt3ahEvKF+XjRcncOGoB+46c8DuaiKAmISGiW4PyjBvY\ngs/ubsnPJ9Lo/fZCFmza53cskXxPTUJCSuMqJfnXdZdy8NgpBr6fyPYDR7OeSURyjJqEhJybrqjM\nu3+4gmMn0+k5Yh7PfPOjdmqL+ERNQkJS3fLFeKfvFVQsUYg3Zm2m5bPTGfzpSqat3eV3NJF8RVeB\nlZA3fd0u3p6zhUVb9pN22nFJuaK0rVOG21tUo3Kpwn7HEwlL53sVWDUJCRtHT6Yx/PskZm/Yw9od\nhwDo16o6/dtUp1KJQj6nEwkvahKSp63cnsrjk9ewbFsqMVERvHrz5XStX97vWCJhQ/eTkDytYVwJ\nPr+7FZ/e1ZJShWO44/0lOmRWJAeoSUhYa1K1JJPubUW5YgW45e0feGnqBg4f19VlRbKLmoSEvXLF\nCvLhgGbUjP0Nw7/fSJvnZ7A8OdXvWCJ5QlBNwsyeMrOVZrbczL4zs4pe3cxsuJkleeMbB8zTx8w2\nekOfgHoTM1vlzTPczCyYbJK/1CpblKkPtWXELY05djKd342cz4gZSYT7PjcRvwW7JjHUOdfQOdcI\n+BL4u1fvDtT2hoHASAAzKwU8DjQDmgKPm1lJb56RwB8D5usWZDbJZ8yMqxpW4N/3taZKqcIMnbKe\nV6cn+R1LJKwF1SScc4cCnhYBfvmzLQEY6zL8AJQwswpAV2Cqc26/c+4AMBXo5o0r5pz7wWX86TcW\n6BlMNsm/6pQryvQ/taPTb8vy0tQNtH5uOk/+ew1HTqT5HU0k7AS9T8LMnjazZKA3/1uTqAQkB0y2\n3audq779LHWRi2JmvHhTI/7c9RLKFyvIu/O20uDxKbwwZb02QYlcgCybhJlNM7PVZxkSAJxzQ5xz\nlYEPgXtzOrCXaaCZJZpZ4p49e3LjLSUMFS8UzT3tazHxrpZ80L8ZDeOK89qMJFo9O12X9xA5T1k2\nCedcJ+dcg7MMk86Y9EPgd97jFKBywLg4r3auetxZ6plletM5F++ci4+Njc3qI4jQunYZvri7FY91\nr8vBY6cYMDaRp79aq7UKkSwEe3RT7YCnCcA67/Fk4HbvKKfmwEHn3A5gCtDFzEp6O6y7AFO8cYfM\nrLl3VNPtwJlNSCQoERHGHe1q8sNfOtKyZmnemrOFHsPncvCozqsQyUyw+ySe9TY9rSTjC/8Br/41\nsBlIAt4C7gZwzu0HngIWe8M/vBreNG9782wCvgkym8hZFS0Yzfv9m3F/h1r8uOMQTf81jeHfb+T0\naa1ViJxJ126SfG3G+t08/dWPJO0+QqtapXnimvrULlfU71giOU7XbhI5D+0vKct3D7alT4uqzEva\nR5dXZjN0yrqsZxTJJ9QkJN+LiDCeTGjAlAfbcnnlEoyYsYmBYxNZnXJQm6Ak31OTEPFcUr4oE+5o\nwfWXV+K7tbu4+tW59Bg+h/mb9vodTcQ32ichchbJ+48yccl2XpuRRPppx41N4njm+kuJitTfVZI3\naJ+ESBAqlyrMQ53rsOgvHelYtyyfLNnO79/8gXlJezl4TIfMSv6hNQmRLDjneH7KekbN2oRzEB1p\n3NAkjieurU+BqEi/44lclPNdk4jKjTAi4czMeLRbXfq1qs7SbQf4cOE2Pl6UzN4jJ3nztiboqvaS\nl2lzk8h5ii1agK71yzO2X1P6tqzG1LW7eGLyGr9jieQorUmIXITHr6nH7sPHGbPgJwrFRHF940rU\n0Ul4kgdpTULkIpgZL/++EfFVSzJq1ia6vDybfu8tZu+RE35HE8lWahIiF6lAVCTjBjbn2wfbcEOT\nOKav203nl2bxztwtumig5BlqEiJBiIqMoG75Yrxw42WM6deUCDOe+nItrZ6bzqIt+7N+AZEQpyYh\nkk3a1Yll8ZBOjO4bT9rp09z0xgLenrNZt02VsKbzJERywE/7fubWdxaSvP8YEZZxIcEnrq1P5VKF\n/Y4mAuiMaxFfVS1dhOl/upLRfeO5qmFFvl+3m2tfm8u6nYf8jiZyQdQkRHJIdGQEHeqW49WbL+ej\nPzbj8PE0egybw9NfrSVdV5eVMKEmIZILWtYsw5f3t+ayyiV4a84W2j4/g29X79Q9tiXkqUmI5JK6\n5Yvx6Z0teeKaehw+foo7P1hCwye/Y0Jist/RRDKlJiGSiyIijL6tqjNnUAf+2bMBJQvHMGjiSl6a\nusHvaCJnlS1Nwsz+ZGbOzMp4z83MhptZkpmtNLPGAdP2MbON3tAnoN7EzFZ58ww3XTVN8rDihaO5\ntXlVvry/NU2qlmT49xvp9spshn+/kf+kHvM7nsh/Bd0kzKwy0AXYFlDuDtT2hoHASG/aUsDjQDOg\nKfC4mZX05hkJ/DFgvm7BZhMJdcUKRjNuYHPu71CLn0+m8dLUDSSMmEfS7iN+RxMBsmdN4mVgEBC4\nBy4BGOsy/ACUMLMKQFdgqnNuv3PuADAV6OaNK+ac+8Fl7MkbC/TMhmwiIS86MoKHu1zCnEEdeL9/\nU/YeOUGPYXP4dvVOv6OJBNckzCwBSHHOrThjVCUgcG/cdq92rvr2s9RF8pU2tWP5972tKVIgkjs/\nWEKvNxew69Bxv2NJPpZlkzCzaWa2+ixDAvAX4O85H/P/ZRpoZolmlrhnz57cfnuRHNWgUnFmPtKe\nm5tW5ofN++n00iyWJ6f6HUvyqSybhHOuk3OuwZkDsBmoDqwws61AHLDUzMoDKUDlgJeJ82rnqsed\npZ5Zpjedc/HOufjY2Njz+ZwiYaV44Wieub4hH/+xOWnpjuten8ejE1ey57AuRS6566I3NznnVjnn\nyjrnqjnnqpGxiaixc24nMBm43TvKqTlw0Dm3A5gCdDGzkt4O6y7AFG/cITNr7h3VdDswKcjPJhL2\nWtQszeR7W9G2dizjE5Np+q9pfL5se9YzimSTnLoz3ddADyAJOAr8AcA5t9/MngIWe9P9wzn3y/WU\n7wbeAwoB33iDSL5Xu1xRxvRrypKf9vPnT1by0PgVrN95hHva16RowWi/40kep6vAioSRlNRj3PvR\nUpZtS+U3BaIY0bsx7epok6tcOF0FViQPqlSiEJ/f3YpRtzYhwqDP6EW8/8NPfseSPExNQiQMdWtQ\nnumPXEnV0oX52xerGTEjiaMndXMjyX5qEiJhqsxvCvDv+1pTI7YIQ6esp8Uz05m5frffsSSPUZMQ\nCWPFCkYz7aF2DOvViFPpp+n77mIeHr+cg8dO+R1N8gg1CZEwFxFhJDSqxPzBHehQtyyfLUuh3dAZ\nvDtvi9/RJA9QkxDJI0oUjmF03yt447YmFC0YxZP/XstD45frxkYSlJw6T0JEfNK1fnk61i3L/eOW\n8fmyFAx4qmcDihTQf3e5cFqTEMmDoiIjGHFLY65qWIHPlqXQ6rnpzEva63csCUNqEiJ5lJkx4pbG\nDOvViKMn0+n99kLu+Wiprv8kF0RNQiSPS2hUicVDOtHj0vJ8tXIH170+j9SjJ/2OJWFCTUIkHyhe\nKJrXezdh1K2N2X7gGO2GzuSpL9fqVqmSJTUJkXykW4MKjOzdmGplivDO3C1c9/o85mzcoyOgJFO6\nwJ9IPjV/0176vbeY46dO06JGaTrXK8fvmsRRvJCuLJsf6AJ/InJOLWuWYfGQTtzctDIbdx/hH1+u\npclTU5m0PNP7fUk+pDUJEQFg1oY9/OWzVaSkHqNlzdLc3qIqTauXplSRGL+jSQ443zUJNQkR+a+D\nR0/x7Lc/8unSFE6mnQagQvGC3HVlTW5rXpWMG0dKXqAmISIX7eCxUyxPTmXDzsO8PXczuw6d4LcV\nivHRgGaU1JpFnqB9EiJy0YoXiqZdnVj+2LYGcwZ14IGOtflxxyE6vTSL5cmpfseTXKQmISLnFBMV\nwUOd6/B678YcO5XOgDGJOhkvH1GTEJHz0uPSCrzdJ569R04wYEwi6afDe1O1nJ+gmoSZPWFmKWa2\n3Bt6BIx7zMySzGy9mXUNqHfzaklmNjigXt3MFnr18WamDZ8iIaZlzTL89arfkvjTAR75ZAXbDxz1\nO5LksOxYk3jZOdfIG74GMLN6QC+gPtANeN3MIs0sEhgBdAfqATd70wI8571WLeAA0D8bsolINuvf\nujrdG5Tn82UptH5uBgPHJnL4uO6El1fl1OamBGCcc+6Ec24LkAQ09YYk59xm59xJYByQYBnH1XUA\nJnrzjwF65lA2EQmCmTHy1iZMvrcVVzeswHdrd9H+hVmsTjnodzTJAdnRJO41s5VmNtrMSnq1SkBy\nwDTbvVpm9dJAqnMu7Yz6WZnZQDNLNLPEPXv2ZMNHEJEL1TCuBK/d0piRvRuTevQkPUfMY+yCrf89\nv0LyhiybhJlNM7PVZxkSgJFATaARsAN4MYfzAuCce9M5F++ci4+Njc2NtxSRTHS/tAJf3t+aUkVi\n+PukNTR5aiqfL9vudyzJJlnez9A51+l8XsjM3gK+9J6mAJUDRsd5NTKp7wNKmFmUtzYROL2IhLi6\n5Ysxf3AHJi3/D898s46Hxq/gma/XcWN8HH1aVKNssYJ+R5SLFOzRTRUCnl4HrPYeTwZ6mVkBM6sO\n1AYWAYuB2t6RTDFk7Nye7DJO+54B3ODN3weYFEw2EcldUZER/K5JHLMHXcnj19SjWKFoRszYRMcX\nZ/Hxom3sPnTc74hyEYK6LIeZvU/GpiYHbAXucM7t8MYNAfoBacCDzrlvvHoP4BUgEhjtnHvaq9cg\nY0d2KWAZcKtzLsv7LOqyHCKha/aGPdw/bhmpRzOOfrqqYQWG/b4RUZE6RctvunaTiISEtPTTLE9O\nZdziZCYu2U6DSsV4/Jr6XFGtlN/R8rXzbRJZ7pMQEQlGVGQE8dVK0aRqSSqVKMTw6Ru5cdQCOtcr\nx7PXX0rp3xTwO6Kcg9b5RCRXmBkPda7Dwr90pGejikxdu4srh85kwuJk0tJ12GyoUpMQkVxVtmhB\nXul1OaP7xhMTFcGgT1cyaOJKdh7Uju1QpCYhIr7oULcci4Z04pZmVfhsWQrdh81m3KJtnEhL9zua\nBFCTEBHfREYY/7ruUj4c0Iz0047Bn63irg+W6qztEKImISK+a1WrDIl/7cyd7Woyfd1u6v7tGyYk\nJmc9o+Q4NQkRCQkxURE82u0SRt3amEvjSjBo4kru+XApSbuP+B0tX1OTEJGQYWZ0a1CBd/teQfcG\n5flq1Q66D5vN23M2+x0t39LJdCISslZtP8h9Hy9l676jVC1dmBsax3Fto4pULV3E72hh73xPptOa\nhIiErEvjijP14Xb8pUdd0tIdL07dwNWvzmXHwWN+R8s31CREJKRFR0YwsG1N5j7ans/vbsnRk+nc\n9cFSDh7V3fByg5qEiIQFM+PyKiV5/Jp6LE9O5Yqnp/HuvC1+x8rz1CREJKzc3qIaY/o1pWKJgjz5\n77Vc//o8lm074HesPEtNQkTCTrs6sXz7YFv+2KY6y5NTue71+UxarvuU5QQ1CREJSwWjIxlyVT3m\nD+5I9TJFeGDccp77dh0Hfj7pd7Q8RYfAikjYS0k9xp3vL2FVykEKx0TSuV45mlQtSe9mVYmMML/j\nhSTddEhE8p2Z63czZv5WVqUcYu+RE7SqVZqXb2qke2yfhZqEiORbzjn+8eVa3p23lQiDZ3/XkOsu\nr0S0bpv6XzqZTkTyLTPj8WvqM35gcyqWKMSgiStp9ex0Zqzb7Xe0sBN0kzCz+8xsnZmtMbPnA+qP\nmVmSma03s64B9W5eLcnMBgfUq5vZQq8+3sxigs0mIvlbsxql+fqBNjx5bX2On0rnD+8tZtDEFew9\ncsLvaGEjqCZhZu2BBOAy51x94AWvXg/oBdQHugGvm1mkmUUCI4DuQD3gZm9agOeAl51ztYADQP9g\nsomIABQrGE2fltWY+ef2tKldhgmJ2+n80ixmrNdaxfkIdk3iLuBZ59wJAOfcL0s9ARjnnDvhnNsC\nJAFNvSHJObfZOXcSGAckmJkBHYCJ3vxjgJ5BZhMR+a9SRWJ4v38z3ro9YzP8/R8tY8ven31OFfqC\nbRJ1gDbeZqJZZnaFV68EBN4xZLtXy6xeGkh1zqWdURcRyVad65XjkztbcDwtnfYvzOSej5YyY/1u\n0k+H90E8OSUqqwnMbBpQ/iyjhnjzlwKaA1cAE8ysRrYmPHumgcBAgCpVquT024lIHlOrbFEm3NGC\nN2Zt5tvVO/lq5Q5qxBZheK/LaVCpuN/xQkqWTcI51ymzcWZ2F/CZyziOdpGZnQbKAClA5YBJ47wa\nmdT3ASXMLMpbmwic/myZ3gTehIxDYLP6DCIiZ7q8SklG3daEAz+f5KNF2xg6ZT1XvzqXDnXL0r91\ndVrVKuN3xJAQ7OamL4D2AGZWB4gB9gKTgV5mVsDMqgO1gUXAYqC2dyRTDBk7tyd7TWYGcIP3un2A\nSUFmExHJUskiMdzTvhZzBrXn2ssqMn3dbnq/vZBm/5rG5BX/4ecTaVm/SB4W1Ml03hf9aKARcBJ4\nxDk33Rs3BOgHpAEPOue+8eqxue0EAAAKSklEQVQ9gFeASGC0c+5pr16DjB3ZpYBlwK2/7BA/F51M\nJyLZ6dDxU0xM3M7w6RtJ9e5Z0bleOYb3upxCMZE+p8s+OuNaRCQIB34+yaKt+5myZiefLU2hVtnf\n8OS19WlQqTjFC0X7HS9oahIiItnk7Tmb+dfXP3LaQXSkcfeVtbinfS1iosL3ohVqEiIi2WjP4RMs\n+ekAw7/fyNodhyhZOJoJd7Sgdrmifke7KLp2k4hINootWoBuDcrz1f2tGXpDQ34+kc7Vr85l7IKt\nnEo/7Xe8HKMmISJyAcyMG+MrM/6O5hQvFM3fJ62h68uzSUk95ne0HKEmISJyES6vUpIFj3VkcPe6\nbN77Mx1fnMmk5SmE+yb8M6lJiIhcpMgI4852Nfn0rpbEREbwwLjl3D9uOUfy0LkVahIiIkFqUrUk\ni4Z0omv9cvx7xX+47Z2FHD2ZNxqFmoSISDYoGB3JG7fF89erfsuybalcPXwuny/bHvY7tdUkRESy\n0YA2NXjy2vrsPHSch8avoOOLs1i385DfsS6azpMQEckBaemnGbvgJ576ai0RZvz+ispcFlecG5tU\nJiLC/I6nk+lERELB2v8c4p9frWV5cipHT6ZTo0wRejevSv/W1X3NpSYhIhJCnHOMnLWJjxZuY/uB\nYzSvUYqPBjT3ba1CZ1yLiIQQs4xrPs36c3v6tqzGD5v3c/3I+STvP+p3tHNSkxARyUWREcbj19Tj\nvg61WJ6cSruhMxg5cxO7Dx/3O9pZqUmIiOQyM+NPXS7hi3taUaVUYZ77dh0tnpnOyJmbOJGW7ne8\nX1GTEBHxSaPKJZj6cDs+/mNz4koW4rlv19F92Bx2HgydtQo1CRERH0VHRtCiZmmm/+lKHutel817\nfqbt8zMYNm1jSJy1rSYhIhICIiOMO9rVZMIdLYgrWYiXp22g7fMzWLRlv6+51CREREJI0+qlmP7I\nlQzr1YiDx05x0xsLeObrH0k/7c/pCkE1CTMbb2bLvWGrmS0PGPeYmSWZ2Xoz6xpQ7+bVksxscEC9\nupkt9OrjzSwmmGwiIuEsoVEl5g/uyGVxxXlj9mb+/MkKXy5DHlSTcM793jnXyDnXCPgU+AzAzOoB\nvYD6QDfgdTOLNLNIYATQHagH3OxNC/Ac8LJzrhZwAOgfTDYRkXAXW7QAX9zTihubxPHZshQGf7qK\n3Ydyd6d2VHa8iJkZcBPQwSslAOOccyeALWaWBDT1xiU55zZ7840DEszsR2/eW7xpxgBPACOzI5+I\nSLgyM/51/aXsPHSc8YnJjE9MpnuD8nT8bTmuurQChWIic/T9s2ufRBtgl3Nuo/e8EpAcMH67V8us\nXhpIdc6lnVEXEcn3oiMjeL9/Mybc0YLLq5Tgm9U7eeSTFaTnwuanLNckzGwaUP4so4Y45yZ5j28G\nPs7OYFlkGggMBKhSpUpuva2IiK+aVi/F+IEteHX6Rsr8pgC/KZAtG4POKct3cM51Otd4M4sCrgea\nBJRTgMoBz+O8GpnU9wElzCzKW5sInP5smd4E3oSMC/xl9RlERPKKmKgI/tTlklx7v+zY3NQJWOec\n2x5Qmwz0MrMCZlYdqA0sAhYDtb0jmWLI2Lk92WXssp8B3ODN3weYhIiI+Co71lV6ccamJufcGjOb\nAKwF0oB7nHPpAGZ2LzAFiARGO+fWeLM9Cowzs38Cy4B3siGbiIgEQfeTEBHJh3Q/CRERCZqahIiI\nZEpNQkREMqUmISIimVKTEBGRTIX90U1mtgf46SJnLwPszcY4OSUccoZDRlDO7BQOGUE5M1PVOReb\n1URh3ySCYWaJ53MImN/CIWc4ZATlzE7hkBGUM1ja3CQiIplSkxARkUzl9ybxpt8BzlM45AyHjKCc\n2SkcMoJyBiVf75MQEZFzy+9rEiIicg75skmYWTczW29mSWY22Ocslc1shpmtNbM1ZvaAV3/CzFLM\nbLk39AiY5zEv+3oz65qLWbea2SovT6JXK2VmU81so/dvSa9uZjbcy7nSzBrnQr5LApbXcjM7ZGYP\nhsKyNLPRZrbbzFYH1C542ZlZH2/6jWbWJ5dyDjWzdV6Wz82shFevZmbHApbrqIB5mni/K0neZ7Fc\nyHnBP+ec/C7IJOP4gHxbzWy5V/dtWWbJOZevBjIuUb4JqAHEACuAej7mqQA09h4XBTYA9ci4x/cj\nZ5m+npe5AFDd+yyRuZR1K1DmjNrzwGDv8WDgOe9xD+AbwIDmwEIffs47gaqhsCyBtkBjYPXFLjug\nFLDZ+7ek97hkLuTsAkR5j58LyFktcLozXmeRl928z9I9F3Je0M85p78LzpbxjPEvAn/3e1lmNeTH\nNYmmQJJzbrNz7iQwDkjwK4xzbodzbqn3+DDwI+e+v3cCMM45d8I5twVIIuMz+SUBGOM9HgP0DKiP\ndRl+IOPOgxVyMVdHYJNz7lwnWubasnTOzQb2n+X9L2TZdQWmOuf2O+cOAFOBbjmd0zn3nfvf/ed/\nIOPOkZnyshZzzv3gMr7lxvK/z5ZjOc8hs59zjn4XnCujtzZwE1nc9jk3lmVW8mOTqAQkBzzfzrm/\nlHONmVUDLgcWeqV7vVX80b9sisDf/A74zsyWWMZ9xgHKOed2eI93AuW8x34v5zNvhhVqyxIufNn5\nnRegHxl/zf6iupktM7NZZtbGq1Xysv0iN3NeyM/Zz+XZBtjlnNsYUAu1ZQnkzyYRkszsN8CnwIPO\nuUPASKAm0AjYQcaqqd9aO+caA92Be8ysbeBI7y8d3w+Xs4xb414LfOKVQnFZ/kqoLLtzMbMhZNxp\n8kOvtAOo4py7HHgY+MjMivmVjzD4OQe4mV//ERNqy/K/8mOTSAEqBzyP82q+MbNoMhrEh865zwCc\nc7ucc+nOudPAW/xvM4hv+Z1zKd6/u4HPvUy7ftmM5P272++cZDSxpc65XV7ekFuWngtddr7lNbO+\nwNVAb6+h4W2+2ec9XkLG9v06XqbATVK5kvMifs6+LE8ziwKuB8b/Ugu1ZRkoPzaJxUBtM6vu/cXZ\nC5jsVxhv2+Q7wI/OuZcC6oHb768DfjlCYjLQy8wKmFl1oDYZO7ZyOmcRMyv6y2Mydmau9vL8cpRN\nH2BSQM7bvSN1mgMHAzat5LRf/ZUWassywIUuuylAFzMr6W1K6eLVcpSZdQMGAdc6544G1GPNLNJ7\nXIOM5bfZy3rIzJp7v9+3B3y2nMx5oT9nv74LOgHrnHP/3YwUasvyV3JzL3moDGQcPbKBjG49xOcs\nrcnYzLASWO4NPYD3gVVefTJQIWCeIV729eTSkQ5kHAGywhvW/LLcgNLA98BGYBpQyqsbMMLLuQqI\nz6WcRYB9QPGAmu/LkoymtQM4RcZ25f4Xs+zI2CeQ5A1/yKWcSWRsu//l93OUN+3vvN+F5cBS4JqA\n14kn40t6E/Aa3om7OZzzgn/OOfldcLaMXv094M4zpvVtWWY16IxrERHJVH7c3CQiIudJTUJERDKl\nJiEiIplSkxARkUypSYiISKbUJEREJFNqEiIikik1CRERydT/AUbWizheemUTAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3GIOONAXKbX7",
        "colab_type": "text"
      },
      "source": [
        "Ejecute el siguiente bloque para observar las acciones del agente y el ambiente"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Db2559BuKYXO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from IPython.display import clear_output\n",
        "from time import sleep\n",
        "\n",
        "# función print_frames\n",
        "def print_frames(frames):\n",
        "    for i, frame in enumerate(frames):\n",
        "        clear_output(wait=True)\n",
        "        print(frame['frame'].getvalue())\n",
        "        print(f\"Timestep: {i + 1}\")\n",
        "        print(f\"State: {frame['state']}\")\n",
        "        print(f\"Action: {frame['action']}\")\n",
        "        print(f\"Reward: {frame['reward']}\")\n",
        "        sleep(.1)\n",
        "        \n",
        "#print_frames(frames) # descomente para llamar a la función print_frames o utilice un bloque de código aparte."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GKLfPCBWU5d3",
        "colab_type": "text"
      },
      "source": [
        "Para nada bueno el desempeño del agente. A nuestro agente aleatorio le toma miles de *timesteps* o *epochs* cometiendo muchas acciones erróneas (es penalizado muchas veces) para tan sólo acercar a un pasajero al destino.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rKBmnzlKYeDU",
        "colab_type": "text"
      },
      "source": [
        "## Aprendizaje por refuerzo con Q-learning\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XAlHkzlJYzbr",
        "colab_type": "text"
      },
      "source": [
        "Inicializamos tabla Q con ceros."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2ZEIB3CqYyge",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "q_table = np.zeros([env.observation_space.n, env.action_space.n])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DfCkfn5DNs2z",
        "colab_type": "text"
      },
      "source": [
        "Entrenamos al agente, es decir, *completamos* la tabla Q con valores que se ajusten a una política.\n",
        "\n",
        "**Hiperparámetros**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hpMTpELTYOFI",
        "colab_type": "text"
      },
      "source": [
        "$\\alpha$ es el coeficiente de aprendizaje en Q-learning, página 873 del libro\n",
        "\n",
        "$\\gamma$ el factor de descuento \n",
        "\n",
        "$\\epsilon$ ¿para qué usamos este coeficiente en el código.\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Sir29BebYu7U",
        "colab_type": "text"
      },
      "source": [
        "Observe las líneas 24 a la 27 y responda la actividad 2) "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kpcJrkfcJuMe",
        "colab_type": "text"
      },
      "source": [
        "## [Algoritmo Q-Learning](https://es.wikipedia.org/wiki/Q-learning)\n",
        "\n",
        "![alt](https://wikimedia.org/api/rest_v1/media/math/render/svg/59db58edf1222b292e40706e503ed5974553606b)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "juiuys_cYnmC",
        "colab_type": "code",
        "outputId": "7e82cad6-0a4e-4537-9a09-dd4d07a06609",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "source": [
        "%%time\n",
        "\"\"\"Entrenando el agente\"\"\"\n",
        "\n",
        "import random\n",
        "from IPython.display import clear_output\n",
        "\n",
        "# Hiperparámetros\n",
        "alpha = 0.1 # tasa de aprendizaje en Q-learning\n",
        "gamma = 0.6 # factor de descuento gamma,\n",
        "epsilon = 0.1 # probabilidad \n",
        "max_iter = 10000 #100001\n",
        "# para graficar métricas\n",
        "all_epochs = []\n",
        "all_penalties = []\n",
        "rewards = []\n",
        "\n",
        "for i in range(1, max_iter):\n",
        "    state = env.reset()\n",
        "\n",
        "    epochs, penalties, reward, = 0, 0, 0\n",
        "    done = False\n",
        "    \n",
        "    while not done:\n",
        "        if random.uniform(0, 1) < epsilon:\n",
        "            action = env.action_space.sample() \n",
        "        else:\n",
        "            action = np.argmax(q_table[state]) \n",
        "\n",
        "        next_state, reward, done, info = env.step(action) \n",
        "        \n",
        "        old_value = q_table[state, action]\n",
        "        \n",
        "        next_max = np.max(q_table[next_state])\n",
        "        \n",
        "        #new_value =  old_value + alpha * (old_value- reward - gamma * next_max)\n",
        "        new_value = (1 - alpha) * old_value + alpha * (reward + gamma * next_max)\n",
        "        \n",
        "        q_table[state, action] = new_value\n",
        "\n",
        "        if reward == -10:\n",
        "            penalties += 1\n",
        "            \n",
        "        rewards.append({'epoch':epochs,'reward':reward})\n",
        "        \n",
        "        state = next_state\n",
        "        epochs += 1\n",
        "        \n",
        "    if i % 100 == 0:\n",
        "        clear_output(wait=True)\n",
        "        print(f\"Episode: {i}\")\n",
        "\n",
        "print(\"Training finished.\\n\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Episode: 9900\n",
            "Training finished.\n",
            "\n",
            "CPU times: user 5.29 s, sys: 203 ms, total: 5.49 s\n",
            "Wall time: 5.41 s\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wo5pl2TqZDxB",
        "colab_type": "text"
      },
      "source": [
        "## Evaluemos el agente\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MbcCJrK5Nwxd",
        "colab_type": "text"
      },
      "source": [
        "Dada la tabla Q aprendida, la ponemos a prueba en el ambiente.\n",
        "\n",
        "Política óptima:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G3Ie8w9TPWFy",
        "colab_type": "text"
      },
      "source": [
        "$\\pi^{*}(s) = arg max_a Q(s,a)$"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y5w5cUe1ZGmu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "\n",
        "total_epochs, total_penalties = 0, 0\n",
        "episodes = 100\n",
        "rewards3 = []\n",
        "frames = [] # para la animación\n",
        "\n",
        "for _ in range(episodes):\n",
        "    state = env.reset()\n",
        "    epochs, penalties, reward = 0, 0, 0\n",
        "    \n",
        "    done = False\n",
        "    \n",
        "    while not done: # mientras el taxi no llegue a destino.\n",
        "        action = np.argmax(q_table[state])\n",
        "        state, reward, done, info = env.step(action)\n",
        "\n",
        "        if reward == -10:\n",
        "            penalties += 1\n",
        "\n",
        "        epochs += 1\n",
        "        rewards3.append({'epoch':epochs,'reward':reward})\n",
        "        \n",
        "        # frames para animacion\n",
        "        frames.append({\n",
        "            'frame': env.render(mode='ansi'),\n",
        "            'state': state,\n",
        "            'action': action,\n",
        "            'reward': reward\n",
        "            }\n",
        "        )\n",
        "\n",
        "    total_penalties += penalties\n",
        "    total_epochs += epochs\n",
        "\n",
        "print(f\"Resultados luego de  {episodes} episodios:\")\n",
        "print(f\"Promedio de timestemps o iteraciones por episodio: {total_epochs / episodes}\")\n",
        "print(f\"Promedio de penalidades por episodio: {total_penalties / episodes}\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wdvp2VEVOKpt",
        "colab_type": "text"
      },
      "source": [
        "Ejecute el siguiente bloque de código si desea visualizar las acciones ejecutadas por el agente. \n",
        "**Nota**: puede demorar un poco.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k_h5EkUFbTCg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# print_frames(frames)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nuqzK4T1i9n6",
        "colab_type": "text"
      },
      "source": [
        "#### [ACTIVIDAD] Grafique la suma acumulada de recompensas a través del tiempo (epochs)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "10C6PJuxZme4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R9z4EaagMalz",
        "colab_type": "text"
      },
      "source": [
        "# ACTIVIDADES\n",
        "\n",
        "## 1) Explique, detalle, como se calcula los 500 estados posibles del problema del *Taxi*. Desglose cada término y valor implicado."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dnsoHwiLNOhj",
        "colab_type": "text"
      },
      "source": [
        "**ESCRIBA AQUI SU RESPUESTA A LA CONSIGNA**\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i7-k4damZBdY",
        "colab_type": "text"
      },
      "source": [
        "## 2) En el código de Q-learning presentado. \n",
        "\n",
        "2.1 Explique que hace las líneas de código entre 24 -27 . Pista: explorar vs explotar.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X1eHQ69MZTL5",
        "colab_type": "text"
      },
      "source": [
        "2.2 ¿Qué pasaría si el valor de $ \\epsilon $ fuera 1?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c_qMqKjGdYcZ",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "\n",
        "## 3.1) Basado en los resultados anteriores, indique cuál agente funcionó mejor (aleatorio vs Q-learning) y justifique.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mrrwCcEbia2b",
        "colab_type": "text"
      },
      "source": [
        "**ESCRIBA AQUI SU TEXTO DE RESPUESTA A LA CONSIGNA "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dpYCGU2RV7dV",
        "colab_type": "text"
      },
      "source": [
        "## 3.2) Compare para este entorno el desempeño y funcionamiento de iteración de valores y políticas vs Q-learning. Utilice código del laboratorio anterior. Escriba sus conclusiones, mientras más completas, mejor."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Yaxj9gr1WSj8",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YvD1YCrCWTH9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UN7cpe_YSZ7q",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T8lGTr54SaJz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WmOmPFlqiU_u",
        "colab_type": "text"
      },
      "source": [
        "## [ OPCIONAL ] 4) Búsqueda en grilla\n",
        "\n",
        "Realice una búsqueda en grilla para determinar los mejores valores para alpha, gamma y epsilon, entre los valores siguientes valores.\n",
        "\n",
        "\n",
        "Valores de alpha a considerar: {0.1, 0.3 ,0.6}\n",
        "\n",
        "Valores de gamma a considerar:  {0.1,0.5, 0.9}\n",
        "\n",
        "Valores de epsilon a considerar: {0.1,0.5, 0.9}\n",
        "\n",
        "\n",
        "* Cree el script para ejecutar este experimento.\n",
        "\n",
        "* Determine los valores de rendimiento a guardar de cada experimento (recompensa promedio, penalización promedio, etc)\n",
        "\n",
        "* Utilice la cantidad de bloques de código y texto que crea necesarios\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9wB2KhRsd0Bs",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fvkx0OtGSuSh",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3bfhYrLTdREu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_sweAdIcVLuR",
        "colab_type": "text"
      },
      "source": [
        "# Referencias\n",
        "\n",
        "Este laboratorio fue inspirado gracias a las siguientes referencias:\n",
        "\n",
        "* Open AI Gym (https://gym.openai.com/)\n",
        "* Taxi environment in Open AI Gym \n",
        " * https://gym.openai.com/envs/Taxi-v2/\n",
        " * https://github.com/openai/gym/blob/master/gym/envs/toy_text/taxi.py\n",
        "* Q-learning from scratch https://www.learndatasci.com/tutorials/reinforcement-q-learning-scratch-python-openai-gym/\n",
        "* Q-learning, wikipedia https://es.wikipedia.org/wiki/Q-learning\n",
        "\n"
      ]
    }
  ]
}