ML0120EN-1.4-Review-LogisticRegressionwithTensorFlow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-DL0120EN-SkillsNetwork/images/IDSN-logo.png\" width=\"300\" alt=\"cognitiveclass.ai logo\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>LOGISTIC REGRESSION WITH TENSORFLOW</h2>\n",
    "Estimated time needed: <b>25</b> minutes\n",
    "\n",
    "<h3>Objective for this Notebook<h3>    \n",
    "<h5> 1. What is different between Linear and Logistic Regression?</h5>\n",
    "<h5> 2. Utilizing Logistic Regression in TensorFlow. </h5>\n",
    "<h5> 3. Training the model </h5>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of Contents\n",
    "\n",
    "Logistic Regression is one of most important techniques in data science. It is usually used to solve the classic classification problem.\n",
    "\n",
    "<div class=\"alert alert-block alert-info\" style=\"margin-top: 20px\">\n",
    "<font size = 3><strong>This lesson covers the following concepts of Logistics Regression:</strong></font>\n",
    "<br>\n",
    "<h2>Table of Contents</h2>\n",
    "\n",
    "<ol>\n",
    "    <li><a href=\"#ref1\">Linear Regression vs Logistic Regression</a></li>\n",
    "    <li><a href=\"#ref2\">Utilizing Logistic Regression in TensorFlow</a></li>\n",
    "    <li><a href=\"#ref3\">Training</a></li>\n",
    "</ol>    \n",
    "</div>\n",
    "<p></p>\n",
    "<br>\n",
    "\n",
    "<hr>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ref1\"></a>\n",
    "\n",
    "<h2>What is different between Linear and Logistic Regression?</h2>\n",
    "\n",
    "While Linear Regression is suited for estimating continuous values (e.g. estimating house price), it is not the best tool for predicting the class in which an observed data point belongs. In order to provide estimate for classification, we need some sort of guidance on what would be the <b>most probable class</b> for that data point. For this, we use <b>Logistic Regression</b>.\n",
    "\n",
    "<div class=\"alert alert-success alertsuccess\" style=\"margin-top: 20px\">\n",
    "<font size=\"3\"><strong>Recall linear regression:</strong></font>\n",
    "<br>\n",
    "<br>\n",
    "Linear regression finds a function that relates a continuous dependent variable, <i>y</i>, to some predictors (independent variables <i>x1</i>, <i>x2</i>, etc.). Simple linear regression assumes a function of the form:\n",
    "<br><br>\n",
    "$$\n",
    "y = w0 + w1 \\times x1 + w2 \\times x2 + \\cdots\n",
    "$$\n",
    "<br>\n",
    "and finds the values of <i>w0</i>, <i>w1</i>, <i>w2</i>, etc. The term <i>w0</i> is the \"intercept\" or \"constant term\" (it's shown as <i>b</i> in the formula below):\n",
    "<br><br>\n",
    "$$\n",
    "Y = W X + b\n",
    "$$\n",
    "<p></p>\n",
    "\n",
    "</div>\n",
    "\n",
    "Logistic Regression is a variation of Linear Regression, useful when the observed dependent variable, <i>y</i>, is categorical. It produces a formula that predicts the probability of the class label as a function of the independent variables.\n",
    "\n",
    "Despite the name logistic <i>regression</i>, it is actually a <b>probabilistic classification</b> model. Logistic regression fits a special s-shaped curve by taking the linear regression and transforming the numeric estimate into a probability with the following function:\n",
    "\n",
    "$$\n",
    "ProbabilityOfaClass = \\theta(y) = \\frac{e^y}{1 + e^y} = exp(y) / (1 + exp(y)) = p \n",
    "$$\n",
    "\n",
    "which produces p-values between 0 (as y approaches minus infinity $-\\infty$) and 1 (as y approaches plus infinity $+\\infty$). This now becomes a special kind of non-linear regression.\n",
    "\n",
    "In this equation, <i>y</i> is the regression result (the sum of the variables weighted by the coefficients), <code>exp</code> is the exponential function and $\\theta(y)$ is the <a href=\"http://en.wikipedia.org/wiki/Logistic_function\">logistic function</a>, also called logistic curve. It is a common \"S\" shape (sigmoid curve), and was first developed for modeling population growth.\n",
    "\n",
    "You might also have seen this function before, in another configuration:\n",
    "\n",
    "$$\n",
    "ProbabilityOfaClass = \\theta(y) = \\frac{1}{1+e^{-y}}\n",
    "$$\n",
    "\n",
    "So, briefly, Logistic Regression passes the input through the logistic/sigmoid function but then treats the result as a probability:\n",
    "\n",
    "<img src=\"https://ibm.box.com/shared/static/kgv9alcghmjcv97op4d6onkyxevk23b1.png\" width=\"400\" align=\"center\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* * *\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ref2\"></a>\n",
    "\n",
    "<h2>Utilizing Logistic Regression in TensorFlow</h2>\n",
    "\n",
    "We begin by installing TensorFlow version 2.9.0 and its required prerequistes.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: grpcio==1.24.3 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (1.24.3)\n",
      "Requirement already satisfied: six>=1.5.2 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from grpcio==1.24.3) (1.16.0)\n",
      "Requirement already satisfied: tensorflow==2.9.0 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (2.9.0)\n",
      "Requirement already satisfied: absl-py>=1.0.0 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (1.4.0)\n",
      "Requirement already satisfied: astunparse>=1.6.0 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (1.6.3)\n",
      "Requirement already satisfied: flatbuffers<2,>=1.12 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (1.12)\n",
      "Requirement already satisfied: gast<=0.4.0,>=0.2.1 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (0.4.0)\n",
      "Requirement already satisfied: google-pasta>=0.1.1 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (0.2.0)\n",
      "Requirement already satisfied: grpcio<2.0,>=1.24.3 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (1.24.3)\n",
      "Requirement already satisfied: h5py>=2.9.0 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (3.8.0)\n",
      "Requirement already satisfied: keras<2.10.0,>=2.9.0rc0 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (2.9.0)\n",
      "Requirement already satisfied: keras-preprocessing>=1.1.1 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (1.1.2)\n",
      "Requirement already satisfied: libclang>=13.0.0 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (16.0.6)\n",
      "Requirement already satisfied: numpy>=1.20 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (1.21.6)\n",
      "Requirement already satisfied: opt-einsum>=2.3.2 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (3.3.0)\n",
      "Requirement already satisfied: packaging in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (23.1)\n",
      "Requirement already satisfied: protobuf>=3.9.2 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (3.19.6)\n",
      "Requirement already satisfied: setuptools in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (67.7.2)\n",
      "Requirement already satisfied: six>=1.12.0 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (1.16.0)\n",
      "Requirement already satisfied: tensorboard<2.10,>=2.9 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (2.9.1)\n",
      "Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (0.34.0)\n",
      "Requirement already satisfied: tensorflow-estimator<2.10.0,>=2.9.0rc0 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (2.9.0)\n",
      "Requirement already satisfied: termcolor>=1.1.0 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (2.3.0)\n",
      "Requirement already satisfied: typing-extensions>=3.6.6 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (4.5.0)\n",
      "Requirement already satisfied: wrapt>=1.11.0 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (1.14.1)\n",
      "Requirement already satisfied: wheel<1.0,>=0.23.0 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from astunparse>=1.6.0->tensorflow==2.9.0) (0.40.0)\n",
      "Requirement already satisfied: google-auth<3,>=1.6.3 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorboard<2.10,>=2.9->tensorflow==2.9.0) (2.23.4)\n",
      "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorboard<2.10,>=2.9->tensorflow==2.9.0) (0.4.6)\n",
      "Requirement already satisfied: markdown>=2.6.8 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorboard<2.10,>=2.9->tensorflow==2.9.0) (3.4.3)\n",
      "Requirement already satisfied: requests<3,>=2.21.0 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorboard<2.10,>=2.9->tensorflow==2.9.0) (2.29.0)\n",
      "Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorboard<2.10,>=2.9->tensorflow==2.9.0) (0.6.1)\n",
      "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorboard<2.10,>=2.9->tensorflow==2.9.0) (1.8.1)\n",
      "Requirement already satisfied: werkzeug>=1.0.1 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorboard<2.10,>=2.9->tensorflow==2.9.0) (2.2.3)\n",
      "Requirement already satisfied: cachetools<6.0,>=2.0.0 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from google-auth<3,>=1.6.3->tensorboard<2.10,>=2.9->tensorflow==2.9.0) (5.3.2)\n",
      "Requirement already satisfied: pyasn1-modules>=0.2.1 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from google-auth<3,>=1.6.3->tensorboard<2.10,>=2.9->tensorflow==2.9.0) (0.3.0)\n",
      "Requirement already satisfied: rsa<5,>=3.1.4 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from google-auth<3,>=1.6.3->tensorboard<2.10,>=2.9->tensorflow==2.9.0) (4.9)\n",
      "Requirement already satisfied: requests-oauthlib>=0.7.0 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.10,>=2.9->tensorflow==2.9.0) (1.3.1)\n",
      "Requirement already satisfied: importlib-metadata>=4.4 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from markdown>=2.6.8->tensorboard<2.10,>=2.9->tensorflow==2.9.0) (4.11.4)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.10,>=2.9->tensorflow==2.9.0) (3.1.0)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.10,>=2.9->tensorflow==2.9.0) (3.4)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.10,>=2.9->tensorflow==2.9.0) (1.26.15)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.10,>=2.9->tensorflow==2.9.0) (2023.5.7)\n",
      "Requirement already satisfied: MarkupSafe>=2.1.1 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from werkzeug>=1.0.1->tensorboard<2.10,>=2.9->tensorflow==2.9.0) (2.1.1)\n",
      "Requirement already satisfied: zipp>=0.5 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard<2.10,>=2.9->tensorflow==2.9.0) (3.15.0)\n",
      "Requirement already satisfied: pyasn1<0.6.0,>=0.4.6 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard<2.10,>=2.9->tensorflow==2.9.0) (0.5.0)\n",
      "Requirement already satisfied: oauthlib>=3.0.0 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.10,>=2.9->tensorflow==2.9.0) (3.2.2)\n",
      "Requirement already satisfied: scikit-learn in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (0.20.1)\n",
      "Collecting scikit-learn\n",
      "  Downloading scikit_learn-1.0.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (24.8 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m24.8/24.8 MB\u001b[0m \u001b[31m37.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: numpy in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (1.21.6)\n",
      "Requirement already satisfied: scipy>=1.1.0 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from scikit-learn) (1.7.3)\n",
      "Collecting joblib>=0.11 (from scikit-learn)\n",
      "  Downloading joblib-1.3.2-py3-none-any.whl (302 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m302.2/302.2 kB\u001b[0m \u001b[31m39.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting threadpoolctl>=2.0.0 (from scikit-learn)\n",
      "  Downloading threadpoolctl-3.1.0-py3-none-any.whl (14 kB)\n",
      "Installing collected packages: threadpoolctl, joblib, scikit-learn\n",
      "  Attempting uninstall: scikit-learn\n",
      "    Found existing installation: scikit-learn 0.20.1\n",
      "    Uninstalling scikit-learn-0.20.1:\n",
      "      Successfully uninstalled scikit-learn-0.20.1\n",
      "Successfully installed joblib-1.3.2 scikit-learn-1.0.2 threadpoolctl-3.1.0\n"
     ]
    }
   ],
   "source": [
    "!pip install grpcio==1.24.3\n",
    "!pip install tensorflow==2.9.0\n",
    "!pip install --upgrade scikit-learn numpy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Restart kernel for latest version of TensorFlow to be activated, then continue to the next step.**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For us to utilize Logistic Regression in TensorFlow, we first need to import the required libraries. To do so, you can run the code cell below.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import time\n",
    "from sklearn.datasets import load_iris\n",
    "from sklearn.model_selection import train_test_split\n",
    "import matplotlib.pyplot as plt\n",
    "TF_ENABLE_ONEDNN_OPTS=0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "if not tf.__version__ == '2.9.0':\n",
    "    print(tf.__version__)\n",
    "    raise ValueError('please upgrade to TensorFlow 2.9.0, or restart your Kernel (Kernel->Restart & Clear Output)')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "IMPORTANT! => Please restart the kernel by clicking on \"Kernel\"->\"Restart and Clear Outout\" and wait until all output disapears. Then your changes will be picked up.\n",
    "\n",
    "After you have restarted the kernel, re-run the previous two cells to import the libraries properly and then continue to the next section.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will load the dataset we are going to use. In this case, we are utilizing the <code>iris</code> dataset, which is inbuilt -- so there's no need to do any preprocessing and we can jump right into manipulating it. We separate the dataset into <i>xs</i> and <i>ys</i>, and then into training <i>xs</i> and <i>ys</i> and testing <i>xs</i> and <i>ys</i>, (pseudo)randomly.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Understanding the Data</h3>\n",
    "\n",
    "<h4><code>Iris Dataset</code>:</h4>\n",
    "This dataset was introduced by British Statistician and Biologist Ronald Fisher, it consists of 50 samples from each of three species of Iris (Iris setosa, Iris virginica and Iris versicolor). In total it has 150 records under five attributes - petal length, petal width, sepal length, sepal width and species. <a href=\"https://archive.ics.uci.edu/ml/datasets/iris\">Dataset source</a>\n",
    "\n",
    "Attributes\n",
    "Independent Variable\n",
    "\n",
    "<ul>\n",
    "    <li>petal length</li>\n",
    "    <li>petal width</li>\n",
    "    <li>sepal length</li>\n",
    "    <li>sepal width</li>\n",
    "</ul>\n",
    "Dependent Variable\n",
    "<ul>    \n",
    "    <li>Species\n",
    "        <ul>\n",
    "            <li>Iris setosa</li>\n",
    "            <li>Iris virginica</li>\n",
    "            <li>Iris versicolor</li>\n",
    "        </ul>\n",
    "    </li>\n",
    "</ul>\n",
    "<br>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "iris = load_iris()\n",
    "iris_X, iris_y = iris.data[:-1,:], iris.target[:-1]\n",
    "iris_y= pd.get_dummies(iris_y).values\n",
    "trainX, testX, trainY, testY = train_test_split(iris_X, iris_y, test_size=0.33, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we define x and y. These variables will hold our iris data (both the features and label matrices) We also need to give them shapes which correspond to the shape of our data. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numFeatures is :  4\n",
      "numLabels is :  3\n"
     ]
    }
   ],
   "source": [
    "# numFeatures is the number of features in our input data.\n",
    "# In the iris dataset, this number is '4'.\n",
    "numFeatures = trainX.shape[1]\n",
    "print('numFeatures is : ', numFeatures )\n",
    "# numLabels is the number of classes our data points can be in.\n",
    "# In the iris dataset, this number is '3'.\n",
    "numLabels = trainY.shape[1]\n",
    "print('numLabels is : ', numLabels )\n",
    "\n",
    "#X = tf.Variable( np.identity(numFeatures), tf.TensorShape(numFeatures),dtype='float32') # Iris has 4 features, so X is a tensor to hold our data.\n",
    "#yGold = tf.Variable(np.array([1,1,1]),shape=tf.TensorShape(numLabels),dtype='float32') # This will be our correct answers matrix for 3 classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "# Set device to CPU\n",
    "tf.config.set_visible_devices([], 'GPU')\n",
    "\n",
    "# Define and work with your TensorFlow constants as before\n",
    "\n",
    "trainX = tf.constant(trainX, dtype='float32')\n",
    "trainY = tf.constant(trainY, dtype='float32')\n",
    "testX = tf.constant(testX, dtype='float32')\n",
    "testY = tf.constant(testY, dtype='float32')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Set model weights and bias</h3>\n",
    "\n",
    "Much like Linear Regression, we need a shared variable weight matrix for Logistic Regression. We initialize both <code>W</code> and <code>b</code> as tensors full of zeros. Since we are going to learn <code>W</code> and <code>b</code>, their initial value does not matter too much. These variables are the objects which define the structure of our regression model, and we can save them after they have been trained so we can reuse them later.\n",
    "\n",
    "We define two TensorFlow variables as our parameters. These variables will hold the weights and biases of our logistic regression and they will be continually updated during training. \n",
    "\n",
    "Notice that <code>W</code> has a shape of [4, 3] because we want to multiply the 4-dimensional input vectors by it to produce 3-dimensional vectors of evidence for the difference classes. <code>b</code> has a shape of [3] so we can add it to the output. TensorFlow variables need to be initialized with values, e.g. with zeros.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "W = tf.Variable(tf.zeros([4, 3]))  # 4-dimensional input and  3 classes\n",
    "b = tf.Variable(tf.zeros([3])) # 3-dimensional output [0,0,1],[0,1,0],[1,0,0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#Randomly sample from a normal distribution with standard deviation .01\n",
    "\n",
    "weights = tf.Variable(tf.random.normal([numFeatures,numLabels],\n",
    "                                       mean=0.,\n",
    "                                       stddev=0.01,\n",
    "                                       name=\"weights\"),dtype='float32')\n",
    "\n",
    "\n",
    "bias = tf.Variable(tf.random.normal([1,numLabels],\n",
    "                                    mean=0.,\n",
    "                                    stddev=0.01,\n",
    "                                    name=\"bias\"))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Logistic Regression model</h3>\n",
    "\n",
    "We now define our operations in order to properly run the Logistic Regression. Logistic regression is typically thought of as a single equation:\n",
    "\n",
    "$$\n",
    "ŷ =sigmoid(WX+b)\n",
    "$$\n",
    "\n",
    "However, for the sake of clarity, we can have it broken into its three main components: \n",
    "\n",
    "-   a weight times features matrix multiplication operation, \n",
    "-   a summation of the weighted features and a bias term, \n",
    "-   and finally the application of a sigmoid function. \n",
    "\n",
    "As such, you will find these components defined as three separate operations below.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Three-component breakdown of the Logistic Regression equation.\n",
    "# Note that these feed into each other.\n",
    "def logistic_regression(x):\n",
    "    apply_weights_OP = tf.matmul(x, weights, name=\"apply_weights\")\n",
    "    add_bias_OP = tf.add(apply_weights_OP, bias, name=\"add_bias\") \n",
    "    activation_OP = tf.nn.sigmoid(add_bias_OP, name=\"activation\")\n",
    "    return activation_OP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we have seen before, the function we are going to use is the <i>logistic function</i> $(\\frac{1}{1+e^{-Wx}})$, which is fed the input data after applying weights and bias. In TensorFlow, this function is implemented as the <code>nn.sigmoid</code> function. Effectively, this fits the weighted input with bias into a 0-100 percent curve, which is the probability function we want.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ref3\"></a>\n",
    "\n",
    "<h2>Training</h2>\n",
    "\n",
    "The learning algorithm is how we search for the best weight vector (${\\\\\\\\bf w}$). This search is an optimization problem looking for the hypothesis that optimizes an error/cost measure.\n",
    "\n",
    "<b>What tells us that our model is bad?</b>  \n",
    "The Cost or Loss of the model, so what we want is to minimize that. \n",
    "\n",
    "<h3>Cost function</h3>\n",
    "Before defining our cost function, we need to define how long we are going to train and how should we define the learning rate.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Number of Epochs in our training\n",
    "numEpochs = 700\n",
    "\n",
    "# Defining our learning rate iterations (decay)\n",
    "learningRate = tf.keras.optimizers.schedules.ExponentialDecay(initial_learning_rate=0.0008,\n",
    "                                          decay_steps=trainX.shape[0],\n",
    "                                          decay_rate= 0.95,\n",
    "                                          staircase=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>What is the cost function in our model?</b>  \n",
    "The cost function we are going to utilize is the Squared Mean Error loss function.\n",
    "\n",
    "<b>How to minimize the cost function?</b>  \n",
    "We can't use <b>least-squares linear regression</b> here, so we will use <a href=\"http://en.wikipedia.org/wiki/Gradient_descent\">gradient descent</a> instead. Specifically, we will use batch gradient descent which calculates the gradient from all data points in the data set.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#Defining our cost function - Squared Mean Error\n",
    "loss_object = tf.keras.losses.MeanSquaredLogarithmicError()\n",
    "optimizer = tf.keras.optimizers.SGD(learningRate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also want some additional operations to keep track of our model's efficiency over time. We can do this like so:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Accuracy metric.\n",
    "def accuracy(y_pred, y_true):\n",
    "# Predicted class is the index of the highest score in prediction vector (i.e. argmax).\n",
    "\n",
    "    correct_prediction = tf.equal(tf.argmax(y_pred, 1), tf.argmax(y_true, 1))\n",
    "\n",
    "    return tf.reduce_mean(tf.cast(correct_prediction, tf.float32))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first wrap computation inside a GradientTape for automatic differentiation. Then we compute gradients and update W and b.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Optimization process. \n",
    "\n",
    "def run_optimization(x, y):\n",
    "    with tf.GradientTape() as g:\n",
    "        pred = logistic_regression(x)\n",
    "        loss = loss_object(pred, y)\n",
    "    gradients = g.gradient(loss, [weights, bias])\n",
    "    optimizer.apply_gradients(zip(gradients, [weights, bias]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we move on to actually running our operations. We will start with the operations involved in the prediction phase (i.e. the logistic regression itself).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can define and run the actual training loop, like this:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 0, training accuracy 0.32, loss 0.140492, change in loss 0.140492\n",
      "step 10, training accuracy 0.32, loss 0.139822, change in loss 0.000669733\n",
      "step 20, training accuracy 0.32, loss 0.139161, change in loss 0.00066106\n",
      "step 30, training accuracy 0.32, loss 0.138509, change in loss 0.000652403\n",
      "step 40, training accuracy 0.32, loss 0.137865, change in loss 0.000643641\n",
      "step 50, training accuracy 0.32, loss 0.13723, change in loss 0.000634864\n",
      "step 60, training accuracy 0.32, loss 0.136604, change in loss 0.000626102\n",
      "step 70, training accuracy 0.32, loss 0.135987, change in loss 0.000617266\n",
      "step 80, training accuracy 0.32, loss 0.135378, change in loss 0.000608459\n",
      "step 90, training accuracy 0.32, loss 0.134779, change in loss 0.000599623\n",
      "step 100, training accuracy 0.32, loss 0.134194, change in loss 0.000584945\n",
      "step 110, training accuracy 0.32, loss 0.133641, change in loss 0.000553176\n",
      "step 120, training accuracy 0.32, loss 0.133095, change in loss 0.000545233\n",
      "step 130, training accuracy 0.32, loss 0.132558, change in loss 0.000537336\n",
      "step 140, training accuracy 0.32, loss 0.132029, change in loss 0.000529438\n",
      "step 150, training accuracy 0.32, loss 0.131507, change in loss 0.000521585\n",
      "step 160, training accuracy 0.32, loss 0.130993, change in loss 0.000513747\n",
      "step 170, training accuracy 0.32, loss 0.130487, change in loss 0.000505969\n",
      "step 180, training accuracy 0.32, loss 0.129989, change in loss 0.000498235\n",
      "step 190, training accuracy 0.32, loss 0.129498, change in loss 0.000490516\n",
      "step 200, training accuracy 0.32, loss 0.129023, change in loss 0.00047569\n",
      "step 210, training accuracy 0.32, loss 0.128571, change in loss 0.000451818\n",
      "step 220, training accuracy 0.32, loss 0.128126, change in loss 0.000445023\n",
      "step 230, training accuracy 0.32, loss 0.127688, change in loss 0.000438273\n",
      "step 240, training accuracy 0.32, loss 0.127256, change in loss 0.000431553\n",
      "step 250, training accuracy 0.32, loss 0.126831, change in loss 0.000424951\n",
      "step 260, training accuracy 0.32, loss 0.126413, change in loss 0.000418365\n",
      "step 270, training accuracy 0.32, loss 0.126001, change in loss 0.000411868\n",
      "step 280, training accuracy 0.32, loss 0.125596, change in loss 0.000405401\n",
      "step 290, training accuracy 0.32, loss 0.125197, change in loss 0.000399008\n",
      "step 300, training accuracy 0.32, loss 0.124812, change in loss 0.000384919\n",
      "step 310, training accuracy 0.32, loss 0.124444, change in loss 0.000367396\n",
      "step 320, training accuracy 0.32, loss 0.124082, change in loss 0.000361823\n",
      "step 330, training accuracy 0.32, loss 0.123726, change in loss 0.000356302\n",
      "step 340, training accuracy 0.32, loss 0.123375, change in loss 0.000350848\n",
      "step 350, training accuracy 0.32, loss 0.12303, change in loss 0.000345461\n",
      "step 360, training accuracy 0.32, loss 0.12269, change in loss 0.000340119\n",
      "step 370, training accuracy 0.32, loss 0.122355, change in loss 0.000334874\n",
      "step 380, training accuracy 0.32, loss 0.122025, change in loss 0.000329666\n",
      "step 390, training accuracy 0.32, loss 0.121701, change in loss 0.000324525\n",
      "step 400, training accuracy 0.32, loss 0.121389, change in loss 0.000311539\n",
      "step 410, training accuracy 0.32, loss 0.12109, change in loss 0.000298955\n",
      "step 420, training accuracy 0.32, loss 0.120796, change in loss 0.000294507\n",
      "step 430, training accuracy 0.32, loss 0.120505, change in loss 0.000290118\n",
      "step 440, training accuracy 0.32, loss 0.12022, change in loss 0.000285752\n",
      "step 450, training accuracy 0.32, loss 0.119938, change in loss 0.000281483\n",
      "step 460, training accuracy 0.32, loss 0.119661, change in loss 0.000277244\n",
      "step 470, training accuracy 0.32, loss 0.119388, change in loss 0.000273064\n",
      "step 480, training accuracy 0.32, loss 0.119119, change in loss 0.000268966\n",
      "step 490, training accuracy 0.32, loss 0.118854, change in loss 0.000264898\n",
      "step 500, training accuracy 0.32, loss 0.118601, change in loss 0.000253133\n",
      "step 510, training accuracy 0.32, loss 0.118357, change in loss 0.000244297\n",
      "step 520, training accuracy 0.32, loss 0.118116, change in loss 0.00024078\n",
      "step 530, training accuracy 0.32, loss 0.117879, change in loss 0.000237338\n",
      "step 540, training accuracy 0.32, loss 0.117645, change in loss 0.000233904\n",
      "step 550, training accuracy 0.32, loss 0.117414, change in loss 0.000230536\n",
      "step 560, training accuracy 0.32, loss 0.117187, change in loss 0.000227228\n",
      "step 570, training accuracy 0.32, loss 0.116963, change in loss 0.000223942\n",
      "step 580, training accuracy 0.32, loss 0.116742, change in loss 0.000220723\n",
      "step 590, training accuracy 0.32, loss 0.116525, change in loss 0.000217535\n",
      "step 600, training accuracy 0.32, loss 0.116318, change in loss 0.00020694\n",
      "step 610, training accuracy 0.32, loss 0.116117, change in loss 0.00020092\n",
      "step 620, training accuracy 0.32, loss 0.115919, change in loss 0.000198156\n",
      "step 630, training accuracy 0.32, loss 0.115723, change in loss 0.000195466\n",
      "step 640, training accuracy 0.32, loss 0.11553, change in loss 0.000192791\n",
      "step 650, training accuracy 0.32, loss 0.11534, change in loss 0.000190146\n",
      "step 660, training accuracy 0.32, loss 0.115153, change in loss 0.000187539\n",
      "step 670, training accuracy 0.32, loss 0.114968, change in loss 0.000185005\n",
      "step 680, training accuracy 0.32, loss 0.114785, change in loss 0.000182465\n",
      "step 690, training accuracy 0.32, loss 0.114605, change in loss 0.000179976\n",
      "final accuracy on test set: 0.32\n"
     ]
    }
   ],
   "source": [
    "# Initialize reporting variables\n",
    "display_step = 10\n",
    "epoch_values = []\n",
    "accuracy_values = []\n",
    "loss_values = []\n",
    "loss = 0\n",
    "diff = 1\n",
    "# Training epochs\n",
    "for i in range(numEpochs):\n",
    "    if i > 1 and diff < .0001:\n",
    "        print(\"change in loss %g; convergence.\"%diff)\n",
    "        break\n",
    "    else:\n",
    "        # Run training step\n",
    "        run_optimization(trainX, trainY)\n",
    "        \n",
    "        # Report occasional stats\n",
    "        if i % display_step == 0:\n",
    "            # Add epoch to epoch_values\n",
    "            epoch_values.append(i)\n",
    "            \n",
    "            pred = logistic_regression(testX)\n",
    "\n",
    "            newLoss = loss_object(pred, testY)\n",
    "            # Add loss to live graphing variable\n",
    "            loss_values.append(newLoss)\n",
    "            \n",
    "            # Generate accuracy stats on test data\n",
    "            acc = accuracy(pred, testY)\n",
    "            accuracy_values.append(acc)\n",
    "            \n",
    "    \n",
    "            # Re-assign values for variables\n",
    "            diff = abs(newLoss - loss)\n",
    "            loss = newLoss\n",
    "\n",
    "            #generate print statements\n",
    "            print(\"step %d, training accuracy %g, loss %g, change in loss %g\"%(i, acc, newLoss, diff))\n",
    "\n",
    "        \n",
    "\n",
    "          \n",
    "\n",
    "# How well do we perform on held-out test data?\n",
    "print(\"final accuracy on test set: %s\" %acc.numpy())\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Why don't we plot the loss to see how it behaves?</b>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkkAAAGwCAYAAAC99fF4AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAA9hAAAPYQGoP6dpAABMa0lEQVR4nO3deVhU9eIG8PcMMMO+CbLI6pKIIqsiKGqlJJm5lkupaWWUlWTeX3Vp8WqF17rpLZMCzUJzyTKzm0toLiQqioyi4hayiCCCsusAw/n9QU6RaMoyh5l5P89znkfOnJnznvOYvJ3zne8RRFEUQURERERNyKQOQERERNQRsSQRERERNYMliYiIiKgZLElEREREzWBJIiIiImoGSxIRERFRM1iSiIiIiJphLHUAXdXQ0IBLly7BysoKgiBIHYeIiIjugiiKqKyshKurK2SyO18rYklqoUuXLsHd3V3qGERERNQC+fn5cHNzu+M2LEktZGVlBaDxJFtbW0uchoiIiO5GRUUF3N3dNb/H74QlqYVu3mKztrZmSSIiItIxdzNUhgO3iYiIiJrBkkRERETUDJYkIiIiomawJBERERE1gyWJiIiIqBksSURERETNYEkiIiIiagZLEhEREVEzWJKIiIiImsGSRERERNQMliQiIiKiZrAkERERETWDJakDSrtwFeXX66SOQUREZNBYkjqYNQdzMSnhAP7v22MQRVHqOERERAaLJamD6etmA2OZDDtOXsaq/TlSxyEiIjJYLEkdTF83W8SO7AUAiNuWBWV+mbSBiIiIDBRLUgc0LcwTD/s5o04tYvbXR1Few/FJRERE2iZ5SVq+fDm8vb1hamqK4OBgpKSk3HbbwsJCTJkyBT179oRMJkNMTMwdP3v9+vUQBAFjxoxp1X61TRAELBrfFx725igou455HJ9ERESkdZKWpA0bNiAmJgaxsbHIyMhAREQEoqKikJeX1+z2KpUKjo6OiI2Nhb+//x0/Ozc3F/PmzUNERESr9ysFa1MTLH8iCHIjGZJPXcbKXy9IHYmIiMigCKKElyhCQ0MRFBSE+Ph4zbpevXphzJgxiIuLu+N7hw4dioCAACxduvSW19RqNYYMGYIZM2YgJSUFZWVl2Lx5c6v2q1KpoFKpND9XVFTA3d0d5eXlsLa2vssjvnerD+TgrR9OwlgmYGN0GAI97NptX0RERPquoqICNjY2d/X7W7IrSbW1tUhPT0dkZGST9ZGRkUhNTW3VZy9YsACOjo54+umn22y/cXFxsLGx0Szu7u6tyni3nhzgiZF+LqhvEPHi2gyU1dRqZb9ERESGTrKSVFJSArVaDScnpybrnZycUFRU1OLP3b9/P1auXInExMQ23e8bb7yB8vJyzZKfn9/ijPeicXySHzw7/T4+aSPHJxEREWmD5AO3BUFo8rMoiresu1uVlZV48sknkZiYCAcHhzbdr0KhgLW1dZNFW6xMTfDplCDIjWXYmVWMxJRsre2biIjIUBlLtWMHBwcYGRndcvWmuLj4lqs8d+u3335DTk4ORo0apVnX0NAAADA2NsaZM2fg7u7e5vvVhj5dbPDOKF/Efn8C/95+BoEedujnZS91LCIiIr0l2ZUkuVyO4OBgJCcnN1mfnJyM8PDwFn2mj48PMjMzoVQqNcujjz6K+++/H0qlEu7u7u2yX22Z0t8DowNcoW4Q8eLaoyipUv39m4iIiKhFJLuSBABz587F1KlTERISgrCwMCQkJCAvLw/R0dEAGscBFRQUICkpSfMepVIJAKiqqsKVK1egVCohl8vh6+sLU1NT9OnTp8k+bG1tAaDJ+r/bb0clCALeH+uHk5cqcL64CjHrlfhqZn8YyVp2e5KIiIhuT9KSNHHiRJSWlmLBggUoLCxEnz59sHXrVnh6egJonDzyr3MXBQYGav6cnp6OtWvXwtPTEzk5OW22347MQmGM+CeC8Oiy/fj1fAk+3nUOrwy/T+pYREREekfSeZJ02b3Ms9Aevs+4iFc2HIMgAF/N6I/B9zlqPQMREZGu0Yl5kqh1xga6YXJ/D4giELNBicLy61JHIiIi0issSTrsnVG+6O1qjavVtXhxbQbq1A1SRyIiItIbLEk6zNTECMufCIKVqTHSc69h8fbTUkciIiLSGyxJOs6zkwU+mND4sN/ElAvYfqJQ4kRERET6gSVJD4zo44xZg7sCAOZtPI7sK1USJyIiItJ9LEl64v8e6on+3vaoUtXj+TVHUVNbL3UkIiIincaSpCeMjWRYNjkQDpYKnLlciTe/P8EH4RIREbUCS5Ie6Wxtik+nBMJIJmBTRgHWpuX9/ZuIiIioWSxJeia0ayf830M9AQD/2nIKx/LLpA1ERESko1iS9NCswV0R6euEWnUDXvj6KK5V10odiYiISOewJOkhQRDw4eP+8OpkjoKy64jZoERDA8cnERER3QuWJD1lbWqC+CeDYWoiw96zV/DfXeekjkRERKRTWJL0WC8Xa7w3xg8A8PEv57D7dLHEiYiIiHQHS5KeGx/shicHND4Id876DOSV1kgdiYiISCewJBmAtx/pjUAPW1TcqEf0mnRcr1VLHYmIiKjDY0kyAHJjGZY/EYROFnKcKqxA7OZMTjRJRET0N1iSDISLjRk+mRIImQBsOlqArw9xokkiIqI7YUkyIOHdHPDaCB8AwL9+PImMvGsSJyIiIuq4WJIMzKzBXTGitzPq1CJe+PooSqpUUkciIiLqkFiSDIwgCPjgsb7o5miBwvIbeGltBurVDVLHIiIi6nBYkgyQlakJPp8aDAu5EQ5kl+Lf209LHYmIiKjDYUkyUN07W+HDx/wBAIkpF7Dl2CWJExEREXUsLEkGLMrPBc8P7QYA+L9vjyGrsELiRERERB0HS5KBmxfZExE9HHCjrgHPrU5HWU2t1JGIiIg6BJYkA2ckE/DxpEC42Zkh72oN5qxXQt3AiSaJiIhYkgh2FnJ8PjUYpiYy7D17BUuSz0odiYiISHIsSQQA6O1qg0Xj+gIAlu0+jx0niyROREREJC2WJNIYE9gFMwZ6AQBe/eYYzhdXShuIiIhIQixJ1MQ/H+6FUG97VKnqMSspHRU36qSOREREJAmWJGrCxEiGT58IgquNKbJLqhGzXokGDuQmIiIDxJJEt3CwVODzqSFQGMvwy+liLNnJgdxERGR4WJKoWX5uNogb5wcA+OSX89h+olDiRERERNrFkkS3NS7IDU8P8gYAzP3mGM4UcSA3EREZDpYkuqM3onwQ3q0TamrVmLX6CMprOJCbiIgMA0sS3ZGxkQzLpgTBzc4MuaU1eGl9BmfkJiIig8CSRH/L3kKOhKkhMDWRYd/ZK1i847TUkYiIiNodSxLdFV9Xayye4A8A+HxvNn5QFkiciIiIqH2xJNFde9TfFdFDugEA/u/b48i8WC5xIiIiovbDkkT35B8P9cQDPp2hqm/ArNVHUFx5Q+pIRERE7YIlie6JkUzA0kkB6OZogcLyG3h+zVGo6tVSxyIiImpzLEl0z6xNTZA4LQRWpsZIz72GtzefhCjyG29ERKRfWJKoRbo6WuKTyYGQCcCGI/lIOpArdSQiIqI2xZJELTa0Z2e8HuUDAFjwv1NIPV8icSIiIqK2w5JErfJsRFeMDewCdYOIF9YeRV5pjdSRiIiI2gRLErWKIAiIG+cHfzcblNXU4Zmkw6i8wUeXEBGR7mNJolYzNTHC51ND0NlKgbOXqxCzXslHlxARkc5jSaI24WxjioRpIVAYy7DrdDE+2HFG6khEREStwpJEbSbA3RaLJ/QFAHy29zdsOnpR4kREREQtx5JEbWp0QBfMvr/x0SWvf5eJo3nXJE5ERETUMixJ1OZeHd4Tw32dUKtuwKykdFwquy51JCIionvGkkRtTiYTsHRiAHycrVBSpcKzSUdQU1svdSwiIqJ7wpJE7cJCYYwV00PQyUKOk5cqMG/jMTTwG29ERKRDWJKo3bjZmeOzqcEwMRKwNbMIS3eelToSERHRXWNJonbVz8se74/1AwB8/Mt5/KAskDgRERHR3ZG8JC1fvhze3t4wNTVFcHAwUlJSbrttYWEhpkyZgp49e0ImkyEmJuaWbTZt2oSQkBDY2trCwsICAQEBWL16dZNt5s+fD0EQmizOzs5tfWj0u8dC3PHckK4AgH98exzpufzGGxERdXySlqQNGzYgJiYGsbGxyMjIQEREBKKiopCXl9fs9iqVCo6OjoiNjYW/v3+z29jb2yM2NhYHDhzA8ePHMWPGDMyYMQM7duxosl3v3r1RWFioWTIzM9v8+OgPrz3k0/iNt/oGPLf6CC5e4zPeiIioYxNEUZRsNG1oaCiCgoIQHx+vWderVy+MGTMGcXFxd3zv0KFDERAQgKVLl/7tfoKCgjBy5EgsXLgQQOOVpM2bN0OpVN51VpVKBZVKpfm5oqIC7u7uKC8vh7W19V1/jiGrVtVjwmcHkFVYAR9nK3z7fDgsFcZSxyIiIgNSUVEBGxubu/r9LdmVpNraWqSnpyMyMrLJ+sjISKSmprbJPkRRxK5du3DmzBkMHjy4yWvnzp2Dq6srvL29MWnSJGRnZ9/xs+Li4mBjY6NZ3N3d2ySjIbFQGGPl9BA4WilwuqgSc9Zl8BlvRETUYUlWkkpKSqBWq+Hk5NRkvZOTE4qKilr12eXl5bC0tIRcLsfIkSPxySefYPjw4ZrXQ0NDkZSUhB07diAxMRFFRUUIDw9HaWnpbT/zjTfeQHl5uWbJz89vVUZD5WprhsQ/PeNt0bYsqSMRERE1S/J7HYIgNPlZFMVb1t0rKysrKJVKVFVVYdeuXZg7dy66du2KoUOHAgCioqI02/r5+SEsLAzdunXDV199hblz5zb7mQqFAgqFolW5qFGAuy0+fMwfL63LQGLKBXRztMSk/h5SxyIiImpCspLk4OAAIyOjW64aFRcX33J16V7JZDJ0794dABAQEICsrCzExcVpStJfWVhYwM/PD+fOnWvVfunujfJ3RfaVaizZeRaxm0+gi50ZIno4Sh2LiIhIQ7LbbXK5HMHBwUhOTm6yPjk5GeHh4W26L1EUmwy6/iuVSoWsrCy4uLi06X7pzl5+sDvGBnaBukHEC2uO4kxRpdSRiIiINCS93TZ37lxMnToVISEhCAsLQ0JCAvLy8hAdHQ2gcRxQQUEBkpKSNO+5+Y20qqoqXLlyBUqlEnK5HL6+vgAaB1iHhISgW7duqK2txdatW5GUlNTkG3Tz5s3DqFGj4OHhgeLiYrz77ruoqKjA9OnTtXfwBEEQsGi8HwrKriPtwlXM/PIwvp8djs5WplJHIyIikrYkTZw4EaWlpViwYAEKCwvRp08fbN26FZ6engAaJ4/865xJgYGBmj+np6dj7dq18PT0RE5ODgCguroaL7zwAi5evAgzMzP4+PhgzZo1mDhxouZ9Fy9exOTJk1FSUgJHR0cMGDAABw8e1OyXtEdhbISEqcEYtzwV2SXVeOarI1g/awDM5ZIPlyMiIgMn6TxJuuxe5lmgv5dTUo1x8am4Wl2L4b5O+OzJYBjJWjeAn4iI6K90Yp4koj/zcrBAwtRgyI1lSD51GXFbOTUAERFJiyWJOowQL3t8+Fjj42ZW/HoBSQdypA1EREQGjSWJOpRH/V3xj4d6AgDmbzmJ7SdaN7EoERFRS7EkUYfzwtBumNTPHQ0i8PL6DBzMvv1M6ERERO2FJYk6HEEQ8O6YPoj0dUJtfQOe/eoITl2qkDoWEREZGJYk6pCMjWT4eHIg+nvZo1JVj+mr0pBXWiN1LCIiMiAsSdRhmZoYIXF6CHycrXClUoVpXxzClcrbz5xORETUlliSqEOzMTPBVzP7w83ODDmlNXhqVRoqb9RJHYuIiAwASxJ1eE7Wpkia2R/2FnKcvFSB51anQ1WvljoWERHpOZYk0gldHS3x5Yx+sJAbIfW3UsSsV0LdwMniiYio/bAkkc7o62aLz6eGwMRIwLYTRXhzcyb4VB0iImovLEmkUwb1cMB/JwVCEIB1afn4YMcZqSMREZGeYkkinfOwnwveH+sHAFi+5zck7suWOBEREekjliTSSZP7e+D/RjQ+vuS9rVnYeCRf4kRERKRvWJJIZz0/pBuejfAGALy+KRM/n+Rz3oiIqO2wJJHOEgQB/3y4Fx4LdoO6QcSL6zJw4Dc+542IiNoGSxLpNEEQEDfO74/nvCUdQebFcqljERGRHmBJIp138zlvYV07oUpVj2lfHMK5y5VSxyIiIh3HkkR64eZz3vzdbHCtpg5PrjyE/Kt8IC4REbUcSxLpDUuFMb6c0R89OlvicoUKT648hOKKG1LHIiIiHcWSRHrFzkKONc+Ewt3eDLmlNZi6Mg1lNbVSxyIiIh3EkkR6x8naFF8/PQCdrRQ4c7kS01cdRpWqXupYRESkY1iSSC95dDLHmmdCYWtugmP5ZZiVdAQ36tRSxyIiIh3CkkR66z4nK3w1oz8s5EZI/a0UL67NQJ26QepYRESkI1iSSK/5u9sicXoIFMYy7My6jFe/OQZ1gyh1LCIi0gEsSaT3wrs5IP7JIBjLBGw5dgmx32dCFFmUiIjozliSyCA84OOE/04KhEwA1h/Ox4L/nWJRIiKiO2JJIoMxsq8LFk/wBwCs2p+Dj5LPSpyIiIg6MpYkMigTgt2wYHRvAMAnv5xH/J7fJE5EREQdFUsSGZxpYV54bYQPAODf208j6UCOtIGIiKhDYkkig/T80G546YHuAIC3fziJb47kS5yIiIg6GpYkMlhzh9+HmQO9AQCvfXccPygLJE5EREQdCUsSGSxBEPDWI70wJdQDogjM/eYYtp8olDoWERF1ECxJZNAEQcC7o/tgfJAb1A0iXlqXgd2ni6WORUREHQBLEhk8mUzA4gl98UhfF9SpRTy3Jh2/niuROhYREUmMJYkIgJFMwJKJAYj0dUJtfQOeSTqMQ9mlUsciIiIJsSQR/c7ESIZPpgRiaE9H3KhrwMwvD+No3jWpYxERkURYkoj+RGFshM+eDEZ4t06orlVj+hdpOH6xTOpYREQkAZYkor8wNTHCiukh6O9lj8ob9Zi6Mg0nCsqljkVERFrGkkTUDHO5Mb6Y0Q9BHrYov16HqSsP4XRRhdSxiIhIi1iSiG7DUmGML2f2h7+7La7V1OGJxEM4d7lS6lhERKQlLElEd2BtaoKkmf3Rp4s1SqtrMTnxEM4XV0kdi4iItIAliehv2JiZYM3ToejlYo2SKhWmJB7EhZJqqWMREVE7Y0kiugu25nJ8/UwoejpZobiysSjllrIoERHpM5YkortkbyHHmmdC0b2zJQrLb2BywkHkldZIHYuIiNoJSxLRPXC0UmDtM6Ho6miBS+U3MDnxIPKvsigREekjliSie9TZ2hTrnx2Aro4WKCi7jkkJLEpERPqIJYmoBTRFyYFFiYhIX7EkEbVQZ2tTrJv1R1GanHgQF6+xKBER6QuWJKJWcPq9KHk7WODitcYrSixKRET6gSWJqJWcrE2x7tkB8OpkzqJERKRHWJKI2oCzTeMVpZtFaeLnHKNERKTrWJKI2oiLjRnWzwprMpib8ygREekuyUvS8uXL4e3tDVNTUwQHByMlJeW22xYWFmLKlCno2bMnZDIZYmJibtlm06ZNCAkJga2tLSwsLBAQEIDVq1e3ar9Ed+vmFaU/pgc4wJm5iYh0lKQlacOGDYiJiUFsbCwyMjIQERGBqKgo5OXlNbu9SqWCo6MjYmNj4e/v3+w29vb2iI2NxYEDB3D8+HHMmDEDM2bMwI4dO1q8X6J74fT79ADdfp9wcuLnfNYbEZEuEkRRFKXaeWhoKIKCghAfH69Z16tXL4wZMwZxcXF3fO/QoUMREBCApUuX/u1+goKCMHLkSCxcuLDV+72poqICNjY2KC8vh7W19V29hwzLld+f8XauuApO1gqse3YAujpaSh2LiMig3cvvb8muJNXW1iI9PR2RkZFN1kdGRiI1NbVN9iGKInbt2oUzZ85g8ODBrdqvSqVCRUVFk4XoThytFFg3awB6OlnhcoUKkxIO4nxxldSxiIjoLklWkkpKSqBWq+Hk5NRkvZOTE4qKilr12eXl5bC0tIRcLsfIkSPxySefYPjw4a3ab1xcHGxsbDSLu7t7qzKSYXCwVGDts6HwcbZCcaUKkxIO4ExRpdSxiIjoLkg+cFsQhCY/i6J4y7p7ZWVlBaVSicOHD+O9997D3LlzsWfPnlbt94033kB5eblmyc/Pb1VGMhydLBtvtfV2tUZJVS0mJRzAyUvlUsciIqK/IVlJcnBwgJGR0S1Xb4qLi2+5ynOvZDIZunfvjoCAALz66quYMGGCZqxRS/erUChgbW3dZCG6W3YWcqx9ZgD83WxwraYOUxIP4fjFMqljERHRHUhWkuRyOYKDg5GcnNxkfXJyMsLDw9t0X6IoQqVSaX2/RH9mY26C1c+EItjTDuXX6/BE4iGk516TOhYREd2GsZQ7nzt3LqZOnYqQkBCEhYUhISEBeXl5iI6OBtB4i6ugoABJSUma9yiVSgBAVVUVrly5AqVSCblcDl9fXwCNY4dCQkLQrVs31NbWYuvWrUhKSmryTba/2y9Re7E2NcFXM/tj5peHkXbhKqatPIQvnuqH0K6dpI5GRER/IWlJmjhxIkpLS7FgwQIUFhaiT58+2Lp1Kzw9PQE0Th7517mLAgMDNX9OT0/H2rVr4enpiZycHABAdXU1XnjhBVy8eBFmZmbw8fHBmjVrMHHixLveL1F7slQY48sZ/fBs0hHsP1+Kp1YdxorpIRjY3UHqaERE9CeSzpOkyzhPErXWjTo1nludjr1nr0BuLEP8E0F4sFfrxuMREdGd6cQ8SUSGztTECAnTgjHc1wm19Q14bnU6fjx2SepYRET0O5YkIgkpjI2w/IkgjA5wRX2DiJfXZ2DDYT4eh4ioI2BJIpKYiZEMSx4PwJRQD4gi8Np3mVj56wWpYxERGTyWJKIOQCYT8N6YPnhucFcAwML/ncJ/d54DhwwSEUmHJYmogxAEAa9H+eDV4fcBAJbsPIv3t2axKBERSYQliagDEQQBLz3YA28/0jjvV2LKBfzj2+OoUzdInIyIyPCwJBF1QDMHeWPxhL4wkgn4Nv0ink06gpraeqljEREZFJYkog7q8RB3JEwNhqmJDHvOXMHkxEMorVJJHYuIyGCwJBF1YA/2csLaZwfA1twEx/LLMOGzA8grrZE6FhGRQWBJIurggjzs8G10OLrYmuFCSTXGxafiREG51LGIiPQeSxKRDuje2RKbXgiHj7MVSqpUmJRwEL+eK5E6FhGRXmNJItIRTtam+CY6DGFdO6FKVY8ZX6Zhc0aB1LGIiPQWSxKRDrE2NcGXM/vhkb4uqFOLiNmgxPI95zmXEhFRO2BJItIxCmMjfDwpEM9GeAMAFm8/g7d+OAF1A4sSEVFbalFJys/Px8WLFzU/p6WlISYmBgkJCW0WjIhuTyYTEDvSF28/4gtBANYczMNzq9NxvVYtdTQiIr3RopI0ZcoU7N69GwBQVFSE4cOHIy0tDf/85z+xYMGCNg1IRLc3c5A3lk8JgtxYhp1ZlzE58SDnUiIiaiMtKkknTpxA//79AQDffPMN+vTpg9TUVKxduxZffvllW+Yjor8R5eeCtc+EwtbcBMr8MoyPT0VOSbXUsYiIdF6LSlJdXR0UCgUAYOfOnXj00UcBAD4+PigsLGy7dER0V0K87PFtdDjc7MyQU1qDcfGpOJp3TepYREQ6rUUlqXfv3vjss8+QkpKC5ORkjBgxAgBw6dIldOrUqU0DEtHduTmXkl8XG1ytrsXkhIPYfoL/00JE1FItKkn//ve/8fnnn2Po0KGYPHky/P39AQBbtmzR3IYjIu3rbGWK9bMG4EGfzlDVN+D5r49iRUo2pwggImoBQWzhv55qtRoVFRWws7PTrMvJyYG5uTk6d+7cZgE7qoqKCtjY2KC8vBzW1tZSxyFqol7dgH/9eAqrD+YCAKaHeeLtUb1hJBMkTkZEJK17+f3doitJ169fh0ql0hSk3NxcLF26FGfOnDGIgkTU0RkbybBgdG/882EfAMBXB3Lx3Op01NTWS5yMiEh3tKgkjR49GklJSQCAsrIyhIaG4j//+Q/GjBmD+Pj4Ng1IRC0jCAJmDe6GT/80RcCkhIMorrwhdTQiIp3QopJ09OhRREREAAC+/fZbODk5ITc3F0lJSfj444/bNCARtc7Ivi5Y92wo7MxNcPxiOcZ+moozRZVSxyIi6vBaVJJqampgZWUFAPj5558xbtw4yGQyDBgwALm5uW0akIhaL9jTHt+/MBDeDhYoKLuO8fGp2Hv2itSxiIg6tBaVpO7du2Pz5s3Iz8/Hjh07EBkZCQAoLi7mIGaiDsrLwQKbng9Hf297VKnqMfPLw/j6EP+nhojodlpUkt5++23MmzcPXl5e6N+/P8LCwgA0XlUKDAxs04BE1HbsLORY/XR/jAvsAnWDiNjvT+D9rVlo4MNxiYhu0eIpAIqKilBYWAh/f3/IZI1dKy0tDdbW1vDx8WnTkB0RpwAgXSaKIj7edR5Ldp4FADzU2wlLJwbCTG4kcTIiovZ1L7+/W1ySbrp48SIEQUCXLl1a8zE6hyWJ9MEPygL8Y+Nx1Kob0NfNBiumhaCztanUsYiI2k27z5PU0NCABQsWwMbGBp6envDw8ICtrS0WLlyIhoaGFoUmIu0bHdAFX//pm2+jP92PEwXlUsciIuoQWlSSYmNjsWzZMixatAgZGRk4evQo3n//fXzyySd466232jojEbWjfl722Dx7ILo5WqCw/AYe++wAfj5ZJHUsIiLJteh2m6urKz777DM8+uijTdb/8MMPeOGFF1BQUNBmATsq3m4jfVN+vQ4vrj2KlHMlEATg9RE+mDW4KwSBjzIhIv3R7rfbrl692uzgbB8fH1y9erUlH0lEErMxM8Gqp/rhyQEeEEUgbttpvPbdcdTW8xY6ERmmFpUkf39/LFu27Jb1y5YtQ9++fVsdioikYWwkw8LRfTB/lC9kAvDNkYuYuvIQrlXXSh2NiEjrWnS7be/evRg5ciQ8PDwQFhYGQRCQmpqK/Px8bN26VfPIEn3G222k73afKcZLazNQpaqHh705Vk4PQQ8nK6ljERG1SrvfbhsyZAjOnj2LsWPHoqysDFevXsW4ceNw8uRJrFq1qkWhiahjub9nZ2x6IRzu9mbIu1qDcctTsftMsdSxiIi0ptXzJP3ZsWPHEBQUBLVa3VYf2WHxShIZiqvVtYhek460C1chE4B/PtwLTw/y5oBuItJJ7X4liYgMh72FHGueDsXEEHc0iMC7P2VxQDcRGQSWJCL6W3JjGRaN98ObI3tpBnQ/ueIQSqtUUkcjImo3LElEdFcEQcAzEV2x8ql+sFIYIy3nKh5dth9ZhRVSRyMiahfG97LxuHHj7vh6WVlZa7IQkQ64OaD7maQjyC2twfj4VHz0uD9G9HGROhoRUZu6pytJNjY2d1w8PT0xbdq09spKRB1EDycr/DB7IAZ1d0BNrRrRa45iSfJZNDS02fdAiIgk16bfbjMk/HYbEVCvbsB7W7Owan8OAGBEb2f853F/WCju6SI1EZHW8NttRKQVxkYyvDOqNxZP6Au5kQzbTxZhfHwq8q/WSB2NiKjVWJKIqNUeD3HHulkD4GCpwOmiSjy67Fek/lYidSwiolZhSSKiNhHsaYcfXxoIvy42uFZTh6kr07Bq/wXwjj4R6SqWJCJqMy42ZtgYHYaxgV2gbhDxrx9PYd7G47hRp/+z8BOR/mFJIqI2ZWpihI8e98dbj/jCSCbgu6MXMfHzAygsvy51NCKie8KSRERtThAEPD3IG0kz+8PW3ATHLpZj1Cf7cSTnqtTRiIjuGksSEbWbgd0d8OOLg+DjbIWSKhUmJx7EmoO5HKdERDqBJYmI2pW7vTk2vRCOkX1dUKcW8ebmE3jtO45TIqKOjyWJiNqdudwYyyYH4vUoH80Dcid+fgCXyjhOiYg6LpYkItIKQRAQPaQbvmoyTulXHPitVOpoRETNYkkiIq2K6OGIH18cBF8Xa5RW1+LJlYewIiWb45SIqMORvCQtX74c3t7eMDU1RXBwMFJSUm67bWFhIaZMmYKePXtCJpMhJibmlm0SExMREREBOzs72NnZYdiwYUhLS2uyzfz58yEIQpPF2dm5rQ+NiG7D3d4c3z0frplP6d2fsjBnvRLVqnqpoxERaUhakjZs2ICYmBjExsYiIyMDERERiIqKQl5eXrPbq1QqODo6IjY2Fv7+/s1us2fPHkyePBm7d+/GgQMH4OHhgcjISBQUFDTZrnfv3igsLNQsmZmZbX58RHR7ZvLG+ZTmj/KFsUzAlmOXMObT/ThfXCl1NCIiAIAgSniNOzQ0FEFBQYiPj9es69WrF8aMGYO4uLg7vnfo0KEICAjA0qVL77idWq2GnZ0dli1bhmnTpgFovJK0efNmKJXKFme/l6cIE9GdpV24ihfXHkVxpQrmciPEjfPD6IAuUsciIj10L7+/JbuSVFtbi/T0dERGRjZZHxkZidTU1DbbT01NDerq6mBvb99k/blz5+Dq6gpvb29MmjQJ2dnZd/wclUqFioqKJgsRtY3+3vb46eUIhHfrhJpaNeasV+KtzSegquc0AUQkHclKUklJCdRqNZycnJqsd3JyQlFRUZvt5/XXX0eXLl0wbNgwzbrQ0FAkJSVhx44dSExMRFFREcLDw1Faevtv2cTFxcHGxkazuLu7t1lGIgIcrRRY/XQoXnqgOwBg9cFcPPbZAeRfrZE4GREZKskHbguC0ORnURRvWddSixcvxrp167Bp0yaYmppq1kdFRWH8+PHw8/PDsGHD8NNPPwEAvvrqq9t+1htvvIHy8nLNkp+f3yYZiegPRjIBr0b2xKoZ/WBrboLjF8vxyCe/YlfWZamjEZEBkqwkOTg4wMjI6JarRsXFxbdcXWqJDz/8EO+//z5+/vln9O3b947bWlhYwM/PD+fOnbvtNgqFAtbW1k0WImof9/fsjP+9NAj+7rYov16Hp786gn/9eJK334hIqyQrSXK5HMHBwUhOTm6yPjk5GeHh4a367A8++AALFy7E9u3bERIS8rfbq1QqZGVlwcXFpVX7JaK242Znjo3PhWHmQG8AwKr9ORi3PBXZV6okTkZEhkLS221z587FihUr8MUXXyArKwuvvPIK8vLyEB0dDaDxFtfNb6TdpFQqoVQqUVVVhStXrkCpVOLUqVOa1xcvXow333wTX3zxBby8vFBUVISioiJUVf3xD+u8efOwd+9eXLhwAYcOHcKECRNQUVGB6dOna+fAieiuyI1leHuUL754KgT2FnKcvFSBRz75FRuP5HPySSJqd5JOAQA0Tia5ePFiFBYWok+fPliyZAkGDx4MAHjqqaeQk5ODPXv2aLZvbrySp6cncnJyAABeXl7Izc29ZZt33nkH8+fPBwBMmjQJ+/btQ0lJCRwdHTFgwAAsXLgQvr6+d52bUwAQadflihuIWa/EgezGL1iMDnDFu2P6wMrUROJkRKRL7uX3t+QlSVexJBFpn7pBxGd7f8NHyWehbhDhYW+OjycHIsDdVupoRKQjdGKeJCKie2UkEzD7/u745rkB6GJrhryrNZgQn4pPd5+HuoH/v0dEbYsliYh0TrCnPbbOicDIvi6obxDxwY4zeGLFQRSWX5c6GhHpEZYkItJJNmYmWDY5EB9M6AtzuREOZl/FiKUp2H6iUOpoRKQnWJKISGcJgoDHQtzx08sR6Otmg/LrdYhecxRvbDqOmtp6qeMRkY5jSSIineftYIFvo8MRPaQbBAFYl5aPRz75FZkXy6WORkQ6jCWJiPSC3FiG16N8sObpUDhZK5B9pRpjl+/H8j0c1E1ELcOSRER6ZWB3B2yfMxgjejujvkHE4u1nMCXxIArKOKibiO4NSxIR6R07CzninwzC4vGNg7oPXbiKEUv3YcuxS1JHIyIdwpJERHpJEAQ83s8dW1+OQIC7LSpv1OPldRl4ZYMSFTfqpI5HRDqAJYmI9JqXgwU2Rofh5Qe6QyYA32cUIGppCg7+/ngTIqLbYUkiIr1nYiTD3Mie+Oa5MHjYm6Og7DomJx5E3NYsqOrVUscjog6KJYmIDEaIV+NM3RND3CGKwOf7sjF62X6cLqqQOhoRdUAsSURkUCwVxvj3hL5ImBqMThZynC6qxKOf7MeKlGw0cKoAIvoTliQiMkiRvZ2xPWYwHvTpjFp1A979KQtTVhzExWs1Ukcjog6CJYmIDJajlQIrpodg0Ti/Js9/++ZIPkSRV5WIDB1LEhEZNEEQMKm/B7bNiUCIpx2qVPX4v2+P49mkdFypVEkdj4gkxJJERATAs5MFNjwXhtejfCA3kmFn1mU8tHQftmUWSh2NiCTCkkRE9DsjmYDoId2w5aWB6OVijavVtXj+66N4ZYMS5dc5ASWRoWFJIiL6Cx9na/wweyBm399NMwFl5JK92HOmWOpoRKRFLElERM2QG8vwj4d88O3z4ejqYIHLFSo8teow3th0HFWqeqnjEZEWsCQREd1BkIcdfno5AjMHegMA1qXl46El+5B6vkTiZETU3liSiIj+hpncCG+P8sX6WQPgbm+GgrLrmLLiEN7+4QRqanlViUhfsSQREd2lAV07YfucwXgi1AMAkHQgF1H/TcEhPiyXSC+xJBER3QMLhTHeG+uH1U/3h6uNKXJLazAx4SDmbznJq0pEeoYliYioBSJ6OGL7K4MxqZ87AODL1ByMWJqCg7yqRKQ3WJKIiFrI2tQEi8b3RdLMxqtKeVdrMCnhIN754QSq+Q04Ip3HkkRE1EqD73PEjlcGY3L/xrFKXx3IxYj/7kPqb/wGHJEuY0kiImoDVqYmiBvnhzVPh6KLrRnyr17HlMRDeGNTJipucLZuIl3EkkRE1IYG9XDAjlcGY+oATwDAurQ8PLRkH3Zztm4incOSRETUxiwVxlg4pg/WzxoAz07mKCy/gRmrDmPuN0qU1dRKHY+I7hJLEhFRO7k5r9Izg7whCMCmowUY9tE+bMsslDoaEd0FliQionZkJjfCm4/44rvnw9G9syVKqlR4/uujiF6djuKKG1LHI6I7YEkiItKCxmfADcKL93eHsUzA9pNFGPbRXmw4nAdRFKWOR0TNYEkiItIShbER5j3UE1teHIS+bjaouFGP177LxJTEQ8gpqZY6HhH9BUsSEZGW+bpaY9Pz4Yh9uBdMTWQ4kF2KEf/dh8/3/oZ6dYPU8YjodyxJREQSMDaS4dnBXbEjZjDCu3XCjboGxG07jdGf7ocyv0zqeEQEliQiIkl5drLA18+EYvH4vrAxM8HJSxUYu3w/3tp8gpNQEkmMJYmISGKCIODxfu7Y9eoQjAvsAlEEVh/MxYP/2Ystxy5xYDeRRFiSiIg6CAdLBT6aGIC1z4Siq4MFrlSq8PK6DEz7Io0Du4kkwJJERNTBhHd3wLaYCLwy7D7IjWVIOVeCyKX78NHPZ3C9Vi11PCKDwZJERNQBKYyNMGdYD+yIGYyIHg6orW/Ax7+cx7CP9mL7iULegiPSApYkIqIOzNvBAkkz+yP+iSC42piioOw6otccxbQv0nC+uErqeER6TRD5vyMtUlFRARsbG5SXl8Pa2lrqOERkAK7XqrF8z3l8vi8btfUNMJYJmDnIGy8/2AOWCmOp4xHphHv5/c2S1EIsSUQkldzSaiz83ynszCoGADhaKfD6CB+MDewCmUyQOB1Rx8aSpAUsSUQktV9OX8a/fjyF3NIaAECghy3+9Whv9HWzlTYYUQfGkqQFLElE1BGo6tVY+esFLPvlPGpq1RAE4PFgd/xjRE84WCqkjkfU4bAkaQFLEhF1JJcrbmDRttP4PqMAAGClMMacYT0wPdwLJkb8jg7RTSxJWsCSREQdUXruVczfcgqZBeUAgG6OFnjzEV/c37OzxMmIOgaWJC1gSSKijkrdIGLjkXx8sOMMSqtrAQBDezrizZG+6N7ZUuJ0RNJiSdICliQi6ugqbtRh2S/nsWr/BdSpRRjLBEwL88KcYT1gY2YidTwiSbAkaQFLEhHpigsl1Xjvpz+mDLC3kGPu8PswqZ87jDleiQwMS5IWsCQRka7Zd/YKFv7vFM79PlP3fU6WeHOkLwbf5yhxMiLtYUnSApYkItJFdeoGrD2UhyU7z6Kspg4A8IBPZ/zz4V4cr0QGgSVJC1iSiEiXldfU4b+7ziHpQA7qGxrHKz05wBNzHuwBOwu51PGI2s29/P6W/Gb08uXL4e3tDVNTUwQHByMlJeW22xYWFmLKlCno2bMnZDIZYmJibtkmMTERERERsLOzg52dHYYNG4a0tLRW7ZeISN/YmJvg7VG++PmVwRjWqzPqG0R8mZqDoR/uwYqUxmfDERk6SUvShg0bEBMTg9jYWGRkZCAiIgJRUVHIy8trdnuVSgVHR0fExsbC39+/2W327NmDyZMnY/fu3Thw4AA8PDwQGRmJgoKCFu+XiEhfdXW0xIrp/bDm6VD4OFuh/Hod3v0pC8OX7MW2zELwZgMZMklvt4WGhiIoKAjx8fGadb169cKYMWMQFxd3x/cOHToUAQEBWLp06R23U6vVsLOzw7JlyzBt2rRW7/cm3m4jIn2jbhDxzZF8/OfnsyipUgEA+nnZ4c2RvvB3t5U2HFEb0YnbbbW1tUhPT0dkZGST9ZGRkUhNTW2z/dTU1KCurg729vat2q9KpUJFRUWThYhInxjJBEzu74E9/xiKlx7oDlMTGQ7nXMPoT/djzvoMXLxWI3VEIq2SrCSVlJRArVbDycmpyXonJycUFRW12X5ef/11dOnSBcOGDWvVfuPi4mBjY6NZ3N3d2ywjEVFHYqkwxquRPbF73lCMC+oCQQB+UF7CA//Zi7itWSi/Xid1RCKtkHzgtiAITX4WRfGWdS21ePFirFu3Dps2bYKpqWmr9vvGG2+gvLxcs+Tn57dJRiKijsrFxgwfPR6AH18chAFd7VFb34DP92VjyAe7sfLXCxzcTXpPspLk4OAAIyOjW67eFBcX33KVpyU+/PBDvP/++/j555/Rt2/fVu9XoVDA2tq6yUJEZAj6dLHBumcH4IunQtCjsyXKauqw8H+nMOyjvfjf8Usc3E16S7KSJJfLERwcjOTk5Cbrk5OTER4e3qrP/uCDD7Bw4UJs374dISEhWtsvEZG+EgQBD/g4YducCMSN84OjlQJ5V2vw4toMjF2eirQLV6WOSNTmjKXc+dy5czF16lSEhIQgLCwMCQkJyMvLQ3R0NIDGW1wFBQVISkrSvEepVAIAqqqqcOXKFSiVSsjlcvj6+gJovMX21ltvYe3atfDy8tJcMbK0tISlpeVd7ZeIiJpnbCTD5P4eeNTfFYkp2UjYlw1lfhke//wAhvXqjNdG+KCHk5XUMYnahOQzbi9fvhyLFy9GYWEh+vTpgyVLlmDw4MEAgKeeego5OTnYs2ePZvvmxg15enoiJycHAODl5YXc3NxbtnnnnXcwf/78u9rv3eAUAEREQHHFDSzddQ4bDudD3SBCJgATgt3wyvD74GJjJnU8olvwsSRawJJERPSH365U4YPtZ7D9ZOPVe4WxDDMHeSN6SDfYmJlInI7oDyxJWsCSRER0q/Tca1i0LQuHc64BAGzNTfDC0G6YFuYFUxMjidMRsSRpBUsSEVHzRFHEzqxi/Hv7aZwvrgIAuNiY4pVh92FcUBcYG0k++wwZMJYkLWBJIiK6M3WDiO+OXsTS5LO4VH4DANC9syXmRfbEQ72d2mxOPKJ7wZKkBSxJRER350adGmsO5mLZ7vMoq2mcrTvA3Rb/N6Inwrs5SJyODA1LkhawJBER3ZuKG3VI2JuNlb9ewPU6NQAgoocD/vFQT/R1s5U2HBkMliQtYEkiImqZ4sobWPbLeaxLy0OduvFX0Ijezpj30H3o3plzLFH7YknSApYkIqLWyb9agyU7z+L7jAKIIiATgHFBbpjzYA+425tLHY/0FEuSFrAkERG1jbOXK/HhjjP4+dRlAICJkYDHQtwx+/7u6GLLCSmpbbEkaQFLEhFR21Lml+HDHWfw6/kSAI1l6fHfy5IryxK1EZYkLWBJIiJqH2kXrmLpzrNI/a0UQGNZmtivsSzxUSfUWixJWsCSRETUvg5ll2LpznM4kN1YluRGMjwW4oboId04ZolajCVJC1iSiIi048BvpVi68ywOXbgKADCWCRgb2AUv3N8d3g4WEqcjXcOSpAUsSURE2nUouxTLdp9HyrnGMUsyARjl74rZ93fHfU6cOoDuDkuSFrAkERFJ42jeNXz6y3nsOl2sWTeitzNm398dfm42EiYjXcCSpAUsSURE0jpRUI7le85j24ki3PxNNvg+R8we2g2hXTtJG446LJYkLWBJIiLqGM5drkT8nt/ww7FLUDc0/krr52WHF+7vjqH3OfJButQES5IWsCQREXUs+Vdr8Nne37DxyEXUqhsAAL1drfH80G6I6uMCIxnLErEkaQVLEhFRx3S54gZWpGTj60N5qKltfJCut4MFnhvcFWODukBhbCRxQpISS5IWsCQREXVs16pr8WVqDr5MzUH59ToAgJO1As9GdMXk/h6wUBhLnJCkwJKkBSxJRES6oVpVj3VpeUhMycblChUAwMbMBNPDvfBUuBfsLeQSJyRtYknSApYkIiLdoqpXY3NGAT7bm40LJdUAAFMTGSaGuOOZiK6cxdtAsCRpAUsSEZFuUjeI2HGyCJ/t/Q3HL5YDAIxkAh7p64LnBneDryv/TddnLElawJJERKTbRFHEgd9KEb/3N80s3kDjXEvRQ7oirGsnTh+gh1iStIAliYhIf5woKMfn+7Lx0/FL+H2qJfh1scGswV0R1ccZxkYyaQNSm2FJ0gKWJCIi/ZNXWoMVv2bjmyP5uFHXONeSu70Znh7ojcf7ucNczm/E6TqWJC1gSSIi0l9Xq2uRdCAHSQdycbW6FgBga26CqQM8MS3MC45WCokTUkuxJGkBSxIRkf67XqvGt+n5SEy5gLyrNQAAubEM4wK74JkIb3TvbCVxQrpXLElawJJERGQ4bn4jLmFfNpT5ZZr1D/h0xrMRXTGgqz0HeesIliQtYEkiIjI8oigiPfcaEvZlIznrMm7+Bu3TxRrPRnTFw34uMOEg7w6NJUkLWJKIiAzbhZJqrPw1G9+mX9QM8naxMcX0cC9M7ucBG3MTiRNSc1iStIAliYiIgMZB3l8fzMVXB3JRUtX42BNzuREeD3HHjIFe8OxkIXFC+jOWJC1gSSIioj9T1auxRXkJK3+9gNNFlQAAQQCG93LCzEHeCPXmuKWOgCVJC1iSiIioOaIoYv/5Uqz4NRt7zlzRrO/tao2ZA73xiL8LFMZGEiY0bCxJWsCSREREf+fc5UqsSs3BpqN/jFtysFRg6gBPPDHAAw6WnG9J21iStIAliYiI7ta16lqsTctD0oEcXK5oHLckN5ZhtL8rZgz05kN1tYglSQtYkoiI6F7VqRuwNbMQX/x6AcculmvWh3rbY8ZALwz3dYaRjOOW2hNLkhawJBERUUuJooijeWVYtf8Ctp0ogvr3p+p2sTXD9HBPTAzhFALthSVJC1iSiIioLRSWX8eag7lYeygP12rqAABmJkZ41N8VTw7whJ+bjcQJ9QtLkhawJBERUVu6Udc4hcAX+/+YQgAA/N1t8WSoB0b5u8LUhN+Kay2WJC1gSSIiovYgiiKO5F7DmoO52JZZhFp147fibMxMMCHYDU+EeqCro6XEKXUXS5IWsCQREVF7K6lS4Zsj+Vh7KA8Xr13XrB/Q1R6T+3tgRB9nzrl0j1iStIAliYiItEXdIGLf2StYczAXu88U4/dx3rAzb7y6NKm/B7rx6tJdYUnSApYkIiKSwqWy6/jmSD42HM5HYfkNzfpQ7z+uLnHs0u2xJGkBSxIREUlJ3SBi79lirD2Uj19OX9ZcXbIxM8HYwC6Y1N8dPs78/fRXLElawJJEREQdRWH5dWw8chEbDuejoOyPsUsB7raY1M8do/xdYaEwljBhx8GSpAUsSURE1NE0NIj49XwJ1h/Ow88nL6P+98tLFnIjPNLXFRP7uyPQ3RaCYLizerMkaQFLEhERdWQlVSp8l954dSm7pFqzvkdnS0zs545xQW6wt5BLmFAaLElawJJERES6QBRFHM65hg2H8/FT5iXcqGucd8nESMBwXyc8HuKOiB6OBvPMOJYkLWBJIiIiXVNxow4/HruEDYfzcfxPD9h1tjbFhGA3TAh2g5eDhYQJ2x9LkhawJBERkS7LKqzAhsP52KwsQNnvz4wDGqcSeDzEHVF+zjCX699gb5YkLWBJIiIifaCqV2NXVjG+OZKPfWevaKYSsFQYY6SfCx4LcUOwp53eDPZmSdICliQiItI3heXXseloAb45ko/c0hrNem8HC0wIdsO4oC5wsTGTMGHrsSRpAUsSERHpK1EUkXbhKjamX8TWzELU1KoBAIIADOrugAnBbniot27O7M2SpAUsSUREZAiqVfXYmlmIjekXkXbhqma9lcIYj/i7YHyQbt2Ou5ff3zItZbqt5cuXw9vbG6ampggODkZKSsptty0sLMSUKVPQs2dPyGQyxMTE3LLNyZMnMX78eHh5eUEQBCxduvSWbebPnw9BEJoszs7ObXhURERE+sFCYYzHQtzxzXNh2PuPoXj5ge7oYmuGSlU91qXlY8JnB/DAf/bik13ncPFazd9/oA6RtCRt2LABMTExiI2NRUZGBiIiIhAVFYW8vLxmt1epVHB0dERsbCz8/f2b3aampgZdu3bFokWL7lh8evfujcLCQs2SmZnZJsdERESkrzw7WWBuZE+k/N/9WPtsKMYHucFcboQLJdX4T/JZDPr3bkxOOIiNR/JRpaqXOm6rSXq7LTQ0FEFBQYiPj9es69WrF8aMGYO4uLg7vnfo0KEICAho9krRTV5eXoiJibnlitP8+fOxefNmKJXKFmfn7TYiIqLG23HbTxTh2/SLOHihFDdbhamJDCN6O2NckBsGdnfoMJNV3svvb8kmQKitrUV6ejpef/31JusjIyORmpra7vs/d+4cXF1doVAoEBoaivfffx9du3a97fYqlQoqlUrzc0VFRbtnJCIi6ugsFMYYH+yG8cFuKCi7js0ZBfju6EVkX6nGZuUlbFZeQmcrBUYHuGJsoBt8XXXnwoJkt9tKSkqgVqvh5OTUZL2TkxOKioradd+hoaFISkrCjh07kJiYiKKiIoSHh6O0tPS274mLi4ONjY1mcXd3b9eMREREuqaLrRlm398du+YOwebZAzEtzBO25iYorlQhMeUCHv44BSOW7sPne39DUfkNqeP+LckHbv91NLwoiu0+Qj4qKgrjx4+Hn58fhg0bhp9++gkA8NVXX932PW+88QbKy8s1S35+frtmJCIi0lWCICDA3RYLRvdB2j+H4fOpwRjR2xlyIxlOF1UibttphC3ahSdWHMS36Rc77PglyW63OTg4wMjI6JarRsXFxbdcXWpvFhYW8PPzw7lz5267jUKhgEKh0GIqIiIi3Sc3luGh3s54qLczymvq8FNmIb7PuIjDOdew/3wp9p8vxZubMzGslxPGBnbB4PscYWIk+TUcABKWJLlcjuDgYCQnJ2Ps2LGa9cnJyRg9erRWs6hUKmRlZSEiIkKr+yUiIjIkNuYmmBLqgSmhHsi/WoPvMwqwOaMA2SXV+N/xQvzveCHsLeQY6eeCMYFdEORhK+n8S5I+uW7u3LmYOnUqQkJCEBYWhoSEBOTl5SE6OhpA4y2ugoICJCUlad5z8xtpVVVVuHLlCpRKJeRyOXx9fQE0Dgg/deqU5s8FBQVQKpWwtLRE9+7dAQDz5s3DqFGj4OHhgeLiYrz77ruoqKjA9OnTtXj0REREhsvd3hwvP9gDLz3QHZkF5diccQlbjl1CSZUKqw/mYvXBXAzq7oA1z4RKllHyGbeXL1+OxYsXo7CwEH369MGSJUswePBgAMBTTz2FnJwc7NmzR7N9c43S09MTOTk5AICcnBx4e3vfss2QIUM0nzNp0iTs27cPJSUlcHR0xIABA7Bw4UJN0bobnAKAiIiobdWrG5D6Wyk2ZxRg+8kiPDPIG3Mje7bpPvhYEi1gSSIiImo/NbX1qFOLsDEzadPP1Yl5koiIiIhux1wufUXpGMPHiYiIiDoYliQiIiKiZrAkERERETWDJYmIiIioGSxJRERERM1gSSIiIiJqBksSERERUTNYkoiIiIiawZJERERE1AyWJCIiIqJmsCQRERERNYMliYiIiKgZLElEREREzZD+Ebs6ShRFAEBFRYXESYiIiOhu3fy9ffP3+J2wJLVQZWUlAMDd3V3iJERERHSvKisrYWNjc8dtBPFuqhTdoqGhAZcuXYKVlRUEQWjTz66oqIC7uzvy8/NhbW3dpp+tK3gOeA5u4nngOQB4Dm7ieWj9ORBFEZWVlXB1dYVMdudRR7yS1EIymQxubm7tug9ra2uD/Y/gJp4DnoObeB54DgCeg5t4Hlp3Dv7uCtJNHLhNRERE1AyWJCIiIqJmsCR1QAqFAu+88w4UCoXUUSTDc8BzcBPPA88BwHNwE8+Dds8BB24TERERNYNXkoiIiIiawZJERERE1AyWJCIiIqJmsCQRERERNYMlqYNZvnw5vL29YWpqiuDgYKSkpEgdqV3t27cPo0aNgqurKwRBwObNm5u8Looi5s+fD1dXV5iZmWHo0KE4efKkNGHbSVxcHPr16wcrKyt07twZY8aMwZkzZ5pso+/nIT4+Hn379tVMDhcWFoZt27ZpXtf3429OXFwcBEFATEyMZp0hnIf58+dDEIQmi7Ozs+Z1QzgHAFBQUIAnn3wSnTp1grm5OQICApCenq55Xd/Pg5eX1y1/DwRBwOzZswFo7/hZkjqQDRs2ICYmBrGxscjIyEBERASioqKQl5cndbR2U11dDX9/fyxbtqzZ1xcvXoyPPvoIy5Ytw+HDh+Hs7Izhw4drnp2nD/bu3YvZs2fj4MGDSE5ORn19PSIjI1FdXa3ZRt/Pg5ubGxYtWoQjR47gyJEjeOCBBzB69GjNP3r6fvx/dfjwYSQkJKBv375N1hvKeejduzcKCws1S2ZmpuY1QzgH165dw8CBA2FiYoJt27bh1KlT+M9//gNbW1vNNvp+Hg4fPtzk70BycjIA4LHHHgOgxeMXqcPo37+/GB0d3WSdj4+P+Prrr0uUSLsAiN9//73m54aGBtHZ2VlctGiRZt2NGzdEGxsb8bPPPpMgoXYUFxeLAMS9e/eKomi458HOzk5csWKFwR1/ZWWl2KNHDzE5OVkcMmSIOGfOHFEUDefvwTvvvCP6+/s3+5qhnIPXXntNHDRo0G1fN5Tz8Gdz5swRu3XrJjY0NGj1+HklqYOora1Feno6IiMjm6yPjIxEamqqRKmkdeHCBRQVFTU5JwqFAkOGDNHrc1JeXg4AsLe3B2B450GtVmP9+vWorq5GWFiYwR3/7NmzMXLkSAwbNqzJekM6D+fOnYOrqyu8vb0xadIkZGdnAzCcc7BlyxaEhITgscceQ+fOnREYGIjExETN64ZyHm6qra3FmjVrMHPmTAiCoNXjZ0nqIEpKSqBWq+Hk5NRkvZOTE4qKiiRKJa2bx21I50QURcydOxeDBg1Cnz59ABjOecjMzISlpSUUCgWio6Px/fffw9fX12COHwDWr1+Po0ePIi4u7pbXDOU8hIaGIikpCTt27EBiYiKKiooQHh6O0tJSgzkH2dnZiI+PR48ePbBjxw5ER0fj5ZdfRlJSEgDD+btw0+bNm1FWVoannnoKgHaP37hNP41aTRCEJj+LonjLOkNjSOfkxRdfxPHjx/Hrr7/e8pq+n4eePXtCqVSirKwM3333HaZPn469e/dqXtf348/Pz8ecOXPw888/w9TU9Lbb6ft5iIqK0vzZz88PYWFh6NatG7766isMGDAAgP6fg4aGBoSEhOD9998HAAQGBuLkyZOIj4/HtGnTNNvp+3m4aeXKlYiKioKrq2uT9do4fl5J6iAcHBxgZGR0SwsuLi6+pS0bipvfaDGUc/LSSy9hy5Yt2L17N9zc3DTrDeU8yOVydO/eHSEhIYiLi4O/vz/++9//Gszxp6eno7i4GMHBwTA2NoaxsTH27t2Ljz/+GMbGxppj1ffz8FcWFhbw8/PDuXPnDObvgouLC3x9fZus69Wrl+ZLPIZyHgAgNzcXO3fuxDPPPKNZp83jZ0nqIORyOYKDgzUj+G9KTk5GeHi4RKmk5e3tDWdn5ybnpLa2Fnv37tWrcyKKIl588UVs2rQJv/zyC7y9vZu8bijn4a9EUYRKpTKY43/wwQeRmZkJpVKpWUJCQvDEE09AqVSia9euBnEe/kqlUiErKwsuLi4G83dh4MCBt0wDcvbsWXh6egIwrH8TVq1ahc6dO2PkyJGadVo9/jYdBk6tsn79etHExERcuXKleOrUKTEmJka0sLAQc3JypI7WbiorK8WMjAwxIyNDBCB+9NFHYkZGhpibmyuKoiguWrRItLGxETdt2iRmZmaKkydPFl1cXMSKigqJk7ed559/XrSxsRH37NkjFhYWapaamhrNNvp+Ht544w1x37594oULF8Tjx4+L//znP0WZTCb+/PPPoijq//Hfzp+/3SaKhnEeXn31VXHPnj1idna2ePDgQfGRRx4RraysNP8OGsI5SEtLE42NjcX33ntPPHfunPj111+L5ubm4po1azTbGMJ5UKvVooeHh/jaa6/d8pq2jp8lqYP59NNPRU9PT1Eul4tBQUGar4Hrq927d4sAblmmT58uimLjV13feecd0dnZWVQoFOLgwYPFzMxMaUO3seaOH4C4atUqzTb6fh5mzpyp+Xvv6OgoPvjgg5qCJIr6f/y389eSZAjnYeLEiaKLi4toYmIiurq6iuPGjRNPnjyped0QzoEoiuKPP/4o9unTR1QoFKKPj4+YkJDQ5HVDOA87duwQAYhnzpy55TVtHb8giqLYttemiIiIiHQfxyQRERERNYMliYiIiKgZLElEREREzWBJIiIiImoGSxIRERFRM1iSiIiIiJrBkkRERETUDJYkIiIiomawJBERtRFBELB582apYxBRG2FJIiK98NRTT0EQhFuWESNGSB2NiHSUsdQBiIjayogRI7Bq1aom6xQKhURpiEjX8UoSEekNhUIBZ2fnJoudnR2Axlth8fHxiIqKgpmZGby9vbFx48Ym78/MzMQDDzwAMzMzdOrUCbNmzUJVVVWTbb744gv07t0bCoUCLi4uePHFF5u8XlJSgrFjx8Lc3Bw9evTAli1b2vegiajdsCQRkcF46623MH78eBw7dgxPPvkkJk+ejKysLABATU0NRowYATs7Oxw+fBgbN27Ezp07m5Sg+Ph4zJ49G7NmzUJmZia2bNmC7t27N9nHv/71Lzz++OM4fvw4Hn74YTzxxBO4evWqVo+TiNqISESkB6ZPny4aGRmJFhYWTZYFCxaIoiiKAMTo6Ogm7wkNDRWff/55URRFMSEhQbSzsxOrqqo0r//000+iTCYTi4qKRFEURVdXVzE2Nva2GQCIb775pubnqqoqURAEcdu2bW12nESkPRyTRER64/7770d8fHyTdfb29po/h4WFNXktLCwMSqUSAJCVlQV/f39YWFhoXh84cCAaGhpw5swZCIKAS5cu4cEHH7xjhr59+2r+bGFhASsrKxQXF7f0kIhIQixJRKQ3LCwsbrn99XcEQQAAiKKo+XNz25iZmd3V55mYmNzy3oaGhnvKREQdA8ckEZHBOHjw4C0/+/j4AAB8fX2hVCpRXV2teX3//v2QyWS47777YGVlBS8vL+zatUurmYlIOrySRER6Q6VSoaioqMk6Y2NjODg4AAA2btyIkJAQDBo0CF9//TXS0tKwcuVKAMATTzyBd955B9OnT8f8+fNx5coVvPTSS5g6dSqcnJwAAPPnz0d0dDQ6d+6MqKgoVFZWYv/+/XjppZe0e6BEpBUsSUSkN7Zv3w4XF5cm63r27InTp08DaPzm2fr16/HCCy/A2dkZX3/9NXx9fQEA5ubm2LFjB+bMmYN+/frB3Nwc48ePx0cffaT5rOnTp+PGjRtYsmQJ5s2bBwcHB0yYMEF7B0hEWiWIoihKHYKIqL0JgoDvv/8eY8aMkToKEekIjkkiIiIiagZLEhEREVEzOCaJiAwCRxYQ0b3ilSQiIiKiZrAkERERETWDJYmIiIioGSxJRERERM1gSSIiIiJqBksSERERUTNYkoiIiIiawZJERERE1Iz/B0G65vLd/BekAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Loss\")\n",
    "plt.plot(loss_values)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Try changing the parameters such as the length of training, and maybe some operations to see how the model behaves. Does it take much longer? How is the performance?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Want to learn more?\n",
    "\n",
    "Also, you can use **Watson Studio** to run these notebooks faster with bigger datasets.**Watson Studio** is IBM’s leading cloud solution for data scientists, built by data scientists. With Jupyter notebooks, RStudio, Apache Spark and popular libraries pre-packaged in the cloud, **Watson Studio** enables data scientists to collaborate on their projects without having to install anything. Join the fast-growing community of **Watson Studio** users today with a free account at [Watson Studio](https://cocl.us/ML0120EN_DSX).This is the end of this lesson. Thank you for reading this notebook, and good luck on your studies.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Thanks for completing this lesson!\n",
    "\n",
    "This is the end of **Logistic Regression with TensorFlow** notebook. Hopefully, now you have a deeper understanding of Logistic Regression and how its structure and flow work. Thank you for reading this notebook and good luck on your studies.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Created by: <a href=\"https://linkedin.com/in/romeo-kienzler-089b4557\"> Romeo Kienzler </a>, <a href=\"https://br.linkedin.com/in/walter-gomes-de-amorim-junior-624726121\">Saeed Aghabozorgi</a> , <a href=\"https://br.linkedin.com/in/walter-gomes-de-amorim-junior-624726121\">Walter Gomes de Amorim Junior</a> ,  Victor Barros Costa\n",
    "\n",
    "Updated to TF 2.X by  <a href=\"https://www.linkedin.com/in/samaya-madhavan\"> Samaya Madhavan </a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Change Log\n",
    "\n",
    "| Date (YYYY-MM-DD) | Version | Changed By | Change Description                                          |\n",
    "| ----------------- | ------- | ---------- | ----------------------------------------------------------- |\n",
    "| 2020-09-21        | 2.0     | Srishti    | Migrated Lab to Markdown and added to course repo in GitLab |\n",
    "| 2023-10-30        | 2.1     | Vandana    | Updated lab instruction                                     |\n",
    "\n",
    "<hr>\n",
    "\n",
    "## <h3 align=\"center\"> © IBM Corporation 2020. All rights reserved. <h3/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "Copyright © 2018 [Cognitive Class](https://cocl.us/DX0108EN_CC). This notebook and its source code are released under the terms of the [MIT License](https://bigdatauniversity.com/mit-license?cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBMDeveloperSkillsNetwork-DL0120EN-SkillsNetwork-20629446&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBMDeveloperSkillsNetwork-DL0120EN-SkillsNetwork-20629446&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBMDeveloperSkillsNetwork-DL0120EN-SkillsNetwork-20629446&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBMDeveloperSkillsNetwork-DL0120EN-SkillsNetwork-20629446&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ).\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python",
   "language": "python",
   "name": "conda-env-python-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}