Logistic Gradient Descent.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#Logistic Regression and Gradient Descent\n",
    "\n",
    "Logistic regression is an excellent tool to know for classification problems. Classification problems are problems where you are trying to classify observations into groups. To make our examples more concrete, we will consider the [Iris dataset](https://archive.ics.uci.edu/ml/datasets/Iris). The iris dataset contains 4 attributes for 3 types of iris plants. The purpose is to classify which plant you have just based on the attributes. To simplify things, we will only consider 2 attributes and 2 classes. Here are the data visually:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": [
       "iVBORw0KGgoAAAANSUhEUgAAAfwAAAFqCAYAAAD/behNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\n",
       "AAALEgAACxIB0t1+/AAAIABJREFUeJzt3X2YXVV96PHvEAIjASZSilFDiQb3Qq2CEuVWLaMEJF4J\n",
       "3FakScjgUKAkUnqVQtRYICBpKFG8XMQZGyAD4WUoPlCUyACOZJRyfQFTUVoW8hIEmyiWzBACkyGT\n",
       "c/84ZyaHJPNyJnNe9/fzPOdhztln77XW3prfWe91mUwGSZJU2/YodwYkSVLxGfAlSUoBA74kSSlg\n",
       "wJckKQUM+JIkpYABX5KkFNiz2AmEEH4O9OTePh1jPCPv2GzgQmArcH2M8dpi50eSpDSqK+Y8/BBC\n",
       "PfBQjPH9uzg2EfgPYAbwCvBvwAkxxt8XLUOSJKVUsZv0Dwf2CSHcG0LoDCEclXfsncCTMcaeGONr\n",
       "wIPA0UXOjyRJqVTsgL8ZWB5jPB5YANwcQhhIc3+2N/UDbAIaCk0ghLBnCGFaCKHo3ROSJFWrYgfJ\n",
       "J4AnAWKMvw4h/DfwZuC3ZIP9fnnf3Q/YONzFQghLgIt3dayzs3McsitJUlWoK/SEYgf804H3AueE\n",
       "EN5Ctla/IXfsceAdIYQ3km0JOBpYPtzFYoxLgCX5n4UQpgHPjGemJUmqNcVu0r8O2D+E8EOgnewP\n",
       "gFNCCGfl+u3PA+4FHgKuizGuL3J+JElKpaKO0i+FgRp+Z2cnU6dOLXd2JEkqhYKb9F14R5KkFDDg\n",
       "S5KUAgZ8SZJSwIAvSVIKGPAlSUoBA74kSSlgwJckVZT+/n6effZZenp6Rv7yKP3zP/8zp59+Ok1N\n",
       "TZx22mk89thjQ373pptuGrd0K4kBX5JUMXp6XuK441aRJBs57LAfc801u79s+pNPPskPfvADVq5c\n",
       "yapVq1i8eDGLFy8e8vutra27nWYlMuBLkirGkiX388ADp9HXdwQbNhzP0qV9vPzyy7t1zf3224/1\n",
       "69fz7W9/m9/97nccdthh3H777cQYOe2002hqauLv/u7vePnll2lpaaG7u5tLL72UrVu3cv755zNn\n",
       "zhxOOeUUvve97wFw8803c8oppzBnzhwuu+wyAJ544gnOOOMMmpubOemkk1i7du1u34vxZsCXJFWM\n",
       "TZsmkh+aenoO5KWXXtqta77pTW+ipaWFn//858yZM4dPfOITPPDAA1x00UVcfPHFrFq1ij//8z9n\n",
       "xYoVLFy4kMmTJ3PRRRfR3t7OgQceSHt7OytXruSqq65i48aN3HnnnYPHp0+fTn9/P08++SRf+MIX\n",
       "aGtr46yzzuKOO+7YzTsx/txSVpJUMWbPnsK3v/0oPT3vBfr58Id/xZQpR+7WNX/zm9+w33778Y//\n",
       "+I8A/OpXv+LMM8+kr6+PJUuWALB161amTZv2uvOefvppPvShDwEwadIkpk+fznPPPceyZcu4/vrr\n",
       "ef755zniiCPIZDIcdNBBfPOb36S+vp7Nmzez77777laei8GAL0mqGCed9EFuvPFnrF79Hfbf/zWW\n",
       "LDmFPfbYvcboGCO33XYbLS0tTJw4kWnTptHQ0MCkSZO44oorePOb38zPfvazwUGCA3vMTJ8+nYcf\n",
       "fphjjz2Wl19+mSeeeIKpU6fS0tLCJZdcwl577cUZZ5zB2rVrWbZsGcuXL2f69OlcffXV/Pa3v93t\n",
       "ezHeDPiSpIpy4okf4MQTx+96xx13HE899RQnn3wy++yzD5lMhkWLFjFlyhQWLVpEf38/dXV1gy0A\n",
       "06dPZ9GiRSxdupQLL7yQefPm0dvby9/+7d9ywAEHkCQJ8+bNY9KkSUyZMoXDDz+cE088kc997nNM\n",
       "mTKFP/3TP+WFF14YvwKME3fLkySp+rhbniRJ2pkBX5KkFDDgS5KUAgZ8SZJSwIAvSVIKGPAlSUoB\n",
       "A74kqaKM9255TU1N/PjHP37dZ5dddhm33357wdd6/PHHueaaawo654477uBrX/tawWmNNwO+JKli\n",
       "vNTTw6rjjmNjkvDjww6js8Dguiuf/vSnueuuuwbf9/X1sWbNGmbPnl3wtQ477DDOOeecgs6pqyt4\n",
       "ynxRuNKeJKli3L9kCac98EC2NrphA/csXcrLn/nMbq1Nf/zxx/P1r3+dLVu2sPfee9PZ2cmHP/xh\n",
       "vvnNb/Lwww+zbds2mpubmTVrFk1NTfzRH/0RPT09XHTRRXzpS19i4sSJbNu2ja997Ws8++yz3Hbb\n",
       "bVx55ZXcfvvttLe3s23bNo455hjOPfdcvvOd73DjjTey1157ccghh/CVr3zldXm5/vrr+d73vsee\n",
       "e+7JjBkzOP/887n66qtZu3Ytr7zyCkuXLmX69Om7dxOHYA1fklQxJm7a9LrAdGBPz27vlrf33nsz\n",
       "c+ZM7rvvPiDbxP7Wt76V559/nltuuYUbbriB1tZWNm3aBMAJJ5zAypUreeihhzjiiCNYuXIl5557\n",
       "Lps2bRqsrb/44otce+213Hrrrdx555289tpr/Nd//Rff+MY3uPHGG7nlllvYf//9ue222wbzEWOk\n",
       "o6OD2267jfb2dp599lnWrFlDXV0dhx566ODue8ViwJckVYwps2fzaEMDAP3Arz78YaZMmbLb1z3l\n",
       "lFO46667+N3vfsemTZvYY489eOyxx2hqauLMM8+kv79/cMObt73tbUC2K2DfffflzDPP5Oabb2bC\n",
       "hAmD13vuued4xzvewV577QXAeeedxx/+8AcOPfRQ9tlnHwA+8IEP8Otf/3rwnGeeeYbDDz988DpH\n",
       "Hnnk4PEdd+orBgO+JKlifPCkk9hy441852/+hn89/3xOufPO3d4tDyBJEjZv3syqVav41Kc+xdvf\n",
       "/naOOuooVq1axcqVKzn++OM5+OCDAQbT+/73v8+MGTNoa2vj+OOPZ8WKFYPXO/jgg3n66afp6+sD\n",
       "4HOf+xwHHnggTz31FK+++ioAP/nJTwZ/PAC8/e1v59FHH6W/v59MJsPDDz88eHw8yjgS+/AlSRXl\n",
       "AyeeyLhul5fzqU99iuXLl7NmzRre8IY38NOf/pRTTz2VV155heOOO45Jkya97vvvec97+MIXvkBL\n",
       "Swvbtm1j8eLFg836BxxwAGeddRZNTU3U1dVxzDHH8Ja3vIVzzz2X0047jT322INDDjmECy64gNWr\n",
       "V1NXV0eSJHziE59g7ty5bNu2jRkzZnDsscfy+OOPl2Rgn7vlSZJUfdwtT5Ik7cyAL0lSChjwJUlK\n",
       "AQO+JEkpYMCXJCkFDPiSJKWAAV+SpBQw4EuSlAJFX2kvhHAQ8AgwM8b4RN7nnwfOAF7IfXR2/nFJ\n",
       "kjR+ihrwQwgTgW8Bm3dx+P1AU4xxbTHzIEmSit+kvxxoAdbv4tiRwOIQwo9CCF8scj4kSUq1ogX8\n",
       "EEIz8EKM8b7cRzuu+3srcDZwDPCREMIni5UXSZLSrphN+qcDmRDCscARwA0hhBNjjL/PHb8qxvgS\n",
       "QAhhNfA+YPVwFwwhLAEuLl6WJUmqTSXZLS+E8AB5g/JCCA3Ao8C7gFeAfwGuizF2jOHa03C3PElS\n",
       "uhS8W17RR+nnqQshzAX2jTGuyPXbPwBsAb4/lmAvSZJGpyQ1/GKyhi9JSqGCa/guvCNJUgqUsklf\n",
       "UhXp7e2lra0TgObmmdTX15c5R5J2hwFf0k56e3uZNesWurqaAGhvX0VHxzyDvlTFbNKXtJO2ts5c\n",
       "sJ8ITKSra/5gbV9SdTLgS5KUAgZ8STtpbp5JY+MqoA/oo7HxJpqbZ5Y7W5J2g334knZSX19PR8c8\n",
       "2truB6C52f57qdoZ8CXtUn19PQsWuMWFVCts0pckKQUM+JIkpYABX5KkFDDgS5KUAgZ8SZJSwIAv\n",
       "SVIKGPAlSUoBA74kSSlgwJckKQUM+JIkpYABX5KkFHAtfakK9fb2Du5P39w8041tJI3IgC9Vmd7e\n",
       "XmbNuoWuriYA2ttX0dHhbnaShmeTvlRl2to6c8F+IjCRrq75g7V9SRqKAV+SpBQw4EtVprl5Jo2N\n",
       "q4A+oI/Gxptobp5Z7mxJqnD24UtVpr6+no6OebS13Q9Ac7P995JGZsCXqlB9fT0LFnyy3NmQVEVs\n",
       "0pckKQUM+JIkpYBN+tIwXOBGUq0w4EtDcIEbSbXEJn1pCC5wI6mWGPAlSUoBA740BBe4kVRL7MOX\n",
       "huACN5JqiQFfGoYL3EiqFTbpS5KUAkWv4YcQDgIeAWbGGJ/I+3w2cCGwFbg+xnhtsfMiSVJaFbWG\n",
       "H0KYCHwL2LyLz68EjgMagb/J/TCQVMV6e3tpbV1Na+tqent7y50dSXmK3aS/HGgB1u/w+TuBJ2OM\n",
       "PTHG14AHgaOLnBdJRTSwUNHChR9n4cKPM2vWLQZ9qYIULeCHEJqBF2KM9+U+qss7vD/Qk/d+E9BQ\n",
       "rLxIKj4XKpIqWzH78E8HMiGEY4EjgBtCCCfGGH9PNtjvl/fd/YCNI10whLAEuLgIeZUkqabVZTKZ\n",
       "oicSQngAOHtg0F6uD/8x4Ciy/fsPAbNjjDs2/Y/m2tOAZzo7O5k6der4ZVpSQbbvPTAfgMbGm9x7\n",
       "QCqeupG/8nqlnIdfF0KYC+wbY1wRQjgPuJdst8J1Ywn2kiqHCxVJla0kNfxisoYvSUqhgmv4Lrwj\n",
       "SVIKGPClMuvu7mbu3MuZO/dyuru7y50dSTXKtfSlMuru7mbatGvo6VkEwD33XMG6decwefLkMudM\n",
       "Uq2xhi+V0cKFrblgn5273tNzAQsXtpY7W5JqkAFfkqQUMOBLZdTSsoCGhiuAPqCPhobltLQsKHe2\n",
       "JNUg+/ClMpo8eTLr1p3DwoVXAtDSYv+9pOIw4EtlNnnyZG699YvlzoakGmeTviRJKWDAlyQpBQz4\n",
       "So0NGzYwY8a5zJhxLhs2bCh3dipeb28vra2raW1d7b72Ug2wD1+psGHDBqZObaG/Pzs4burUy3j+\n",
       "+YVMmTKlzDmrTNt3vmsCoL19lTvfSVXOGr5S4YQTltLf/w8MLHDT3/9lTjhhabmzVbHa2jpzwT57\n",
       "v7q65tPW1lnubEnaDQZ8SZJSwICvVLj77i8zYcJlDCxwM2HCUu6++8vlzlbFam6eSWPjKgbuV2Pj\n",
       "TTQ3zyx3tiTtBvvwlQpTpkzh+ecXcsIJfw9kfwDYfz+0+vp6Ojrm0dZ2PwDNzfbfS9WuLpPJlDsP\n",
       "uyWEMA14prOzk6lTp5Y7O5IklUJdoSfYpC9JUgoY8CVJSgH78JUavb29g1PLmptnFqVPeixplCJf\n",
       "kmTAVyqUYiGZsaThAjeSSsUmfaVCKRaSGUsaLnAjqVQM+JIkpYABX6lQioVkxpKGC9xIKhXn4Ss1\n",
       "HLQnqYYUPA/fgC9JUvVx4R1JkrQzA74kSSngPPwUqaW+4loqiySVggE/JWppgZdaKosklYpN+ilR\n",
       "Swu81FJZJKlUDPiSJKWAAT8lammBl1oqiySVivPwU6SWBrrVUlkkaQzGf+GdEMIHgfOBA/MSyMQY\n",
       "jyk4e0VgwJckpVDBAX80o/RvBK4G/gMY+HVQ3c0CkiSlzGgC/isxxmvGcvEQwgRgBZCQ/ZGwIMb4\n",
       "WN7xzwNnAC/kPjo7xvjEWNJSunR3d7NwYSsALS0LmDx5clHOqdSug0rNl6TKNWTADyH8Cdkmg7Uh\n",
       "hPOAfwW2DhyPMf5mFNc/AdgWY/xICKERWAr8r7zj7weaYoxrx5J5pVN3dzfTpl1DT88iAO655wrW\n",
       "rTtn2AA+lnMqdb5/peZLUmUbbpT+D4E1wDHAuUAn0JX3GlGM8S7g7NzbacDGHb5yJLA4hPCjEMIX\n",
       "R51rpdrCha25wJ2dh9/Tc8FgzX08z6nU+f6Vmi9JlW3IGn6McRpACOGAGOOL+cdyA+VGJcbYH0Jo\n",
       "A/4COHmHw7cC1wCbgDtDCJ+MMa4e6lohhCXAxaNNW5IkZQ05Sj+EcDDZFoDVwP/MOzQRWB1jPKyQ\n",
       "hEIIbwJ+Arwzxvhq7rP9Y4wv5f5eCPxRjPGyAq87DUfpp8r25vkLAGhoWF5Ak/7oz9nedD4fgMbG\n",
       "myqi6bxS8yWppMZ1lP6lwEeBt/D6JvytwN2juXgIoQmYGmNcBrwKbCM3wj+E0AA8GkJ4F/AK2a6D\n",
       "6wrMv1Jo8uTJrFt3DgsXXglAS8vwgXus59TX19PRMY+2tvsBaG6ujKBaqfmSVNlGMw//CzHGfxrL\n",
       "xUMIbwDagClkWwaWAfsC+8YYV4QQ5gKfB7YA348xXjKGNKZhDV+SlC7jt/BOCOFisrXxOnYx7z7G\n",
       "eGmhiRWDAV+SlEIFB/zhRulvzr3eR3Z63UtkR9nPBMJYcidJkspjuFH6XwUIIXwaODrG2Jt7/y3g\n",
       "wdJkT+OpVIu1jGWBm1KkMZbyl+KeleJ+1Zre3l4629oAmNnc7BgGaTQymcywryRJnkiSZFLe+4Yk\n",
       "SeJI55XqlSTJtCRJMs8991xGQ3v11VczjY3XZaAvA32ZxsbrMq+++uq4p7Nx48ZMQ8Nlg+k0NFyW\n",
       "2bhxY9nTGEv5S3HPSnG/as2rr76aua6xMdMHmT7IXNfYWJT/LUsVruB4OZrtcb8FPBJC+GoI4Urg\n",
       "EeDrxf0ZovFWqsVaxrLATSnSGEv5S3HPSnG/ak1nWxtNXV25Owbzu7oGa/uShjZiwI8xfg2YD6wH\n",
       "ngc+FWP0XyRJkqrJUFX/JElm5/77mSRJTsv9d+B12liaE4rxskl/dLY3T2/JwJYSNOln0yluk/7o\n",
       "0xhL+Utxz0pxv2rNQJP+FshssUlf6VVwvBxuWt4lMcaLc8vi7mpa3ulF/i0yKk7LGz0H7Tlor1Y4\n",
       "aE8a33n4zcC9Mcb1u5mpojLgS5JSaFyX1j0e+EoI4SXgXuB+YM3AOviSJKl6DDloL8Y4N8Z4MNlF\n",
       "d34B/CXwkxBCp1vZSpJUXYar4QMQY3wmhNANdAN/AD5JNvhfXuS8SUOq1P54pZtjC1TJhgz4IYQP\n",
       "kW3WPx74Y+AHwH3A8hjji6XJnrSz7dvDNgHQ3r5qxO1hx3KOVIje3l5umTWLpq7s5qKr2tuZ19Hh\n",
       "/8ZUMYabh/8g8EHgb2OM02OMZ8UYbzfYq9wqdREdpZsLAqnSDdekfyLZ2v0NIYT/Jjtw774Y489K\n",
       "kjNJkjRuhpyWly+E8Dbg47nXu4FfxBj/qsh5GxWn5aXP9ub5+QA0Nt5UQJP+6M+RCjHQpD8/16R/\n",
       "U2OjTfoqpnGdlgdACKEeOAQ4EKgHXmMXC/FIpVJfX09Hxzza2u4HoLl55MA9lnOkQtTX1zOvo4P7\n",
       "c8348xy0pwoz3MI7/wf4EPB24CGyg/Y6Y4y/LF32RmYNX5KUQuNaw38BOBd4JMa4dcxZkiRJZTdk\n",
       "wI8xLi1lRiRJUvGM2Iev0ijFojBjSaNSN3ZxEZ3akfbFatJe/kJ5v3bDWLbYq6RXLWyPu30b1r4M\n",
       "9BVlG9axpLF969bsOZWydWsp7pdKY2Cr2z7I9KVwq9u0l79Q3q/XKTheDhdILx7mddFYEivGqxYC\n",
       "fkvL3bnglcm9tmRaWu4uexpz5izb6Zw5c5aNa77GohT3S6Vxd0tLpm/7g8xsgczdLS3lzlbJpL38\n",
       "hfJ+vU7B8XK4lfbqdvi7bhd/S5KkalDoL4QkSfZIkmT6WH5dFONVCzX87U3UWzKwpchN+qNPY3uT\n",
       "fvacymvSL979UmkMNNFuydXW0tZEm/byF8r79ToFx8sRV9oLIZwLLAUmsb1m/58xxncX+bfIqNTK\n",
       "PHwH7RXGQXu1I+2DsNJe/kJ5vwYV3NI+moC/DjiGbND/EvBR4LAY4xcLzl4R1ErAlySpAAUH/OH6\n",
       "8Af8Psb4NPAL4D0xxjbgzwtNSJIklc9oAv7LIYSPAb8EZocQ3gxMKW62VAy9vb20tq6mtXU1vb29\n",
       "FZPOhg0bmDHjXGbMOJcNGzYULV+SRq+7u5vL587l8rlz6e7uLlo6vb29rG5tZXVra1H/XdLoFt75\n",
       "O+AM4O+BvwYeB5YUMU8qgu27xTUB0N6+qii7xRWazoYNG5g6tYX+/isBmDr1Mp5/fiFTpvibUiqX\n",
       "7u5urpk2jUU9PQBccc89nLNu3biP4RnYYbApt8PgqvZ2dxgsohFr+DHGXwGLgCOAS4E3xhi/XuyM\n",
       "aXy1tXXmgvBEYCJdXfMHB72VM50TTlhKf/8/DH6/v//LnHCCqzpL5dS6cCGLenpy/6+EC3p6aF24\n",
       "cNzT6Wxro6mrazCd+V1dgwPyNP5GDPghhOOAZ4EVQBvwVAjhg0XOlyRJGk8jzdtLkuSxJEmOyHs/\n",
       "I0mSh8cyB7AYr1qYh18KpZq7Xmg669evz0yYcNHg9ydMuCizfv36cc+XpNHbuHFj5rKGhsH57pc1\n",
       "NBRlDQ7n1e+WoszDfyTGeORIn5WL0/JGr1Rz1wtNZ8OGDYPN+Hff/WX776UK0N3dPdiMv6ClpWhr\n",
       "cDivfsyKMg//arKD+1qBfuBU4FBgOUCM8acFZ3McGfAlSSlUcMAfzSj99wAZ4P/kJZIB/in3/mOF\n",
       "JipJkkprxIAfY/xoCfIhSZKKaMSAn2syXwG8DTgauBn46xjjM6M4d0Lu3IRsq8CCGONjecdnAxcC\n",
       "W4HrY4zXjqEMFWcsfeW1tDZ8oevvp/1+laoPs9A+2VLlqxTppL2feCzlT/s9q0kjjepLkuTeJEmO\n",
       "T5JkbW6nvDOTJPnhaEYEJklyUpIk1+b+bkyS5F/zjk1MkuTXSZI05P7+aZIkBxU66rDSRulvH6Xe\n",
       "l4G+UY2GH8s5lWr7DnvZsoy0w17a79fAKOU+yPQVcZTywKjrgXRGGnVdqnyVIp1SlaVSjaX8ab9n\n",
       "VaLgUfqjCaiP5P67Nu+zX4w2gSRJJuT++5kkSVbmff7eJEnuyXt/ZZIkJxdagEoL+C0td+cCUSb3\n",
       "2pJpabl73M+pVHPmLNupLHPmLBvy+2m/X3e3tGT6thckswUyd7e0jHs6y+bM2SmdZXPmlD1fpUin\n",
       "VGWpVGMpf9rvWZUoOOCPZtDeKyGEweHvIYSPAKNe8DjG2B9CaAP+Ajg579D+QE/e+01Aw3DXCiEs\n",
       "AS4ebdqSJClnpF8ESZJ8IEmSXyRJsin33+eSJPkfhf6ySJLkTUmSrEuS5A259+9JkmR13vErkyT5\n",
       "yzFct6Jq+GNZ4KZUi+KUwvYm/WxZRt+kn877VaqFRwpdSKVU+SpFOmlf3GUs5U/7PasSBdfwR5yH\n",
       "DxBC2IvswLs9gMdjjH2j+TERQmgCpsYYl4UQ9gf+HXhXjLE3hDAReAw4CtgMPATMjjGuL+QHSyXO\n",
       "w0/7IDQH7RXGQXsO2is2B+3VpPFdeCc3iv4/YoxPhRD+guyueT8HLo0xbh3p4iGEN5Bdf38K2b0R\n",
       "lgH7AvvGGFeEEE4ALiL7Q+K6GGNLoQWoxIAvSVKRjV/ADyGcD8wBPgNMAH5MdqvcdwN1McbPjT2f\n",
       "48eAL0lKoYID/nC75Z0GNObmzc8D7srNkz8PmDW2/EmSpHIYbpT+thjj5tzfHwNaAGKMmRDCyB3/\n",
       "Kkgt9UmrMhXaJ1vJfbilKItjGFRzhhrNlyTJw0mSvDFJkqlJkvQlSfLm3Od/kiTJo2MZIViMV6WN\n",
       "0h+LWlpIRpWp0IVUKnnhlVKUxYWHVAUKjpfDBdKTc9PofpskyTdzn306SZInkyQ5bSyJFeNVCwG/\n",
       "lhaSUWUqdCGVSl54pRRlceEhVYGC4+WQTfoxxm+HEP4fcGCM8Re5j18BzowxrilF64MkSRonY/mV\n",
       "UEmvWqjh19JCMqpMhS6kUskLr5SiLC48pCpQcLwc1cI7laxWpuU5aE/F5qC90X9/rOeMhYP2NEbj\n",
       "u/BONaiVgC9JUgHGdR6+JEmqEQZ8SZJSYDTb40pKoUI326lktVQWqNw++UrNl7IM+JJ20t3dzTXT\n",
       "prGopweAK+65h3PWravKQFlLZYFsUL1l1iyauroAWNXezryOjrIH10rNl7azSV/STloXLmRRTw8T\n",
       "yW5zeUFPz2ANudrUUlkAOtvaaOrqGizP/K6uwVp1OVVqvrSdAV+SpBQw4EvayYKWFq5oaKAP6AOW\n",
       "NzSwoKWl3Nkak1oqC2T7xlc1Ng6W56bGRmY2N5c5V5WbL23nPHxJu1RLA91qqSxQuYPjKjVfNcqF\n",
       "dyRJSgEX3pEkSTsz4EuSlALOw5eGUUt9kqUoS6n6ymvpuUilYsCXhlBLC4mUoiylWuCmlp6LVEo2\n",
       "6UtDqKWFREpRllItcFNLz0UqJQO+JEkpYMCXhlBLC4mUoiylWuCmlp6LVErOw5eGUUuDwxy0J9UU\n",
       "F96RJCkFXHhHkiTtzGl5UhUqRZP2WNKwqV2qXAZ8qcqUYh76WNJwfrxU2WzSl6pMKeahjyUN58dL\n",
       "lc2AL0lSChjwpSpTinnoY0nD+fFSZXNanlSFHLQnpZ7z8CVJSgHn4UuSpJ0VbVpeCGEicD1wCLA3\n",
       "cFmM8bt5xz8PnAG8kPvo7BjjE8XKjyRJaVbMefinAi/EGJtCCG8E/h34bt7x9wNNMca1RcyDNKiW\n",
       "+pftX69M3mNVsmIG/NuBb+f+3gPYusPxI4HFIYQpwOoY4+VFzItSrpYWhXFRnMrkPValK1offoxx\n",
       "c4zx5RDCfmSD/5d3+MqtwNnAMcBHQgifLFZepFpaFMZFcSqT91iVrqhL64YQDgbuAK6JMbbvcPiq\n",
       "GONLue+tBt4HrB7hekuAi4uQVUmSalrRpuWFEN4ErAE+G2N8YIdjDcCjwLuAV4B/Aa6LMXaMIZ1p\n",
       "OC1PIxhobp2fa269qbGxaptbx1KWWip/pfIeq8QqZx5+COEq4NNAzPt4BTApxrgihDAX+DywBfh+\n",
       "jPGSMaYzDQO+RqGWBlQ5aK8yeY9VQpUT8EvFgC9JSiEX3pEkSTsz4EuSlAJFHaUvjYb9noXr7u6m\n",
       "deFCABa0tDB58uQy50hSpTPgq6xcrKRw3d3dXDNtGot6egC44p57OGfdOoO+pGHZpK+ycrGSwrUu\n",
       "XMiinp7Be3ZBT89gbV+ShmLAlyQpBQz4KquZzc2samykD+gju1jJzObmMueqsi1oaeGKhobBe7a8\n",
       "oYEFLS3lzpakCmcfvsqqvr6eeR0d3J9rxp/noL0RTZ48mXPWrePKXDP+OQ7akzQKLrwjSVL1ceEd\n",
       "SZK0MwO+JEkpYB9+lert7aWtrROA5uaZqev3TvtiPWkvf6XyuaiSGfCrUG9vL7Nm3UJXVxMA7e2r\n",
       "6OiYl5rkUysIAAALeElEQVR/XNK+WE/ay1+pfC6qdDbpV6G2ts5csM8uvdLVNX+wtp8GaV+sJ+3l\n",
       "r1Q+F1U6A74kSSlgwK9Czc0zaWxcBbmlVxobb6K5eWa5s1UyaV+sJ+3lr1Q+F1U65+FXKQftpXtw\n",
       "VNrLX6l8LiqhgufhG/AlSao+LrwjSZJ25rQ8SVWnFE3nNs+r1hjwJVWVUsx3d069apFN+pKqSinm\n",
       "uzunXrXIgC9JUgoY8CVVlVLMd3dOvWqR0/IkVR0H7UnOwy93diRJKgXn4UuSpJ0Z8CVJSgEDviRJ\n",
       "KWDAlyQpBQz4kiSlgAFfkqQUMOBLkpQCBnxJklLAgC9JUgoY8CVJSoE9i3nxEMJE4HrgEGBv4LIY\n",
       "43fzjs8GLgS2AtfHGK8tZn4kSUqrYtfwTwVeiDEeDcwCvjFwIPdj4ErgOKAR+JsQwkFFzo9qRG9v\n",
       "L6tbW1nd2kpvb2+5syNJFa+oNXzgduDbub/3IFuTH/BO4MkYYw9ACOFB4Oi870u71Nvbyy2zZtHU\n",
       "1QXAqvZ25nV0uJuZJA2jqDX8GOPmGOPLIYT9yAb/L+cd3h/oyXu/CWgoZn5UGzrb2mjq6mIiMBGY\n",
       "39U1uI2pJGnXil3DJ4RwMHAHcE2MsT3vUA+wX977/YCNI1xrCXDxeOdRkqRaV5fJZIp28RDCm4A1\n",
       "wGdjjA/scGwi8BhwFLAZeAiYHWNcX2Aa04BnOjs7mTp16nhkWxVuoEl/fq5J/6bGRpv0JaVNXaEn\n",
       "FLuGv5hsM/1FIYSLcp+tACbFGFeEEM4D7iXbtXBdocFe6VRfX8+8jg7uzzXjz2tuNthL0giKWsMv\n",
       "BWv4kqQUKriG78I7kiSlgAFfkqQUMOBLkpQCBnxJklLAgC9JUgoY8CVJSgEDviRJKWDAlyQpBQz4\n",
       "kiSlgAFfkqQUMOBLkpQCBnxJklLAgC9JUgoY8CVJSgEDviRJKWDAlyQpBQz4kiSlgAFfkqQUMOBL\n",
       "kpQCBnxJklLAgC9JUgoY8CVJSgEDviRJKWDAlyQpBQz4kiSlgAFfkqQUMOBLkpQCBnxJklLAgC9J\n",
       "UgoY8CVJSgEDviRJKWDAlyQpBQz4kiSlgAFfkqQUMOBLkpQCBnxJklJgz2InEEI4Crg8xvixHT7/\n",
       "PHAG8ELuo7NjjE8UOz+SJKVRUQN+CGERMB94eReH3w80xRjXFjMPkiSp+E36TwJ/CdTt4tiRwOIQ\n",
       "wo9CCF8scj4kSUq1otbwY4x3hBCmDXH4VuAaYBNwZwjhkzHG1WNIZgLAhg0bxpZJSZKqzMyZM6cB\n",
       "z8cYt472nKL34Q/jqhjjSwAhhNXA+4BhA34IYQlw8a6OnXrqqeOdP0mSKtUzwNuAdaM9oSwBP4TQ\n",
       "ADwaQngX8ApwDHDdSOfFGJcAS3a41t5AL3Ao0D/eea0CAw89rSy/5bf86ZTmskO2/M8XckKpAn4G\n",
       "IIQwF9g3xrgi12//ALAF+H6MsWMsF44xbgkhEGN8avyyWz1yZV9X7nyUi+W3/JY/neVPc9lhsPyj\n",
       "bs6HEgT83AP5UO7vW/M+v5VsP74kSSoyF96RJCkFDPiSJKVArQT8S8qdgTJKc9nB8lv+dEtz+dNc\n",
       "dhhD+esymUwxMiJJkipIrdTwJUnSMAz4kiSlgAFfkqQUMOBLkpQCBnxJklLAgC9JUgqUc7e8MQkh\n",
       "HAQ8AsyMMT6R9/ls4EJgK3B9jPHaMmWxqIYp/+eBM4AXch+dnX+8FoQQfg705N4+HWM8I+9YTT//\n",
       "Ecqehmf/JWA2MBH4RozxhrxjNf3sYcTy1/TzDyF8BmjOvX0DcDjwprzdVmv6+Y+i/KN+/lUV8EMI\n",
       "E4FvAZt38fmVwAyyu+/9WwjhOzHG35c+l8UzVPlz3g80xRjXljZXpRFCqAeIMX5sF8dq+vkPV/ac\n",
       "Wn/2HwX+LMb4oRDCJGBR3rGafvYwfPlzavr5537c3AAQQvgGcG1esKv55z9c+XNG/fyrrUl/OdAC\n",
       "rN/h83cCT8YYe2KMrwEPAkeXOnMlMFT5AY4EFocQfpTbibDWHA7sE0K4N4TQGUI4Ku9YrT//4coO\n",
       "tf/sPw78MoTwr8B3ge/kHav1Zw/Dlx9q//kDEEKYAbx7hxp8Gp4/MGT5oYDnXzUBP4TQDLwQY7wv\n",
       "91Fd3uH92d7cCbAJaChR1kpihPJDdufBs4FjgI+EED5ZwuyVwmZgeYzxeGABcHMIYeB/v7X+/Icr\n",
       "O9T+s/9jsv+onUyu/HnHav3Zw/Dlh9p//gMWA0t2+CwNz3/ArsoPBTz/qgn4wOnAcSGEB4AjgBty\n",
       "/dmQfeD75X13P2BjifNXbMOVH+CqGOOLuV+5q4H3lSOTRfQEuX/oYoy/Bv4beHPuWK0//+HKDrX/\n",
       "7P8A3Bdj3Jrrm+wNIRyYO1brzx6GLz/U/vMnhDAZSGKMXTscSsPzH678UMDzr5o+/Bhj48DfuaB3\n",
       "dl4/zePAO0IIbyRbGzqabPN3zRiu/CGEBuDREMK7yPZjHQNcV5aMFs/pwHuBc0IIbyH7y35D7lit\n",
       "P/8hy56SZ/8g8L+BK3PlnwS8mDtW688ehil/Sp4/ZJ9r5y4+T8PzhyHKX+jzr6Ya/o7qQghzQwhn\n",
       "5X7ZnAfcCzwEXBdj3FU/dy3JL38P8EXgAeCHwK9ijB3lzd64uw7YP4TwQ6CdbBA8JSXPf7iy1/yz\n",
       "jzGuBtaGEH5Ktv/6s8BfpeTZj1T+mn/+OQnw1MCbFP7bP1T5C3r+7pYnSVIKVHMNX5IkjZIBX5Kk\n",
       "FDDgS5KUAgZ8SZJSwIAvSVIKGPAlSUqBqll4R9KuhRBOJjsXd0+yP+JvjDF+dZzTWAJkYoyX7PD5\n",
       "thhj0SoOuZ3QDo0xfn2oPEgaHWv4UhULIbwV+CpwXIzxCODPgDm5QDmeyrVgx5FkVxYsZx6kmmAN\n",
       "X6puB5LdI30SsDHGuDm3f3YvQAjhA2S3D92H7JrsZ8cY14UQ1gC/BD4E1AOfizHeH0L4U+D/AvsC\n",
       "BwFfizFeXWimQgizgEtyeXsGOCvG+GIIYR1wI3B8Ls+nxRh/nku3DZhAdinZWcCJZDeLyYQQns1d\n",
       "+oMhhH8D3gqstLYvjZ41fKmKxRh/AdwFPB1C+EkI4XJgQozxqRDCXsC1wNwY45FkA/+K3KkZYM/c\n",
       "56eS3YxpInAG8JUY4wfJrsu9NPf9HXdnHFII4Y+BZcDHY4zvB+4D/ikv3T/EGI8CWsnuAAbZ/b7/\n",
       "Icb4PrJLiE6IMf4n2e2gW2KMbbk8HAR8lGzN/4Lc/vCSRsGAL1W5GONngUPIBsdDgB+HEP6C7Prb\n",
       "bwe+G0JYC1wOvC3v1Nbc+f8OrAfeA/w9sE9uX+2lZGvhhToK+BNgTS7dc4BD844PrPX9GHBAbuOT\n",
       "Q/LWAL+e7T8w6vL+zgD3xBhfizH+N9kWiwPGkD8plWzSl6pYbu/rfWKMt5NtEm8LIZxJtqa+GHg6\n",
       "V2smhLAHMCXv9P68v/fIvb+d7Pa73yW7Uc9f5Y4X0n++B/BgjPGkXLr1vH4L0968a9bl0s1vQdix\n",
       "NSE/7f4dPh91y4OUdtbwpeq2GVgWQvgTgBBCHfBu4Odktw49IITwkdx3/xq4Ofd3HdmmfEIIM4DJ\n",
       "ZPv0jwUujjF+l2zT+cAPhUIC60+BPwshvCP3/h/Y3qS/kxjjS8CTuX5/gHlsD/KvkR0HMJBnSWNk\n",
       "DV+qYjHGNSGES4G7c33wdWSbzC+NMW4NIXwauCpXy+4BPpM7NQMcGkJ4JPf3X8UYt+Wmvj0YQtgA\n",
       "/Aj4T7LdABmGqOWHEDblvV0XY3xPCOGvgX8JIUwAngPm7+LU/Gt+Brg+hLAUeBR4Nff5D8mOL/jd\n",
       "cHmQNDK3x5VSKITwAPCFGONPy50XgBDChcCKGOOGEMJfkh1o+Oly50uqJdbwJVWC3wD3hxBeA14k\n",
       "OwZB0jiyhi9JUgo4aE+SpBQw4EuSlAIGfEmSUsCAL0lSChjwJUlKgf8PE7G/2DZvnT8AAAAASUVO\n",
       "RK5CYII=\n"
      ],
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11219c5d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn import datasets\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "%matplotlib inline\n",
    "sns.set(style='ticks', palette='Set2')\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import math\n",
    "from __future__ import division\n",
    "\n",
    "data = datasets.load_iris()\n",
    "X = data.data[:100, :2]\n",
    "y = data.target[:100]\n",
    "X_full = data.data[:100, :]\n",
    "\n",
    "setosa = plt.scatter(X[:50,0], X[:50,1], c='b')\n",
    "versicolor = plt.scatter(X[50:,0], X[50:,1], c='r')\n",
    "plt.xlabel(\"Sepal Length\")\n",
    "plt.ylabel(\"Sepal Width\")\n",
    "plt.legend((setosa, versicolor), (\"Setosa\", \"Versicolor\"))\n",
    "sns.despine()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wow! This is nice - the two classes are completely separate. Now this obviously is a toy example, but let's now think about how to create a learning algorithm to give us the probability that given Sepal Width and Sepal Length the plant is Setosa. So if our algorithm returns .9 we place 90% probability on the plant being Setosa and 10% probability on it being Versicolor.\n",
    "\n",
    "#Logisitic Function\n",
    "\n",
    "So we want to return a value between 0 and 1 to make sure we are actually representing a probability. To do this we will make use of the logistic function. The logistic function mathematically looks like this: $$y = \\frac{1}{1 + e^{-x}}$$ Let's take a look at the plot:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": [
       "iVBORw0KGgoAAAANSUhEUgAAAekAAAFbCAYAAADxzHbpAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\n",
       "AAALEgAACxIB0t1+/AAAIABJREFUeJzt3XlwnOWB5/Fvt7p135IlW7fPxwc+MAaMzRkCgXDnmmHY\n",
       "ZEJCJhMmM7M7W7WVma2dsLVbu1UzRWZmk8kBSSbHTC7CEY4ESAwJ2ARjDuMD+7ElW4cl676llvp6\n",
       "9w/JRBhbkm1J79vdv09Vl/T222p+buT++Xnft5/H5zgOIiIi4j1+twOIiIjImamkRUREPEolLSIi\n",
       "4lEqaREREY9SSYuIiHiUSlpERMSjZlXSxpjLjTEvnuH+24wxrxljXjHG3Df38URERFLXjCVtjPlv\n",
       "wMNAxmn3B4GvADcA1wB/Zowpm4+QIiIiqWg2I+l64COA77T71wD11toBa20E2AlcPcf5REREUtaM\n",
       "JW2tfQyInmFXPjAwZXsIKDjXAMaYgDGmzhgTONefFRERSWYXUowDQN6U7Tygb7ofMMY8AHz5TPt2\n",
       "7NhxAVFE5HzU1U18bWx0M4VIyjr9CPX7XEhJHwZWGmOKgBEmDnX/43Q/YK19AHhg6n3GmDrg+AXk\n",
       "EBGRFBB3HELRMMORcYaj44xGw4xEwoxMfj9xixCKhgnFIoSiEUKxMACfX3MVNbnFLv8Jzt25lLQD\n",
       "YIy5G8i11j5sjPkb4DkmDpt/x1p7ch4yiohIkoo7DsORcQbCoSm3MQYjIYbCYwxFxhmKjDEUGWM4\n",
       "EsZh9otCZaYFyEwLkp+eRbo/bR7/FPPH5/YqWKdG0jt27KCqqsrVLCKpRoe7Zb6Nx6L0jI3QMz5M\n",
       "99gIfeOj9I5PfO0bH6U/HCLmxKd9juxAOnnBDHKDmeQGM8gNZJATTH/3a3Ygg5xAOtmBdLICQbLS\n",
       "0skKBPD7PD8VyLwe7hYREWE8FqUjNEjH6CCdY0N0hobpGhuiKzTMYGTsjD/jw0dhehY1uUUUpGdR\n",
       "mJ5NQXoWBemZFKRnkZ+eSf5kKQcSdBQ8F1TSIiIyK+FYlLbRgYnbSD+towO0jw7QOz76vsf68VGc\n",
       "mcOanMWUZOZQkpFLaWYOxRk5FGdOFHKa90e6rlNJi4jI+4SiEZqHe2kc7uHEcB8tI/20jw6+75xw\n",
       "YXoWpqCcxdn5LM7Kpywrj7KsPEoyckjzq4QvlEpaRCTFxR2Hk6MDNAx20TDYTeNQDx2hwffUcWZa\n",
       "gOX5pVTlFFKZU0hFdiEVOQVkB9Jdy50KVNIiIikm5sRpHu7lSH8nRwY6aRjsIhSLvLs/My3AqoJy\n",
       "avOKqcstoSa3mJLMHPy+Ga9zkjmmkhYRSXKO49A5NsQ7fSd5p6+dIwMdjMX+MJFkWVYem/KrWZ5X\n",
       "yvL8UhZnF6iQPUIlLSKShCLxGEcHOnm7p5X9va30jI+8u688K4/LCspZVVDGqsJyCtKzXEwq01FJ\n",
       "i4gkifFYlP29rbzZ3cKBvjbGJ0fLWWlBNpdUs7ZoCWuKFlOametyUpktlbSISAKLxGPs62nl9a4m\n",
       "9ve1EYnHAFiUmcuG8ko2lFSyMr9MV1onKJW0iEiCiTsO9QOd7O5q5I2u5ncv+irPyueS0mouWVRD\n",
       "ZXYhPp1XTngqaRGRBDEYDvFKx3F2tdfTOTYMTHxO+aolK7i8rE7FnIRU0iIiHuY4DkcHu3ixzbK3\n",
       "5wRxxyHoT2Nr2VKuKF/KqoKyRJijWs6TSlpExIMi8RivdzWxo9XSMtIHQGV2IVctWc5li5aSE9Qk\n",
       "IqlAJS0i4iGhaISXTh7lN62HGYyM4cPH5tJqrq8wLM9fpMPZKUYlLSLiAcORMXa0Wn578gij0QiZ\n",
       "aUFuqFzDdRWrKMnMcTueuEQlLSLiolA0zK9PHOY3bYcZj0XJDWRwR+1Grq1YqXmxRSUtIuKGcCzK\n",
       "C22W504cYjQaJj+Yye21G7h68QrS0/TWLBP0myAisoDijsNrnY083riX/nCI7EA6d9Vt4rqKVWSo\n",
       "nOU0+o0QEVkg9QNd/OzYGzQN9xLw+bmpei03Va0lS4e15SxU0iIi82wwPMajx9/k1c5GAC5dVMtd\n",
       "dZt0QZjMSCUtIjJP4o7DzvYGHm/cy2g0TE1uEX+8fAvL8xe5HU0ShEpaRGQetI8O8oOjr9Iw2E1m\n",
       "WoA/WnYJ11as1Oxgck5U0iIicyjuxNnRavlF0z4i8RiXlNbwiWWbKczIdjuaJCCVtIjIHOkIDfL9\n",
       "IxOj57xgBp8x29hcWu12LElgKmkRkQvkOA6/7zzOT+pfZzwe5ZLSGu5evoW89Ey3o0mCU0mLiFyA\n",
       "UDTCf9S/xp6uJjLTgtxntnFpWZ3bsSRJqKRFRM5T41APDx/eSffYCEvzSrhv9XZKM3PdjiVJRCUt\n",
       "InIedrY38OP6PcScODdXr+O2mvWk+XXltswtlbSIyDmIxGP8tOENXm6vJzuQzn2rt7GuqMLtWJKk\n",
       "VNIiIrPUPz7KNw+9zPGhHqpzivjztVfp8LbMK5W0iMgstAz38bWDv6U/HGJrWR33rLhMq1XJvNNv\n",
       "mIjIDPb3tvLwoV2Mx6N8ZOkmbqxcg8/nczuWpACVtIjINF5otfzs2JsE/H4+v+YqTU4iC0olLSJy\n",
       "Bo7j8ETj2zx74h3yg5n8xbprqMsrcTuWpBiVtIjIaeJOnP+o38PO9gbKsvL464uu0wVi4gqVtIjI\n",
       "FJF4jO8cfoW3elqoyS3iL9ddR76m9xSXqKRFRCaFY1G+/s5LHOpvZ1VBGfevvYasQNDtWJLCVNIi\n",
       "IsB4LMq/HvwddqCDjcWVfG7NlQT9aW7HkhSnkhaRlDcei/K1g7/lyEAnm0qq+Nzq7QRU0OIBKmkR\n",
       "SWljsQhfO/A7jg52srmkmvtWb9cc3OIZKmkRSVmReIyvH3yJo4OdXFJaw2fNNhW0eIpKWkRSUiwe\n",
       "56FDO7EDHWwqqeKzq7eR5lNBi7foN1JEUk7ccfjekd+zr7eVNYWLJw5xq6DFg/RbKSIpxXEcfly/\n",
       "h9e6mlieX8oX1l6tq7jFs1TSIpJSnmrez0vt9VTnFPHFddeSoZWsxMNU0iKSMna1N/BM8wFKM3P5\n",
       "q4uuJTuQ7nYkkWmppEUkJRzsa+Pfj75GTiCdv1p3LfnpWW5HEpmRSlpEkl7LcB/fOrQTv8/H/Wuv\n",
       "oTw73+1IIrOikhaRpNY3PsrXDv6WcCzKZ1dvY0XBIrcjicyaSlpEklY4FuWb77xEfzjER5dezObS\n",
       "GrcjiZwTlbSIJCXHcfjh0d00DveyrXwZH6xc7XYkkXOmkhaRpPTciUO81tXEsrxS/mTFpfh8Prcj\n",
       "iZwzlbSIJJ19Pa080biXovRs/nztVZqsRBKWSlpEkkpHaJDv2FcI+NO4f93VFOijVpLApp1qxxjj\n",
       "B74ObADGgfustQ1T9t8F/B3gAN+11n5zHrOKiEwrHIvy0KGdjMUi3GuuoCa32O1IIhdkppH0nUC6\n",
       "tXYb8CXgwdP2fwW4AdgO/FdjTMHcRxQRmZnjOPyofg8nRvq5evEKtpYtdTuSyAWbqaS3A88CWGt3\n",
       "A1tO2x8BCoEswMfEiFpEZMHtbG/g953Hqc0t5hPLL3E7jsicmGlm+XxgcMp2zBjjt9bGJ7cfBN4A\n",
       "RoBHrbWDpz/BVMaYB4Avn2dWEZEzah7u5ScNr5MdSOfP1lypC8Ukacw0kh4E8qY+/lRBG2NqgC8C\n",
       "tUAdUG6M+dh0T2atfcBa65t6A3RMSkTO21g0wkOHdhJ14nzGXEFpZq7bkUTmzEwlvQv4MIAxZiuw\n",
       "b8q+TCAGjE8WdycTh75FRBbMjxtep2tsmBur1rC+uNLtOCJzaqbD3Y8DNxhjdk1u32uMuRvItdY+\n",
       "bIz5PvCKMWYMqAe+N39RRUTe69XO47zaeZy63GLuqN3gdhyROTdtSVtrHeALp919ZMr+fwL+aR5y\n",
       "iYhMqys0xI/q95CZFuC+1dsJ6Dy0JCFNZiIiCScaj/Htw7sYj0W5e8WlLMrKm/mHRBKQSlpEEs4z\n",
       "zQdoHO5la1mdPg8tSU0lLSIJ5dhgN79qeYeSjBz+ePmlbscRmVcqaRFJGOFYlH878nvA4dOrtpIV\n",
       "CLodSWReqaRFJGE8enwvnaEhrq9czarCcrfjiMw7lbSIJIRDfe389uQRlmQXcGfdRrfjiCwIlbSI\n",
       "eF4oGub7R17F7/Nx76orNO2npAyVtIh43qPH99IXHuXD1euozdPyk5I6VNIi4mm2v4OX2+upzC7k\n",
       "5up1bscRWVAqaRHxrHAsyg+O7saHj0+tulyziknKUUmLiGf9omkf3WPD3FC1mrq8ErfjiCw4lbSI\n",
       "eNLxwW52tFrKMnO5rWa923FEXKGSFhHPicXj/ODobhwcPrnyctLTZlqwTyQ5qaRFxHN+3XqYttEB\n",
       "rlq8QpOWSEpTSYuIp3SPDfN0837ygpncVbfJ7TgirlJJi4hnOI7Dj+tfJxKP8fFlF5MTTHc7koir\n",
       "VNIi4hlv9bRwoK+N1YXlXLaozu04Iq5TSYuIJ4SiEX7a8AYBn58/WX4pPp/P7UgirlNJi4gnPNW0\n",
       "j/5wiJuq11Gene92HBFPUEmLiOtaR/p5se0IZZm53FS91u04Ip6hkhYRVzmOw08aXieOwyeWX6IV\n",
       "rkSmUEmLiKte727myEAnG4orWV9c6XYcEU9RSYuIa8ZiER499hYBn59PLNvsdhwRz1FJi4hrftVy\n",
       "kL7wKDdWrWFRVp7bcUQ8RyUtIq7oCA3ymxOHKcrI1jrRImehkhYRVzx67C2iTpyPL92sBTREzkIl\n",
       "LSIL7lBfO2/3trIyv4zNpdVuxxHxLJW0iCyouBPnkWNv4gM+vmyzZhYTmYZKWkQW1M72Y7SO9rO1\n",
       "fBm1ecVuxxHxNJW0iCyYUDTMk01vk+EPcGftBrfjiHieSlpEFswvWw4yFBnnQ9VrKczIdjuOiOep\n",
       "pEVkQXSPDfNCq6UoI5sbKle7HUckIaikRWRBPNH4NlEnzl11G/WRK5FZUkmLyLxrHOphT1cTNblF\n",
       "XLqozu04IglDJS0i88pxHB49/hYAH1u6Gb8+ciUyayppEZlXB/raODLQyUVFFZjCcrfjiCQUlbSI\n",
       "zJuYE+fR43vx4eMjSze5HUck4aikRWTe/L7jOCdHB9hWvozKnEK344gkHJW0iMyLcCzKU037CPrT\n",
       "uL12vdtxRBKSSlpE5sULbUfoD4f4YOVqTVwicp5U0iIy50YiYZ47cZDsQDo3Vq1xO45IwlJJi8ic\n",
       "e771HUajEW6qXkt2IN3tOCIJSyUtInNqIBxiR6ulMD2L65ascjuOSEJTSYvInHqm+QCReIxbatZr\n",
       "+k+RC6SSFpE50xka4uX2esqy8thevsztOCIJTyUtInPmqaZ9xB2HO2o3kObX24vIhdLfIhGZE60j\n",
       "/ezpaqI6p4jNpTVuxxFJCippEZkTTzbtwwHuqNugRTRE5ohKWkQuWNNQL3t7TrA0r4SLiircjiOS\n",
       "NFTSInLBnmx6G4A7ajfi0yhaZM6opEXkgtQPdHGg7ySmoJw1RYvdjiOSVFTSInLeHMfhF5Oj6Ntr\n",
       "N7icRiT5qKRF5Lwd7u/gyEAnFxUtYUXBIrfjiCSdaacDMsb4ga8DG4Bx4D5rbcOU/ZcCDwI+oBX4\n",
       "lLU2PH9xRcQrHMfhqeZ9ANxeu9HlNCLJaaaR9J1AurV2G/AlJgoZAGOMD3gI+LS19ipgB7B0voKK\n",
       "iLcc6m+nYbCbjSVV1OYVux1HJCnNVNLbgWcBrLW7gS1T9q0CeoC/Mcb8Fii01tr5CCki3uI4Dk82\n",
       "TYyib6tZ73IakeQ1U0nnA4NTtmOTh8ABSoFtwFeBDwLXG2Oum/uIIuI1B/tOcnyoh00lVVTnFrkd\n",
       "RyRpzbREzSCQN2Xbb62NT37fA9SfGj0bY55lYqT94tmezBjzAPDl804rIq5zHIenTo2iazWKFplP\n",
       "M42kdwEfBjDGbAX2Tdl3DMg1xiyf3L4KODDdk1lrH7DW+qbe0HlskYRyoK+NxuFeNpdWU5WjUbTI\n",
       "fJppJP04cIMxZtfk9r3GmLuBXGvtw8aYzwI/mryIbJe19lfzGVZE3DVxLno/PuBWnYsWmXfTlrS1\n",
       "1gG+cNrdR6bsfxG4fB5yiYgH7ettpXm4ly2lNVTmFLodRyTpaTITEZkVx3F4uvkAPuAWjaJFFoRK\n",
       "WkRm5UBfG83DvWwuraEip8DtOCIpQSUtIjNyHIenm/YDcEvNRS6nEUkdKmkRmdHBvpPvXtGtc9Ei\n",
       "C0clLSLTmjgXPTGK1hXdIgtLJS0i0zo1u9jFJRpFiyw0lbSInJXjODzTPDFHkc5Fiyw8lbSInNXh\n",
       "/g6ODU2sdKU5ukUWnkpaRM7q1LnoW6o1ihZxg0paRM7oSH8H9YNdXFRUofWiRVyikhaRM3qmReei\n",
       "RdymkhaR92kY7OJwfwdrChezLL/U7TgiKUslLSLvoyu6RbxBJS0i73F8qJuDfSdZVVDGyoIyt+OI\n",
       "pDSVtIi8xy+bDwIaRYt4gUpaRN7VMtzHvt5WluWVYgrK3Y4jkvJU0iLyrl9OuaLb5/O5nEZEVNIi\n",
       "AkDbyABvdbdQk1vMuqIlbscREVTSIjLpVy0HcYBbqtdpFC3iESppEaEzNMSeriYqswvZUFLldhwR\n",
       "maSSFhGebXkHB4eba9bh1yhaxDNU0iIpLu7E+X3nMcqz8rmktNrtOCIyhUpaJMWFYhHijsPN1Wvx\n",
       "+/SWIOIl+hspksLiTpzxWJTSzBwuK6tzO46InEYlLZLCxmIRHBxuqlpHmkbRIp4TcDvAKVde6XYC\n",
       "kdTi4NB9MggE+eQVy92OI5JyGhtnfoz+6SySokLRCMDkZ6J1RbeIF3lmJL1zJ1Tp45kiC2IkEubv\n",
       "9jzB9z56C0UZ2bP6F72ILDyNpEVS0IttlrFYlKxAEI2iRbxLJS2SYsaiEXa0WXIC6WSkBd2OIyLT\n",
       "UEmLpJjfnTzKaDTM9ZWr8WkULeJpKmmRFBKORfl162Gy0oJ8oGKV23FEZAYqaZEU8nJ7PUORMa6r\n",
       "WEVWIN3tOCIyA5W0SIqIxGM8f+IQGf4A11cat+OIyCyopEVSxCsdx+gPh7imYiW5wUy344jILKik\n",
       "RVJALB7nuZZ3CPrT+GDlarfjiMgsqaRFUsCrncfpGR/hqsXLKUjPcjuOiMySSlokycWcOM+2HCTg\n",
       "83Nj1Vq344jIOVBJiyS517ua6BwbZlv5Mooyst2OIyLnQCUtksTiTpxfNh/E7/PxoWqNokUSjUpa\n",
       "JIm92d1Ce2iQK8qWUpqZ63YcETlHKmmRJBV3HJ5pPoAPHzdVr3M7joicB5W0SJJ6u+cEbaMDXF5W\n",
       "S1lWnttxROQ8qKRFkpDz7igabtYoWiRhqaRFktD+3jZaRvrYsqiWxdkFbscRkfOkkhZJMo7j8EzL\n",
       "AQA+rFG0SEJTSYskmYN9J2kc6mFzSTUVOYVuxxGRC6CSFkkijuPwdPN+AG6pvcjlNCJyoVTSIknk\n",
       "nf6THB/qYVNJFVU5RW7HEZELpJIWSRKO4/B008S56Ftr1rucRkTmgkpaJEkc6m/n2FA3G0uqqM7V\n",
       "KFokGaikRZLAxLnoU6NonYsWSRYqaZEkcLi/g4bBLjYUV1KTW+x2HBGZI4Hpdhpj/MDXgQ3AOHCf\n",
       "tbbhDI97COix1v7tvKQUkbNyHIenTl3RrVG0SFKZaSR9J5Burd0GfAl48PQHGGM+D1wEOHMfT0Rm\n",
       "cqi//d1RdF1eidtxRGQOzVTS24FnAay1u4EtU3caY7YBlwHfAnzzEVBEzs5xHJ5qmhhF31arK7pF\n",
       "ks1MJZ0PDE7Zjk0eAscYswT4e+CLqKBFXHGw7yTHhrrZVFKlc9EiSWjac9JMFPTUNe781tr45Pcf\n",
       "A0qBXwKLgWxjzCFr7Q/O9mTGmAeAL59/XBE5Zeq5aH0uWiQ5zVTSu4DbgEeMMVuBfad2WGu/CnwV\n",
       "wBjzp8Dq6Qp68mceAB6Yep8xpg44fo65RVLegb42God6uLikWp+LFklSM5X048ANxphdk9v3GmPu\n",
       "BnKttQ+f9lhdOCayQHQuWiQ1TFvS1loH+MJpdx85w+O+P5ehRGR6+3pbaRru5ZLSGiq10pVI0tJk\n",
       "JiIJJu44PNm0Dx86Fy2S7FTSIgnmze5mToz0c1lZHRU5BW7HEZF5pJIWSSAxJ86TTfvx49MoWiQF\n",
       "qKRFEshrnY10hAbZtngZZVl5M/+AiCQ0lbRIgojGYzzdvJ+Az88t1ZqjWyQVqKRFEsQrHcfoHhvh\n",
       "qiUrKM7McTuOiCwAlbRIAgjHojzTfICgP42bq9e5HUdEFohKWiQB/O7kUfrDIa6rWEVBepbbcURk\n",
       "gaikRTwuFA3zq5aDZKUFualqrdtxRGQBqaRFPO7XJw4zEg3zoeq15AQz3I4jIgtIJS3iYYPhEL9p\n",
       "PUx+MJMPVBi344jIAlNJi3jYL1sOMh6PcmvNejLSZloPR0SSjUpaxKO6x4Z56WQ9pZm5bF+8zO04\n",
       "IuIClbSIRz3VtI+YE+eO2g0E/GluxxERF6ikRTyoZbiP3Z2NVOUUsmVRrdtxRMQlKmkRD3r0+Fs4\n",
       "wEeXXozf53M7joi4RCUt4jEH+9o41N/O2sLFrC1a4nYcEXGRSlrEQ+JOnMeO78UHfGTpxW7HERGX\n",
       "qaRFPGR3ZyMnRvrZWraU6twit+OIiMtU0iIeEY5F+UXjPoL+NG6v2+B2HBHxAJW0iEfsaLP0hUe5\n",
       "vsJQnKGlKEVEJS3iCQPhEL9qOUhuIIMPVWsRDRGZoJIW8YBfNO5jPBbl9roNZAfS3Y4jIh6hkhZx\n",
       "WctwH690NFCRXcCVi5e7HUdEPEQlLeIix3H42bE3cICPL9tMmk9/JUXkD/SOIOKit3tOcGSgk/XF\n",
       "FZq4RETeRyUt4pJIPMbPj7+F3+fjY5q4RETOQCUt4pIXWi1dY8Ncs2Qli7ML3I4jIh6kkhZxQd/4\n",
       "KM80HyA3kMFtNevdjiMiHqWSFnHBY8ffYjwe5a6lG8kJZrgdR0Q8SiUtssCODHTyWlcTtbnFbCvX\n",
       "R65E5OxU0iILKObE+Un96wDcvXyL1ooWkWmppEUW0Esnj9I62s+28mUszS91O46IeJxKWmSBDIZD\n",
       "PNm0j6y0IHfVbXQ7jogkAJW0yAL5+fG3GI1GuL12A/npWW7HEZEEoJIWWQCH+trZ3dlIbW4x11as\n",
       "dDuOiCQIlbTIPIvEY/yoYQ8+fNyz4jL8mp9bRGZJ7xYi8+zZlnfoDA1xXcVKavOK3Y4jIglEJS0y\n",
       "jzpCgzzbcpDC9Cxur9XFYiJyblTSIvPEcRz+4+geok6cP1p+CVmBoNuRRCTBqKRF5smujmPYgQ7W\n",
       "F1dwcUm123FEJAGppEXmQf/4KD8/9iaZaUHuWXEZPs0sJiLnQSUtMsccx+FH9XsIxSJ8dOnFFGVk\n",
       "ux1JRBKUSlpkjr3e3czbva2sKijjysVaQENEzp9KWmQODUfG+GnD6wT9aXxy5eVaQENELohKWmQO\n",
       "/aThDYYi49xRu4GyrDy344hIglNJi8yRPV1N7OlqYlleKddXGrfjiEgSUEmLzIH+8VF+XL+HdH8a\n",
       "nzZbNfWniMwJvZOIXCDHcfjh0d2MRMN8bOlmyrPy3Y4kIklCJS1ygV5ub+BA30nWFi7m6iUr3I4j\n",
       "IklEJS1yAbpCQ/z82JtkB4J8atVWTVoiInNKJS1ynmLxON+2rzAej3L38ks1aYmIzDmVtMh5erJp\n",
       "H41DPWwtq+Oysjq344hIElJJi5yHQ33tPHfiHRZl5nL38kvdjiMiSSow3U5jjB/4OrABGAfus9Y2\n",
       "TNl/N/DXQBTYD9xvrXXmL66I+4bCY3zXvoLP5+O+1dvJ1BKUIjJPZhpJ3wmkW2u3AV8CHjy1wxiT\n",
       "Bfwv4Fpr7ZVAAXDrfAUV8QLHcfj+0VcZjIxxZ91G6vJK3I4kIklsppLeDjwLYK3dDWyZsm8MuMJa\n",
       "Oza5HQBCc55QxEOeP3GI/b1trClczA2Va9yOIyJJbtrD3UA+MDhlO2aM8Vtr45OHtbsAjDF/CeRY\n",
       "a38z3ZMZYx4AvnwBeUVcY/s7eLzxbQrTs/iM2abFM0Rk3s1U0oPA1FUC/Nba+KmNyXPW/wCsAD46\n",
       "03/MWvsA8MDU+4wxdcDxWaUVcclAOMS3D+/C54PPrb6S/PRMtyOJSAqY6XD3LuDDAMaYrcC+0/Z/\n",
       "C8gA7ppy2FskqcTicR46tJPByBgfXXoxKwoWuR1JRFLETCPpx4EbjDG7JrfvnbyiOxd4HfgM8BLw\n",
       "gjEG4F+stU/MV1gRNzzWuJf6wS4uKa3h+gqtbiUiC2fakp487/yF0+4+MuX7tDlPJOIhr3Yc5zet\n",
       "hynPyueTKy/XtJ8isqA0mYnIWRwf7OaHR3eTlRbk/rVXk6XPQ4vIAlNJi5xB3/go3zj0MjHH4XNr\n",
       "trM4W8tPisjCU0mLnCYci/LNd15iIBzio0s3sa6owu1IIpKiVNIiU8Qdh+8feZXG4V6uKFvKBytX\n",
       "ux1JRFKYSlpkiscb9/J6dzPL8xdxz8rLdKGYiLhKJS0y6cW2Izx/4hDlWfncv/Zqgn59eEFE3KWS\n",
       "FgHe7jnBTxveIC+YyV+uu5bcYIbbkUREVNIixwa7efjwLgJ+P19cdw2LsnLdjiQiAqikJcWdGOnj\n",
       "qwdfJBaP87nV27X0pIh4ikpaUlZHaJB/2f8io9EIf7pqKxtLqtyOJCLyHippSUm94yP88/4XGIyM\n",
       "8cfLL2Fr+VK3I4mIvI9KWlLOQDjEP+9/kd7xUe6o3cB1WjRDRDxKJS0pZSAc4sF9O+gIDXJj1Rpu\n",
       "rl7ndiQRkbOaaalKkaTRPz7KV/bvoCM0xI1Va/hI3SZNViIinqaSlpTQN1nQnaEhPlS1lrvqNqqg\n",
       "RcTzVNKS9LpCw/zzgRfoHhvm5up13FG7QQUtIglBJS1J7cRIH/+y/0UGI2PcWnMRt9asV0GLSMJQ\n",
       "SUvSqh/o5GsHf0coFuGPll3CByp1FbeIJBaVtCSlfT2tPHR4JzEnzmfMFVxeps9Bi0jiUUlL0nmh\n",
       "1fKzY28S8Pu5f+3VrC+udDuSiMh5UUlL0og5cX7W8Ca/PXmE/GAmf7HuGs3FLSIJTSUtSSEUjfDt\n",
       "wzs50HeSyuxCvrjuGoozc9yOJSJyQVTSkvDaRgb45qGX6QgNclHREu5bfSVZgaDbsURELphKWhLa\n",
       "m93NfO+Fh5ptAAAKMUlEQVTIq4zHotxQuYa7lm4kzafZbkUkOaikJSHF4nGeaHqb508cIsMf4HOr\n",
       "t7NlUa3bsURE5pRKWhJO99gw3zn8CseGuinLyuMLa66iIqfQ7VgiInNOJS0J5Y2uZn54dDehWIQt\n",
       "pTX8p5WXkRVIdzuWiMi8UElLQghFIzxy7E12dTSQ7k/jUysvZ1v5Mk3xKSJJTSUtnneor50fHH2V\n",
       "3vFRqnOKuG/1NhZnF7gdS0Rk3qmkxbPGohEea9zL704exY+PW6ov4sM16wj409yOJiKyIFTS4jmO\n",
       "47C35wQ/bXiDvvAoFdkFfHrVFdTmFbsdTURkQamkxVO6x4b5ScPr7O9tI83n55bqi7i5Zh1BjZ5F\n",
       "JAWppMUTwrEoz504xHMn3iESj2EKyvmTFVt07llEUppKWlwVdxxe62zk8ca99IdD5Acz+eTKy7hs\n",
       "UZ2u3BaRlKeSFlc4jsPh/g6eaNxL43AvAZ+fm6vXcVPVWjI177aICKCSFhfUD3Txi6a3OTLQCcCl\n",
       "i2q5q24TJVq1SkTkPVTSsmDqBzr5ZctBDvadBOCioiXcUbeRmlxdtS0iciYqaZlXjuNwsO8kv2o5\n",
       "SP1gFwCrCsq4o3YjKwoWuZxORMTbVNIyL8KxKK91NfFCq6V1tB+A9cUV3FS1TuUsIjJLKmmZU71j\n",
       "I7zUXs9LJ+sZiY7jx8eli2r5UNVaqnOL3I4nIpJQVNJywWJOnP29bbx8sp6DfW04QE4gnZuq13Lt\n",
       "klUUZWS7HVFEJCGppOW8tQz3sbuzkde6GhkIhwBYmlfClYtXcNmiWtLT9OslInIh9C4q56QrNMyb\n",
       "3c3s7mx891xzdiDItUtWcdWS5VTl6JC2iMhcUUnLjDpGB3mrp4U3uptpHu4DIM3nZ1NJFZeXLWV9\n",
       "cYXm1hYRmQcqaXmfWDxOw2AX+3pb2dfbSkdoCAC/z8e6oiVsLq3h4pIqcoIZLicVEUluKmnBcRy6\n",
       "xoY51NfOwf6T2P52xmJRADL8ATaVVLGxpIqNxZUqZhGRBaSSTkGnSvnoQCdHBjqwA530jY++u78s\n",
       "M5fLy5awobgSU1iuQ9kiIi5RSaeAsViE5uE+jg920zDUzbHBLoYi4+/uzw1ksLmkmjVFi1lTuIRF\n",
       "WbkuphURkVNU0kkmFA3TMtLPieE+mod7aRzupX10AGfKY4oystlSWsOKgkWYgnKWZBdoWUgREQ9S\n",
       "SSeocCxKR2iIttF+2kYHaBsZoG20n+6xkfc8LsMfYEV+GXV5xdTllbAsv5TiDK02JSKSCFTSHhaJ\n",
       "x+geG6YrNEzn2BBdoSE6QkN0hAbpnXIO+ZS8YAZrChdTlVNEdW4h1TlFLM7Ox+/zu5BeREQulEra\n",
       "JY7jMBwZpz8com98lN7xEXonv/aMjdA9NsxgZOyMP1uYnoUpKGdxdj5LsvOpyC6kIruAvPTMBf5T\n",
       "iIjIfFJJzyHHcQjFIgxHxhmKjDEUHmMwMs5QJMRAeIyBcIjBya8D4RBRJ37G5/HjozgzG5NdTmlm\n",
       "DmVZeSzKzGNRVi5lmXlkBoIL/CcTERE3TFvSxhg/8HVgAzAO3GetbZiy/zbgfwBR4LvW2m/PY9YF\n",
       "dapIx6IRQrEIoWiE0Wh44hYLMxqNMBoZZyQaZiQyzvDk1/h7LtF6P7/PR34wk6rcIgrTsyhMz6Yo\n",
       "I5viyVtRRg6FGVmk6RC1iEjKm2kkfSeQbq3dZoy5HHhw8j6MMUHgK8AWYBTYZYx50lrbOZ+BF8Kx\n",
       "wW7+4e3nZ6jbCT58ZAeC5AYzKMvKJTeQQU4wg/xgJnnpmeQFM8gLZlKQnkV+MJOcYAZ+XUktIiKz\n",
       "MFNJbweeBbDW7jbGbJmybw1Qb60dADDG7ASuBn4+H0EXUnlWPh+oMDg4ZKWlkxUIkpkWJDuQPuUW\n",
       "JCeYQWZaUKUrIiLzYqaSzgcGp2zHjDF+a218ct/AlH1DQMF5ZEgDaG9vP48fnT/bMsrfe0ds8jYe\n",
       "BaKMM3H8XyQZnDjhdgKR1HP99dfXASestdGzPWamkh4E8qZsnypomCjoqfvygL7pnswY8wDw5TPt\n",
       "u+eee2aIIiJzLWNyKvbrr3c3h0iKOg4sBRrP9oCZSnoXcBvwiDFmK7Bvyr7DwEpjTBEwwsSh7n+c\n",
       "7smstQ8AD0y9zxiTAYwBK5gYq8r0Tv1PlenpdZo9vVazo9dpdvQ6zd5xYNrjWD7HOfvlUcYYH3+4\n",
       "uhvgXuASINda+7Ax5lbg7wE/8B1r7TfOJ6UxxrHW6sTuLOi1mh29TrOn12p29DrNjl6n2ZvNazXt\n",
       "SNpa6wBfOO3uI1P2Pw08fd4JRURE5Kz0YVwRERGPUkmLiIh4lFdK+n+6HSCB6LWaHb1Os6fXanb0\n",
       "Os2OXqfZm/G1mvbCMREREXGPV0bSIiIichqVtIiIiEeppEVERDxKJS0iIuJRKmkRERGPUkmLiIh4\n",
       "1EwLbCwIY0wa8BUm5gVPB/7eWvusu6m8yxizGngVKLPWht3O40XGmALg35lYnS0d+Btr7avupvIO\n",
       "Y4yfP8zLPw7cZ61tcDeVNxljgsB3gVogA/jf1tqn3E3lXcaYMuAN4Hpr7ZGZHp+KjDF/y8TiVUHg\n",
       "a9ba75/tsV4ZSX8SCFhrrwTuBNa4nMezjDH5wINMrBwmZ/dfgF9ba68FPg38q6tpvOdOIN1auw34\n",
       "EhO/U3Jm9wBd1tqrgZuAr7mcx7Mm/0HzLSZWRpQzMMZcC1wx+XfvWmDZdI/3SknfCLQaY54GHgZ+\n",
       "4XIeT5pclexbwN8CIZfjeN0/AQ9Nfh9Er9fptgPPAlhrdwNb3I3jaY8wsdofTLxnRl3M4nX/CHwD\n",
       "OOl2EA+7EdhvjHkCeAp4croHL/jhbmPMZ4H/fNrdXUDIWnurMeZq4N+AaxY6m5ec5XVqAn5ird1n\n",
       "jAHQcnCc9bX6tLX2DWPMYuCHwF8vfDJPywcGp2zHjDF+a23crUBeZa0dATDG5DFR2P/d3UTeZIz5\n",
       "NBNHHJ6fPJyr96czWwRUA7cyMYp+Elh9tgd7YlpQY8yPgUestY9Nbp+01i5xOZbnGGOO8ocFwrcC\n",
       "uycP58oZGGPWAz8G/qu19jm383iJMeZB4FVr7SOT2y3W2mqXY3mWMaYaeAz4V2vt91yO40nGmN8B\n",
       "zuRtE2CBO6y1Ha4G8xhjzP9l4h8zX5nc3gt80FrbfabHe+LCMWAn8GHgMWPMRiZGjHIaa+3KU98b\n",
       "Y44zcdhEzsAYs5aJUc/HrbX73c7jQbuYuHDlEWPMVmCfy3k8yxhTDjwP3G+tfdHtPF5lrX336Kcx\n",
       "5kXg8yroM9rJxJG9rxhjKoAcoOdsD/ZKST8MfMMY8/vJ7T93M0yCcP8QiLf9Hyau6v5/k6cG+q21\n",
       "d7kbyVMeB24wxuya3L7XzTAe93dAAfD3xphT56Zvttbq4k05Z9baZ4wxVxtjXmPiGof7rbVnfT/3\n",
       "xOFuEREReT+vXN0tIiIip1FJi4iIeJRKWkRExKNU0iIiIh6lkhYREfEolbSIiIhHqaRFREQ86v8D\n",
       "AVL4PIwC7VIAAAAASUVORK5CYII=\n"
      ],
      "text/plain": [
       "<matplotlib.figure.Figure at 0x111084510>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x_values = np.linspace(-5, 5, 100)\n",
    "y_values = [1 / (1 + math.e**(-x)) for x in x_values]\n",
    "plt.plot(x_values, y_values)\n",
    "plt.axhline(.5)\n",
    "plt.axvline(0)\n",
    "sns.despine()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see why this is a great function for a probability measure. The y-value represents the probability and only ranges between 0 and 1. Also, for an x value of zero you get a .5 probability and as you get more positive x values you get a higher probability and more negative x values a lower probability.\n",
    "\n",
    "#Make use of your data\n",
    "\n",
    "Okay - so this is nice, but how the heck do we use it? Well we know we have two attributes - Sepal length and Sepal width - that we need to somehow use in our logistic function. One pretty obvious thing we could do is: \n",
    "\n",
    "$$x = \\beta_{0} + \\beta{1}SW + \\beta_{2}SL $$ \n",
    "\n",
    "Where SW is our value for sepal width and SL is our value for sepal length. For those of you familiar with [Linear Regression](http://en.wikipedia.org/wiki/Linear_regression) this looks very familiar. Basically we are assuming that x is a linear combination of our data plus an intercept. For example, say we have a plant with a sepal width of 3.5 and a sepal length of 5 and some oracle tells us that $\\beta_{0} = 1$, $\\beta_{1} = 2$, and $\\beta_{2} = 4$. This would imply: \n",
    "\n",
    "$$x = 1 + (2 * 3.5) + (4 * 5) = 28$$ \n",
    "\n",
    "Plugging this into our logistic function gives: \n",
    "\n",
    "$$\\frac{1}{1 + e^{-28}} = .99$$ \n",
    "\n",
    "So we would give a 99% probability to a plant with those dimensions as being Setosa.\n",
    "\n",
    "#Learning\n",
    "\n",
    "Okay - makes sense. But who is this oracle giving us our $\\beta$ values? Good question! This is where the learning in machine learning comes in :). We will learn our $\\beta$ values.\n",
    "\n",
    "##Step 1 - Define your cost function\n",
    "\n",
    "If you have been around machine learning, you probably hear the phrase \"cost function\" thrown around. Before we get to that, though, let's do some thinking. We are trying to choose $\\beta$ values in order to maximize the probability of correctly classifying our plants. That is just the definition of our problem. Let's say someone did give us some $\\beta$ values, how would we determine if they were good values or not? We saw above how to get the probability for one example. Now imagine we did this for all our plant observations - all 100. We would now have 100 probability scores. What we would hope is that for the Setosa plants, the probability values are close to 1 and for the Versicolor plants the probability is close to 0.\n",
    "\n",
    "But we don't care about getting the correct probability for just one observation, we want to correctly classify all our observations. If we assume our data are [independent and identically distributed](http://en.wikipedia.org/wiki/Independent_and_identically_distributed_random_variables), we can just take the product of all our individually calculated probabilities and that is the value we want to maximize. So in math: $$\\prod_{Setosa}\\frac{1}{1 + e^{-(\\beta_{0} + \\beta{1}SW + \\beta_{2}SL)}}\\prod_{Versicolor}1 - \\frac{1}{1 + e^{-(\\beta_{0} + \\beta{1}SW + \\beta_{2}SL)}}$$ If we define the logistic function as: $$h(x) = \\frac{1}{1 + e^{-x}}$$ and x as: $$x = \\beta_{0} + \\beta{1}SW + \\beta_{2}SL$$ This can be simplified to: $$\\prod_{Setosa}h(x)\\prod_{Versicolor}1 - h(x)$$ \n",
    "\n",
    "The $\\prod$ symbol means take the product for the observations classified as that plant. Here we are making use of the fact that are data are labeled, so this is called supervised learning. Also, you will notice that for Versicolor observations we are taking 1 minus the logistic function. That is because we are trying to find a value to maximize, and since Versicolor observations should have a probability close to zero, 1 minus the probability should be close to 1. So now we know that we want to maximize the following: $$\\prod_{Setosa}h(x)\\prod_{Versicolor}1 - h(x)$$ \n",
    "\n",
    "So we now have a value we are trying to maximize. Typically people switch this to minimization by making it negative: $$-\\prod_{Setosa}h(x)\\prod_{Versicolor}1 - h(x)$$ Note: minimizing the negative is the same as maximizing the positive. The above formula would be called our cost function.\n",
    "\n",
    "##Step 2 - Gradients\n",
    "\n",
    "So now we have a value to minimize, but how do we actually find the $\\beta$ values that minimize our cost function? Do we just try a bunch? That doesn't seem like a good idea...\n",
    "\n",
    "This is where [convex optimization](http://en.wikipedia.org/wiki/Convex_optimization) comes into play. We know that the logistic cost function is [convex](http://en.wikipedia.org/wiki/Convex_function) - just trust me on this. And since it is convex, it has a single global minimum which we can converge to using [gradient descent](http://en.wikipedia.org/wiki/Gradient_descent).\n",
    "\n",
    "Here is an image of a convex function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"http://www.me.utexas.edu/~jensen/ORMM/models/unit/nonlinear/subunits/terminology/graphics/convex1.gif\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import Image\n",
    "Image(url=\"http://www.me.utexas.edu/~jensen/ORMM/models/unit/nonlinear/subunits/terminology/graphics/convex1.gif\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you can imagine, that this curve is our cost function defined above and that if we just pick a point on the curve, and then follow it down to the minimum we would eventually reach the minimum, which is our goal. [Here](http://vis.supstat.com/2013/03/gradient-descent-algorithm-with-r/) is an animation of that. That is the idea behind gradient descent.\n",
    "\n",
    "So the way we follow the curve is by calculating the gradients or the first derivatives of the cost function with respect to each $\\beta$. So lets do some math. First realize that we can also define the cost function as:\n",
    "\n",
    "$$-\\sum_{i=1}^{100}y_{i}log(h(x_{i})) + (1-y_{i})log(1-h(x_{i}))$$\n",
    "\n",
    "This is because when we take the log our product becomes a sum. See [log rules](http://www.mathwords.com/l/logarithm_rules.htm). And if we define $y_{i}$ to be 1 when the observation is Setosa and 0 when Versicolor, then we only do h(x) for Setosa and 1 - h(x) for Versicolor. So lets take the derivative of this new version of our cost function with respect to $\\beta_{0}$. Remember that our $\\beta_{0}$ is in our x value. So remember that the derivative of log(x) is $\\frac{1}{x}$, so we get (for each observation): \n",
    "\n",
    "$$\\frac{y_{i}}{h(x_{i})} + \\frac{1-y_{i}}{1-h(x_{i})}$$ \n",
    "\n",
    "And using the [quotient rule](https://www.math.hmc.edu/calculus/tutorials/quotient_rule/) we see that the derivative of h(x) is:\n",
    "\n",
    "$$\\frac{e^{-x}}{(1+e^{-x})^{2}} = \\frac{1}{1+e^{-x}}(1 - \\frac{1}{1+e^{-x}}) = h(x)(1-h(x))$$ \n",
    "\n",
    "And the derivative of x with respect to $\\beta_{0}$ is just 1. Putting it all together we get:\n",
    "\n",
    "$$\\frac{y_{i}h(x_{i})(1-h(x_{i}))}{h(x_{i})} - \\frac{(1-y_{i})h(x_{i})(1-h(x_{i}))}{1-h(x_{i})}$$ \n",
    "\n",
    "Simplify to: \n",
    "\n",
    "$$y_{i}(1-h(x_{i})) - (1 - y_{i})h(x_{i}) = y_{i}-y_{i}h(x_{i}) - h(x_{i})+y_{i}h(x_{i}) = y_{i} - h(x_{i})$$\n",
    "\n",
    "Bring in the neative and sum and we get the partial derivative with respect to $\\beta_0$ to be:\n",
    "\n",
    "$$\\sum_{i=1}^{100}h(x_{i}) - y_{i}$$ \n",
    "\n",
    "Now the other partial derivaties are easy. The only change is now the derivative for $x_{i}$ is no longer 1. For $\\beta_{1}$ it is $SW_{i}$ and for $\\beta_{2}$ it is $SL_{i}$. So the partial derivative for $\\beta_{1}$ is:\n",
    "\n",
    "$$\\sum_{i=1}^{100}(h(x_{i}) - y_{i})SW_{i}$$ \n",
    "\n",
    "For $\\beta_{2}$: \n",
    "\n",
    "$$\\sum_{i=1}^{100}(h(x_{i}) - y_{i})SL_{i}$$\n",
    "\n",
    "##Step 3 - Gradient Descent\n",
    "\n",
    "So now that we have our gradients, we can use the gradient descent algorithm to find the values for our $\\beta$s that minimize our cost function. The gradient descent algorithm is very simple:\n",
    "* Initially guess any values for your $\\beta$ values\n",
    "* Repeat until converge:\n",
    "    * $\\beta_{i} = \\beta_{i} - (\\alpha *$ gradient with respect to $\\beta_{i})$ for $i = 0, 1, 2$ in our case\n",
    "\n",
    "Here $\\alpha$ is our learning rate. Basically how large of steps to take on our cost curve. What we are doing is taking our current $\\beta$ value and then subtracting some fraction of the gradient. We subtract because the gradient is the direction of greatest increase, but we want the direction of greatest decrease, so we subtract. In other words, we pick a random point on our cost curve, check to see which direction we need to go to get closer to the minimum by using the negative of the gradient, and then update our $\\beta$ values to move closer to the minimum. Repeat until converge means keep updating our $\\beta$ values until our cost value converges - or stops decreasing - meaning we have reached the minimum. Also, it is important to update all the $\\beta$ values at the same time. Meaning that you use the same previous $\\beta$ values to update all the next $\\beta$ values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##Gradient Descent Tricks\n",
    "I think most of this are from [Andrew Ng's machine learning course](https://www.coursera.org/course/ml)\n",
    "* Normalize variables:\n",
    "    * This means for each variable subtract the mean and divide by standard deviation.\n",
    "* Learning rate:\n",
    "    * If not converging, the learning rate needs to be smaller - but will take longer to converge\n",
    "    * Good values to try ..., .001, .003, .01, .03, .1, .3, 1, 3, ...\n",
    "* Declare converges if cost decreases by less than $10^{-3}$ (this is just a decent suggestion)\n",
    "* Plot convergence as a check\n",
    "\n",
    "#Lets see some code\n",
    "\n",
    "Below is code that implements everything we discussed. It is vectorized, though, so things are represented as vectors and matricies. It should still be fairly clear what is going on (I hope...if not, please let me know and I can put out a version closer to the math). Also, I didn't implement an intercept (so no $\\beta_{0}$) feel free to add this if you wish :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def logistic_func(theta, x):\n",
    "    return float(1) / (1 + math.e**(-x.dot(theta)))\n",
    "def log_gradient(theta, x, y):\n",
    "    first_calc = logistic_func(theta, x) - np.squeeze(y)\n",
    "    final_calc = first_calc.T.dot(x)\n",
    "    return final_calc\n",
    "def cost_func(theta, x, y):\n",
    "    log_func_v = logistic_func(theta,x)\n",
    "    y = np.squeeze(y)\n",
    "    step1 = y * np.log(log_func_v)\n",
    "    step2 = (1-y) * np.log(1 - log_func_v)\n",
    "    final = -step1 - step2\n",
    "    return np.mean(final)\n",
    "def grad_desc(theta_values, X, y, lr=.001, converge_change=.001):\n",
    "    #normalize\n",
    "    X = (X - np.mean(X, axis=0)) / np.std(X, axis=0)\n",
    "    #setup cost iter\n",
    "    cost_iter = []\n",
    "    cost = cost_func(theta_values, X, y)\n",
    "    cost_iter.append([0, cost])\n",
    "    change_cost = 1\n",
    "    i = 1\n",
    "    while(change_cost > converge_change):\n",
    "        old_cost = cost\n",
    "        theta_values = theta_values - (lr * log_gradient(theta_values, X, y))\n",
    "        cost = cost_func(theta_values, X, y)\n",
    "        cost_iter.append([i, cost])\n",
    "        change_cost = old_cost - cost\n",
    "        i+=1\n",
    "    return theta_values, np.array(cost_iter)\n",
    "def pred_values(theta, X, hard=True):\n",
    "    #normalize\n",
    "    X = (X - np.mean(X, axis=0)) / np.std(X, axis=0)\n",
    "    pred_prob = logistic_func(theta, X)\n",
    "    pred_value = np.where(pred_prob >= .5, 1, 0)\n",
    "    if hard:\n",
    "        return pred_value\n",
    "    return pred_prob"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#Put it to the test\n",
    "\n",
    "So here I will use the above code for our toy example. I initalize our $\\beta$ values to all be zero, then run gradient descent to learn the $\\beta$ values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-1.52645347  1.39922382]\n"
     ]
    }
   ],
   "source": [
    "shape = X.shape[1]\n",
    "y_flip = np.logical_not(y) #flip Setosa to be 1 and Versicolor to zero to be consistent\n",
    "betas = np.zeros(shape)\n",
    "fitted_values, cost_iter = grad_desc(betas, X, y_flip)\n",
    "print(fitted_values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So I get a value of -1.5 for $\\beta_1$ and a value of 1.4 for $\\beta_2$. Remember that $\\beta_1$ is my coefficient for Sepal Length and $\\beta_2$ for Sepal Width. Meaning that as sepal width becomes larger I would have a stronger prediction for Setosa and as Sepal Length becomes larger I have more confidence it the plant being Versicolor. Which makes sense when looking at our earlier plot.\n",
    "\n",
    "Now let's make some predictions (Note: since we are returning a probability, if the probability is greater than or equal to 50% then I assign the value to Setosa - or a value of 1):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1,\n",
       "       1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0])"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predicted_y = pred_values(fitted_values, X)\n",
    "predicted_y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "And let's see how accurate we are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "99"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sum(y_flip == predicted_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cool - we got all but 1 right. So that is pretty good. But again note: this is a very simple example, where getting all correct is actually pretty easy and we are looking at training accuracy. But that is not the point - we just want to make sure our algorithm is working.\n",
    "\n",
    "We can do another check by taking a look at how our gradient descent converged:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": [
       "iVBORw0KGgoAAAANSUhEUgAAAfsAAAFqCAYAAAAdsfM0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\n",
       "AAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8XNd93/3P7DPAYF9JgvtySHETF0USKWuXLMlWrdhu\n",
       "G9lPHCuR4zp24rr2K1H8xDEb56kTt0prJ4+aWE5iO3XcxqkUN7asjZIli7LoSCJFcTskuAhcAGLH\n",
       "YB0AM9M/ZkANKZICSAwu5t7v+/XCa3DnDu78LjjE995zzz3Hl8lkEBEREffyO12AiIiIFJbCXkRE\n",
       "xOUU9iIiIi6nsBcREXE5hb2IiIjLKexFRERcLlioDRtj/MAjwDogCTxorT2SW9cA/M+8l18N/J61\n",
       "9puFqkdERMSrChb2wH1A2Fq7xRhzLfBw7jmstWeAWwCMMdcDXwEeLWAtIiIinlXIZvytwJMA1tqd\n",
       "wObzX2CM8QHfAD5lrdXoPiIiIgVQyLAvBxJ5y6lc036+e4G91trDl/MGxpigMWaRMaaQLRQiIiJF\n",
       "rZAhmQDK8pb91tr0ea/5KPDfJrMxY8w24MsXWrd9+/bLqU9ERKRY+aby4kKG/Q6yZ+4/MMZcB+y5\n",
       "wGs2W2t/PpmNWWu3AdvynzPGLAKOXVGVIiIiLlfIsH8cuMMYsyO3/IAx5n4gbq191BhTB/RNxxsl\n",
       "RkemYzMiIiKuVLCwz3W4+9R5Tx/KW98BbJyO92pOdHAVy6ZjUyIiIq7jikF1Tg72OF2CiIjIrOWK\n",
       "sD8xoLAXERG5GFeEfetwH2PplNNliIiIzEquCPt0Rmf3IiIiF+OKsAc43t/ldAkiIiKzkmvC/pjC\n",
       "XkRE5IJcEfaxQJDjAwp7ERGRC3FF2DeVVtE+3M/gWNLpUkRERGYdl4R9JYDO7kVERC7AHWEfrwLU\n",
       "SU9ERORC3BH2uTN7ddITERF5J1eEfVkoSnWkhOP93WQyGafLERERmVVcEfYAi8pq6B8boTs55HQp\n",
       "IiIis4qrwh503V5EROR8rgn7xfFs2Ou6vYiIyLlcE/YLyqrx4dOZvYiIyHlcE/bRQIi5JRW8NdBF\n",
       "KpN2uhwREZFZwzVhD9nr9qPpFK1DfU6XIiIiMmu4LuxBnfRERETyuSrsF5epk56IiMj5XBX2c0sq\n",
       "CPkDOrMXERHJ46qwD/j9LIhXc3qwj2Rq3OlyREREZgVXhT3AorJq0mQ4MdDtdCkiIiKzguvCXoPr\n",
       "iIiInMt1Yb+orBZQj3wREZEJrgv72mgppcGIzuxFRERyXBf2Pp+PxWXVdCUHSYyOOF2OiIiI41wX\n",
       "9vB2U/5bAzq7FxERcWXYnx1cJ6GwFxERcWXYTwyb25zocLgSERER57ky7OOhCPNKKjna38lYOuV0\n",
       "OSIiIo5yZdgDLK+oZyyd4i31yhcREY9zbdivqKgH4FBfu8OViIiIOMu1Yb9cYS8iIgK4OOzLw1Hm\n",
       "lFRwJNFBKp12uhwRERHHuDbsIduUP5pOcVz324uIiIcFC7VhY4wfeARYBySBB621R/LWXwM8DPiA\n",
       "U8DHrLWj01nDiop6Xmg9zOG+dpaW103npkVERIpGIc/s7wPC1totwENkgx0AY4wP+CbwcWvte4Dt\n",
       "wOLpLkDX7UVERAob9luBJwGstTuBzXnrVgBdwH8wxvwUqLTW2ukuoCIcoyFWTnOig1RG1+1FRMSb\n",
       "Chn25UAibzmVa9oHqAW2AH8O3A7cZoy5pRBFrKioJ5kap2WguxCbFxERmfUKds2ebNCX5S37rbUT\n",
       "p9ddQPPE2bwx5kmyZ/7PX2xjxphtwJenWsSKinp+1tbMob52FucmyBEREfGSQp7Z7wDuATDGXAfs\n",
       "yVt3FIgbY5bmlt8D7L3Uxqy126y1vvwvJnGdf2JwncO6bi8iIh5VyDP7x4E7jDE7cssPGGPuB+LW\n",
       "2keNMb8B/H2us94Oa+1PClFEZaSE+micw30dpDNp/D5X320oIiLyDgULe2ttBvjUeU8fylv/PHBt\n",
       "od4/34rKBl5qO8KJgV4WllXPxFuKiIjMGp44zX37FrwzDlciIiIy8zwR9poUR0REvMwTYV8dKaU2\n",
       "Wkpzop207rcXERGP8UTYA6yoaGBofIxTg31OlyIiIjKjPBT2um4vIiLe5Jmw1zj5IiLiVZ4J+9po\n",
       "nJpIKYf72klnMk6XIyIiMmM8E/aQPbsfHB+ldUjX7UVExDs8FfYT1+1tr67bi4iId3gs7BsAjZMv\n",
       "IiLe4qmwr42WUhUu4ZCu24uIiId4Kux9Ph8rqxoZGE9yYqDH6XJERERmhKfCHmBN1RwA9vaccrgS\n",
       "ERGRmeG5sF9VOQc/Pt7sPu10KSIiIjPCc2FfGgqzpLyW4/1dDIyNOF2OiIhIwXku7AHWVs8lA+zv\n",
       "aXO6FBERkYLzZNivqZ4LoKZ8ERHxBE+G/bySSirDMfb1tGrKWxERcT1Phr3P52NN9VwGx5Mc7+92\n",
       "uhwREZGC8mTYA6ypyjbl71VTvoiIuJxnw35lZSMBn583exT2IiLibp4N+1gwxLLyOloGukmMDjtd\n",
       "joiISMF4NuwhewsewL6eVocrERERKRxPh/3qKt2CJyIi7ufpsJ9TUk5NpJT9Pa2kdAueiIi4lKfD\n",
       "fuIWvOHUGEcTnU6XIyIiUhCeDnvIuwVPvfJFRMSlPB/2prKBoM+v++1FRMS1PB/2kUCQFRX1nBzs\n",
       "pSc55HQ5IiIi087zYQ9vT4yzT035IiLiQgp7NAueiIi4m8IeaIiVUx+Nc7C3jfF0yulyREREppXC\n",
       "PmdN9VxGUuM0JzqcLkVERGRaKexz1lU3AbCr84TDlYiIiEwvhX3Oisp64sEIr3eeIK3R9ERExEUU\n",
       "9jkBn58NtfNJjI1wuE9N+SIi4h7BQm3YGOMHHgHWAUngQWvtkbz1nwN+A5hI1k9aaw8Vqp7J2FS7\n",
       "gJ+1NfNaZwumssHJUkRERKZNwcIeuA8IW2u3GGOuBR7OPTdhI/Cr1tpdBaxhSlZU1lMWyjbl/8rS\n",
       "Tfh9avgQEZHiV8g02wo8CWCt3QlsPm/9JuCLxpifGWMeKmAdkxbw+dlQM59+NeWLiIiLFDLsy4FE\n",
       "3nIq17Q/4fvAJ4FbgRuMMe8rYC2TtqluAQCvdrzlcCUiIiLTo5DN+AmgLG/Zb63N7+b+dWttAsAY\n",
       "82NgA/Dji23MGLMN+HIB6jzH8op6ykJRdnWd4FeWbSagpnwRESlyhUyyHcA9AMaY64A9EyuMMRXA\n",
       "m8aYUmOMj+zZ/auX2pi1dpu11pf/BSye7qIDPj8ba+fTP5bkcF/7dG9eRERkxhUy7B8HRowxO8h2\n",
       "zvucMeZ+Y8wnrLV9wEPA88CLwF5r7ZMFrGVKNtVmm/Jf62hxuBIREZErV7BmfGttBvjUeU8fylv/\n",
       "fbLX7Wed5RV1lIei2V75asoXEZEipxS7AH9ugJ2B8SSHetWULyIixU1hfxGbc035r3aqV76IiBQ3\n",
       "hf1FLMs15e/qPElKY+WLiEgRU9hfhN/nZ2PtAgbHk9jeM06XIyIictkU9pewOTfAzmud6pUvIiLF\n",
       "S2F/CUvL66gIx9jVeYJUWk35IiJSnBT2l+D3+dhYO5/B8VFsn5ryRUSkOCns38XZXvkaK19ERIqU\n",
       "wv5dLCmvoypSwmudLSRT406XIyIiMmUK+3fh9/nY0rCEkdS4OuqJiEhRUthPwpaGJfiAHW1HnC5F\n",
       "RERkyhT2k1AbjbOyspHmRAdtQwmnyxEREZkShf0kbW1cCsCOMzq7FxGR4qKwn6Sra5ooDYb5+Zlj\n",
       "uudeRESKisJ+kkL+ANfWL6J/bIQ3u085XY6IiMikKeynYKIp/yU15YuISBFR2E9BU2kVi+LV7O1u\n",
       "pSc55HQ5IiIik6Kwn6KtjUvJkOHnZ445XYqIiMikKOyn6Jq6hYT8AV4+c4R0JuN0OSIiIu9KYT9F\n",
       "sWCYTbUL6BgZ4HBfu9PliIiIvCuF/WW4YaKjnkbUExGRIqCwvwzLyutoiJWxq+sEQ+OjTpcjIiJy\n",
       "SQr7y+Dz+djSsJSxdIpftB93uhwREZFLUthfpusbFuPHp6Z8ERGZ9RT2l6kiHGNtzTxODPZwLNHp\n",
       "dDkiIiIXpbC/ArfMWQHAs6cOOlyJiIjIxSnsr8DKygaaSit5vfMEXSODTpcjIiJyQQr7K+Dz+bh9\n",
       "3krSZHjutHW6HBERkQtS2F+ha+oWUhGO8VJbM8PjY06XIyIi8g4K+ysU9Ae4ec4KRlLj7NBseCIi\n",
       "Mgsp7KfBjXOWEfIHeO6UJZVJO12OiIjIORT20yAeirClYQldyUF2dZ5wuhwREZFzKOynyW1zDT7g\n",
       "mVMHyWg2PBERmUUU9tOkoaScddXzON7fxdF+DbIjIiKzh8J+Gt0+byUAz57UIDsiIjJ7KOyn0fKK\n",
       "ehbEq9jVdZKO4QGnyxEREQEKGPbGGL8x5i+NMS8bY543xiy9yOu+aYz5aqHqmEkTg+xkyPDcaZ3d\n",
       "i4jI7FDIM/v7gLC1dgvwEPDw+S8wxnwSWAO4pkfb5tqFVIZj7Gg7qrnuRURkVihk2G8FngSw1u4E\n",
       "NuevNMZsAX4J+CvAV8A6ZlTA7+fWuYZkepwXWg87XY6IiEhBw74cSOQtp4wxfgBjzBzgD4HPMMmg\n",
       "N8ZsM8Zk8r+AY9Nd9HS4cc4ySoIhnjl5QEPoioiI4woZ9gmgLP+9rLUTw8t9GKgFngB+D/iIMeZj\n",
       "l9qYtXabtdaX/wUsLkThVyoWDHP7vFUMjo9qghwREXFcIcN+B3APgDHmOmDPxApr7Z9bazdba28B\n",
       "/gT4e2vtdwtYy4y7ba6hNBjh2VMHdO1eREQcVciwfxwYMcbsINs573PGmPuNMZ+4wGtd00FvQjQY\n",
       "4r1NqxgaH9N99yIi4qhgoTZsrc0Anzrv6UMXeN13ClWD026eu4JnTh1k++mD3DrPEA9FnC5JREQ8\n",
       "SIPqFFAkEOSu+VcxkhrnmZMHnC5HREQ8SmFfYDc2LqMiHOO505bE6IjT5YiIiAcp7AssHAhy9/yr\n",
       "GE2nePrkfqfLERERD1LYz4AbGpdRFSnhp62H6RsddrocERHxGIX9DAj5A9wzfw1j6RQ/ObHP6XJE\n",
       "RMRjFPYzZEvDYmoipfystZme5JDT5YiIiIco7GdI0B/gfQvWMJ5J80TLXqfLERERD1HYz6DrGhbT\n",
       "ECvjpbYjnB7sc7ocERHxiHcNe2PMHRd47oOFKcfdAj4/H1q8gTQZ/uHoa2Qyrhs4UEREZqGLjqBn\n",
       "jPkVIAL8kTHmS2Rnp8sAIeCLwGMzUqHLrKuex1VVc9jf08ob3ae4uqbJ6ZJERMTlLnVmXw7cAsRz\n",
       "jzfnHq8jG/ZyGXw+H/9myUb8Ph8/OPo6Y+mU0yWJiIjLXfTM3lr7TeCbxpjbrLXbJ543xlRYa3XB\n",
       "+QrMKangljkr2H7asv3UQe6av9rpkkRExMUm00GvxBjzp8aYMmPMAeCoMeYzhS7M7d6/cC1loQhP\n",
       "tOyjV7fiiYhIAU0m7L8M/C3wb4FfAAuBBwpZlBeUBMN8YOF6kulxHj++2+lyRETExSZ165219iDw\n",
       "PuCfrbUDZDvpyRXa2riE+aVVvNJ+nKOJTqfLERERl5pM2J8xxvwFcA3wpDHmYaClsGV5g9/n598u\n",
       "3QTA/zryKmndiiciIgUwmbC/n2zz/c25s/rDuedkGiyvqOeauoUcH+jmlfZjTpcjIiIuNJmwHyB7\n",
       "+92fGmN+SPbe+8GCVuUxH1x8NSF/gMeP7WZ4fMzpckRExGUmE/ZfA+4EvkO2o96twJ8VsiivqY6U\n",
       "cvf81STGRnjs2C6nyxEREZe56H32ee4ENlhrUwDGmB8Bmsllmr23aRWvdrzFi23NbKpbwMrKRqdL\n",
       "EhERl5jMmX2Acw8KgsB4YcrxrqA/wK+tuA4fPv7u8E6SKf2KRURkekwm7L8H/NQY89vGmN8Bnge+\n",
       "X9iyvGlRWQ13Nq2ic2SQf9K99yIiMk0uGfbGmCrgUeArZAfT+TjwiLX2/yt8ad70/gVraIiV8/zp\n",
       "QzT3tTtdjoiIuMBFw94YswE4AGyy1j5hrf0C8BTZXvnrZ6pArwkHgvzaimsB+O7hnYyqOV9ERK7Q\n",
       "pc7sHwZ+xVr75MQT1trfJztU7sOFLszLlpbXcetcw5nhfv655U2nyxERkSJ3qbCvstb+9PwnrbVP\n",
       "AXUFq0gA+MCi9dRG4zxz8iDH+jWUroiIXL5LhX3QGPOO9bnnNDZ+gUUCQT62/FoyZPjuoZ2a915E\n",
       "RC7bpcL+RbIz3p3vS8CrhSlH8pnKBm6as5zTQ3386C0154uIyOW51KA6vw88YYz5f8iOje8HNgLt\n",
       "wL+agdoE+OCiq9nXc5qnTu5nZWUjq6o02I6IiEzNRc/srbUJ4EbgN4HXgFeAX7fWbrXWds1QfZ4X\n",
       "DYb4xMob8Pv8/I19mb7RYadLEhGRInPJ4XKttWlge+5LHLKorIZfXrSefzy2i7+1P+d31tyC3+dz\n",
       "uiwRESkSkxlBT2aB2+etZG31XA70tvHkif1OlyMiIkVEYV8kfD4fH19xHZXhGP/nrT0aXU9ERCZN\n",
       "YV9E4qEoD67cCsC3Dr7MwFjS4YpERKQYKOyLzPKKeu5duJae0SG+c+gVMpmM0yWJiMgsp7AvQnfP\n",
       "v4qVlQ3s6T7F9tPW6XJERGSWu2Rv/CuRG2nvEWAdkAQetNYeyVv/IeD3gAzwPWvtNwpVi9v4fX5+\n",
       "3WzhK6//hP99bBcLSqtYUdngdFkiIjJLFfLM/j4gbK3dAjxE3uQ5xpgA8FXgNuB64LeMMdUFrMV1\n",
       "KsIxfjN3/f4vD7xEx/CAwxWJiMhsVciw3wo8CWCt3QlsnlhhrU0BK621/WQn1QkAowWsxZVWVDbw\n",
       "kaXXMDie5JH9LzA8PuZ0SSIiMgsVMuzLgUTecip/Yh1rbdoY80FgF/A8MHSpjRljthljMvlfwLFC\n",
       "FF5M3jNnGbfOXcHpoT7+2u4gnUk7XZKIiMwyhQz7BFCW/165EfnOstY+BswDIsDHLrUxa+02a60v\n",
       "/wtYPN1FF6MPL9nIVZWNvNl9msePv+F0OSIiMssUMux3APcAGGOuA/ZMrDDGlBtjXjDGhK21GWAQ\n",
       "0Byulyng8/OJVTfQECvj6ZMHePnMUadLEhGRWaSQYf84MGKM2UG2c97njDH3G2M+kZtk538ALxpj\n",
       "fgakc8tymUqCYT591U2UBEN87/AvaO7rcLokERGZJXzFPCiLMWYRcGz79u00NTU5Xc6scKCnjW/s\n",
       "fZ7SUJiHrn4vtdG40yWJiMj0m9JsaBpUx2VWVTXyb5Zuon8sydfffI6EpsQVEfE8hb0L3TJ3BXfN\n",
       "v4r2kQG+sfenDI/rrkYRES9T2LvUfQvX857GZZwY7OEv9r3AaGrc6ZJERMQhCnuX8vl8fGTZZjbV\n",
       "LqA50cGjB3eQSusefBERL1LYu5jf5+cBcz2rKhvZ032K7x5+hXQRd8gUEZHLo7B3uZA/wL+76j0s\n",
       "Lqvhlfbj/ODo65oWV0TEYxT2HhANhPjM6puZU1LBc6ctP2p50+mSRERkBinsPSIeivDZNbdQEynl\n",
       "Ry17+eHxN3SGLyLiEQp7D6mKlPCFdbdTF43zxIl9PHZ8twJfRMQDFPYeUx0t5fPrbqchVs7TJw/w\n",
       "D7qGLyLiegp7D6qKlPD5dbedvYb//SOvqpe+iIiLKew9qiIc4/Nrb6OptJIXWg/zPw7/gnRG9+GL\n",
       "iLiRwt7DysJRPrf2NhbEq9hx5gjfOfSKBt4REXEhhb3HxUMRPrf2trP34T+y/wVGUmNOlyUiItNI\n",
       "YS+UBMP8+zW3srpqDnt7Wnl4z7P0abY8ERHXUNgLANFgiE9fdRNbG5bSMtDDn+5+mrahPqfLEhGR\n",
       "aaCwl7MCfj+/uvyXuHfBWrqSg/zpG8/Q3NfudFkiInKFFPZyDp/Px/sXruVjy69lJDXGf33zOV7v\n",
       "bHG6LBERuQIKe7mgrY1L+czqmwj4/XzzwEs8dWK/Bt8RESlSCnu5qNVVc/nCutupCMd47PhuvnVw\n",
       "B8nUuNNliYjIFCns5ZIWxKv54oa7WFpex6udLXztjafpGB5wuiwREZkChb28q4pwjP+w9lZumrOc\n",
       "k4O9/KfdT7K/p9XpskREZJIU9jIpQX+Ajyy7hl9dfi2jqXG+sfenPHVS1/FFRIqBwl6m5IbGpXx+\n",
       "3e1UhKM8dmw3jx7cwfD4qNNliYjIJSjsZcqWlNeevY7/WmcLf7zrJxzr73S6LBERuQiFvVyWiVnz\n",
       "7p6/mq6RQb72xjM8dWK/psoVEZmFFPZy2QJ+P/ctWs+/X3srZaEojx3fzZ/vfZ6ExtUXEZlVFPZy\n",
       "xVZWNvKlDXezpmou+3vb+KPXf6Le+iIis4jCXqZFWTjKZ1bfxL9espGh8VG+vvd5vt/8qgbhERGZ\n",
       "BRT2Mm18Ph+3z1vJQ1ffyZySCn7aeog/ev0JDvWecbo0ERFPU9jLtFsQr+b/3XAX7226iq6RQR5+\n",
       "czv/84jO8kVEnKKwl4II+QN8cPHV/N76O2iMlfP86UN85fUnOKQpc0VEZpzCXgpqcXktf7Dxbu5s\n",
       "WkXnyCB/tudZ/r75XxjSQDwiIjNGYS8FF/IH+NDiDfzu+jtoiJXzQuthvvzqj/hF+3ENtysiMgMU\n",
       "9jJjlpTX8qWNd/OBhesZTo3x1/Zl/uubz9E2lHC6NBERVwsWasPGGD/wCLAOSAIPWmuP5K2/H/gs\n",
       "MA68CfyWtVaneS4X9Ae4Z8Fqfql+Id9vfpW9Paf5yutPcGfTKu6ev5pwoGAfSRERzyrkmf19QNha\n",
       "uwV4CHh4YoUxJgZ8BbjZWnsDUAG8v4C1yCxTG43zmdU38clV76EsFOWJE/v4j6//mNc6WtS0LyIy\n",
       "zQoZ9luBJwGstTuBzXnrRoDrrbUjueUgoDFWPcbn87Gxdj7bNr+PO+atpDs5xDcPvsR/2fMsx/u7\n",
       "nC5PRMQ1Chn25UD+xdhUrmkfa23GWtsBYIz5baDUWvtsAWuRWSwaCPHhJRvZtul9rK9pojnRwVd3\n",
       "P8Xf2pfpSQ45XZ6ISNEr5AXSBFCWt+y31qYnFnLB/zVgGfChd9uYMWYb8OVprlFmkYZYOb911Y3Y\n",
       "3jP84OjrvNJ+nNc6T3Bn0yrubFpFNBByukQRkaJUyLDfAdwL/MAYcx2w57z1f0W2Of+XJ9Mxz1q7\n",
       "DdiW/5wxZhFwbBpqlVnEVDbwxQ3v5ednjvFPx9/gxy17ebH1MHfNX81Nc5YT8gecLlFEpKj4CtUZ\n",
       "yhjj4+3e+AAPAJuAOPBq7uvFvB/5urX2n6b4HouAY9u3b6epqemKa5bZZyQ1xjMnD/LsqQOMpMap\n",
       "CpfwvgVr2NKwhIBfd46KiGf5pvTiYu75rLD3joGxJE+d3M/zpw8xlk5RH41z78J1bK5biN83pc+8\n",
       "iIgbKOzFvXqTQzxxYh8/a2smnckwp6SCe+avZlPdAgI+nemLiGco7MX9OoYH+HHLm+xsP06aDPXR\n",
       "OHfNX8219YsI6pq+iLifwl68o2N4gKdP7mfHmaOkMmmqIyW8t+kqtjYuVUc+EXEzhb14T09yiKdP\n",
       "HuBnbc2MpVOUh6LcMtdw05xllIYiTpcnIjLdFPbiXYnREZ49dZAXWg8zkhoj7A+wpWEJt81bSX2s\n",
       "7N03ICJSHBT2IsPjY7zU1sxzpy3dySF8wPqaJu6Yt4ql5bX41INfRIrblP6IaYoxcaVYMMQdTau4\n",
       "dZ7h9Y4Wnjl1kN1dJ9nddZIF8WpunrOca+oWapY9EfEEndmLJ2QyGZoTHTx76iBvdJ0iQ4bSYJgt\n",
       "DUu5ac5y6mJxp0sUEZkKndmLnM/n87G8op7lFfV0jwzyYlszL7U188ypAzx76gCrq+Zy89zlrK6a\n",
       "g1/364uIyyjsxXOqo6Xct2g971uwhtc7W/jp6cPs7TnN3p7TVIZjbG1YypbGJdRGdbYvIu6gZnwR\n",
       "oGWgm5fajrCz/TgjqTEAVlU2srVxKVfXNOmefRGZbdSMLzJVC+LVfGRZNR9evIHXOlt4qe0IB3rb\n",
       "ONDbRmkwzOa6hVxfv5hFZTXqyS8iRUdhL5InHAhyfcMSrm9YQttQHzvOHOWVM8d4ofUwL7QepiFW\n",
       "xrX1i7m2fpGa+UWkaKgZX+RdpDJpDvS08Ur7MXZ3nWQsnQJgWXkd19YvYmPtfOKhqMNViojHqBlf\n",
       "ZDoFfH7WVM9lTfVchsfHeL2zhZ3tx7F9Z2hOdPD95ldZVdXI5rqFXF3TREkw7HTJIiLnUNiLTEEs\n",
       "GGJr41K2Ni6lOznIax0t/EvHW+zraWVfTyvf8/lZXTWHzXULWVs9j1gw5HTJIiIKe5HLVR0p5Y6m\n",
       "VdzRtIqO4X7+paOF1zrf4o3uU7zRfYqgz8/KykY21M5nffU8ysJq6hcRZ+iavcg0Oz3Yx66uFnZ1\n",
       "nuTEYA8APnwsr6hjQ8181tc0URMtdbhKESlymghHZLboGB5gV9cJdnWe4Gh/59nnm0orWVs9j/XV\n",
       "81hYVoNft/OJyNQo7EVmo97kEG90nWJP90kO9p5hPJMGoDwUZW31PNZVz2VlZSNRXecXkXensBeZ\n",
       "7UZSYxzoaWNP9yne7D5F/1gSAL/Px7LyOtZUZXv/zy2p0CA+InIhuvVOZLaLBkJsqJ3Phtr5pDNp\n",
       "jvV3sa+7lb09pznU186hvnYeO76bqnAJq6vnsKqykZWVjcRDEadLF5EipDN7kVkmMTrC/p5W9vWc\n",
       "Zl9PG4Pj2bN+HzA/XsWqymz4L6uo05j9It6lM3uRYlYejnJdw2Kua1hMOpOmZaCH/T1tHOht5Uii\n",
       "k5aBHp46uZ+QP8DS8lpMRQOmsoFF8RoCfk3PKyLvpLAXmcX8Pj+LympYVFbDPQtWk0yNc7ivnf29\n",
       "rRzsOcPB3uwXb0HEH2RpRV02/CvqWRCvVviLCKCwFykqkUDw7NC9AP2jIxzqa8f2neFQ7xn297Sy\n",
       "v6cVgLA/wJLyWpaV17Oiop7FZTWEA/ovL+JF+p8vUsTKwlE21S1gU90CAPpGhznUe4bDiQ4O97W/\n",
       "feZPdoz/BfEqlpXXsbS8jqXltZSHY06WLyIzRGEv4iIV4RjX1C/imvpFAAyMJWlOdNDc187hRAdv\n",
       "9XdzrL+LZ04dBKA2GmdZeS1LyutYUlbL3NIKAj41/Yu4jcJexMXioQhX1zRxdU32bpVkapzj/V0c\n",
       "SXRwJNGbE52TAAAPj0lEQVTJ0f4OXmk/zivtx4Fs0//CeA2Ly2tYXFbLkrIaKiMlDu6BiEwHhb2I\n",
       "h0QCQUxltvc+QDqToW0owZFEB8f6uzjW30lzop3DifazP1MZjrGorIaF8RoWlVWzMF5Nqe73Fykq\n",
       "CnsRD/P7fMwtrWBuaQXvmbMMgOHxMd4a6MqFfxfHEp3s7jrJ7q6TZ3+uNhpnYTwb/Avi1cyPV2nA\n",
       "H5FZTGEvIueIBUOszI3YN6EnOcRb/V0cH+jmrf4u3hro5rXOFl7rbDn7mppIKQvi1SyIVzE/XkVT\n",
       "aRWV4ZiG+xWZBRT2IvKuqiIlVEVKuLp2PgCZTIbOkUFODHbTMtBDy0A3LQPd2Rn+uk6c/bl4MJIL\n",
       "/kqa4lXML62iMVau+/9FZpjCXkSmzOfzUReLUxeLs7E2e9tfJpOhd3SYloFuTgz0cHKwlxODPRzo\n",
       "beNAb9vZnw34/MwpKWduSSVNpZXMy32pFUCkcBT2IjItfD7f2RaA9TVvz1UxPD6aDf7cAcCpoV5O\n",
       "D/ZycrCXX3S8/fMlwRBzSiqZV1LBnJJsP4K5JZWUh6MO7I2IuyjsRaSgYsEwyyvqWV5Rf/a5dCZD\n",
       "58gAp3Khf3qwl1NDfRxLdHIk0XHOz5cGI7mWgAoaS8qZkzsYUEuAyOQVPOyNMX7gEWAdkAQetNYe\n",
       "Oe81JcAzwK9ba22haxIRZ/l9PupjZdTHytiQ6wcAMJZOcWY4wenBPk4P9XF6sJfWoT6OJDppPu8g\n",
       "IBoI0hgrp7GknIbcY2OsnLpYmWYDFDnPTJzZ3weErbVbjDHXAg/nngPAGLMZ+EtgLlC88+2KyBUL\n",
       "+QM0lWZ78uebOAhoHUrQOtRHW+7x5GAvxwe6z3mtDx810VIaYmU0xMqoj5Xnvi+nKlKCX60B4kEz\n",
       "EfZbgScBrLU7c+GeL0w2/P9uBmoRkSJ0sYOAdCZN18ggbcMJ2oYSZx/bh/vZ19PKvtykQBOCPj91\n",
       "uRaF+mg8+300u1wVieHXUMHiUjMR9uVAIm85ZYzxW2vTANbalwGMMTNQioi4iT8X3nWxMtZWzztn\n",
       "3fD4KGeG+2kf7ufMcIIzw/1nl1uH+t6xraDPT020lLponNpoWfZug2icumgZtdFSzRgoRW0mPr0J\n",
       "oCxv+WzQT4UxZhvw5ekqSkTcLRYMs6ishkVlNec8n8lkGBhL0j7ST8fwAO3D/bnv++kYGeDMcD/Q\n",
       "+o7tlYei1MXi1ESyBwE10VJqc49VkRJNICSz2kyE/Q7gXuAHxpjrgD2XsxFr7TZgW/5zxphFwLEr\n",
       "K09EvMTn81EWjlIWjrK0vO4d64fGR+kcGaBjeICOkewBQOfIAJ0jgxxLdHGEznf8jJ/sbYc10VJq\n",
       "IqVU5x4nlqsiJQTVaVAcNBNh/zhwhzFmR275AWPM/UDcWvvoDLy/iMiklQTDuWF/q9+xLpVJ05Mc\n",
       "Ohv+XSMDdCUHz35/uK+dQxfYpo/s9MPVkVKqIyVU5R6ro7nHSAmlwYhuJZSC8WUyxdsBfuLMfvv2\n",
       "7TQ1Nb3by0VECmosnaI7OUj3yBBdyUG6Rwazj8lBukYG6RkdIn2Rv7khfyA7KFG4JHdAkD0oqIrE\n",
       "qIqUUBkuoTQY1gGBTJjSB0E9TkREpknIH6Ahlr3v/0LSmTR9oyPZA4Lk0NkDg57ccu/oEO3D/Zfc\n",
       "fmV4IvxjVE48hkuojMSoCpdQHo7qkoG8g8JeRGSG+H3+s0MKL73Ia8bSKXqSQ29/jQ7RmxyiZ3SY\n",
       "nmT2+0N97Rd9Dx8QD0WoCMeoDMeoCJdQEY5SmXusCMeoCMd0UOAxCnsRkVkk5A+cHV3wYsbTKfpG\n",
       "R+gdHaI3OZx9HB0++33f6AidIwOcHOy95HuVBiN5BwBRysMxykPRswcD5aHsoy4fFD+FvYhIkQn6\n",
       "A9me/tHSS75uZHyMvtFhekeH6XvH1wh9udaC0xcYdyBfwOenLBTJHQBEKcsdFJxdzn1fFooQD0U0\n",
       "ONEspLAXEXGpaDBENBiioeTCfQgmjKbGSYyNkBgdITE6TN9Y7nF0JPf8MInREVqHErSkey65LR8+\n",
       "4qEwZbmDgOwBQJTycPZxYrksFKEsFKEkGNEQxjNAYS8i4nHhQJDaQJzaaPySr8tkMiRT4/SNDdM/\n",
       "OkJiLEl/7iChf2wk7/skfaPD79piAG8fHMRDUeLByNnWgdLQ29/HQxHiwejZ78P+gC4rTJHCXkRE\n",
       "JsXn873dWnCROw7ypdJpBsaTZw8GBnIHB/25x+xykoFc68GFhjG+kKDPfzb4S4MRSkNh4sHsAUJp\n",
       "MHzO86XBCPFQmJJg2NOXFxT2IiJSEAG//2zv/8lIZdIMjo0ykDsA6B9LMjieZGBslIHxEQbHkrl1\n",
       "SQbGR+kaGeRk6tKdEPOVBEOUBLMHBKXBMKWhCCUX+H7isTwcIx6KXO7uzyoKexERmRUCPn+20184\n",
       "ClRM6mdS6XTeAUGSwbEkg+Oj2QOD8SRD49mDh8GxUYbGRxkcT3J6qI+xdOpdt+3HxxfW337BYZWL\n",
       "jcJeRESKVsDvz94yOMnWgwmjqfFc+GcPDCa+z3/MZDI0XOIWyGKisBcREc8JB4KEA0EqIyVOlzIj\n",
       "vNtbQURExCMU9iIiIi6nsBcREXE5hb2IiIjLKexFRERcTmEvIiLicgp7ERERl1PYi4iIuJzCXkRE\n",
       "xOUU9iIiIi6nsBcREXE5hb2IiIjLKexFRERcTmEvIiLicgp7ERERl1PYi4iIuJzCXkRExOUU9iIi\n",
       "Ii6nsBcREXE5hb2IiIjLKexFRERcTmEvIiLicgp7ERERl1PYi4iIuJzCXkRExOWChdqwMcYPPAKs\n",
       "A5LAg9baI3nr7wW+BIwDf2Ot/VahahEREfGyQp7Z3weErbVbgIeAhydWGGNCwJ8BdwA3Ab9pjKkv\n",
       "YC0iIiKeVciw3wo8CWCt3Qlszlu3Cmi21vZZa8eAl4AbC1iLiIiIZxWsGR8oBxJ5yyljjN9am86t\n",
       "68tb1w9UXMZ7BADa2touu0gREZFic9ttty0CTlprxyfz+kKGfQIoy1ueCHrIBn3+ujKg51IbM8Zs\n",
       "A758oXUf/ehHL79KERGR4nMMWAwcn8yLCxn2O4B7gR8YY64D9uStOwgsN8ZUAYNkm/D/86U2Zq3d\n",
       "BmzLf84YEwFGgGVAaroKLzIT/+Bepf337v57ed9B+6/9h5OTfbEvk8kUpApjjI+3e+MDPABsAuLW\n",
       "2keNMe8H/pBsv4G/ttb+98t8n4y11jcdNRcj7b/236v77+V9B+2/9n9q+1+wM3trbQb41HlPH8pb\n",
       "/yPgR4V6fxEREcnSoDoiIiIup7AXERFxOTeE/X90ugCHaf+9zcv77+V9B+2/9n8KCtZBT0RERGYH\n",
       "N5zZi4iIyCUo7EVERFxOYS8iIuJyCnsRERGXU9iLiIi4nMJeRETE5Qo5EU5BGWP8vD32fhJ40Fp7\n",
       "xNmqCs8Ycy3wJ9baW4wxy4BvA2lgL/Dp3DDFrmSMCQF/AywEIsAfAwfwyO/AGBMAHgVWABng35H9\n",
       "7H8bD+w/gDGmHngNuI3sPn8b7+z767w9NfhR4Kt4a/9/n+zkaiHgL8hOtvZtPLD/xphfAz6eW4wB\n",
       "64EbgK8zyf0v5jP7+4CwtXYL8BDwsMP1FJwx5nfJ/rGP5J76M+CL1tobAR/wAadqmyEfBTpy+3sX\n",
       "8P+T/Xf3yu/g/UDaWnsD8AfAf8JD+5872PsrsjNl+vDQ598YEwWw1t6S+/oNvLX/NwPX5/7e3wws\n",
       "wUOffWvtdyb+7YFXgd8mO5HcpPe/mMN+K/AkgLV2J7DZ2XJmRDPwQbL/sAAbrbUv5r7/CXC7I1XN\n",
       "nB+Q/YBD9rM7hod+B9baHwKfzC0uAnqATV7Zf7LTYP93oDW37Jl/e7JnciXGmKeMMdtz04Z7af/v\n",
       "BN40xvwT8M/A/8Fbn30AjDGbgaustd9iivtfzGFfDiTyllO5pn3XstY+BoznPZU/veEAUDGzFc0s\n",
       "a+2gtXbAGFNGNvj/gHM/w174HaSMMd8m23z3PTzyGTDGfJxsq87Tuad8eGTfcwaB/2ytfS/Zyzff\n",
       "O2+92/e/juwU6R8mu/9/j7f+/Sd8kbeHyZ3S/hdzOCaAsrxlv7U27VQxDsnf3zKg16lCZooxZj7w\n",
       "HPBda+338eDvwFr7ccAA3wKieavcvP8PAHcYY54Hrga+QzYAJrh53yE7Pfj3AKy1h4EuoCFvvdv3\n",
       "vxN42lo7bq09BIxwbri5ff8xxlQCK6y1L+SemtLfvmIO+x3APQC5Jq09zpbjiF3GmJty398NvHip\n",
       "Fxc7Y0wD8DTwu9bab+ee9szvwBjzq7lOSgDDQAp41Qv7b629yVp7c+6a5W7gY8CTXtj3nAfI9Usy\n",
       "xswl+8f9aQ/t/0tk++lM7H8JsN1D+w9wI7A9b3lKf/uKtjc+8DjZI/0dueUHnCxmhk30uPw88Kgx\n",
       "JgzsB/7RuZJmxBfJHs3/oTFm4tr9Z4FveOR38I/At40xL5DtkfxZ4CDe+gxMyOCtz/9fA39rjJn4\n",
       "g/4A2bN7T+y/tfbHxpgbjTG/IHuS+lvAcTyy/zkrgPw7zqb0+desdyIiIi5XzM34IiIiMgkKexER\n",
       "EZdT2IuIiLicwl5ERMTlFPYiIiIup7AXERFxOYW9iIcYY9K5xwpjzOPTuN3n877fNV3bFZHpobAX\n",
       "8aYqssPOTpeJkbyw1m6Yxu2KyDQo5hH0ROTyfQOYa4z539baDxljPkZ2RD4/2fniP22tTRpjOshO\n",
       "qdkA/BLZWedW55Yt2VkYvwZgjPm5tfZ6Y0zaWus3xpSQnZJ5HdlxvP+LtfbvcpPa3EX2gGMJ2THP\n",
       "Pz1jey7iQTqzF/Gm3wZO54J+NfAg2fnCNwAdwBdyr6sBvmqt3QhcD4zk5hRfBsSAu621vwNgrb3+\n",
       "vPfYRnamurXArcA2Y8za3LrryR4orAPuzdUgIgWiM3sRb8qfHvMWYDmw0xgDECZ7dj9hJ4C19mfG\n",
       "mC5jzKeBlbmfiV/iPW4Bfj33s13GmB8CN5OdsfJla+0ggDHmKFA9DfskIhehsBcRP/AP1trPAhhj\n",
       "4uT9bbDWJnPP/yuyc2n/N+BvyJ71+96xtXO36ztveWK7I3nPZ95lOyJyhdSML+JN47wdvC8Av2yM\n",
       "qTPG+Mhel/+dC/zMbWQPCr4DnCE75WYgty5ljAmc9/rngN8AMMbUAh8AnkfBLjLjFPYi3jIxzeUZ\n",
       "oMUYs91a+wbZM/bngL259X9y3ush29nufmPMvwB/BfwQWJxb90NgtzEmkvczfwRUG2P2kD2g+GNr\n",
       "7e7cek23KTKDNMWtiIiIy+nMXkRExOUU9iIiIi6nsBcREXE5hb2IiIjLKexFRERcTmEvIiLicgp7\n",
       "ERERl/u/PUl8JICi+8EAAAAASUVORK5CYII=\n"
      ],
      "text/plain": [
       "<matplotlib.figure.Figure at 0x111bc2d10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(cost_iter[:,0], cost_iter[:,1])\n",
    "plt.ylabel(\"Cost\")\n",
    "plt.xlabel(\"Iteration\")\n",
    "sns.despine()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see that as we ran our algorithm, we continued to decrease our cost function and we stopped right at about when we see the decrease in cost to level out. Nice - everything seems to be working!\n",
    "\n",
    "Lastly, another nice check is to see how well a packaged version of the algorithm does:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "99"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn import linear_model\n",
    "logreg = linear_model.LogisticRegression()\n",
    "logreg.fit(X, y_flip)\n",
    "sum(y_flip == logreg.predict(X))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cool - they also get 99 / 100 correct. Looking good :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#Advanced Optimization \n",
    "\n",
    "So gradient descent is one way to learn our $\\beta$ values, but there are some other ways too. Basically these are more advanced algorithms that I won't explain, but that can be easily run in Python once you have defined your cost function and your gradients. These algorithms are:\n",
    "\n",
    "* BFGS\n",
    "    * http://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.optimize.fmin_bfgs.html\n",
    "* L-BFGS: Like BFGS but uses limited memory\n",
    "    * http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.fmin_l_bfgs_b.html\n",
    "* Conjugate Gradient\n",
    "    * http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.fmin_cg.html\n",
    "\n",
    "Here are the very high level advantages / disadvantages of using one of these algorithms over gradient descent:\n",
    "\n",
    "* Advantages\n",
    "    * Don't need to pick learning rate\n",
    "    * Often run faster (not always the case)\n",
    "    * Can numerically approximate gradient for you (doesn't always work out well)\n",
    "* Disadvantages\n",
    "    * More complex\n",
    "    * More of a black box unless you learn the specifics\n",
    "    \n",
    "The one I hear most about these days is L-BFGS, so I will use it as my example. To use the others, all you do is replace the scipy function with the one in the links above. All the arguments remain the same. Also, I will now use all 4 features as opposed to just 2.\n",
    "\n",
    "##L-BFGS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ -1.39630462,   5.3512917 ,  -9.41860088, -10.84876254])"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from scipy.optimize import fmin_l_bfgs_b\n",
    "#normalize data\n",
    "norm_X = (X_full - np.mean(X_full, axis=0)) / np.std(X_full, axis=0)\n",
    "myargs = (norm_X, y_flip)\n",
    "betas = np.zeros(norm_X.shape[1])\n",
    "lbfgs_fitted = fmin_l_bfgs_b(cost_func, x0=betas, args=myargs, fprime=log_gradient)\n",
    "lbfgs_fitted[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above are the $\\beta$ values we have learned. Now let's make some predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "100"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lbfgs_predicted = pred_values(lbfgs_fitted[0], norm_X, hard=True)\n",
    "sum(lbfgs_predicted == y_flip)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A perfect 100 - not bad."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##Compare with Scikit-Learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "100"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn import linear_model\n",
    "logreg = linear_model.LogisticRegression()\n",
    "logreg.fit(norm_X, y_flip)\n",
    "sum(y_flip == logreg.predict(norm_X))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##Compare with our implementation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "100"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fitted_values, cost_iter = grad_desc(betas, norm_X, y_flip)\n",
    "predicted_y = pred_values(fitted_values, norm_X)\n",
    "sum(predicted_y == y_flip)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So with all 4 features we all get a perfect accuracy, which is to be expected given that the classes are linearlly seperable. So no surprise here, but it is nice to know things are working :). Note: This example doesn't really let L-BFGS shine. The purpose of this post, though, isn't to evaluate advanced optimization techniques. If this is your interest try running some tests with much larger data with many more features and less seperable classes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I hope this little tutorial helped you understand in some depth logistic regression. It is a powerful tool that is good to know. It can even become more powerful with things like [regularization](http://en.wikipedia.org/wiki/Regularization_%28mathematics%29).\n",
    "\n",
    "Even more so, I hope this helped explain the steps of how a learning algorithm might be designed. Having a grasp on what a cost function is and how to minimize it with techniques such as gradient descent can really help understand some of the machine learning literature. \n",
    "\n",
    "Anyway - if you have any questions or comments. I would love to hear them!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}