AB testing with Python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A/B testing: A step-by-step guide in Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook we'll go over the process of analysing an A/B test, from formulating a hypothesis, testing it, and finally interpreting results. For our data, we'll use a <a href='https://www.kaggle.com/zhangluyuan/ab-testing?select=ab_data.csv'>dataset from Kaggle</a> which contains the results of an A/B test on what seems to be 2 different designs of a website page (old_page vs. new_page). Here's what we'll do:\n",
    "\n",
    "1. [Designing our experiment](#1.-Designing-our-experiment)\n",
    "2. [Collecting and preparing the data](#2.-Collecting-and-preparing-the-data)\n",
    "3. [Visualising the results](#3.-Visualising-the-results)\n",
    "4. [Testing the hypothesis](#4.-Testing-the-hypothesis)\n",
    "5. [Drawing conclusions](#5.-Drawing-conclusions)\n",
    "\n",
    "To make it a bit more realistic, here's a potential **scenario** for our study:\n",
    "\n",
    "> Let's imagine you work on the product team at a medium-sized **online e-commerce business**. The UX designer worked really hard on a new version of the product page, with the hope that it will lead to a higher conversion rate. The product manager (PM) told you that the **current conversion rate** is about **13%** on average throughout the year, and that the team would be happy with an **increase of 2%**, meaning that the new design will be considered a success if it raises the conversion rate to 15%.\n",
    "\n",
    "Before rolling out the change, the team would be more comfortable testing it on a small number of users to see how it performs, so you suggest running an **A/B test** on a subset of your user base users."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "## 1. Designing our experiment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Formulating a hypothesis\n",
    "\n",
    "First things first, we want to make sure we formulate a hypothesis at the start of our project. This will make sure our interpretation of the results is correct as well as rigorous.\n",
    "\n",
    "Given we don't know if the new design will perform better or worse (or the same?) as our current design, we'll choose a <a href=\"https://en.wikipedia.org/wiki/One-_and_two-tailed_tests\">**two-tailed test**</a>:\n",
    "\n",
    "$$H_0: p = p_0$$\n",
    "$$H_a: p \\ne p_0$$\n",
    "\n",
    "where $p$ and $p_0$ stand for the conversion rate of the new and old design, respectively. We'll also set a **confidence level of 95%**:\n",
    "\n",
    "$$\\alpha = 0.05$$\n",
    "\n",
    "The $\\alpha$ value is a threshold we set, by which we say \"if the probability of observing a result as extreme or more ($p$-value) is lower than $\\alpha$, then we reject the null hypothesis\". Since our $\\alpha=0.05$ (indicating 5% probability), our confidence (1 - $\\alpha$) is 95%.\n",
    "\n",
    "Don't worry if you are not familiar with the above, all this really means is that whatever conversion rate we observe for our new design in our test, we want to be 95% confident it is statistically different from the conversion rate of our old design, before we decide to reject the Null hypothesis $H_0$. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Choosing the variables\n",
    "\n",
    "For our test we'll need **two groups**:\n",
    "* A `control` group - They'll be shown the old design\n",
    "* A `treatment` (or experimental) group - They'll be shown the new design\n",
    "\n",
    "This will be our *Independent Variable*. The reason we have two groups even though we know the baseline conversion rate is that we want to control for other variables that could have an effect on our results, such as seasonality: by having a `control` group we can directly compare their results to the `treatment` group, because the only systematic difference between the groups is the design of the product page, and we can therefore attribute any differences in results to the designs.\n",
    "\n",
    "For our *Dependent Variable* (i.e. what we are trying to measure), we are interested in capturing the `conversion rate`. A way we can code this is by  each user session with a binary variable:\n",
    "* `0` - The user did not buy the product during this user session\n",
    "* `1` - The user bought the product during this user session\n",
    "\n",
    "This way, we can easily calculate the mean for each group to get the conversion rate of each design."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Choosing a sample size\n",
    "\n",
    "It is important to note that since we won't test the whole user base (our <a href=\"https://www.bmj.com/about-bmj/resources-readers/publications/statistics-square-one/3-populations-and-samples\">population</a>), the conversion rates that we'll get will inevitably be only *estimates* of the true rates.\n",
    "\n",
    "The number of people (or user sessions) we decide to capture in each group will have an effect on the precision of our estimated conversion rates: **the larger the sample size**, the more precise our estimates (i.e. the smaller our confidence intervals), **the higher the chance to detect a difference** in the two groups, if present.\n",
    "\n",
    "On the other hand, the larger our sample gets, the more expensive (and impractical) our study becomes.\n",
    "\n",
    "*So how many people should we have in each group?*\n",
    "\n",
    "The sample size we need is estimated through something called <a href=\"https://research.usu.edu//irb/wp-content/uploads/sites/12/2015/08/A_Researchers_Guide_to_Power_Analysis_USU.pdf\">*Power analysis*</a>, and it depends on a few factors:\n",
    "* **Power of the test** ($1 - \\beta$) - This represents the probability of finding a statistical difference between the groups in our test when a difference is actually present. This is usually set at 0.8 as a convention (here's more info on <a href=\"https://en.wikipedia.org/wiki/Power_of_a_test\">statistical power</a>, if you are curious)\n",
    "* **Alpha value** ($\\alpha$) - The critical value we set earlier to 0.05\n",
    "* **Effect size** - How big of a difference we expect there to be between the conversion rates\n",
    "\n",
    "Since our team would be happy with a difference of 2%, we can use 13% and 15% to calculate the effect size we expect. \n",
    "\n",
    "Luckily, **Python takes care of all these calculations for us**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Packages imports\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import scipy.stats as stats\n",
    "import statsmodels.stats.api as sms\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from math import ceil\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "# Some plot styling preferences\n",
    "plt.style.use('seaborn-whitegrid')\n",
    "font = {'family' : 'Helvetica',\n",
    "        'weight' : 'bold',\n",
    "        'size'   : 14}\n",
    "\n",
    "mpl.rc('font', **font)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4720\n"
     ]
    }
   ],
   "source": [
    "effect_size = sms.proportion_effectsize(0.13, 0.15)    # Calculating effect size based on our expected rates\n",
    "\n",
    "required_n = sms.NormalIndPower().solve_power(\n",
    "    effect_size, \n",
    "    power=0.8, \n",
    "    alpha=0.05, \n",
    "    ratio=1\n",
    "    )                                                  # Calculating sample size needed\n",
    "\n",
    "required_n = ceil(required_n)                          # Rounding up to next whole number                          \n",
    "\n",
    "print(required_n)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'd need **at least 4720 observations for each group**. \n",
    "\n",
    "Having set the `power` parameter to 0.8 in practice means that if there exists an actual difference in conversion rate between our designs, assuming the difference is the one we estimated (13% vs. 15%), we have about 80% chance to detect it as statistically significant in our test with the sample size we calculated."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "## 2. Collecting and preparing the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great stuff! So now that we have our required sample size, we need to collect the data. Usually at this point you would work with your team to set up the experiment, likely with the help of the Engineering team, and make sure that you collect enough data based on the sample size needed.\n",
    "\n",
    "However, since we'll use a dataset that we found online, in order to simulate this situation we'll:\n",
    "1. Download the <a href='https://www.kaggle.com/zhangluyuan/ab-testing?select=ab_data.csv'>dataset from Kaggle</a>\n",
    "2. Read the data into a pandas DataFrame\n",
    "3. Check and clean the data as needed\n",
    "4. Randomly sample `n=4720` rows from the DataFrame for each group *****\n",
    "\n",
    "***Note**: Normally, we would not need to perform step 4, this is just for the sake of the exercise\n",
    "\n",
    "Since I already downloaded the dataset, I'll go straight to number 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0\n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0\n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0\n",
       "3   853541  2017-01-08 18:28:03.143765  treatment     new_page          0\n",
       "4   864975  2017-01-21 01:52:26.210827    control     old_page          1"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('ab_data.csv')\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 294478 entries, 0 to 294477\n",
      "Data columns (total 5 columns):\n",
      " #   Column        Non-Null Count   Dtype \n",
      "---  ------        --------------   ----- \n",
      " 0   user_id       294478 non-null  int64 \n",
      " 1   timestamp     294478 non-null  object\n",
      " 2   group         294478 non-null  object\n",
      " 3   landing_page  294478 non-null  object\n",
      " 4   converted     294478 non-null  int64 \n",
      "dtypes: int64(2), object(3)\n",
      "memory usage: 11.2+ MB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>landing_page</th>\n",
       "      <th>new_page</th>\n",
       "      <th>old_page</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>group</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>control</th>\n",
       "      <td>1928</td>\n",
       "      <td>145274</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>treatment</th>\n",
       "      <td>145311</td>\n",
       "      <td>1965</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "landing_page  new_page  old_page\n",
       "group                           \n",
       "control           1928    145274\n",
       "treatment       145311      1965"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# To make sure all the control group are seeing the old page and viceversa\n",
    "\n",
    "pd.crosstab(df['group'], df['landing_page'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are **294478 rows** in the DataFrame, each representing a user session, as well as **5 columns** :\n",
    "* `user_id` - The user ID of each session\n",
    "* `timestamp` - Timestamp for the session\n",
    "* `group` - Which group the user was assigned to for that session {`control`, `treatment`}\n",
    "* `landing_page` - Which design each user saw on that session {`old_page`, `new_page`}\n",
    "* `converted` - Whether the session ended in a conversion or not (binary, `0`=not converted, `1`=converted)\n",
    "\n",
    "We'll actually only use the `group` and `converted` columns for the analysis.\n",
    "\n",
    "Before we go ahead and sample the data to get our subset, let's make sure there are no users that have been sampled multiple times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 3894 users that appear multiple times in the dataset\n"
     ]
    }
   ],
   "source": [
    "session_counts = df['user_id'].value_counts(ascending=False)\n",
    "multi_users = session_counts[session_counts > 1].count()\n",
    "\n",
    "print(f'There are {multi_users} users that appear multiple times in the dataset')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are, in fact, users that appear more than once. Since the number is pretty low, we'll go ahead and remove them from the DataFrame to avoid sampling the same users twice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The updated dataset now has 286690 entries\n"
     ]
    }
   ],
   "source": [
    "users_to_drop = session_counts[session_counts > 1].index\n",
    "\n",
    "df = df[~df['user_id'].isin(users_to_drop)]\n",
    "print(f'The updated dataset now has {df.shape[0]} entries')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sampling\n",
    "\n",
    "Now that our DataFrame is nice and clean, we can proceed and sample `n=4720` entries for each of the groups. We can use pandas' `DataFrame.sample()` method to do this, which will perform Simple Random Sampling for us. \n",
    "\n",
    "**Note**: I've set `random_state=22` so that the results are reproducible if you feel like following on your own Notebook: just use `random_state=22` in your function and you should get the same sample as I did."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "control_sample = df[df['group'] == 'control'].sample(n=required_n, random_state=22)\n",
    "treatment_sample = df[df['group'] == 'treatment'].sample(n=required_n, random_state=22)\n",
    "\n",
    "ab_test = pd.concat([control_sample, treatment_sample], axis=0)\n",
    "ab_test.reset_index(drop=True, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>763854</td>\n",
       "      <td>2017-01-21 03:43:17.188315</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>690555</td>\n",
       "      <td>2017-01-18 06:38:13.079449</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>861520</td>\n",
       "      <td>2017-01-06 21:13:40.044766</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>630778</td>\n",
       "      <td>2017-01-05 16:42:36.995204</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>656634</td>\n",
       "      <td>2017-01-04 15:31:21.676130</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9435</th>\n",
       "      <td>908512</td>\n",
       "      <td>2017-01-14 22:02:29.922674</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9436</th>\n",
       "      <td>873211</td>\n",
       "      <td>2017-01-05 00:57:16.167151</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9437</th>\n",
       "      <td>631276</td>\n",
       "      <td>2017-01-20 18:56:58.167809</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9438</th>\n",
       "      <td>662301</td>\n",
       "      <td>2017-01-03 08:10:57.768806</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9439</th>\n",
       "      <td>944623</td>\n",
       "      <td>2017-01-19 10:56:01.648653</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>9440 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      user_id                   timestamp      group landing_page  converted\n",
       "0      763854  2017-01-21 03:43:17.188315    control     old_page          0\n",
       "1      690555  2017-01-18 06:38:13.079449    control     old_page          0\n",
       "2      861520  2017-01-06 21:13:40.044766    control     old_page          0\n",
       "3      630778  2017-01-05 16:42:36.995204    control     old_page          0\n",
       "4      656634  2017-01-04 15:31:21.676130    control     old_page          0\n",
       "...       ...                         ...        ...          ...        ...\n",
       "9435   908512  2017-01-14 22:02:29.922674  treatment     new_page          0\n",
       "9436   873211  2017-01-05 00:57:16.167151  treatment     new_page          0\n",
       "9437   631276  2017-01-20 18:56:58.167809  treatment     new_page          0\n",
       "9438   662301  2017-01-03 08:10:57.768806  treatment     new_page          0\n",
       "9439   944623  2017-01-19 10:56:01.648653  treatment     new_page          1\n",
       "\n",
       "[9440 rows x 5 columns]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ab_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 9440 entries, 0 to 9439\n",
      "Data columns (total 5 columns):\n",
      " #   Column        Non-Null Count  Dtype \n",
      "---  ------        --------------  ----- \n",
      " 0   user_id       9440 non-null   int64 \n",
      " 1   timestamp     9440 non-null   object\n",
      " 2   group         9440 non-null   object\n",
      " 3   landing_page  9440 non-null   object\n",
      " 4   converted     9440 non-null   int64 \n",
      "dtypes: int64(2), object(3)\n",
      "memory usage: 368.9+ KB\n"
     ]
    }
   ],
   "source": [
    "ab_test.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "control      4720\n",
       "treatment    4720\n",
       "Name: group, dtype: int64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ab_test['group'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great, looks like everything went as planned, and we are now ready to analyse our results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "## 3. Visualising the results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first thing we can do is to calculate some **basic statistics** to get an idea of what our samples look like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/renatofillinich/opt/anaconda3/lib/python3.7/site-packages/numpy/core/_methods.py:217: RuntimeWarning: Degrees of freedom <= 0 for slice\n",
      "  keepdims=keepdims)\n",
      "/Users/renatofillinich/opt/anaconda3/lib/python3.7/site-packages/numpy/core/_methods.py:186: RuntimeWarning: invalid value encountered in true_divide\n",
      "  arrmean, rcount, out=arrmean, casting='unsafe', subok=False)\n",
      "/Users/renatofillinich/opt/anaconda3/lib/python3.7/site-packages/numpy/core/_methods.py:209: RuntimeWarning: invalid value encountered in double_scalars\n",
      "  ret = ret.dtype.type(ret / rcount)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style  type=\"text/css\" >\n",
       "</style><table id=\"T_ac2728e4_a0ec_11ea_bdda_acde48001122\" ><thead>    <tr>        <th class=\"blank level0\" ></th>        <th class=\"col_heading level0 col0\" >conversion_rate</th>        <th class=\"col_heading level0 col1\" >std_deviation</th>        <th class=\"col_heading level0 col2\" >std_error</th>    </tr>    <tr>        <th class=\"index_name level0\" >group</th>        <th class=\"blank\" ></th>        <th class=\"blank\" ></th>        <th class=\"blank\" ></th>    </tr></thead><tbody>\n",
       "                <tr>\n",
       "                        <th id=\"T_ac2728e4_a0ec_11ea_bdda_acde48001122level0_row0\" class=\"row_heading level0 row0\" >control</th>\n",
       "                        <td id=\"T_ac2728e4_a0ec_11ea_bdda_acde48001122row0_col0\" class=\"data row0 col0\" >0.123</td>\n",
       "                        <td id=\"T_ac2728e4_a0ec_11ea_bdda_acde48001122row0_col1\" class=\"data row0 col1\" >0.329</td>\n",
       "                        <td id=\"T_ac2728e4_a0ec_11ea_bdda_acde48001122row0_col2\" class=\"data row0 col2\" >0.005</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_ac2728e4_a0ec_11ea_bdda_acde48001122level0_row1\" class=\"row_heading level0 row1\" >treatment</th>\n",
       "                        <td id=\"T_ac2728e4_a0ec_11ea_bdda_acde48001122row1_col0\" class=\"data row1 col0\" >0.126</td>\n",
       "                        <td id=\"T_ac2728e4_a0ec_11ea_bdda_acde48001122row1_col1\" class=\"data row1 col1\" >0.331</td>\n",
       "                        <td id=\"T_ac2728e4_a0ec_11ea_bdda_acde48001122row1_col2\" class=\"data row1 col2\" >0.005</td>\n",
       "            </tr>\n",
       "    </tbody></table>"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x1048d7110>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "conversion_rates = ab_test.groupby('group')['converted']\n",
    "\n",
    "std_p = lambda x: np.std(x, ddof=0)              # Std. deviation of the proportion\n",
    "se_p = lambda x: stats.sem(x, ddof=0)            # Std. error of the proportion (std / sqrt(n))\n",
    "\n",
    "conversion_rates = conversion_rates.agg([np.mean, std_p, se_p])\n",
    "conversion_rates.columns = ['conversion_rate', 'std_deviation', 'std_error']\n",
    "\n",
    "\n",
    "conversion_rates.style.format('{:.3f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Judging by the stats above, it does look like **our two designs performed very similarly**, with our new design performing slightly better, approx. **12.3% vs. 12.6% conversion rate**.\n",
    "\n",
    "Plotting the data will make these results easier to grasp:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAggAAAGjCAYAAABJz+3TAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzde1hU5d4+8BtQRI4iAilpGDmACsihtFErBDLxkLpRZKdttagwDymiiaSJhIoV4iEVUUMbgZ87RE3TLR54NZAQTWsHqOUBTEkQkYMwwKzfH17O67hAp3fPMAP7/lwX17Xn+zyz5ruw2XOz1rPWGAiCIICIiIjoEYa6boCIiIj0DwMCERERiTAgEBERkQgDAhEREYkwIBAREZEIAwIRERGJMCAQteDYsWN45513MHDgQPTr1w9Dhw5FeHg4Ll26pOvWtKKkpATOzs5ISUnRdStPtXXrVmRmZv7H21m3bh2GDRumgY6I2h8GBKJmfPbZZwgLCwMAREREICEhAdOmTcP58+fxt7/9Dbm5uTruUPO6deuGLVu24LXXXtN1K0+1bds2jQQEImpZB103QKRv9u7dix07dmDGjBmYM2eOylhQUBAmTJiAiIgIHD16FB07dtRRl+qTy+UwNjZ+6jwTExO88sorrdBR89Ttsz3j74D0CY8gED1m06ZNcHR0xKxZs0RjlpaWmD9/PlxdXXHjxg1lff/+/Rg7dizc3Nzw8ssvY9GiRSgrK1OOp6enw9nZGT///DNmzJgBT09PvPjii4iKioJcLgcADB8+HG+88YboNT/88EMMHToUCoUCAFBYWIj33nsPXl5e8PHxwZw5c1BcXKycn5ubC2dnZxw9ehTvvPMOvLy8ADz48ImPj4efnx/69+8PX19fxMfHo7GxEUDzpxiuXr2KmTNnwsfHB+7u7vj73/+O7Oxslf6cnZ2RmJiI9evX45VXXoGbmxsmTJiAgoKCJ/6enZ2dsWnTJkRHR8PLywvnzp0DAOTk5OAf//gHfHx84OXlhb/97W/IyMhQeV5ZWRn27NkDZ2dnlJSUAAAqKirwySefQCqVwsPDA2+99RZycnKe2MNDhw8fxqhRo9C/f3/4+flh69ateHiT2X/84x8YPHgwmpqaVJ4TFhaGIUOGiOoP3b9/H9HR0Rg4cCC8vb0xe/ZsfPfddyo9f/zxxxg1ahQyMzPh7++PpUuXAgAaGhqwdu1a5b+Vn58f1q9fj4aGBuX2p0yZgokTJ6q85m+//QZnZ2ekp6cD+N//7s6cOYOwsDB4enpi8ODBWL16tcq2iJrDgED0iFu3buH333/HsGHDYGjY/NsjICAAmzdvhqOjIwBAJpNh/vz5cHJywqpVqzB79mzk5uZi0qRJqKysVHnue++9h+eeew4rV66Ev78/du/ejeTkZADA6NGjceXKFVy+fFk5v7q6GidPnsTo0aNhaGiIixcvIiQkBPfu3cPSpUsRGRmJ33//HcHBwSgtLVV5rcjISDz33HOIj48HACQkJGDHjh2YOnUq4uPjMXLkSCQlJWHDhg3N7ucff/yB4OBgXLp0CfPmzUNcXBy6du2K6dOn48iRIypzt27diuzsbERERGDRokW4fPky5s2b99Tf97Zt23Dt2jXExMRAIpGguLgYoaGhMDQ0xLJlyxATEwNbW1ssXLhQGUy2bNkCKysrSKVSbNmyBd26dUNtbS0mT56MrKwsfPjhh1i1ahWsrKwwffp0nDp16ok9/Pnnn4iKisLIkSMRFxcHd3d3xMXFYf369QCAcePGoaysDD/++KPyOVVVVTh16hRGjx4NIyOjZrc7a9Ys7Nu3D++//z5iYmJgZGSETz75pNnf87JlyzBt2jT84x//AADMnz8fmzdvxqhRo/Dll19i/PjxSEpKwkcfffTU32lLvUgkEqxevRpvvvkmtm3bpgwjRC0SiEjp3LlzgkQiEWQymVrza2pqBC8vLyEsLEyl/ttvvwnOzs7CV199JQiCIHz77beCRCIRVq5cqTLP19dXCA0NFQRBEK5duyZIJBJhw4YNyvG9e/cKEolEKCoqEgRBEKZPny6MGDFCqK+vV865ffu24ObmJsTFxQmCIAinT58WJBKJEBkZqfJaY8aMEaZOnapS27hxo7Bq1SpBEAShuLhYkEgkwq5duwRBEITIyEjBzc1NKC0tVc5XKBRCcHCwEBAQoKxJJBLh1VdfFerq6pS1hIQEQSKRCOXl5S3+7iQSiTB8+HChsbFRWTty5IgQGBgoVFdXK2s1NTWi34tUKhUWLlyofLx582ahf//+wtWrV5W1pqYmYdy4cUJQUFCLPaxdu1aQSCTCqVOnVOpvv/22MGDAAKG+vl6ora0VPD09haioKOX4P//5T5V/l8dlZ2cLEolEOHDggEo9NDRUkEgkQnFxsSAIgrBw4UJBIpEIeXl5yjl5eXmCRCIRtm/frvLc1NRUQSKRCGfOnBEEQRAmT54sTJgwQWXO5cuXBYlEInz77beCIPzvf3erV69Wmbd8+XLBxcVFuHnzZou/GyIeQSB6xMO/BgU1v8Ps/PnzqK6uxt/+9jeV+vPPPw8nJyecPXtWpR4YGKjyuEePHrh37x4AoFevXhgwYIDKX+eHDh1C3759IZFIIJfLkZubC19fXzQ0NKCmpgY1NTXo3LkzPDw8kJ+fr7Ltx9cT+Pj4IDs7G5GRkTh69CgqKirwwQcfYMGCBc3u26lTp/DKK6/Azs5OWTMwMIC/vz+uXbuG8vJyZX3YsGHo1KmT8rGDgwMAiI6gPG7w4MEqf4H7+/vjwIEDMDIyQmFhIY4dO6Y8wtHSoXwAyMrKQr9+/dCtWzfl7+X+/ft45ZVX8PPPP6O+vr7F53bt2hWDBw9WqY0aNQq1tbX4/fff0blzZwwfPhz/+te/lIflDx48iH79+kEikTS7zdOnT8PIyAh+fn4q9cf//QHAzMwMPj4+yscPj3g8/t9UQEAAAIj+m1LH66+/LtqWQqHAL7/88pe3Rf89uEiR6BH29vYAoHJO/3G3bt3CoUOH8Oqrr+L27dsAHnzQP87MzAy1tbUqNVNTU5XHBgYGyrUFADBmzBhER0ejpKQEXbp0walTpxAeHg7gwTn2hoYGJCUlISkpSfR6D095PNS5c2eVx4sWLYKjoyO+//57zJkzB42NjXBzc8P8+fMxcOBA0fbKysrQvXv3ZvcLAGpra2FjY9PifgFQ2bfmPN5jRUUFPv30Uxw9ehQmJiZ49tln4eHh8cRtAMDt27dx7do15XqLx929e1f5b/s4W1tbUc3a2hoAlP9+48aNQ3p6OrKzs+Hm5obTp0+3GKwAoLy8HBYWFiqhCXhwpcjjHv8dlJWVwcLCAhYWFir1R3/vLWkp2D6+j1ZWVgAerJMgagkDAtEj7Ozs4OjoiGPHjmH+/Pno0EH8FvnnP/+JdevWwdfXF+bm5gAefLA97o8//sCLL774l14/MDAQK1aswJEjR9C1a1c0NTVh1KhRAB58QBgYGGDSpEnK2qMe/zB6XFNTE9566y1MmTIF9+/fx48//ogvvvgCYWFhOH36tGi+ubl5i/tlaGioDAeatHLlSpw+fRq7du2Cu7s7AKCmpgapqalPfJ65uTkGDBiAiIiIZscffuA3586dO6Law/UcD4+evPjii3j22Wdx8OBBlJSUwMDAAKNHj25xmyYmJqiuroZCoVBZy/L4OpGW9qWmpkZ0RcPDRbGPHtF53N27d5utV1dXqzx+2Ic2/g2p/eApBqLHTJs2DdeuXVMuHnxUcXExduzYgSFDhuC5557DgAED0LFjR3z33Xcq83Jzc3H79m3Roeunsba2xpAhQ/Cvf/0Lhw4dwtChQ5X/J25ubo6+ffvi5s2b8PHxUf70798fGzduxP/8z/+0uN379+9jwIAB+PrrrwE8+Kv11Vdfxbhx41BTU4OqqirRc7y9vZGVlaXy4dLY2IhDhw7By8tLdNRAE/7973/D3d1dGQ4AqHUlwsCBA1FcXIy+ffuq/G4OHz6M7du3P/HSwdu3b6ucnhEEAfv374ejoyOeffZZAA+OiIwdOxaZmZnYu3cvXn31VXTt2rXFbfbt2xeNjY04efKkSv3RqzFa4uPjA4VCgYMHD6rUDxw4AAMDA+V/U2ZmZiqneQCoLKR81PHjx1Ue79u3D8bGxiq/Z6LH8QgC0WOCg4Nx7tw5xMXF4aeffoKfnx8sLCxw8eJFJCcnw9zcHLGxsQAenL9+++23sXXrVhgaGmLIkCGoqKjApk2b4OLigjFjxvzl1x8zZgzCw8NhZGSE1atXq4xFRETg3XffxYIFC/Daa6+hrq4OaWlpuHTpEiIjI1vc5sNAsG7dOtTW1sLFxQUlJSXYtGkTXnrpJdjY2CgvvXto5syZCA4OxltvvYXJkyfDwsICe/bswc2bN7FixYq/vF/qcHd3x759+7Blyxb07t0b+fn5+Ne//gUzMzP8+uuvKC4uRs+ePWFubo5///vfOHbsGKRSKaZPn479+/dj+vTpCA4ORqdOnXD8+HHs27cPcXFxT3xNMzMzzJkzB++//z5sbGxw8OBB/PTTT8qrGB4aO3YsNmzYgPPnz2Pz5s1P3OYbb7yB9evXY+HChfjwww9ha2uLffv2KS/9fHgKpjm+vr7w9PTE0qVLcePGDTg7O6OwsBCJiYmYNGkSevXqBeDBUY3jx48jNTUVvr6++Omnn7Br165mt5mYmAi5XA6JRIKsrCx89913+OCDD5RHwIiaw4BA9BgDAwOsWrUKUqkUqampWLZsGRobG+Hg4ICgoCC8++676NKli3J+REQEunbtitTUVGRkZKBLly7w9fXF3Llz/083vfHz84OpqSkMDQ1Fi9xefvllbN++HV9++SUWLlwIExMTDBgwADKZDE5OTk/cbmxsLGJjYyGTyVBVVQU7OzuMGDFCdDOoh/r27Yvk5GTEx8cjJiYGhoaG6N+/P7Zu3aqyqE6TFi5ciPr6eiQmJioD144dO5CWlobt27fj9OnT6NmzJ959912sXr0a8+bNw+HDh2Fvb4/U1FTExcVh+fLlUCgUcHJywpo1azBixIgnvqajoyOmT5+OdevW4caNG3B0dMSXX34Jf39/lXk9e/aEm5sbysrKnnpDKVNTU6SlpSEuLg4bN27E/fv3MXToUMyaNQuxsbHKNQDNMTIyQmJiIj7//HN88803qKqqgoODA2bMmIH33ntPOW/KlCkoLi5GfHw8Vq1ahf79++Pzzz9XXir5qJiYGOzcuRNbtmxRXjY6bdq0J+4DkYGg7nJtIqL/YtXV1XjllVfwwQcfqHxQN6e8vBz//ve/4eHhoRIGYmJicPToUdEhf21JT0/HokWLcPDgwacGSKLH8QgCEdETVFdX4/jx4zh+/DgMDQ0xadKkpz6nsbERH374Idzd3RESEgITExPk5+cjJSXliVc/EOkTBgQioieorq5GZGQkbG1tERcXB0tLy6c+x97eHtu2bcOGDRuwbNky1NfX47nnnsMnn3yiVsAg0gc8xUBEREQivMyRiIiIRBgQiIiISIQBgYiIiEQYEIiIiEiEAYGIiIhEGBCIiIhIhAGBiIiIRBgQiIiISIQBgYiIiEQYEIiIiEiEAYGIiIhEGBCIiIhIhAGBiIiIRBgQiIiISIQBgYiIiEQYEIiIiEikg64b0Cf5+fm6boGIiKhVeXt7N1tnQHhMS78oIiKi9uZJfxjzFAMRERGJMCAQERGRCAMCERERiTAgEBERkQgDAhEREYkwIBAREZEIAwIRERGJMCAQERGRCAMCERERiTAgEBERkQgDAhEREYkwIBAREZGI3gaEsrIyhIWFwdPTE76+vpDJZE99TnR0NOLj40X1nJwcjB07Fm5ubhg+fDgyMzO10TIREVG7obcBITw8HHK5HCkpKYiKikJ8fHyLH+wKhQI//PADMjIyRGNXrlzBjBkz8OabbyI9PR1+fn6IiIhAeXm5tneBiIiozdLLr3suLCxEXl4esrKyYGtrCxcXFwQHByMtLQ3+/v4qc+/fv49Bgwahrq6u2W19/fXXGDZsGKZNmwYAiIiIQHZ2Ns6dOyfaFhERET2gl0cQ8vLyIJFIYGtrq6x5e3sjNzdXNLdTp05ITU1FRkYG+vXrJxrPzs7GiBEjlI8NDAyQkZHBcEBERPQEehkQSkpK4ODgoFKzs7NDfX09qqurVeqGhoZwdXWFq6srzMzMVMbkcjmuX7+O8vJyTJgwAS+++CImTZqEH3/8Uev7QERE1Jbp5SmGmpoamJiYqNRMTU2VY+bm5mptp7KyEgCwdu1azJ8/H05OTti3bx9CQ0Oxd+9eODo6ip5TUFDwnzVPRETUDuhlQLCwsMCdO3dUanK5HABgZWWl9nYUCgUAIDQ0FOPGjQMAuLu749SpUzh48CBmzJgheo6rq+v/tW0iIqI2JT8/v8UxvTzFYGtri1u3bqnUSktLYW1tLTqy8CTW1tYwNDRE7969Vep9+vRBWVmZRnolIiJqj/QyIEilUhQVFalcipiTkwOpVPqXtmNsbAwXFxcUFhYqa4Ig4NKlS3j++ec11i8REVF7o5enGFxcXODm5obIyEjMnTsX58+fh0wmQ3JyMoAHixitra1FixKbM336dCxbtgzdu3eHs7Mzvv32W1RWVmLs2LHa3g0iIqI2Sy8DAgAkJCRg8eLFCA4ORrdu3RAbGwsvLy8AgJ+fH1asWIHx48c/dTujR49GXV0d1qxZgz///BP9+/dHUlKS2gsdiYiI/hsZCIIg6LoJfZGfnw9vb29dt0FERNQqnvS5p5drEIiIiEi3GBCIiIhIhAGBiIiIRBgQiIiISIQBgYiIiEQYEIiIiEiEAYGIiIhEGBCIiIhIhAGBiIiIRBgQiIiISIQBgYiIiEQYEIiIiEiEAYGIiIhEGBCIiIhIhAGBiIiIRBgQiIiISIQBgYiIiEQYEIiIiEiEAYGIiIhEGBCIiIhIhAGBiIiIRBgQiIiISIQBgYiIiEQYEIiIiEiEAYGIiIhEGBCIiIhIRG8DQllZGcLCwuDp6QlfX1/IZLKnPic6Ohrx8fEtjl+7dg0eHh7Izs7WZKtERETtjt4GhPDwcMjlcqSkpCAqKgrx8fHIzMxsdq5CocAPP/yAjIyMFrcnCAKioqJQV1enrZaJiIjajQ66bqA5hYWFyMvLQ1ZWFmxtbeHi4oLg4GCkpaXB399fZe79+/cxaNCgp37wp6amQhAEbbZNRETUbujlEYS8vDxIJBLY2toqa97e3sjNzRXN7dSpE1JTU5GRkYF+/fo1u71bt25h3bp1iI6O1lrPRERE7YleHkEoKSmBg4ODSs3Ozg719fWorq6Gubm5sm5oaAhXV1cAgJmZWbPb+/TTTzF58mQ8//zz2muaiIioHdHLIwg1NTUwMTFRqZmamirH/or9+/fjxo0bCA0N1Vh/RERE7Z1eHkGwsLDAnTt3VGpyuRwAYGVlpfZ2KioqsHLlSmzYsAEdO3ZU6zkFBQXqN0pERNRO6WVAsLW1Fa03KC0thbW1tejIwpNcvHgRZWVlmDJlikr9vffeg5+fHxISEkTPeXi6goiIqL3Lz89vcUwvA4JUKsUXX3yB8vJy2NjYAABycnIglUr/0nbc3d1x8OBBlVpgYCCWLVuGIUOGaKxfIiKi9kYvA4KLiwvc3NwQGRmJuXPn4vz585DJZEhOTgbwYBGjtbV1i4sSH+rcuTOcnJxE9e7du8Pe3l4rvRMREbUHerlIEQASEhLQ1NSE4OBgJCYmIjY2Fl5eXgAAPz8/HD58WMcdEhERtV8GAu8epJSfnw9vb29dt0FERNQqnvS5p7dHEIiIiEh3GBCIiIhIhAGBiIiIRBgQiIiISIQBgYiIiEQYEIiIiEiEAYGIiIhEGBCIiIhIhAGBiIiIRBgQiIiISIQBgYiIiEQYEIiIiEiEAYGIiIhEGBCIiIhIhAGBiIiIRBgQiIiISIQBgYiIiEQYEIiIiEiEAYGIiIhEGBCIiIhIhAGBiIiIRDrougEiIm25Hu2m6xaINKLXkp9b/TV5BIGIiIhEGBCIiIhIhAGBiIiIRBgQiIiISISLFFuBd8QOXbdApBH5q9/WdQtE1Er09ghCWVkZwsLC4OnpCV9fX8hksqc+Jzo6GvHx8aL6tm3bMGzYMPTv3x8jR47EiRMntNAxERFR+6G3ASE8PBxyuRwpKSmIiopCfHw8MjMzm52rUCjwww8/ICMjQzS2f/9+JCQkYO7cudizZw+kUilmz56NmzdvansXiIiI2iy9PMVQWFiIvLw8ZGVlwdbWFi4uLggODkZaWhr8/f1V5t6/fx+DBg1CXV1ds9vKyMhAUFAQRo8eDQCIjIxEZmYmMjIyEBYWpvV9ISIiaov+UkC4fv067ty5A0NDQ1hbW8PBwQGGhpo/CJGXlweJRAJbW1tlzdvbGzt37hTN7dSpE1JTUwEAixcvFo3X1dXB3d1d+djAwABdunTB7du3Nd43ERFRe/HUgHD8+HF8++23+PHHH3Hv3j2VMTMzM7z44osICgoS/WX/nygpKYGDg4NKzc7ODvX19aiuroa5ubmybmhoCFdXV2U/j3t87cKFCxdQVFSEkJAQjfVLRETU3rQYEH7++Wd89tlnUCgUCAgIwNSpU9GrVy9YWVnBwMAAVVVVuHr1Ki5cuICkpCR89dVXWLhwIQYOHPgfN1VTUwMTExOVmqmpqXLs0YDwVxw4cABLly6Fk5MTxo4d2+ycgoKC/9O2if4btLX3h/hPBqK2SRfvvRYDQlJSEj755BP069ev2XEbGxvY2NjA29sb06ZNw6+//ort27drJCBYWFjgzp07KjW5XA4AsLKy+svbq6qqwuLFi3H48GH4+fkhNjYWxsbGzc59eDRCs/K0sE2i1qed94f2XNd1A0Qaoq33Xn5+fotjLQaEhISEv/Qiffv2xerVq//Sc1pia2uL3NxclVppaSmsra1FRxae5s6dOwgJCcHdu3fx5ZdfYuTIkRrpkYiIqD1Ta5FiRUUFkpKScOnSpWavFtixQ7M3ApJKpfjiiy9QXl4OGxsbAEBOTg6kUulf3lZcXBxqamqQnp4uWtdAREREzVMrIMydOxcXL15EYGDg//n8/1/h4uICNzc3REZGYu7cuTh//jxkMhmSk5MBPFjEaG1t3eyixEcpFAocOXIE06dPhyAIKCkpUY5ZWlrC0tJSq/tBRETUVqkVEM6ePYtt27bBx8dH2/0oJSQkYPHixQgODka3bt0QGxsLLy8vAICfnx9WrFiB8ePHP3Ebd+/eRXV1NdauXYu1a9eqjM2cOROzZs3SWv9ERERtmVoBoUePHjAyMtJ2Lyrs7e2RlJTU7FhRUVGz9cfvk9C1a9cW5xIREVHL1LrL0cKFCxETE4MLFy6goaEBCoVC5YeIiIjaF7WOIMybNw/19fUIDg5udrytXRtNRERET6ZWQNi8ebO2+yAiIiI9olZAeOmllwAA9+7dw7Vr19C5c2f07NkTnTp10mpzREREpBtqBYS6ujosX74c6enpEAQBwINbH4eEhOCjjz5Cx44dtdokERERtS61AsKaNWvw66+/IjU1Fa6urqirq0NeXh5WrFgBuVze7LcoEhERUdul1lUMhw4dwtKlS+Hh4QFjY2NYWlrCz88P0dHROHDggLZ7JCIiolamVkCorKyEoaF4qqWlJWpqajTeFBEREemWWgFh6NChWLNmDSoqKpS16upqfPXVV8q7GxIREVH7odYahCVLlmDWrFkYOnQoevfujY4dO+LKlSuwt7fHpk2btN0jERERtTK1AkK3bt2QkpKCM2fO4OLFi5DL5XB0dMTQoUNb/RbMREREpH0tBoTi4mI4ODjA0NAQxcXFAB58P4K9vb1yzh9//AEA6Nmzp5bbJCIiotbUYkAICAjAsWPH0KNHDwQEBMDAwEA0RxAEGBgY8FbLRERE7UyLAeHo0aPKowVHjx5ttYaIiIhI91q8isHBwUG5vmD9+vUwMzODg4ODyk/nzp2xdevWVmuWiIiIWscTFymuX78eAJCRkQFLS0tYWFiojF+7dg2ZmZlYsmSJ9jokIiKiVvfEgJCdnQ3gwVqD/Px8GBsbq4wbGxsjMjJSe90RERGRTjwxIOzatQsAMGXKFKxZswY2Njat0hQRERHp1lPvpNjY2IhffvkFlZWVrdEPERER6YGnBoQOHTogICAAe/fubY1+iIiISA+odSdFY2NjbN++Hf/zP/+DF154QXRPhLi4OK00R0RERLqhVkBoamrCyJEjtd0LERER6Qm1AsKKFSu03QcRERHpEbUCAgCcOXMGX3/9Na5evQoTExM4Oztj6tSp6NOnjzb7IyIiIh146iJFAMjMzMT06dNhY2ODyZMnY9SoUbh37x7GjRuHzMxMbfdIRERErUytIwgbNmzAp59+ivHjxytrU6dOxfbt25GQkAB/f3+tNUhEREStT60jCL///js8PDxE9cGDB+Pq1aua7gkAUFZWhrCwMHh6esLX1xcymeypz4mOjkZ8fLyonpiYiCFDhsDLywvz5s3DvXv3tNEyERFRu6FWQOjVqxeOHTsmqp8+fRrPPPOMxpsCgPDwcMjlcqSkpCAqKgrx8fEtns5QKBT44YcfkJGRIRrLyMjA1q1bsXz5cshkMlRUVGDRokVa6ZmIiKi9UOsUw/z58/Hhhx/ip59+gre3Nzp27IgLFy7g+++/R3R0tMabKiwsRF5eHrKysmBrawsXFxcEBwcjLS1NdDrj/v37GDRoEOrq6prdVnJyMkJDQ+Hr6wsAiIqKwsiRI3H79m3Y2tpqvHciIqL2QK0jCK+++ir27t2LLl264Pvvv8fu3btRX1+Pr7/+WmVdgqbk5eVBIpGofIB7e3sjNzdXNLdTp05ITU1FRkYG+vXrpzJWVVWFgoICDBkyRFlzcnKClZUV8vLyNN43ERFRe6H2ZY5OTk5YuHAhiouLYWBgAEdHR5iammqlqZKSEjg4OKjU7OzsUF9fj+rqapibmyvrhoaGcHV1BQCYmZmpPOfGjRsQBKHZbZWVlWmldyIiovZArYBQXV2NpUuX4vvvv4cgCAAAIyMjjB49GkuWLEHnzp012lRNTQ1MTExUag/DSE1NjUpAeFPtuJQAACAASURBVNp2gAdHGR7fVm1tbbPPKSgo+KvtEv3XaGvvD7OnTyFqE3Tx3lMrICxbtgyXL1/Gtm3b4O7ujrq6Opw7dw4rV67EZ599hpiYGI02ZWFhgTt37qjU5HI5AMDKyuovbQcA6urqYGxsrKzX19fD0tKy2ec8PBqhWTydQe2Ddt4f2nNd1w0QaYi23nv5+fktjqm1BiErKwvLli3DoEGDYGpqiq5du8LPzw/R0dE4cuSIxhp9yNbWFrdu3VKplZaWwtraWnRk4WnbAYCbN2+KttWjR4//vFEiIqJ2Sq2AYG1tDSMjI1HdwsLiL31gq0sqlaKoqAjl5eXKWk5ODqRS6V/ajrW1Nfr27YucnBxlraioCFVVVfDx8dFYv0RERO2NWgEhMjISn332Gc6ePYvGxkYADz5oV6xYgdmzZ0OhUCh/NMHFxQVubm6IjIxEYWEh0tLSIJPJMHnyZAAPFjE+XF/wNJMmTcLGjRtx8uRJnDlzBvPnz8f48ePVXsdARET030itNQgzZsxAU1MT/v73v8PAwABGRkZoamqCIAg4e/YsoqKilHM1tZAiISEBixcvRnBwMLp164bY2Fh4eXkBAPz8/LBixQq1LrGcOHEibt++jQULFkAul+ONN97A4sWLNdIjERFRe6VWQNi+fbu2+xCxt7dHUlJSs2NFRUXN1nfu3CmqGRgYYObMmZg5c6ZG+yMiImrP1AoIL730EoAHdzi8evUqOnfujD59+nChHxERUTulVkCorKzEnDlzkJubCxsbG9TV1aG2thavv/46li9frryckIiIiNoHtRYprly5EkZGRsjKysKpU6dw5swZpKeno6SkROP3QCAiIiLdUysgnDx5EuHh4bCzs1PWXFxc8PHHH+P48eNaa46IiIh0Q62AIJfLm701sZGRERoaGjTeFBEREemWWgHB398fMTExKCwsVNauX7+OVatWqXxTIhEREbUPai1SjIqKwieffIJx48ahc+fO6NixI+7du4eBAwdi6dKl2u6RiIiIWtlTA4JCoUBVVRXi4uIQHh6OS5cuQS6Xw9HREX369GmNHomIiKiVPTUgGBgY4I033sDu3bvxwgsv8N4HRERE/wWeugbBwMAAkyZNwrZt25Tfw0BERETtm1prEH777Tfk5OQgKysLzz33HAwMDFTGZTKZVpojIiIi3VArILi7u8Pd3V3bvRAREZGeUCsgPPpFR3fv3oWBgQGsrKy01hQRERHplloBoaGhAevWrUNKSgqqq6sBADY2Npg0aRI+/PBD0SkHIiIiatvUCghxcXE4cuQIPv74Y7i7u6Ourg7nzp3DV199hfr6eoSHh2u7TyIiImpFagWE7777Dl9++SVefvllZc3NzQ3du3fH0qVLGRCIiIjaGbVutWxsbIyuXbuK6r169YJCodB4U0RERKRbagWE2bNnY/Xq1SgtLVXWKisrsW7dOrzzzjtaa46IiIh0Q61TDJs3b8Yff/yB1157DXZ2djA2NsbNmzfR1NSEX375ReU+CCdOnNBWr0RERNRK1AoIYWFh2u6DiIiI9EiLAeHy5ct44YUXAADjxo1Ta2OPfh00ERERtV0tBoTPPvsMxsbGmDx5MoYMGdLivQ4aGxtx4sQJpKSkwNjYGBs3btRas0RERNQ6WgwI27dvx/HjxxEfH4+5c+eiX79+eO6552BhYQFBEFBZWYlr167hl19+gZOTE+bMmYNXXnmlNXsnIiIiLXniGgRfX1/4+vqioKAAOTk5uHjxIi5fvozGxkbY2dlhzJgxWLZsGZycnFqrXyIiImoFai1SdHV1haurq7Z7ISIiIj2h1n0QiIiI6L+L3gaEsrIyhIWFwdPTE76+vir3WnhcVlYWAgMD4eHhgZCQEBQVFamMp6SkICAgAN7e3ggNDcWNGze03T4REVGbprcBITw8HHK5HCkpKYiKikJ8fDwyMzNF865evYpZs2Zh4sSJSE9Ph4eHB0JDQ1FbWwvgwY2b4uLiEBERgZ07d0KhUGDGjBm8RTQREdET6GVAKCwsRF5eHlauXAkXFxf4+fkhODgYaWlporkpKSkYNGgQpk6dCicnJyxYsAAKhUJ5R8d9+/Zh1KhReP3119G3b198+umnKCwsxJUrV1p5r4iIiNqOFhcprl+/Xu2NzJw5UyPNPJSXlweJRAJbW1tlzdvbGzt37mx27ptvvql8bGhoCE9PT+Tm5iIwMBCVlZWwsbFRjpuYmAAA5HK5RnsmIiJqT1oMCI+ep6+rq8OhQ4fwwgsvoF+/fmhqasKFCxdw+/ZtBAUFabypkpISODg4qNTs7OxQX1+P6upqmJubP3Xuwy+WGjJkCDZv3ozx48ejd+/eSEhIQI8ePdCnTx+N901ERNRetBgQVqxYofzfCxYswNtvv41FixYpa4IgYPny5WhoaNB4UzU1Ncq/9B8yNTVVjj0aEFqa+3ANwltvvYX9+/dj7NixMDAwgCAI+Pzzz9Ghg1pXeBIREf1XUutTMjMzE6mpqSo1AwMDBAcH46233sLSpUs12pSFhQXu3LmjUnt4SsDKykqlbm5ujrq6OpVafX09LC0tAQCLFi1CXV0dNm/eDBsbGxw+fBiLFi2Co6Mj3NzcRK9dUFCgyV0halfa2vvDTNcNEGmILt57agUEc3NznD9/HhKJRKX+888/i/561wRbW1vk5uaq1EpLS2FtbS16PTs7O9y8eVM0t0ePHqiqqsKBAweQnJyMgQMHAgDc3Nxw7tw5HDhwoNmAoJ0bQuVpYZtEra+t3TDtuq4bINIQbb338vPzWxxTKyBMnz4dMTExuHLlCjw9PWFgYICzZ89i165dmDNnjsYafUgqleKLL75AeXm5coFhTk4OpFJps3Ozs7MxZcoUAEBDQwPOnDmDlStXokOHDjAwMBB90VSnTp3QqVMnjfdNRETUXqgVEKZOnQpHR0ckJycjPT0djY2NcHJywqpVqzB8+HCNN+Xi4gI3NzdERkZi7ty5OH/+PGQyGZKTkwE8WJhobW0NMzMzBAUFYdy4cZDJZPD09ERiYiKsra0hlUphZGQEX19fLF++HJGRkbC1tcXx48eRl5eHjz/+WON9ExERtRcGgiAIum6iOaWlpVi8eDHy8vLQrVs3fPTRRxg9ejQAwNnZGStWrMD48eMBAEeOHMHnn3+OW7duYcCAAYiJiUHPnj0BAFVVVUhISMCRI0dw7949ODk5Yd68ec0ejcjPz4e3t7fG98U7YofGt0mkC/mr39Z1C3/J9WjxaUSitqjXkp+1st0nfe6pFRAEQcDWrVuxZ88eFBcXY9++fdixYwcGDBiAMWPGaLxhXWFAIHoyBgQi3dBFQFDrToobNmzAzp07MXXqVCgUCgiCAFdXV8TExOCbb77RaLNERESke2oFhN27d2Pp0qWYMGECDA0fPGXChAlYsmSJcl0AERERtR9qBYTKykr06tVLVO/Tp4/yjoVERETUfqgVEF588UXs3btX+fjhZYO7d+9Gv379tNMZERER6Yxalzl+8sknCA0NxQ8//IDGxkZER0ejuLgYdXV12LJli7Z7JCIiolamVkDo1asXDhw4gAMHDqCoqAiNjY14/fXXMXr0aJiZ8WamRERE7Y1aAWH9+vWYOnWqytcqA8C9e/ewdetWvPPOO1ppjoiIiHTjiQEhIyMDwIPLHM3NzdGlSxeV8cuXL2Pnzp0MCERERO3MEwNCfHw8gAc3SkpKSoKRkZHKuLGxMd59913tdUdEREQ68cSAkJWVBQAYNmwYUlJSYG9v3ypNERERkW499TLHxsZGmJiYoKGhoTX6ISIiIj3w1IDQoUMHPPvsszh+/Hhr9ENERER6QK2rGFxdXfHFF18gNzcXL7zwgvJGSQ/NmTNHK80RERGRbqgVEM6ePQs3NzdUVlYiPz9fZezxsEBERERtn1oBYefOndrug4iIiPSIWt/FAAAXLlzAsmXL8O6776K0tBS7d+/GtWvXtNkbERER6YhaAeHQoUOYMmUKqqurcfr0adTW1qKgoABjx45FTk6OtnskIiKiVqZWQNiwYQMWLFiA1atXw9DwwVOWLFmC0NBQfP7551ptkIiIiFqfWgHh2rVrGDhwoKju7++Py5cva7wpIiIi0i21AsLzzz+Pn3/+WVQ/d+4cnnnmGY03RURERLql1lUMERER+Oijj3Dx4kUoFApkZGTgzz//xMGDB7Fy5Upt90hEREStTK0jCIMHD8b/+3//D1VVVXBxcUFWVhbq6+uxc+dOjBgxQts9EhERUStT6whCVlYWhgwZgpiYGG33Q0RERHpArYAwc+ZMmJmZ4fXXX8fIkSPx0ksv8Q6KRERE7ZhaASEnJwcnTpzA0aNHMWPGDHTu3BlvvPEGRo4cCU9PT233SERERK1MrYBgbm6OUaNGYdSoUZDL5cjJyUFmZibCwsJgamqKY8eOabtPIiIiakVq32r5oevXr+PixYu4fPkyqqur0b17d230RURERDqkVkDIycnBZ599Bn9/f4wZMwZHjx7F8OHDcfToUchkMq00VlZWhrCwMHh6esLX1/eJr5OVlYXAwEB4eHggJCQERUVFov7Hjh0LNzc3DB8+HJmZmVrpmYiIqL1Q6xTDtGnT0L9/f/z973/HiBEjWuWoQXh4ODp06ICUlBTcuHEDCxcuhL29Pfz9/VXmXb16FbNmzcK8efMwdOhQ7N69G6GhoTh06BBMTU1x5coVzJgxA7Nnz8aQIUOwZ88eREREIDMzEzY2NlrfDyIiorZIrYCQmZmJZ599Vtu9KBUWFiIvLw9ZWVmwtbWFi4sLgoODkZaWJgoIKSkpGDRoEKZOnQoAWLBgAb777jucOHECgYGB+PrrrzFs2DBMmzYNwIObPmVnZ+PcuXOibREREdEDagUEMzMzrF69GpcuXUJdXZ1ofMeOHRptKi8vDxKJBLa2tsqat7c3du7c2ezcN998U/nY0NAQnp6eyM3NRWBgILKzs7Fw4ULluIGBATIyMjTaLxERUXujVkCYO3cuLl68iMDAQJibm2u7J5SUlMDBwUGlZmdnh/r6elRXV6v00NLc0tJSyOVyXL9+HeXl5ZgwYQKuXr0KJycnzJs3Dy+99JLW94OIiKitUisgnD17Ftu2bYOPj4+2+wEA1NTUwMTERKVmamqqHHs0ILQ0t7a2FpWVlQCAtWvXYv78+XBycsK+ffsQGhqKvXv3wtHRUfTaBQUFGt4bovajrb0/zHTdAJGG6OK9p1ZA6NGjB4yMjLTdi5KFhQXu3LmjUpPL5QAAKysrlbq5ubnotEd9fT0sLS2hUCgAAKGhoRg3bhwAwN3dHadOncLBgwcxY8YM0Wu7urpqbD/+V54WtknU+rTz/tCe67pugEhDtPXey8/Pb3FMrcscFy5ciJiYGFy4cAENDQ1QKBQqP5pma2uLW7duqdRKS0thbW0tOlpgZ2eHmzdviuZ2794d1tbWMDQ0RO/evVXG+/Tpg7KyMo33TURE1F6oFRDmzZuHgoICBAcHw93dHf369VP50TSpVIqioiKUl5crazk5OZBKpc3Ozc7OVj5uaGjAmTNnIJVKYWxsDBcXFxQWFirHBUHApUuX8Pzzz2u8byIiovZCrVMMmzdv1nYfKlxcXODm5obIyEjMnTsX58+fh0wmQ3JyMoAHCxOtra1hZmaGoKAgjBs3DjKZDJ6enkhMTIS1tbUyTEyfPh3Lli1D9+7d4ezsjG+//RaVlZUYO3Zsq+4TERFRW6JWQHi44v/evXu4du0aOnfujJ49e6JTp05aaywhIQGLFy9GcHAwunXrhtjYWHh5eQEA/Pz8sGLFCowfPx59+vRBfHw8Pv/8c8TFxWHAgAHYuHGjcs3E6NGjUVdXhzVr1uDPP/9E//79kZSU1CpXYxAREbVVagWEuro6LF++HOnp6RAEAcCDKwVCQkLw0UcfoWPHjhpvzN7eHklJSc2OPX4r5YCAAAQEBLS4rQkTJmDChAka7Y+IiKg9UysgrFmzBr/++itSU1Ph6uqKuro65OXlYcWKFZDL5Vi8eLG2+yQiIqJWpNYixUOHDmHp0qXw8PCAsbExLC0t4efnh+joaBw4cEDbPRIREVErUysgVFZWwtBQPNXS0hI1NTUab4qIiIh0S62AMHToUKxZswYVFRXKWnV1Nb766ivlwkEiIiJqP9Rag7BkyRLMmjULQ4cORe/evdGxY0dcuXIFzzzzDDZu3KjtHomIiKiVqRUQunXrhpSUFJw5cwYXL16EXC6Ho6Mjhg4d2qq3YCYiIqLW8dRTDJcvX0ZxcTEAwMfHB0FBQejSpQv69+/PcEBERNROtRgQmpqasGTJEowePRo//fSTst7Q0IDly5cjICAAMpmsVZokIiKi1tXiKYYtW7bg2LFj2LZtG15++WVl3czMDD/++CP27duH5cuXo2vXrhgxYkSrNEtERESto8WAsGfPHkRERKiEg4eMjIwwbtw41NbWYtu2bQwIRERE7UyLpxj++OMPuLm5PfHJUqkUly5d0nhTREREpFstBgQLCwvcuXPniU++e/euVr6HgYiIiHSrxYDw8ssvY8uWLcovZ2rON998o/ymRyIiImo/WgwIc+fOxYULFzBt2jScPHkSd+/ehUKhQFlZGU6ePInp06fj+PHj+Oijj1qzXyIiImoFLS5SfPbZZ7Fr1y58+umnCA0NhYGBgXJMEAR4eXlh165d6NOnT6s0SkRERK3niXdS7N27N5KTk/HHH3+goKAAtbW1sLCwgIuLC5555pnW6pGIiIhamVq3Wu7Rowd69Oih7V6IiIhIT6j1bY5ERET034UBgYiIiEQYEIiIiEiEAYGIiIhEGBCIiIhIhAGBiIiIRBgQiIiISIQBgYiIiEQYEIiIiEhEbwNCWVkZwsLC4OnpCV9fX8hkshbnZmVlITAwEB4eHggJCUFRUVGz865duwYPDw9kZ2drq20iIqJ2QW8DQnh4OORyOVJSUhAVFYX4+HhkZmaK5l29ehWzZs3CxIkTkZ6eDg8PD4SGhqK2tlZlniAIiIqKQl1dXWvtAhERUZullwGhsLAQeXl5WLlyJVxcXODn54fg4GCkpaWJ5qakpGDQoEGYOnUqnJycsGDBAigUCpw4cUJlXmpqKgRBaKU9ICIiatv0MiDk5eVBIpHA1tZWWfP29kZubm6zcwcPHqx8bGhoCE9PT5W5t27dwrp16xAdHa3dxomIiNoJvQwIJSUlcHBwUKnZ2dmhvr4e1dXVas0tLy9XPv70008xefJkPP/889prmoiIqB3Ry4BQU1MDExMTlZqpqalyTJ25D9cg7N+/Hzdu3EBoaKgWOyYiImpfOui6geZYWFjgzp07KjW5XA4AsLKyUqmbm5uLFh7W19fD0tISFRUVWLlyJTZs2ICOHTuq9doFBQX/QedE7Vtbe3+Y6boBIg3RxXtPLwOCra2taL1BaWkprK2tRUcL7OzscPPmTdHcHj164OLFiygrK8OUKVNUxt977z34+fkhISFB9Nqurq4a2otH5Wlhm0StTzvvD+25rusGiDREW++9/Pz8Fsf08hSDVCpFUVGRyjqCnJwcSKXSZuc+el+DhoYGnDlzBlKpFO7u7jh48CAyMjKUPwCwbNkyREZGan9HiIiI2ii9PILg4uICNzc3REZGYu7cuTh//jxkMhmSk5MBPFiYaG1tDTMzMwQFBWHcuHGQyWTw9PREYmIirK2tIZVKYWRkBCcnJ9H2u3fvDnt7+9beLSIiojZDL48gAEBCQgKampoQHByMxMRExMbGwsvLCwDg5+eHw4cPAwD69OmD+Ph47NixAyEhIaioqMDGjRthZGSky/aJiIjaNL08ggAA9vb2SEpKanbs8VspBwQEICAgQK3ttnQbZiIiIvpfensEgYiIiHSHAYGIiIhEGBCIiIhIhAGBiIiIRBgQiIiISIQBgYiIiEQYEIiIiEiEAYGIiIhEGBCIiIhIhAGBiIiIRBgQiIiISIQBgYiIiEQYEIiIiEiEAYGIiIhEGBCIiIhIhAGBiIiIRBgQiIiISIQBgYiIiEQYEIiIiEiEAYGIiIhEGBCIiIhIhAGBiIiIRBgQiIiISIQBgYiIiEQYEIiIiEiEAYGIiIhE9DYglJWVISwsDJ6envD19YVMJmtxblZWFgIDA+Hh4YGQkBAUFRWpjG/btg3Dhg1D//79MXLkSJw4cULL3RMREbVtehsQwsPDIZfLkZKSgqioKMTHxyMzM1M07+rVq5g1axYmTpyI9PR0eHh4IDQ0FLW1tQCA/fv3IyEhAXPnzsWePXsglUoxe/Zs3Lx5s7V3iYiIqM3Qy4BQWFiIvLw8rFy5Ei4uLvDz80NwcDDS0tJEc1NSUjBo0CBMnToVTk5OWLBgARQKhfIoQUZGBoKCgjB69Gj06dMHkZGRsLGxQUZGRivvFRERUduhlwEhLy8PEokEtra2ypq3tzdyc3ObnTt48GDlY0NDQ3h6eirn1tXVwd3dXTluYGCALl264Pbt21rcAyIioratg64baE5JSQkcHBxUanZ2dqivr0d1dTXMzc2fOre0tBQARGsXLly4gKKiIoSEhGipeyIiorZPL48g1NTUwMTERKVmamqqHFNn7sM1CI86cOAApk+fDicnJ4wdO1bDXRMREbUfenkEwcLCAnfu3FGpyeVyAICVlZVK3dzcHHV1dSq1+vp6WFpaKh9XVVVh8eLFOHz4MPz8/BAbGwtjY+NmX7ugoEATu0DULrW194eZrhsg0hBdvPf0MiDY2tqK1huUlpbC2tpadLTAzs5OdEVCaWkpevToAQC4c+cOQkJCcPfuXXz55ZcYOXLkE1/b1dVVA3vwuDwtbJOo9Wnn/aE913XdAJGGaOu9l5+f3+KYXp5ikEqlKCoqQnl5ubKWk5MDqVTa7Nzs7Gzl44aGBpw5c0Y5Ny4uDjU1NUhPT39qOCAiIqIH9DIguLi4wM3NDZGRkSgsLERaWhpkMhkmT54M4MHCxIdrEYKCgnDy5EnIZDL8+uuviIiIgLW1NaRSKRQKBY4cOYKQkBAIgoCSkhLlz71793S5i0RERHpNLwMCACQkJKCpqQnBwcFITExEbGwsvLy8AAB+fn44fPgwAKBPnz6Ij4/Hjh07EBISgoqKCmzcuBFGRka4e/cuqqursXbtWvj5+an8JCcn63L3iIiI9JperkEAAHt7eyQlJTU79vitlAMCAhAQECCa17VrV9FcIiIiejq9PYJAREREusOAQERERCIMCERERCTCgEBEREQiDAhEREQkwoBAREREIgwIREREJMKAQERERCIMCERERCTCgEBEREQiDAhEREQkwoBAREREIgwIREREJMKAQERERCIMCERERCTCgEBEREQiDAhEREQkwoBAREREIgwIREREJMKAQERERCIMCERERCTCgEBEREQiDAhEREQkwoBAREREIgwIREREJMKAQERERCIMCERERCSitwGhrKwMYWFh8PT0hK+vL2QyWYtzs7KyEBgYCA8PD4SEhKCoqEhlPDExEUOGDIGXlxfmzZuHe/fuabt9IiKiNk1vA0J4eDjkcjlSUlIQFRWF+Ph4ZGZmiuZdvXoVs2bNwsSJE5Geng4PDw+EhoaitrYWAJCRkYGtW7di+fLlkMlkqKiowKJFi1p7d4iIiNoUvQwIhYWFyMvLw8qVK+Hi4gI/Pz8EBwcjLS1NNDclJQWDBg3C1KlT4eTkhAULFkChUODEiRMAgOTkZISGhsLX1xeurq6IiorC0aNHcfv27VbeKyIiorZDLwNCXl4eJBIJbG1tlTVvb2/k5uY2O3fw4MHKx4aGhvD09ERubi6qqqpQUFCAIUOGKMednJxgZWWFvLw87e4EERFRG6aXAaGkpAQODg4qNTs7O9TX16O6ulqtueXl5bhx4wYEQWh2vKysTDvNExERtQMddN1Ac2pqamBiYqJSMzU1VY6Zm5s/dW5tbS1qamoAAJ06dWp2vDn5+fn/cf+PS5zUT+PbJNIFbbw/tGrk17rugEgjbuvgvaeXAcHCwgJ37txRqcnlcgCAlZWVSt3c3Bx1dXUqtfr6elhaWsLCwgIAUFdXB2NjY9H447y9vTXSPxERUVunl6cYbG1tcevWLZVaaWkprK2tRUcL7OzscPPmTdHc7t27K9cwNDfeo0cPLXRORETUPuhlQJBKpSgqKkJ5ebmylpOTA6lU2uzc7Oxs5eOGhgacOXMGUqkU1tbW6Nu3L3JycpTjRUVFqKqqgo+Pj3Z3goiIqA3Ty4Dg4uICNzc3REZGorCwEGlpaZDJZJg8eTKABwsTH64vCAoKwsmTJyGTyfDrr78iIiIC1tbWyjAxadIkbNy4ESdPnsSZM2cwf/58jB8/XmUdA9FD69atQ0hIiK7bICLSOb0MCACQkJCApqYmBAcHIzExEbGxsfDy8gIA+Pn54fDhwwCAPn36ID4+Hjt27EBISAgqKiqwceNGGBkZAQAmTpyIKVOmYMGCBXj//ffh7u6OxYsX62y/SLvi4+MxZcoUXbdBpDOvvPIK0tPT29y2n6SxsRHOzs7NXupO2qOXixQBwN7eHklJSc2OPX4r5YCAAAQEBDQ718DAADNnzsTMmTM13iO1LQ0NDejYsaOu2yDSCf73T3+V3h5BoPbvt99+w5QpU+Du7o5hw4bhm2++AQCcP38eEydOhLu7O9544w2VO2iuW7cO77//PjZt2gSpVApPT08sXboUgiBg3bp12LRpE3788UcMGzYMADBs2DBs3boVkyZNwgcffPDU7RO1ZcOGDUNpaSkWLVqEKVOmwNPTE3v37sXgwYNx8OBBAMD+/fsxfPhwuLm5ISgoCKdPn1Y+v7y8HLNnz4aPjw98fHzwzjvv4OrVq6Jtr1u3Drm5uXB2dsb333+PYcOGwcvLC2vXrsXRo0fh7+8PDw8PzJ8/H01NTcrtf/3113jttdfg7u6Ot99+G4WFhcoxZ2dn6JfW5gAACiBJREFU7N37/9u7/5gmzj8O4O8K1jCVBGVA/MMJSltGqkSRWZewkWnUZRG6OXBGx8TshwTFaZw/IGET2YYmBkvI3NTNiolLZCDGwExNGGQrxrj4a0hhnUOWrRMjrkqAStfP9w/CfWXHNqy6or5ficndPXef566h9O3Tu4dqLF++HEajEYsWLcK5c+cAAPHx/Y+Kv/766wEZwXhsCVEAeDweSUlJkfz8fPnxxx+lurpa4uPjpba2VmbOnClFRUXS3NwsJ06ckNmzZ0tVVZWIiFgsFomPj5d169ZJc3OzVFRUiF6vl7q6OnG73bJt2zZ59dVXxeVyiYhISkqKJCYmSmVlpbhcLrl+/fq/1l+6dGnAXheie+FyueTZZ5+Vffv2ic1mE4PBIOnp6XL27Fm5deuWNDY2islkktraWnE4HGKxWMRoNMqvv/4qIiLr16+XFStWSFNTk1y8eFEyMzMlMzNTVdvtdsupU6dEp9NJRkaGnD9/XqxWq+h0Opk/f77Y7Xax2+2SkJAgNTU1IiLy1Vdfybx586S+vl6ampqkoKBAkpKSpLu7W0REdDqdmEwm+frrr+XSpUuybNkyefnll0VE5MqVK6LT6eT48ePS1dX137+wj6kR+xUDPdpqamrg8XhQUFCA4OBgTJs2DU6nE06nE5MmTcLWrVsB9N+w2tbWhi+++AJpaWkAgLFjx+LDDz9ESEgIDAYDDhw4gNbWVjz//PMYN24cxowZg6ioKKWvtLQ0mM1mAEBZWdm/1id6WEVFRWHUqFEICwvD+PHj4fP58P777yMuLg4AsGfPHrzzzjtYuHAhgP7/tdvtdhw/fhxvvfUWZs6cicTEROj1egDAggULsHfvXlXtO+eR2bJlC6ZPnw6DwYCioiKsWrUKJpMJQP/76/LlywCATz75BJs3b0ZycjIAoKCgAN988w3q6urw4osvAgDeeOMNLFiwAED/aMHGjRsBQHksPTw8HGPHjn1wLyANwoBAAeF0OmE0GhEc/P8fwfXr1yM3N1f1CKrBYEBZWZmyPmXKFISEhCjr48aNg8fj+du+YmNjleXW1tZ/rU/0KJk2bZqy7HA4cObMGezcuVPZ5vV6lX3MZjOOHTuG8vJytLe349KlS0NOKjdU/YHJ6O6c2j44OBi3b99Gd3c32tvbkZubC41Go7T39fWhvb1dWR8IMkD/hHn/9L6mB48BgQKiu7t7yBumPB6ParvH4xk0QdaoUXd368ydv5CGU5/oUXLnz7/X68WWLVswZ86cQfuEhoair68PS5cuRUhICFJTU5GamorW1lbs37//H+v/dSr7od6fXq8XAFBSUoLo6OhBbRMmTFCWB54+o5GBAYECIjo6Gg0NDRAR5RfY6tWrUV9fjxkzZgza98yZM8qQ5/3o9+zZsw+sPtFIFhMTg87OTkydOhUAICJYt24d0tLSEBERgZaWFtjtdkycOBEA8N13392XfkNDQxEeHo6bN28qfff29iI7OxvvvvsuwsLC7ks/dH/xKQYKiNTUVLjdbhQXF6OlpQWff/45vv32Wxw7dgw//PADSkpK4HA4UFVVhcOHD2PlypXDqqvVanHt2jVcuXJlyPbXXnvtnuoTjXRarRZOpxNut1vVlpWVBavVipqaGjgcDhQWFuL06dOYMWMGwsPDERQUhKqqKvz00084ePAgjhw5gp6eHmXq+4Haf/zxx12fV1ZWFnbv3o36+no0NTXhvffeg8vlGlY4DwoKQnBwMBwOh+ov+tKDw4BAAREaGoo9e/agsbERS5YsQVVVFSwWC2JjY1FaWgqbzYYlS5agrKwMmzdvRkpKyrDqpqSk4ObNm8jOzh6yffLkyfdUn2ikW7x4McrLy4cMyQsXLkRubi6Ki4uRkZGBlpYW7Nu3DxMmTEBkZCQ++OADHDhwAOnp6WhubobVasXo0aOVexYGah85cuSuz2vlypUwm83YtGkTli9fjp6eHnz22WeD/pDe39FoNFi8eDF27tyJhoaGu+6b/KMREQn0SRAREdHIwhEEIiIiUmFAICIiIhUGBCIiIlJhQCAiIiIVBgQiIiJSYUAgIiIiFQYEIiIiUmFAICIiIhUGBCIiIlJhQCAiIiIVBgQiIiJSYUAgIiIiFQYEIiIiUmFAICIiIhUGBCIiIlJhQCAiIiIVBgQiIiJSCQ70CRDRw62jowO7d+9GfX093G43oqKiMG/ePKxevRqhoaGBPj0i8pNGRCTQJ0FED6erV68iPT0dTz31FLKzs/Hkk0+ipaUFFosFWq0WFRUV0Gq1gT5NIvIDAwIR+W3NmjXo6OjAoUOHMHr0aGX71atX8cILL6CwsBBms3nQMX19fYP2JaKRifcgEJFfOjs7cfLkSbz99tuqD/zIyEjs3bsXcXFxqKysxEsvvYT9+/cjMTER58+fh9frRXFxMUwmE5KSkpCTk4PffvtNOV6v18NutyvrlZWVSE5OVpbnz5+PiooKJCcnY86cOdi+fTtu377931w40WOCAYGI/HLx4kX4fD4kJCQM2W4ymWAwGAAAbW1tOHXqFKxWK4xGI0pLS1FbW4uPP/4Y5eXlGDNmDFatWjXsD3mXy4Xa2lp8+umn2LVrF2w2GywWy327NiJiQCAiP7ndbgAYdCNifn4+jEaj8i8zMxMA4PP5sGPHDsTHxwMArFYr8vLy8Nxzz0Gv1+Ojjz7CtWvX0NDQMKy++/r6UFhYiLi4OMydOxdr1qzB4cOH7/MVEj3eGBCIyC9PPPEEAOD69evKtrVr1+Lo0aM4evQozGazMiIwceJEhIWFAQB++eUX9PT0YNasWcpxWq0WMTEx+Pnnn4fVd0REBCZNmqSsT58+HV1dXejs7Lzn6yKifnzMkYj8MnXqVABAc3MzIiMjAfR/cEdERAAAurq6lH01Go2y7PF4AEB130Jvby9CQkKG7GvgmAF/fTKit7dX1Q8R3RuOIBCRX6Kjo/H000/j4MGDqraOjg7U1dUNedzkyZMRFBSECxcuKNtu3LiBy5cvQ6/XA+gPDz09PUp7W1vboBq///47bty4oayfO3cO4eHhyigFEd07BgQi8lt+fj5Onz6NDRs24Pvvv4fT6UR1dTUyMjIwZcqUIY8ZP348UlNTsW3bNjQ2NuLChQvYtGkT9Ho9Zs+eDaB/dMJms8Hr9aK1tRXV1dWDavz555/Iy8uDw+GAzWZDWVkZ0tPTH/TlEj1W+BUDEflt1qxZ+PLLL1FaWoo333wTPp8PMTExWLt2LWJjY1FUVDTkcXl5edi+fTtycnLg8/nwzDPPoLS0VGkvKCjA1q1bkZSUBJ1Oh6ysLBw6dEhpj4iIQEJCAjIzM6HRaPDKK68gJyfngV8v0eOEEyUR0UOlsrISJSUlw37igYj8w68YiIiISIUBgYiIiFT4FQMRERGpcASBiIiIVBgQiIiISIUBgYiIiFQYEIiIiEiFAYGIiIhUGBCIiIhI5X+EOspraUYdcAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 576x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(8,6))\n",
    "\n",
    "sns.barplot(x=ab_test['group'], y=ab_test['converted'], ci=False)\n",
    "\n",
    "plt.ylim(0, 0.17)\n",
    "plt.title('Conversion rate by group', pad=20)\n",
    "plt.xlabel('Group', labelpad=15)\n",
    "plt.ylabel('Converted (proportion)', labelpad=15);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The conversion rates for our groups are indeed very close. Also note that the conversion rate of the `control` group is lower than what we would have expected given what we knew about our avg. conversion rate (12.3% vs. 13%). This goes to show that there is some variation in results when sampling from a population.\n",
    "\n",
    "So... the `treatment` group's value is higher. **Is this difference *statistically significant***?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "## 4. Testing the hypothesis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last step of our analysis is testing our hypothesis. Since we have a very large sample, we can use the <a href=\"https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval#Normal_approximation_interval\">normal approximation</a> for calculating our $p$-value (i.e. z-test). \n",
    "\n",
    "Again, Python makes all the calculations very easy. We can use the `statsmodels.stats.proportion` module to get the $p$-value and confidence intervals:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from statsmodels.stats.proportion import proportions_ztest, proportion_confint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "control_results = ab_test[ab_test['group'] == 'control']['converted']\n",
    "treatment_results = ab_test[ab_test['group'] == 'treatment']['converted']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "z statistic: -0.34\n",
      "p-value: 0.732\n",
      "ci 95% for control group: [0.114, 0.133]\n",
      "ci 95% for treatment group: [0.116, 0.135]\n"
     ]
    }
   ],
   "source": [
    "n_con = control_results.count()\n",
    "n_treat = treatment_results.count()\n",
    "successes = [control_results.sum(), treatment_results.sum()]\n",
    "nobs = [n_con, n_treat]\n",
    "\n",
    "z_stat, pval = proportions_ztest(successes, nobs=nobs)\n",
    "(lower_con, lower_treat), (upper_con, upper_treat) = proportion_confint(successes, nobs=nobs, alpha=0.05)\n",
    "\n",
    "print(f'z statistic: {z_stat:.2f}')\n",
    "print(f'p-value: {pval:.3f}')\n",
    "print(f'ci 95% for control group: [{lower_con:.3f}, {upper_con:.3f}]')\n",
    "print(f'ci 95% for treatment group: [{lower_treat:.3f}, {upper_treat:.3f}]')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "## 5. Drawing conclusions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since our $p$-value=0.732 is way above our $\\alpha$=0.05, we cannot reject the null hypothesis $H_0$, which means that our new design did not perform significantly different (let alone better) than our old one :(\n",
    "\n",
    "Additionally, if we look at the confidence interval for the `treatment` group ([0.116, 0.135], i.e. 11.6-13.5%) we notice that:\n",
    "1. It includes our baseline value of 13% conversion rate\n",
    "2. It does not include our target value of 15% (the 2% uplift we were aiming for)\n",
    "\n",
    "What this means is that it is more likely that the true conversion rate of the new design is similar to our baseline, rather than the 15% target we had hoped for. This is further proof that our new design is not likely to be an improvement on our old design, and that unfortunately we are back to the drawing board! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}