sampling_soln.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Random Sampling\n",
    "=============\n",
    "\n",
    "Copyright 2016 Allen Downey\n",
    "\n",
    "License: [Creative Commons Attribution 4.0 International](http://creativecommons.org/licenses/by/4.0/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import print_function, division\n",
    "\n",
    "import numpy\n",
    "import scipy.stats\n",
    "\n",
    "import matplotlib.pyplot as pyplot\n",
    "\n",
    "from ipywidgets import interact, interactive, fixed\n",
    "import ipywidgets as widgets\n",
    "\n",
    "# seed the random number generator so we all get the same results\n",
    "numpy.random.seed(18)\n",
    "\n",
    "# some nicer colors from http://colorbrewer2.org/\n",
    "COLOR1 = '#7fc97f'\n",
    "COLOR2 = '#beaed4'\n",
    "COLOR3 = '#fdc086'\n",
    "COLOR4 = '#ffff99'\n",
    "COLOR5 = '#386cb0'\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Part One\n",
    "========\n",
    "\n",
    "Suppose we want to estimate the average weight of men and women in the U.S.\n",
    "\n",
    "And we want to quantify the uncertainty of the estimate.\n",
    "\n",
    "One approach is to simulate many experiments and see how much the results vary from one experiment to the next.\n",
    "\n",
    "I'll start with the unrealistic assumption that we know the actual distribution of weights in the population.  Then I'll show how to solve the problem without that assumption.\n",
    "\n",
    "Based on data from the [BRFSS](http://www.cdc.gov/brfss/), I found that the distribution of weight in kg for women in the U.S. is well modeled by a lognormal distribution with the following parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "weight = scipy.stats.lognorm(0.23, 0, 70.8)\n",
    "weight.mean(), weight.std()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's what that distribution looks like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xs = numpy.linspace(20, 160, 100)\n",
    "ys = weight.pdf(xs)\n",
    "pyplot.plot(xs, ys, linewidth=4, color=COLOR1)\n",
    "pyplot.xlabel('weight (kg)')\n",
    "pyplot.ylabel('PDF')\n",
    "None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`make_sample` draws a random sample from this distribution.  The result is a NumPy array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_sample(n=100):\n",
    "    sample = weight.rvs(n)\n",
    "    return sample"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's an example with `n=100`.  The mean and std of the sample are close to the mean and std of the population, but not exact."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample = make_sample(n=100)\n",
    "sample.mean(), sample.std()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to estimate the average weight in the population, so the \"sample statistic\" we'll use is the mean:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sample_stat(sample):\n",
    "    return sample.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One iteration of \"the experiment\" is to collect a sample of 100 women and compute their average weight.\n",
    "\n",
    "We can simulate running this experiment many times, and collect a list of sample statistics.  The result is a NumPy array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_sampling_distribution(n=100, iters=1000):\n",
    "    stats = [sample_stat(make_sample(n)) for i in range(iters)]\n",
    "    return numpy.array(stats)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next line runs the simulation 1000 times and puts the results in\n",
    "`sample_means`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_means = compute_sampling_distribution(n=100, iters=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the distribution of the sample means.  This distribution shows how much the results vary from one experiment to the next.\n",
    "\n",
    "Remember that this distribution is not the same as the distribution of weight in the population.  This is the distribution of results across repeated imaginary experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pyplot.hist(sample_means, color=COLOR5)\n",
    "pyplot.xlabel('sample mean (n=100)')\n",
    "pyplot.ylabel('count')\n",
    "None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The mean of the sample means is close to the actual population mean, which is nice, but not actually the important part."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_means.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The standard deviation of the sample means quantifies the variability from one experiment to the next, and reflects the precision of the estimate.\n",
    "\n",
    "This quantity is called the \"standard error\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "std_err = sample_means.std()\n",
    "std_err"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use the distribution of sample means to compute a \"90% confidence interval\", which contains 90% of the experimental results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "conf_int = numpy.percentile(sample_means, [5, 95])\n",
    "conf_int"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'd like to see what happens as we vary the sample size, `n`.  The following function takes `n`, runs 1000 simulated experiments, and summarizes the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_sampling_distribution(n, xlim=None):\n",
    "    \"\"\"Plot the sampling distribution.\n",
    "    \n",
    "    n: sample size\n",
    "    xlim: [xmin, xmax] range for the x axis \n",
    "    \"\"\"\n",
    "    sample_stats = compute_sampling_distribution(n, iters=1000)\n",
    "    se = numpy.std(sample_stats)\n",
    "    ci = numpy.percentile(sample_stats, [5, 95])\n",
    "    \n",
    "    pyplot.hist(sample_stats, color=COLOR2)\n",
    "    pyplot.xlabel('sample statistic')\n",
    "    pyplot.xlim(xlim)\n",
    "    text(0.03, 0.95, 'CI [%0.2f %0.2f]' % tuple(ci))\n",
    "    text(0.03, 0.85, 'SE %0.2f' % se)\n",
    "    pyplot.show()\n",
    "    \n",
    "def text(x, y, s):\n",
    "    \"\"\"Plot a string at a given location in axis coordinates.\n",
    "    \n",
    "    x: coordinate\n",
    "    y: coordinate\n",
    "    s: string\n",
    "    \"\"\"\n",
    "    ax = pyplot.gca()\n",
    "    pyplot.text(x, y, s,\n",
    "                horizontalalignment='left',\n",
    "                verticalalignment='top',\n",
    "                transform=ax.transAxes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's a test run with `n=100`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_sampling_distribution(100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can use `interact` to run `plot_sampling_distribution` with different values of `n`.  Note: `xlim` sets the limits of the x-axis so the figure doesn't get rescaled as we vary `n`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sample_stat(sample):\n",
    "    return sample.mean()\n",
    "\n",
    "slider = widgets.IntSlider(min=10, max=1000, value=100)\n",
    "interact(plot_sampling_distribution, n=slider, xlim=fixed([55, 95]))\n",
    "None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other sample statistics\n",
    "\n",
    "This framework works with any other quantity we want to estimate.  By changing `sample_stat`, you can compute the SE and CI for any sample statistic.\n",
    "\n",
    "**Exercise 1**: Fill in `sample_stat` below with any of these statistics:\n",
    "\n",
    "* Standard deviation of the sample.\n",
    "* Coefficient of variation, which is the sample standard deviation divided by the sample standard mean.\n",
    "* Min or Max\n",
    "* Median (which is the 50th percentile)\n",
    "* 10th or 90th percentile.\n",
    "* Interquartile range (IQR), which is the difference between the 75th and 25th percentiles.\n",
    "\n",
    "NumPy array methods you might find useful include `std`, `min`, `max`, and `percentile`.\n",
    "Depending on the results, you might want to adjust `xlim`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sample_stat(sample):\n",
    "    # TODO: replace the following line with another sample statistic\n",
    "    return sample.mean()\n",
    "\n",
    "slider = widgets.IntSlider(min=10, max=1000, value=100)\n",
    "interact(plot_sampling_distribution, n=slider, xlim=fixed([0, 100]))\n",
    "None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "STOP HERE\n",
    "---------\n",
    "\n",
    "We will regroup and discuss before going on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Part Two\n",
    "========\n",
    "\n",
    "So far we have shown that if we know the actual distribution of the population, we can compute the sampling distribution for any sample statistic, and from that we can compute SE and CI.\n",
    "\n",
    "But in real life we don't know the actual distribution of the population.  If we did, we wouldn't be doing statistical inference in the first place!\n",
    "\n",
    "In real life, we use the sample to build a model of the population distribution, then use the model to generate the sampling distribution.  A simple and popular way to do that is \"resampling,\" which means we use the sample itself as a model of the population distribution and draw samples from it.\n",
    "\n",
    "Before we go on, I want to collect some of the code from Part One and organize it as a class.  This class represents a framework for computing sampling distributions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Resampler(object):\n",
    "    \"\"\"Represents a framework for computing sampling distributions.\"\"\"\n",
    "    \n",
    "    def __init__(self, sample, xlim=None):\n",
    "        \"\"\"Stores the actual sample.\"\"\"\n",
    "        self.sample = sample\n",
    "        self.n = len(sample)\n",
    "        self.xlim = xlim\n",
    "        \n",
    "    def resample(self):\n",
    "        \"\"\"Generates a new sample by choosing from the original\n",
    "        sample with replacement.\n",
    "        \"\"\"\n",
    "        new_sample = numpy.random.choice(self.sample, self.n, replace=True)\n",
    "        return new_sample\n",
    "    \n",
    "    def sample_stat(self, sample):\n",
    "        \"\"\"Computes a sample statistic using the original sample or a\n",
    "        simulated sample.\n",
    "        \"\"\"\n",
    "        return sample.mean()\n",
    "    \n",
    "    def compute_sampling_distribution(self, iters=1000):\n",
    "        \"\"\"Simulates many experiments and collects the resulting sample\n",
    "        statistics.\n",
    "        \"\"\"\n",
    "        stats = [self.sample_stat(self.resample()) for i in range(iters)]\n",
    "        return numpy.array(stats)\n",
    "    \n",
    "    def plot_sampling_distribution(self):\n",
    "        \"\"\"Plots the sampling distribution.\"\"\"\n",
    "        sample_stats = self.compute_sampling_distribution()\n",
    "        se = sample_stats.std()\n",
    "        ci = numpy.percentile(sample_stats, [5, 95])\n",
    "    \n",
    "        pyplot.hist(sample_stats, color=COLOR2)\n",
    "        pyplot.xlabel('sample statistic')\n",
    "        pyplot.xlim(self.xlim)\n",
    "        text(0.03, 0.95, 'CI [%0.2f %0.2f]' % tuple(ci))\n",
    "        text(0.03, 0.85, 'SE %0.2f' % se)\n",
    "        pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following function instantiates a `Resampler` and runs it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def interact_func(n, xlim):\n",
    "    sample = weight.rvs(n)\n",
    "    resampler = Resampler(sample, xlim=xlim)\n",
    "    resampler.plot_sampling_distribution()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's a test run with `n=100`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "interact_func(n=100, xlim=[50, 100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can use `interact_func` in an interaction:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "slider = widgets.IntSlider(min=10, max=1000, value=100)\n",
    "interact(interact_func, n=slider, xlim=fixed([50, 100]))\n",
    "None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 2**: write a new class called `StdResampler` that inherits from `Resampler` and overrides `sample_stat` so it computes the standard deviation of the resampled data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here\n",
    "\n",
    "class StdResampler(Resampler):   \n",
    "    \"\"\"Computes the sampling distribution of the standard deviation.\"\"\"\n",
    "    \n",
    "    def sample_stat(self, sample):\n",
    "        \"\"\"Computes a sample statistic using the original sample or a\n",
    "        simulated sample.\n",
    "        \"\"\"\n",
    "        return sample.std()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test your code using the cell below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def interact_func2(n, xlim):\n",
    "    sample = weight.rvs(n)\n",
    "    resampler = StdResampler(sample, xlim=xlim)\n",
    "    resampler.plot_sampling_distribution()\n",
    "    \n",
    "interact_func2(n=100, xlim=[0, 100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When your `StdResampler` is working, you should be able to interact with it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "slider = widgets.IntSlider(min=10, max=1000, value=100)\n",
    "interact(interact_func2, n=slider, xlim=fixed([0, 100]))\n",
    "None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "STOP HERE\n",
    "---------\n",
    "\n",
    "We will regroup and discuss before going on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Part Three\n",
    "==========\n",
    "\n",
    "We can extend this framework to compute SE and CI for a difference in means.\n",
    "\n",
    "For example, men are heavier than women on average.  Here's the women's distribution again (from BRFSS data):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "female_weight = scipy.stats.lognorm(0.23, 0, 70.8)\n",
    "female_weight.mean(), female_weight.std()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And here's the men's distribution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "male_weight = scipy.stats.lognorm(0.20, 0, 87.3)\n",
    "male_weight.mean(), male_weight.std()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I'll simulate a sample of 100 men and 100 women:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "female_sample = female_weight.rvs(100)\n",
    "male_sample = male_weight.rvs(100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The difference in means should be about 17 kg, but will vary from one random sample to the next:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "male_sample.mean() - female_sample.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's the function that computes Cohen's effect size again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def CohenEffectSize(group1, group2):\n",
    "    \"\"\"Compute Cohen's d.\n",
    "\n",
    "    group1: Series or NumPy array\n",
    "    group2: Series or NumPy array\n",
    "\n",
    "    returns: float\n",
    "    \"\"\"\n",
    "    diff = group1.mean() - group2.mean()\n",
    "\n",
    "    n1, n2 = len(group1), len(group2)\n",
    "    var1 = group1.var()\n",
    "    var2 = group2.var()\n",
    "\n",
    "    pooled_var = (n1 * var1 + n2 * var2) / (n1 + n2)\n",
    "    d = diff / numpy.sqrt(pooled_var)\n",
    "    return d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The difference in weight between men and women is about 1 standard deviation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "CohenEffectSize(male_sample, female_sample)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can write a version of the `Resampler` that computes the sampling distribution of $d$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CohenResampler(Resampler):\n",
    "    def __init__(self, group1, group2, xlim=None):\n",
    "        self.group1 = group1\n",
    "        self.group2 = group2\n",
    "        self.xlim = xlim\n",
    "        \n",
    "    def resample(self):\n",
    "        n, m = len(self.group1), len(self.group2)\n",
    "        group1 = numpy.random.choice(self.group1, n, replace=True)\n",
    "        group2 = numpy.random.choice(self.group2, m, replace=True)\n",
    "        return group1, group2\n",
    "    \n",
    "    def sample_stat(self, groups):\n",
    "        group1, group2 = groups\n",
    "        return CohenEffectSize(group1, group2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can instantiate a `CohenResampler` and plot the sampling distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "resampler = CohenResampler(male_sample, female_sample)\n",
    "resampler.plot_sampling_distribution()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This example demonstrates an advantage of the computational framework over mathematical analysis.  Statistics like Cohen's $d$, which is the ratio of other statistics, are relatively difficult to analyze.  But with a computational approach, all sample statistics are equally \"easy\".\n",
    "\n",
    "One note on vocabulary: what I am calling \"resampling\" here is a specific kind of resampling called \"bootstrapping\".  Other techniques that are also considering resampling include permutation tests, which we'll see in the next section, and \"jackknife\" resampling.  You can read more at <http://en.wikipedia.org/wiki/Resampling_(statistics)>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}