03-Data-Exploration_II.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "Environmental Data Analytics | John Fay and Luana Lima | Developed by Kateri Salk  \n",
    "Spring 2024\n",
    "\n",
    "---\n",
    "# 03: Data Exploration - Part 2 (in Python)\n",
    "*Environmental Data Analytics / John Fay*<br>\n",
    "*Spring 2021*\n",
    "\n",
    "## Objectives\n",
    "1. Import and explore datasets in R\n",
    "2. Graphically explore datasets in R\n",
    "3. Apply data exploration skills to a real-world example dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## IMPORT AND VIEW SUMMARIES\n",
    "\n",
    "As in R, we often begin our scripts by importing whatever packages we need. The \"Pandas\" package is a Python data analytics library. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Load packages\n",
    "import pandas as pd "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Read the USGS dataset into a dataframe object\n",
    "USGSFlowData = pd.read_csv(\"./data/Processed/USGS_Site02085000_Flow_Processed.csv\")\n",
    "\n",
    "#View the first 5 records\n",
    "USGSFlowData.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Examine characteristics of our dataframe..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Display the data type of our USGSFlowData object\n",
    "type(USGSFlowData) #\"class\" in R"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Reveal the column names \n",
    "USGSFlowData.columns #\"colnames\" in R"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Reveal the structure of our dataframe\n",
    "USGSFlowData.info() #\"str\" in R"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Reveal the dimensions of our dataframe\n",
    "USGSFlowData.shape #\"dim\" in R"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "USGSFlowData['discharge_mean_approval'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check our date column and set it to a datetime object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Reveal datatype of the datetime column\n",
    "USGSFlowData['datetime'].dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Change it to a proper datetime object\n",
    "USGSFlowData['datetime'] = pd.to_datetime(USGSFlowData['datetime'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Reveal datatype of the datetime column\n",
    "USGSFlowData['datetime'].dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Visualization\n",
    "\n",
    "### Data visualization packages\n",
    "Python has many data visualization packages. `Pandas` itself is able to do some quick visualizaitons. A package called \"`Matplotlib`\" has perhaps been around the longest and mimics the syntax used in MatLab. Some of the more powerful ones include `Plotly`, `Dash`, and `Bokeh`, which allow for some interactive plots. However, for those with experience in R's *ggplot*, a package called `plotnine` allows us to mimic that format.\n",
    "\n",
    "### Plotting with Pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Bar plot - need to run the \"value_counts\" to compute # of records in each group\n",
    "USGSFlowData['discharge_mean_approval'].value_counts().plot(kind='bar');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Histogram of mean discharge\n",
    "USGSFlowData['discharge_mean'].plot(kind='hist',bins=10);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Line plot\n",
    "date_mask = USGSFlowData['datetime']>pd.to_datetime(\"2010/01/01\")\n",
    "USGSFlowData[date_mask].plot(\n",
    "    kind='line',\n",
    "    x='datetime',\n",
    "    y='gage_height_mean',\n",
    "    legend=False,\n",
    "    figsize=(20,3)\n",
    ");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plotting with Plotnine/ggplot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Install plotnine (install if needed)\n",
    "try: \n",
    "    from plotnine import *\n",
    "except:\n",
    "    !pip install plotnine\n",
    "    from plotnine import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Construct the plots\n",
    "(   ggplot(USGSFlowData, aes(x='discharge_mean_approval')) + \n",
    "    geom_bar()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(USGSFlowData) +\n",
    "  geom_histogram(aes(x = 'discharge_mean'), binwidth = 10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(USGSFlowData) +\n",
    "  geom_freqpoly(aes(x = 'gage_height_mean'), bins = 50) +\n",
    "  geom_freqpoly(aes(x = 'gage_height_min'), bins = 50, color = \"darkgray\") +\n",
    "  geom_freqpoly(aes(x = 'gage_height_max'), bins = 50, linetype = 'dashed') +\n",
    "  scale_x_continuous(limits = (0, 10)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "myPlot = (\n",
    "    ggplot(USGSFlowData, aes(x='datetime',y='discharge_mean')) + \n",
    "          geom_line(color='blue',size=0.25) +\n",
    "          theme_xkcd()\n",
    "         )\n",
    "myPlot"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}