wrangle_report.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Objective of project"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Report requirement:\n",
    "- Create a **300-600** word written report called *wrangle_report.pdf or wrangle_report.html* that briefly describes your wrangling efforts. This is to be framed as an internal document.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary of my findings during initial assessment of the 3 datasets\n",
    "\n",
    "#### 1. Twitter archive enhanced file\n",
    "\n",
    "**Dirty Data**\n",
    "1. Changing the data type of timestamp from object to datetime, and tweet_id to object. \n",
    "2. Truncating the html codes in 'source' column to four categories: 'Twitter for iPhone', 'Vine - Make a Scene', 'Twitter Web Client', 'TweetDeck'. \n",
    "3. Renaming column name from 'name' to 'dog_name' to make it more comprehensible.\n",
    "4. Grouping up the names that start with lowercase to None, except for the ones where names could be recovered from the text column. \n",
    "5. The rating_denominator should always be 10. I will update all the cells from whatever they are to 10. \n",
    "6. Delete retweeted_status_id,retweeted_status_user_id,retweeted_status_timestamp, in_reply_to_status_id,in_reply_to_user_id as this information does not bring additional values to the analysis.\n",
    "\n",
    "\n",
    "**Messy Data** \n",
    "1. Flattening the data in columns \"doggo\", \"puppo\",\"floofer\", \"pupper\" by using pandas MELT function. The data will be regrouped to a new column called \"growth_stage\". Investigate if each tweet has more than one growth_stage first before rolling out the changes. \n",
    "\n",
    "\n",
    "#### 2. Image Predictions File \n",
    "**Dirty Data**\n",
    "1. Renaming column names from  p1/2/3,p1/2/3_conf,p1/2/3_dog to breed_pred_1, breed_pred_1_conf, breed_pred_1_outcome,breed_pred_2, breed_pred_2_conf, breed_pred_2_outcome,breed_prediction_3, breed_pred_3_conf, breed_pred_3_outcome to give these columns more meanings. \n",
    "2. Updating all the breeds of dogs in column p1, p2, p3 to lowercase to maintain consistency. \n",
    "3. Changing the data type of tweet_id to object\n",
    "\n",
    "#### 3. Twitter API data\n",
    "**Dirty Data**\n",
    "1. Changing the data type of tweet_id to object\n",
    "\n",
    "\n",
    "\n",
    "#### Merging the tables above \n",
    "**Messy Data** \n",
    "1. After each table has been cleaned up, I will merge all these tables together. \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Written Report##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">During my initial inspection on the 3 datasets, I started with some quick wins, such as changing data types, renaming columns and looking for outliers in each column to understand what do the outliers mean before I proceeded on to cleaning up data. \n",
    ">\n",
    ">Given that a lot of data is dirty, I had to decide what data I actually **should** clean up without compromising the integrity of data. For example, a lot of dogs do not have any names, or do not have . I could have deleted all the rows with no names but then I would reduce the size of data significantly. It was important to understand what kind of data I wanted to look into closely first for further analysis before making a judgement too soon, too early. The question I constantly asked myself was *'does it really make a difference if I clean up this data?'*\n",
    ">\n",
    ">During the actual cleanup, I often had to think about how should I write my code to make it easily comprehensible and optimal. I played around with a lot of functions and dissected the datasets in different ways before deciding on what methods I wanted to use. A good example would be deciding on whether to use left joints or inner joints to merge the 3 datasets.  \n",
    ">\n",
    ">*Please see below all the notes **in bold** I took during the assessment process and the rationale behind my decisions*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Assessing data "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. twitter_archive_enhance file "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Notes to change data type:*\n",
    ">**I will change the datatype of timestamp to datetime, and tweet_id to string.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*During inspection, I noticed that the names of dogs are weird...*\n",
    ">**It is weird that articles (a/an/the) have been used for names. What I found was that all the names that start with lowercase are indeed errors as a result of extraction of wrong data. I was able to recover some of the names of the dog. I would rename the rest of them to None for those that had no mentions of names in the text column.**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*I noticed the 4 columns of growth stage of dogs should be grouped up to just one column*\n",
    ">**I want to flatten these 4 columns using MELT function and put the data under a new category called \"stage_of_life\".**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. image_predictions file \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*I wanted to make sure the characters in the dog breed column is standardised as I would conduct data analysis using this information*\n",
    ">**I noticed that the breeds of dogs can be in lowercase or uppercase. I decided to make everything in lowercase.** "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*To avoid confusing myself or audience, I decided to use more descriptive names for columns*\n",
    "\n",
    ">**I also find that the names of p1,p1_conf,p1_dog,p2,p2_conf,p2_dog,p3,p3_conf,p3_dog unintuitive. I will rename them from p1/2/3,p1/2/3_conf,p1/2/3_dog to breed_pred_1, breed_pred_1_conf, breed_pred_1_outcome,breed_pred_2, breed_pred_2_conf, breed_pred_2_outcome,breed_prediction_3, breed_pred_3_conf, breed_pred_3_outcome.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*During inspection, I couldn't quite figure out what does the info in column p1/2/3_dog mean. So I filtered the data, looked at the \"breed\" in the p1/2/3 column, and realised they were all random objects. So I looked at the photo of via url link and realised that photos were taken when dogs were holding objects*\n",
    ">**Looking at the breeds of dogs in column p1/p2/03, it is obvious that a lot of names are wrong as they are some random objects. It appears that when p1/p2/p3 is showing some random objects, the p1_dog/p2_dog/p3_dog column will show \"False\" as the outcome. I decided not to clean up this part but I will use this information for furth anaylsis**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Twitter API data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*I wanted to make sure I understand why some tweets wouldn't have any favourite counts.*\n",
    ">**The data looks clean. But there some tweets do not have any favorite_count. They can be outliers so further investigation is required.**\n",
    "<br>\n",
    ">**I will also change the data type of tweet_id to object**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">**Information looks normal. Seems like all the tweets without favourite counts are all reweets. I decided not to take any actions**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cleaning data "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Twitter_archive_enhanced file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*When cleaning the names, I noticed that more names can be recovered via the text in the \"text\" column.*\n",
    ">**Most names have been updated. However, I noticed that I could potentially find the names via the text column for some dogs' name with *None*.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">**I need to clean up row 603,2166,2277 and 2269**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">**There are 89 rows with dog_name starting with lowercase need to be updated to None**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*I wanted to make sure I knew what exactly I was deleting when grouping up the 4 columns of growth stages of dogs, so I looked at things in more details to see if there were any odd ones*\n",
    ">**I initially wanted to use the melt function to flatten the data and combine the 4 columns into just one. However, I noticed that 14 samples have more than one growth_stage.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Despite finding the weird data, I decided against complicating the matter and putting too much time to find granular information*\n",
    ">**It seems like maybe I can look up the text content one by one to decide whether I can tidy up the 14 odd samples with multiple growth stages. However, given most data shows *None* anyway, and most dogs are actually not attributed to any growth stage, I decided it is not worth to effort to go through the data one by one. We won't be able to conduct a meaningful anaylsis anyway since most data is missing. I will simply go with the MELT function to sort data.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Image Predictions File"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I didn't encounter any major problems cleaning up this dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Twitter API data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I didn't encounter any major problems cleaning up this dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Merging the three datasets "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*I wanted to make sure I was using the right type of joint to join to datasets*\n",
    ">**I used left join to merge the 3 datasets but decided that this is not the best option for me to conduct analysis as I do not want to have too many nulls in certain columns. I decided I would merge the 3 datasets using inner joins instead.**  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">**After using inner joints, 272 tweets have been excluded. This is alright and I will use this data set to conduct further analysis**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}