Issue_Embeddings/notebooks/01_AcquireData.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Running This Notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook should be run using the [github/mdtok container on DockerHub](https://cloud.docker.com/u/github/repository/docker/github/mdtok).  The Dockerfile that defines this container is located at the root of this repository named: [cpu.Dockerfile](https://github.com/machine-learning-apps/IssuesLanguageModel/blob/master/cpu.Dockerfile)\n",
    "\n",
    "This will ensure that you are able to run this notebook properly as many of the dependencies in this project are rapidly changing.  To run this notebook using this container, the commands are:\n",
    "\n",
    "Get the container: `docker pull github\\mdtok`\n",
    "\n",
    "Run the container: `docker run --it --net=host -v <host_dir>:/ds github/mdtok bash`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mdparse.parser import transform_pre_rules, compose\n",
    "import pandas as pd\n",
    "from tqdm import tqdm_notebook\n",
    "from fastai.text.transform import defaults"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Source of Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [GHArchive project](https://www.gharchive.org/) ingests large amounts of data from GitHub repositories.  This data is stored in [BigQuery](https://cloud.google.com/bigquery/docs/) for public consumption.  \n",
    "\n",
    "For this project, we gathered over 18 million GitHub issues by executing [this query](https://console.cloud.google.com/bigquery?sq=1073071082706:8b10cab0a54b4884b8bf948e70f2f22f). This query attempts to remove duplicate issues where the content of the issue is roughly the same.\n",
    "\n",
    "This query results in over 18 Million GitHub issues.  The results of this query are split into 100 csv files for free download on the following Google Cloud Storage Bucket:\n",
    "\n",
    "`https://storage.googleapis.com/issue_label_bot/language_model_data/0000000000{00-99}.csv.gz`, each file contains approximately 180,000 issues and is 55MB compressed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preview Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Download Sample\n",
    "\n",
    "The below dataframe illustrates what the format of the raw data looks like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>url</th>\n",
       "      <th>repo</th>\n",
       "      <th>title</th>\n",
       "      <th>title_length</th>\n",
       "      <th>body</th>\n",
       "      <th>body_length</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4354</th>\n",
       "      <td>https://github.com/codegangsta/cli/issues/405</td>\n",
       "      <td>codegangsta/cli</td>\n",
       "      <td>Help does not display command aliases</td>\n",
       "      <td>39</td>\n",
       "      <td>Hello, xxxlnbrk First I would like to say that your project is awesome and it is the best CLI library I have used! Keep up the good work! Thank you for that. xxxlnbrk I have a question related to latest version. Recently I have noticed that when displaying help it does no longer show aliases next to the command names.  xxxlnbrk When command is defined in the following way, no aliases are shown in the help output xxxlnbrk `{Name:     xxxdblqtenginx-startxxxdblqte, xxxlnbrk  Aliases:  []string{xxxdblqtengstartxxxdblqte, xxxdblqtenstartxxxdblqte}, xxxlnbrk  Category: cat(NginxCategory),}` xxxlnbrk When deprecated ShortName is used instead Aliases, help does display alias properly. xxxlnbrk `{ xxxlnbrk     Name:     xxxdblqtenginx-startxxxdblqte, xxxlnbrk     ShortName: xxxdblqtengstartxxxdblqte, xxxlnbrk     Category: cat(NginxCategory), xxxlnbrk     ... xxxlnbrk }` xxxlnbrk Is this change in behaviour intentional and we should define a custom help template or did it simply slipped during refactoring?</td>\n",
       "      <td>945</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                url             repo  \\\n",
       "4354  https://github.com/codegangsta/cli/issues/405  codegangsta/cli   \n",
       "\n",
       "                                      title  title_length  \\\n",
       "4354  Help does not display command aliases            39   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       body  \\\n",
       "4354  Hello, xxxlnbrk First I would like to say that your project is awesome and it is the best CLI library I have used! Keep up the good work! Thank you for that. xxxlnbrk I have a question related to latest version. Recently I have noticed that when displaying help it does no longer show aliases next to the command names.  xxxlnbrk When command is defined in the following way, no aliases are shown in the help output xxxlnbrk `{Name:     xxxdblqtenginx-startxxxdblqte, xxxlnbrk  Aliases:  []string{xxxdblqtengstartxxxdblqte, xxxdblqtenstartxxxdblqte}, xxxlnbrk  Category: cat(NginxCategory),}` xxxlnbrk When deprecated ShortName is used instead Aliases, help does display alias properly. xxxlnbrk `{ xxxlnbrk     Name:     xxxdblqtenginx-startxxxdblqte, xxxlnbrk     ShortName: xxxdblqtengstartxxxdblqte, xxxlnbrk     Category: cat(NginxCategory), xxxlnbrk     ... xxxlnbrk }` xxxlnbrk Is this change in behaviour intentional and we should define a custom help template or did it simply slipped during refactoring?   \n",
       "\n",
       "      body_length  \n",
       "4354          945  "
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(f'https://storage.googleapis.com/issue_label_bot/language_model_data/000000000000.csv.gz').sample(5)\n",
    "\n",
    "df.head(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Illustrate Markdown Parsing Using `mdparse`\n",
    "\n",
    "[mdparse](https://github.com/machine-learning-apps/mdparse) is a library that parses markdown text and annotates the text with fields with meta-data for deep learning. Below is an illustration of `mdparse` at work.  The parsed and annotated text can be seen in the `clean_body` field:\n",
    "\n",
    "The changes are often subtle, but can make a big difference with regard to feature extraction for language modeling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e625afd2956d46f697c5c1b4b5a1e0a3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=5), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.7/site-packages/pandas/core/indexing.py:190: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  self._setitem_with_indexer(indexer, value)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>body</th>\n",
       "      <th>clean_body</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4354</th>\n",
       "      <td>Hello, xxxlnbrk First I would like to say that your project is awesome and it is the best CLI library I have used! Keep up the good work! Thank you for that. xxxlnbrk I have a question related to latest version. Recently I have noticed that when displaying help it does no longer show aliases next to the command names.  xxxlnbrk When command is defined in the following way, no aliases are shown in the help output xxxlnbrk `{Name:     xxxdblqtenginx-startxxxdblqte, xxxlnbrk  Aliases:  []string{xxxdblqtengstartxxxdblqte, xxxdblqtenstartxxxdblqte}, xxxlnbrk  Category: cat(NginxCategory),}` xxxlnbrk When deprecated ShortName is used instead Aliases, help does display alias properly. xxxlnbrk `{ xxxlnbrk     Name:     xxxdblqtenginx-startxxxdblqte, xxxlnbrk     ShortName: xxxdblqtengstartxxxdblqte, xxxlnbrk     Category: cat(NginxCategory), xxxlnbrk     ... xxxlnbrk }` xxxlnbrk Is this change in behaviour intentional and we should define a custom help template or did it simply slipped du...</td>\n",
       "      <td>Hello, First I would like to say that your project is awesome and it is the best CLI library I have used! Keep up the good work! Thank you for that. I have a question related to latest version. Recently I have noticed that when displaying help it does no longer show aliases next to the command names. \\n When command is defined in the following way, no aliases are shown in the help output xxxcdb xxxjson xxxcde \\n When deprecated ShortName is used instead Aliases, help does display alias properly. xxxcdb xxxjson xxxcde \\n Is this change in behaviour intentional and we should define a custom help template or did it simply slipped during refactoring?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52889</th>\n",
       "      <td>Like height, custom colors (e.g. adding alpha), vibration length and custom layouts, because not including &lt;&gt; makes this keyboard rather not THAT useful for me.</td>\n",
       "      <td>Like height, custom colors (e.g. adding alpha), vibration length and custom layouts, because not including &lt;&gt; makes this keyboard rather not THAT useful for me.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>151496</th>\n",
       "      <td>As a user, I can see a list of all the Open tasks (created by security guard) so I can take select any of those tasks to complete it. xxxlnbrk  xxxlnbrk I should see a list of entries made by Security guard with trailer's location as 'Open.'  xxxlnbrk  xxxlnbrk As a user I can see following details on my dashboard: xxxlnbrk  xxxlnbrk - Trailer Number xxxlnbrk - Status xxxlnbrk - Trip Number xxxlnbrk - Shunt Pending (Yes/No) xxxlnbrk - Confirm Shunt (Yes/No) xxxlnbrk - Zone = Open xxxlnbrk - Notes  xxxlnbrk - Seal Number xxxlnbrk  xxxlnbrk # Acceptance Tests xxxlnbrk  xxxlnbrk * Verify that a shunt driver user is getting following options: xxxlnbrk &gt;* Assign zone xxxlnbrk &gt;* Raise conflict xxxlnbrk  xxxlnbrk * Verify that a Shunt Driver can sort the list using any of the below mentioned parameters: xxxlnbrk  xxxlnbrk * Verify that the entries made by Security User are appearing under Shunt Driver's dashboard with Zone as open. xxxlnbrk  xxxlnbrk * Verify that entries with Zone as Op...</td>\n",
       "      <td>As a user, I can see a list of all the Open tasks (created by security guard) so I can take select any of those tasks to complete it. \\n I should see a list of entries made by Security guard with trailer's location as 'Open.' \\n As a user I can see following details on my dashboard: \\n xxxlistB Trailer Number Status Trip Number Shunt Pending (Yes / No) Confirm Shunt (Yes / No) Zone = Open Notes Seal Number xxxlistE \\n xxxhl Acceptance Tests \\n xxxlistB Verify that a shunt driver user is getting following options: \\n Assign zone Raise conflict Verify that a Shunt Driver can sort the list using any of the below mentioned parameters: \\n Verify that the entries made by Security User are appearing under Shunt Driver's dashboard with Zone as open. \\n Verify that entries with Zone as Open will get displayed under separate list \\n xxxlistE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>168828</th>\n",
       "      <td>When rendering a simple markdown with latex equation as html xxxlnbrk Simple reproducible example: xxxlnbrk `$$\\\\hat{f}(x) = \\\\sum\\\\limits_{i=1}^N \\\\hat{\\\\alpha_i}\\\\; y_i\\\\; K(x,x_i) + \\\\hat{\\\\beta_i}$$` xxxlnbrk VS-code produces:  xxxlnbrk ```html xxxlnbrk &lt;script type=xxxdblqtetext/x-mathjax-configxxxdblqte&gt; xxxlnbrk           MathJax.Hub.Config({xxxdblqteextensionsxxxdblqte:[xxxdblqtetex2jax.jsxxxdblqte],xxxdblqtejaxxxxdblqte:[xxxdblqteinput/TeXxxxdblqte,xxxdblqteoutput/HTML-CSSxxxdblqte],xxxdblqtemessageStylexxxdblqte:xxxdblqtenonexxxdblqte,xxxdblqtetex2jaxxxxdblqte:{xxxdblqteprocessEnvironmentsxxxdblqte:false,xxxdblqteprocessEscapesxxxdblqte:true,xxxdblqteinlineMathxxxdblqte:[[xxxdblqte$xxxdblqte,xxxdblqte$xxxdblqte],[xxxdblqte\\\\\\\\(xxxdblqte,xxxdblqte\\\\\\\\)xxxdblqte]],xxxdblqtedisplayMathxxxdblqte:[[xxxdblqte$$xxxdblqte,xxxdblqte$$xxxdblqte],[xxxdblqte\\\\\\\\[xxxdblqte,xxxdblqte\\\\\\\\]xxxdblqte]]},xxxdblqteTeXxxxdblqte:{xxxdblqteextensionsxxxdblqte:[xxxdblqteAMSmath.jsxxxdblqte,xxxd...</td>\n",
       "      <td>When rendering a simple markdown with latex equation as html Simple reproducible example: xxxcdb $$ \\ \\ hat xxxjson (x) = \\ \\ sum \\ \\ limits_ xxxjson ^N \\ \\ hat xxxjson \\ \\ ; y_i \\ \\ ; K(x,x_i) + \\ \\ hat xxxjson $$ xxxcde \\n VS-code produces: \\n xxxcdb lang-html xxxhtml xxxhtml xxxhtml &lt; / script&gt; \\n xxxcde \\n The last particular statement: \\n xxxcdb \" root \" : \" file: xxxfilepath \" xxxcde \\n Makes it not rendering correctly in Firefox. It works in chrome. VS-code works in both browsers.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40010</th>\n",
       "      <td>### Versions and Environment xxxlnbrk **Vuetify:** 1.2.1 xxxlnbrk **Vue:** 2.5.17 xxxlnbrk **Browsers:** Chrome 68.0.3440.106 xxxlnbrk **OS:** Mac OS 10.12.6 xxxlnbrk ### Steps to reproduce xxxlnbrk Position the mouse pointer exactly between two checkboxes and click, or try checking a checkbox by clicking in the lower half of a checkbox.  xxxlnbrk ### Expected Behavior xxxlnbrk One would expect the checkbox to be checked where the mouse pointer is, i.e the tip of the arrow or the end of the index finger if the icon is a hand. This also happens in HTML checkboxes.  xxxlnbrk ### Actual Behavior xxxlnbrk The checkbox to the bottom of the mouse pointer is selected. Either because the checkbox seems to correspond to the center of the mouse index, or the surface area of the checkbox is too large and overlaps other check box areas since it does not scale when heigh values lower. This also happens on other low value heights. I suspect the latter one to be the issue, as checkboxes can also ...</td>\n",
       "      <td>xxxhm Versions and Environment \\n Vuetify: 1.2.1 Vue: 2.5.17 Browsers: Chrome xxunk OS: Mac OS 10.12.6 \\n xxxhm Steps to reproduce \\n Position the mouse pointer exactly between two checkboxes and click, or try checking a checkbox by clicking in the lower half of a checkbox. \\n xxxhm Expected Behavior \\n One would expect the checkbox to be checked where the mouse pointer is, i.e the tip of the arrow or the end of the index finger if the icon is a hand. This also happens in HTML checkboxes. \\n xxxhm Actual Behavior \\n The checkbox to the bottom of the mouse pointer is selected. Either because the checkbox seems to correspond to the center of the mouse index, or the surface area of the checkbox is too large and overlaps other check box areas since it does not scale when heigh values lower. This also happens on other low value heights. I suspect the latter one to be the issue, as checkboxes can also be checked when clicking slightly outside of the checkbox. \\n xxxhm Reproduction Link \\...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           body  \\\n",
       "4354    Hello, xxxlnbrk First I would like to say that your project is awesome and it is the best CLI library I have used! Keep up the good work! Thank you for that. xxxlnbrk I have a question related to latest version. Recently I have noticed that when displaying help it does no longer show aliases next to the command names.  xxxlnbrk When command is defined in the following way, no aliases are shown in the help output xxxlnbrk `{Name:     xxxdblqtenginx-startxxxdblqte, xxxlnbrk  Aliases:  []string{xxxdblqtengstartxxxdblqte, xxxdblqtenstartxxxdblqte}, xxxlnbrk  Category: cat(NginxCategory),}` xxxlnbrk When deprecated ShortName is used instead Aliases, help does display alias properly. xxxlnbrk `{ xxxlnbrk     Name:     xxxdblqtenginx-startxxxdblqte, xxxlnbrk     ShortName: xxxdblqtengstartxxxdblqte, xxxlnbrk     Category: cat(NginxCategory), xxxlnbrk     ... xxxlnbrk }` xxxlnbrk Is this change in behaviour intentional and we should define a custom help template or did it simply slipped du...   \n",
       "52889                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Like height, custom colors (e.g. adding alpha), vibration length and custom layouts, because not including <> makes this keyboard rather not THAT useful for me.   \n",
       "151496  As a user, I can see a list of all the Open tasks (created by security guard) so I can take select any of those tasks to complete it. xxxlnbrk  xxxlnbrk I should see a list of entries made by Security guard with trailer's location as 'Open.'  xxxlnbrk  xxxlnbrk As a user I can see following details on my dashboard: xxxlnbrk  xxxlnbrk - Trailer Number xxxlnbrk - Status xxxlnbrk - Trip Number xxxlnbrk - Shunt Pending (Yes/No) xxxlnbrk - Confirm Shunt (Yes/No) xxxlnbrk - Zone = Open xxxlnbrk - Notes  xxxlnbrk - Seal Number xxxlnbrk  xxxlnbrk # Acceptance Tests xxxlnbrk  xxxlnbrk * Verify that a shunt driver user is getting following options: xxxlnbrk >* Assign zone xxxlnbrk >* Raise conflict xxxlnbrk  xxxlnbrk * Verify that a Shunt Driver can sort the list using any of the below mentioned parameters: xxxlnbrk  xxxlnbrk * Verify that the entries made by Security User are appearing under Shunt Driver's dashboard with Zone as open. xxxlnbrk  xxxlnbrk * Verify that entries with Zone as Op...   \n",
       "168828  When rendering a simple markdown with latex equation as html xxxlnbrk Simple reproducible example: xxxlnbrk `$$\\\\hat{f}(x) = \\\\sum\\\\limits_{i=1}^N \\\\hat{\\\\alpha_i}\\\\; y_i\\\\; K(x,x_i) + \\\\hat{\\\\beta_i}$$` xxxlnbrk VS-code produces:  xxxlnbrk ```html xxxlnbrk <script type=xxxdblqtetext/x-mathjax-configxxxdblqte> xxxlnbrk           MathJax.Hub.Config({xxxdblqteextensionsxxxdblqte:[xxxdblqtetex2jax.jsxxxdblqte],xxxdblqtejaxxxxdblqte:[xxxdblqteinput/TeXxxxdblqte,xxxdblqteoutput/HTML-CSSxxxdblqte],xxxdblqtemessageStylexxxdblqte:xxxdblqtenonexxxdblqte,xxxdblqtetex2jaxxxxdblqte:{xxxdblqteprocessEnvironmentsxxxdblqte:false,xxxdblqteprocessEscapesxxxdblqte:true,xxxdblqteinlineMathxxxdblqte:[[xxxdblqte$xxxdblqte,xxxdblqte$xxxdblqte],[xxxdblqte\\\\\\\\(xxxdblqte,xxxdblqte\\\\\\\\)xxxdblqte]],xxxdblqtedisplayMathxxxdblqte:[[xxxdblqte$$xxxdblqte,xxxdblqte$$xxxdblqte],[xxxdblqte\\\\\\\\[xxxdblqte,xxxdblqte\\\\\\\\]xxxdblqte]]},xxxdblqteTeXxxxdblqte:{xxxdblqteextensionsxxxdblqte:[xxxdblqteAMSmath.jsxxxdblqte,xxxd...   \n",
       "40010   ### Versions and Environment xxxlnbrk **Vuetify:** 1.2.1 xxxlnbrk **Vue:** 2.5.17 xxxlnbrk **Browsers:** Chrome 68.0.3440.106 xxxlnbrk **OS:** Mac OS 10.12.6 xxxlnbrk ### Steps to reproduce xxxlnbrk Position the mouse pointer exactly between two checkboxes and click, or try checking a checkbox by clicking in the lower half of a checkbox.  xxxlnbrk ### Expected Behavior xxxlnbrk One would expect the checkbox to be checked where the mouse pointer is, i.e the tip of the arrow or the end of the index finger if the icon is a hand. This also happens in HTML checkboxes.  xxxlnbrk ### Actual Behavior xxxlnbrk The checkbox to the bottom of the mouse pointer is selected. Either because the checkbox seems to correspond to the center of the mouse index, or the surface area of the checkbox is too large and overlaps other check box areas since it does not scale when heigh values lower. This also happens on other low value heights. I suspect the latter one to be the issue, as checkboxes can also ...   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     clean_body  \n",
       "4354                                                                                                                                                                                                                                                                                                                                                             Hello, First I would like to say that your project is awesome and it is the best CLI library I have used! Keep up the good work! Thank you for that. I have a question related to latest version. Recently I have noticed that when displaying help it does no longer show aliases next to the command names. \\n When command is defined in the following way, no aliases are shown in the help output xxxcdb xxxjson xxxcde \\n When deprecated ShortName is used instead Aliases, help does display alias properly. xxxcdb xxxjson xxxcde \\n Is this change in behaviour intentional and we should define a custom help template or did it simply slipped during refactoring?  \n",
       "52889                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Like height, custom colors (e.g. adding alpha), vibration length and custom layouts, because not including <> makes this keyboard rather not THAT useful for me.  \n",
       "151496                                                                                                                                                              As a user, I can see a list of all the Open tasks (created by security guard) so I can take select any of those tasks to complete it. \\n I should see a list of entries made by Security guard with trailer's location as 'Open.' \\n As a user I can see following details on my dashboard: \\n xxxlistB Trailer Number Status Trip Number Shunt Pending (Yes / No) Confirm Shunt (Yes / No) Zone = Open Notes Seal Number xxxlistE \\n xxxhl Acceptance Tests \\n xxxlistB Verify that a shunt driver user is getting following options: \\n Assign zone Raise conflict Verify that a Shunt Driver can sort the list using any of the below mentioned parameters: \\n Verify that the entries made by Security User are appearing under Shunt Driver's dashboard with Zone as open. \\n Verify that entries with Zone as Open will get displayed under separate list \\n xxxlistE  \n",
       "168828                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             When rendering a simple markdown with latex equation as html Simple reproducible example: xxxcdb $$ \\ \\ hat xxxjson (x) = \\ \\ sum \\ \\ limits_ xxxjson ^N \\ \\ hat xxxjson \\ \\ ; y_i \\ \\ ; K(x,x_i) + \\ \\ hat xxxjson $$ xxxcde \\n VS-code produces: \\n xxxcdb lang-html xxxhtml xxxhtml xxxhtml < / script> \\n xxxcde \\n The last particular statement: \\n xxxcdb \" root \" : \" file: xxxfilepath \" xxxcde \\n Makes it not rendering correctly in Firefox. It works in chrome. VS-code works in both browsers.  \n",
       "40010   xxxhm Versions and Environment \\n Vuetify: 1.2.1 Vue: 2.5.17 Browsers: Chrome xxunk OS: Mac OS 10.12.6 \\n xxxhm Steps to reproduce \\n Position the mouse pointer exactly between two checkboxes and click, or try checking a checkbox by clicking in the lower half of a checkbox. \\n xxxhm Expected Behavior \\n One would expect the checkbox to be checked where the mouse pointer is, i.e the tip of the arrow or the end of the index finger if the icon is a hand. This also happens in HTML checkboxes. \\n xxxhm Actual Behavior \\n The checkbox to the bottom of the mouse pointer is selected. Either because the checkbox seems to correspond to the center of the mouse index, or the surface area of the checkbox is too large and overlaps other check box areas since it does not scale when heigh values lower. This also happens on other low value heights. I suspect the latter one to be the issue, as checkboxes can also be checked when clicking slightly outside of the checkbox. \\n xxxhm Reproduction Link \\...  "
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.set_option('max_colwidth', 1000)\n",
    "\n",
    "df['clean_body'] = ''\n",
    "for i, b in tqdm_notebook(enumerate(df.body), total=len(df)):\n",
    "    try:\n",
    "        df['clean_body'].iloc[i] = compose(transform_pre_rules+defaults.text_pre_rules)(b)\n",
    "    except:\n",
    "        print(f'error at: {i}')\n",
    "        break\n",
    "        \n",
    "df[['body', 'clean_body']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Download And Pre-Process Data\n",
    "\n",
    "We download the data from GCP and pre-process this data before saving to disk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai.text.transform import ProcessPoolExecutor, partition_by_cores\n",
    "import numpy as np\n",
    "from fastai.core import parallel\n",
    "from itertools import chain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "transforms = transform_pre_rules + defaults.text_pre_rules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def process_dict(dfdict, _):\n",
    "    \"\"\"process the data, but allow failure.\"\"\"\n",
    "    t = compose(transforms)\n",
    "    title = dfdict['title']\n",
    "    body = dfdict['body']\n",
    "    try:\n",
    "        text = 'xxxfldtitle '+ t(title) + ' xxxfldbody ' + t(body)\n",
    "    except:\n",
    "        return None\n",
    "    return {'url': dfdict['url'], 'text':text}\n",
    "\n",
    "\n",
    "def download_data(i, _):\n",
    "    \"\"\"Since the data is in 100 chunks already, just do the processing by chunk.\"\"\"\n",
    "    fn = f'https://storage.googleapis.com/issue_label_bot/language_model_data/{str(i).zfill(12)}.csv.gz'\n",
    "    dicts = [process_dict(d, 0) for d in pd.read_csv(fn).to_dict(orient='rows')]\n",
    "    df = pd.DataFrame([d for d in dicts if d])\n",
    "    df.to_csv(f'/ds/IssuesLanguageModel/data/1_processed_csv/processed_part{str(i).zfill(4)}.csv', index=False)\n",
    "    return df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: The below procedure took over 30 hours on a [p3.8xlarge](https://aws.amazon.com/ec2/instance-types/p3/) instance on AWS with 32 Cores and 64GB of Memory.  You may have to change the number of workers based on your memory and compute constraints."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfs = parallel(download_data, list(range(100)), max_workers=31)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number of rows in pre-processed data: 18,620,833\n"
     ]
    }
   ],
   "source": [
    "dfs_rows = sum([x.shape[0] for x in dfs])\n",
    "print(f'number of rows in pre-processed data: {dfs_rows:,}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "del dfs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cached pre-processed data\n",
    "\n",
    "Since ~19M GitHub issues take a long time to pre-process, the pre-processed files are available here:\n",
    "\n",
    "`https://storage.googleapis.com/issue_label_bot/pre_processed_data/1_processed_csv/processed_part00{00-99}.csv`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Partition Data Into Train/Validation Set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set aside random 10 files (out of 100) as the Validation set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('/ds/IssuesLanguageModel/data/1_processed_csv/processed_part0039.csv'),\n",
       " PosixPath('/ds/IssuesLanguageModel/data/1_processed_csv/processed_part0095.csv'),\n",
       " PosixPath('/ds/IssuesLanguageModel/data/1_processed_csv/processed_part0097.csv'),\n",
       " PosixPath('/ds/IssuesLanguageModel/data/1_processed_csv/processed_part0029.csv'),\n",
       " PosixPath('/ds/IssuesLanguageModel/data/1_processed_csv/processed_part0064.csv')]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from pathlib import Path\n",
    "from random import shuffle\n",
    "\n",
    "# shuffle the files\n",
    "p = Path('/ds/IssuesLanguageModel/data/1_processed_csv/')\n",
    "files = p.ls()\n",
    "shuffle(files)\n",
    "\n",
    "# show a preview of files\n",
    "files[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rows in train_df:, 16,762,799\n",
      "rows in valid_df:, 1,858,033\n"
     ]
    }
   ],
   "source": [
    "valid_df = pd.concat([pd.read_csv(f) for f in files[:10]]).dropna().drop_duplicates()\n",
    "train_df = pd.concat([pd.read_csv(f) for f in files[10:]]).dropna().drop_duplicates()\n",
    "\n",
    "print(f'rows in train_df:, {train_df.shape[0]:,}')\n",
    "print(f'rows in valid_df:, {valid_df.shape[0]:,}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "valid_df.to_hdf('/ds/IssuesLanguageModel/data/2_partitioned_df/valid_df.hdf')\n",
    "train_df.to_hdf('/ds/IssuesLanguageModel/data/2_partitioned_df/train_df.hdf')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Location of Train/Validaiton DataFrames\n",
    "\n",
    "You can download the above saved dataframes (in hdf format) from Google Cloud Storage:\n",
    "\n",
    "**train_df.hdf (9GB)**: \n",
    "\n",
    "`https://storage.googleapis.com/issue_label_bot/pre_processed_data/2_partitioned_df/train_df.hdf`\n",
    "\n",
    "**valid_df.hdf (1GB)**\n",
    "\n",
    "`https://storage.googleapis.com/issue_label_bot/pre_processed_data/2_partitioned_df/valid_df.hdf`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}