scraping-github-topics-repos.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scraping the top repos for Topics on GitHub\n",
    "\n",
    "TODO (Intro):\n",
    "- Introduction to web scraping.\n",
    "- Introduction to Github (contains millions of open source repositories) and the problem statement.\n",
    "- Tools we are using \n",
    " - Python\n",
    " - requests\n",
    " - BeautifulSoup4\n",
    " - Pandas\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Outline:\n",
    "Here're the steps we will follow:\n",
    "- We're going to scrape : https://github.com/topics\n",
    "- We'll get a list of topics. For each topic, we will get topic title, topic page URL and topic description.\n",
    "- For each topic, we'll get top 25 repositories in the topic from topic page.\n",
    "- For each repository, we'll grab the repo name, username, stars and repo URL\n",
    "- For each topic, we'll create a csv file in the following format:\n",
    "\n",
    "```\n",
    "Repo Name,Username,Stars ,Repo URL\n",
    "three.js,mrdoob,69700,https://github.com/mrdoob/three.js\n",
    "libgdx,libgdx,18300,https://github.com/libgdx/libgdx\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scrape the list of topics from GitHub\n",
    "\n",
    "Explain\n",
    "\n",
    "- use requests to download the page\n",
    "- use bs4 to parse and extract information\n",
    "- convert to pandas dataframe\n",
    "\n",
    "Let's write a function to download the page"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# importing required libraries\n",
    "\n",
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_topics_page():\n",
    "    topics_url = 'https://github.com/topics'\n",
    "    response = requests.get(topics_url)\n",
    "    if response.status_code != 200:\n",
    "        raise Exception(\"Failed to load page {}\".format(topic_url))\n",
    "    doc = BeautifulSoup(response.text, 'html.parser')\n",
    "    return doc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc = get_topics_page()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "bs4.BeautifulSoup"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create some helper functions to parse information from the page.\n",
    "\n",
    "\n",
    "To get topic titles we can `p` tags with the `class` ...\n",
    "\n",
    "![](https://i.imgur.com/sklG1ai.png?1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_topic_titles(doc):\n",
    "    selection_class = 'f3 lh-condensed mb-0 mt-1 Link--primary'\n",
    "    topic_title_tags = doc.find_all('p', {'class' : selection_class})\n",
    "    \n",
    "    topic_titles = []\n",
    "    \n",
    "    for tag in topic_title_tags:\n",
    "        topic_titles.append(tag.text)\n",
    "    \n",
    "    return topic_titles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`get_topic_titles` can be used to get the title from the topics page"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "titles = get_topic_titles(doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "30"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(titles)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['3D', 'Ajax', 'Algorithm', 'Amp', 'Android']"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titles[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly we have designed functions for descriptions and URLs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_topic_descs(doc):\n",
    "    desc_selection_class = 'f5 color-text-secondary mb-0 mt-1'\n",
    "    topic_desc_tags = doc.find_all('p', {'class' : desc_selection_class})\n",
    "    \n",
    "    topic_descs = []\n",
    "    \n",
    "    for desc in topic_desc_tags:\n",
    "        topic_descs.append(desc.text.strip())\n",
    "        \n",
    "    return topic_descs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`get_topic_descs` can be used to get description of the repositories."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TODO  : Example and explanation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "descs = get_topic_descs(doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "30"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(descs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['3D modeling is the process of virtually developing the surface and structure of a 3D object.',\n",
       " 'Ajax is a technique for creating interactive web applications.',\n",
       " 'Algorithms are self-contained sequences that carry out a variety of tasks.',\n",
       " 'Amp is a non-blocking concurrency framework for PHP.',\n",
       " 'Android is an operating system built by Google designed for mobile devices.']"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "descs[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_topic_urls(doc):\n",
    "    topic_link_tags = doc.find_all('a', {'class' : 'd-flex no-underline'})     \n",
    "    \n",
    "    base_url = 'https://github.com'\n",
    "    \n",
    "    topic_urls = []\n",
    "    \n",
    "    for url in topic_link_tags:\n",
    "        topic_urls.append(base_url + url['href'])\n",
    "    \n",
    "    return topic_urls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`get_topic_urls` can be used to get the urls of the respective topic page"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's put this all together into a single function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def scrape_topics():\n",
    "    topics_url = 'https://github.com/topics'\n",
    "    response = requests.get(topics_url)\n",
    "    if response.status_code != 200:\n",
    "        raise Exception(\"Failed to load page {}\".format(topic_url))\n",
    "    doc = BeautifulSoup(response.text, 'html.parser')\n",
    "        \n",
    "    topics_dict = {\n",
    "        'title': get_topic_titles(doc),\n",
    "        'description': get_topic_descs(doc),\n",
    "        'urls': get_topic_urls(doc)\n",
    "    }\n",
    "    \n",
    "    return pd.DataFrame(topics_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>description</th>\n",
       "      <th>urls</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3D</td>\n",
       "      <td>3D modeling is the process of virtually develo...</td>\n",
       "      <td>https://github.com/topics/3d</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Ajax</td>\n",
       "      <td>Ajax is a technique for creating interactive w...</td>\n",
       "      <td>https://github.com/topics/ajax</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Algorithm</td>\n",
       "      <td>Algorithms are self-contained sequences that c...</td>\n",
       "      <td>https://github.com/topics/algorithm</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Amp</td>\n",
       "      <td>Amp is a non-blocking concurrency framework fo...</td>\n",
       "      <td>https://github.com/topics/amphp</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Android</td>\n",
       "      <td>Android is an operating system built by Google...</td>\n",
       "      <td>https://github.com/topics/android</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Angular</td>\n",
       "      <td>Angular is an open source web application plat...</td>\n",
       "      <td>https://github.com/topics/angular</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Ansible</td>\n",
       "      <td>Ansible is a simple and powerful automation en...</td>\n",
       "      <td>https://github.com/topics/ansible</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>API</td>\n",
       "      <td>An API (Application Programming Interface) is ...</td>\n",
       "      <td>https://github.com/topics/api</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Arduino</td>\n",
       "      <td>Arduino is an open source hardware and softwar...</td>\n",
       "      <td>https://github.com/topics/arduino</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>ASP.NET</td>\n",
       "      <td>ASP.NET is a web framework for building modern...</td>\n",
       "      <td>https://github.com/topics/aspnet</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Atom</td>\n",
       "      <td>Atom is a open source text editor built with w...</td>\n",
       "      <td>https://github.com/topics/atom</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Awesome Lists</td>\n",
       "      <td>An awesome list is a list of awesome things cu...</td>\n",
       "      <td>https://github.com/topics/awesome</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Amazon Web Services</td>\n",
       "      <td>Amazon Web Services provides on-demand cloud c...</td>\n",
       "      <td>https://github.com/topics/aws</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Azure</td>\n",
       "      <td>Azure is a cloud computing service created by ...</td>\n",
       "      <td>https://github.com/topics/azure</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Babel</td>\n",
       "      <td>Babel is a compiler for writing next generatio...</td>\n",
       "      <td>https://github.com/topics/babel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Bash</td>\n",
       "      <td>Bash is a shell and command language interpret...</td>\n",
       "      <td>https://github.com/topics/bash</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Bitcoin</td>\n",
       "      <td>Bitcoin is a cryptocurrency developed by Satos...</td>\n",
       "      <td>https://github.com/topics/bitcoin</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Bootstrap</td>\n",
       "      <td>Bootstrap is an HTML, CSS, and JavaScript fram...</td>\n",
       "      <td>https://github.com/topics/bootstrap</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Bot</td>\n",
       "      <td>A bot is an application that runs automated ta...</td>\n",
       "      <td>https://github.com/topics/bot</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>C</td>\n",
       "      <td>C is a general purpose programming language th...</td>\n",
       "      <td>https://github.com/topics/c</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>Chrome</td>\n",
       "      <td>Chrome is a web browser from the tech company ...</td>\n",
       "      <td>https://github.com/topics/chrome</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>Chrome extension</td>\n",
       "      <td>Google Chrome Extensions are add-ons that allo...</td>\n",
       "      <td>https://github.com/topics/chrome-extension</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>Command line interface</td>\n",
       "      <td>A CLI, or command-line interface, is a console...</td>\n",
       "      <td>https://github.com/topics/cli</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Clojure</td>\n",
       "      <td>Clojure is a dynamic, general-purpose programm...</td>\n",
       "      <td>https://github.com/topics/clojure</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>Code quality</td>\n",
       "      <td>Automate your code review with style, quality,...</td>\n",
       "      <td>https://github.com/topics/code-quality</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>Code review</td>\n",
       "      <td>Ensure your code meets quality standards and s...</td>\n",
       "      <td>https://github.com/topics/code-review</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>Compiler</td>\n",
       "      <td>Compilers are software that translate higher-l...</td>\n",
       "      <td>https://github.com/topics/compiler</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>Continuous integration</td>\n",
       "      <td>Automatically build and test your code as you ...</td>\n",
       "      <td>https://github.com/topics/continuous-integration</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>COVID-19</td>\n",
       "      <td>The coronavirus disease 2019 (COVID-19) is an ...</td>\n",
       "      <td>https://github.com/topics/covid-19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>C++</td>\n",
       "      <td>C++ is a general purpose and object-oriented p...</td>\n",
       "      <td>https://github.com/topics/cpp</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     title                                        description  \\\n",
       "0                       3D  3D modeling is the process of virtually develo...   \n",
       "1                     Ajax  Ajax is a technique for creating interactive w...   \n",
       "2                Algorithm  Algorithms are self-contained sequences that c...   \n",
       "3                      Amp  Amp is a non-blocking concurrency framework fo...   \n",
       "4                  Android  Android is an operating system built by Google...   \n",
       "5                  Angular  Angular is an open source web application plat...   \n",
       "6                  Ansible  Ansible is a simple and powerful automation en...   \n",
       "7                      API  An API (Application Programming Interface) is ...   \n",
       "8                  Arduino  Arduino is an open source hardware and softwar...   \n",
       "9                  ASP.NET  ASP.NET is a web framework for building modern...   \n",
       "10                    Atom  Atom is a open source text editor built with w...   \n",
       "11           Awesome Lists  An awesome list is a list of awesome things cu...   \n",
       "12     Amazon Web Services  Amazon Web Services provides on-demand cloud c...   \n",
       "13                   Azure  Azure is a cloud computing service created by ...   \n",
       "14                   Babel  Babel is a compiler for writing next generatio...   \n",
       "15                    Bash  Bash is a shell and command language interpret...   \n",
       "16                 Bitcoin  Bitcoin is a cryptocurrency developed by Satos...   \n",
       "17               Bootstrap  Bootstrap is an HTML, CSS, and JavaScript fram...   \n",
       "18                     Bot  A bot is an application that runs automated ta...   \n",
       "19                       C  C is a general purpose programming language th...   \n",
       "20                  Chrome  Chrome is a web browser from the tech company ...   \n",
       "21        Chrome extension  Google Chrome Extensions are add-ons that allo...   \n",
       "22  Command line interface  A CLI, or command-line interface, is a console...   \n",
       "23                 Clojure  Clojure is a dynamic, general-purpose programm...   \n",
       "24            Code quality  Automate your code review with style, quality,...   \n",
       "25             Code review  Ensure your code meets quality standards and s...   \n",
       "26                Compiler  Compilers are software that translate higher-l...   \n",
       "27  Continuous integration  Automatically build and test your code as you ...   \n",
       "28                COVID-19  The coronavirus disease 2019 (COVID-19) is an ...   \n",
       "29                     C++  C++ is a general purpose and object-oriented p...   \n",
       "\n",
       "                                                urls  \n",
       "0                       https://github.com/topics/3d  \n",
       "1                     https://github.com/topics/ajax  \n",
       "2                https://github.com/topics/algorithm  \n",
       "3                    https://github.com/topics/amphp  \n",
       "4                  https://github.com/topics/android  \n",
       "5                  https://github.com/topics/angular  \n",
       "6                  https://github.com/topics/ansible  \n",
       "7                      https://github.com/topics/api  \n",
       "8                  https://github.com/topics/arduino  \n",
       "9                   https://github.com/topics/aspnet  \n",
       "10                    https://github.com/topics/atom  \n",
       "11                 https://github.com/topics/awesome  \n",
       "12                     https://github.com/topics/aws  \n",
       "13                   https://github.com/topics/azure  \n",
       "14                   https://github.com/topics/babel  \n",
       "15                    https://github.com/topics/bash  \n",
       "16                 https://github.com/topics/bitcoin  \n",
       "17               https://github.com/topics/bootstrap  \n",
       "18                     https://github.com/topics/bot  \n",
       "19                       https://github.com/topics/c  \n",
       "20                  https://github.com/topics/chrome  \n",
       "21        https://github.com/topics/chrome-extension  \n",
       "22                     https://github.com/topics/cli  \n",
       "23                 https://github.com/topics/clojure  \n",
       "24            https://github.com/topics/code-quality  \n",
       "25             https://github.com/topics/code-review  \n",
       "26                https://github.com/topics/compiler  \n",
       "27  https://github.com/topics/continuous-integration  \n",
       "28                https://github.com/topics/covid-19  \n",
       "29                     https://github.com/topics/cpp  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scrape_topics()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Get Top 25 repositories from a topic page"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_topic_page(topic_url):\n",
    "    # download the page\n",
    "    response3 = requests.get(topic_url)\n",
    "    \n",
    "    # checking successful response\n",
    "    if response3.status_code != 200:\n",
    "        raise Exception('Failed to load Page {}'.format(topic_url))\n",
    "    \n",
    "    # parse using beatuifulSoup\n",
    "    topic_docs = BeautifulSoup(response3.text,'html.parser')\n",
    "    \n",
    "    return topic_docs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "topic_doc = get_topic_page(\"https://github.com/topic/3d\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TODO: Talk about the h3 tags"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(topic_doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<h3 class=\"f3 color-text-secondary text-normal lh-condensed\">\n",
       "<a data-ga-click=\"Explore, go to repository owner, location:explore feed\" data-hydro-click='{\"event_type\":\"explore.click\",\"payload\":{\"click_context\":\"REPOSITORY_CARD\",\"click_target\":\"OWNER\",\"click_visual_representation\":\"REPOSITORY_OWNER_HEADING\",\"actor_id\":null,\"record_id\":97088,\"originating_url\":\"https://github.com/topics/3d\",\"user_id\":null}}' data-hydro-click-hmac=\"4bdbc49d3c05ae7f70b531fbce709a384200b0768554e0172950286a8db30940\" data-view-component=\"true\" href=\"/mrdoob\">\n",
       "            mrdoob\n",
       "</a>          /\n",
       "          <a class=\"text-bold\" data-ga-click=\"Explore, go to repository, location:explore feed\" data-hydro-click='{\"event_type\":\"explore.click\",\"payload\":{\"click_context\":\"REPOSITORY_CARD\",\"click_target\":\"REPOSITORY\",\"click_visual_representation\":\"REPOSITORY_NAME_HEADING\",\"actor_id\":null,\"record_id\":576201,\"originating_url\":\"https://github.com/topics/3d\",\"user_id\":null}}' data-hydro-click-hmac=\"517d3d5cb9d89752156923904a4238816bc9b51ab7772f3e3644ce897d8dd4e5\" data-view-component=\"true\" href=\"/mrdoob/three.js\">\n",
       "            three.js\n",
       "</a> </h3>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "topic_doc.find('h3')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_repo_info(h3_tags, repo_star):\n",
    "    #returns all the required information about the repository\n",
    "    \n",
    "    a_tags = h3_tags.find_all('a')\n",
    "    username = a_tags[0].text.strip()\n",
    "    repo_name = a_tags[1].text.strip()\n",
    "    repo_url = base_url + a_tags[1]['href']\n",
    "    stars = parse_star_count(repo_star.text.strip())\n",
    "    \n",
    "    return username, repo_name, stars, repo_url"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TODO : Show an example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_topic_repos(topic_docs):\n",
    "    \n",
    "    # get h1 tags containing repo title, repo URL and username\n",
    "    h3_selection_class = 'f3 color-text-secondary text-normal lh-condensed'\n",
    "    repo_tags = topic_docs.find_all('h3', {'class': h3_selection_class})\n",
    "    \n",
    "    # get star tags\n",
    "    stars_selection_class = \"social-count float-none\"\n",
    "    repo_stars = topic_docs.find_all('a',{'class' : stars_selection_class})\n",
    "    \n",
    "    topic_repos_dict = {\n",
    "        'username': [],\n",
    "        'repo_name': [],\n",
    "        'stars' : [],\n",
    "        'repo_url': []\n",
    "    }\n",
    "    \n",
    "    # get repo info\n",
    "    for i in range(len(repo_tags)):\n",
    "        repo_info = get_repo_info(repo_tags[i],repo_stars[i])\n",
    "        topic_repos_dict['username'].append(repo_info[0])\n",
    "        topic_repos_dict['repo_name'].append(repo_info[1])\n",
    "        topic_repos_dict['stars'].append(repo_info[2])\n",
    "        topic_repos_dict['repo_url'].append(repo_info[3])\n",
    "    \n",
    "    return pd.DataFrame(topic_repos_dict)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TODO - Show an example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "def scrape_topic(topic_url, path):\n",
    "#     fname = topic_name + \".csv\"\n",
    "    if os.path.exists(path):\n",
    "        print(\"The file {} already exists. Skipping...\".format(path))\n",
    "        return\n",
    "    topic_df = get_topic_repos((get_topic_page(topic_url)))\n",
    "    \n",
    "    topic_df.to_csv(path, index=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TODO - Show an example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Putting it all together\n",
    "\n",
    "- We have a function to get the list of topics\n",
    "- We have a function to create a CSV file for scraped repos from a topic page.\n",
    "- Let's create a function to put them together\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "def scrape_topic_repos():\n",
    "    print(\"Scraping list of topics\")\n",
    "    topics_df = scrape_topics()\n",
    "    \n",
    "    os.makedirs('data', exist_ok=True)\n",
    "    \n",
    "    for index, row in topics_df.iterrows():\n",
    "        print(\"Scraping top repositories for {}\".format(row['title']))\n",
    "        scrape_topic(row['urls'], 'data/{}.csv'.format(row['title']))\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's run it to scrape the top repos for all the topics on the first page of https://github.com/topics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scrape_topic_repos()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can check that the CSVs were created properly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read and display a csv using pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References and Future Work\n",
    "\n",
    "Summary of what we did\n",
    "\n",
    "References to link I found useful\n",
    "\n",
    "Ideas for future work"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}