English_Wikipedia_NLP_AWS.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploring Wikipedia clickstream data: English Wiki in December 2018    \n",
    "\n",
    "## Defining community topics with NLP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Introduction   \n",
    "\n",
    "This notebook contains natural language processing of the article communities found in the English Wikipedia clickstream dataset for December 2018.  \n",
    "This is the 5th part of a project about the usage patterns of Wikipedia.  \n",
    "The preceding parts are:  \n",
    "1. [Data quality analysis of available datasets](data_quality_analysis.ipynb)  \n",
    "2. [Exploratory data analysis of the English Wikipedia clickstream dataset for December 2018](English_Wikipedia_EDA.ipynb)  \n",
    "3. [Graph modeling of the English Wikipedia clickstream dataset for December 2018](English_Wikipedia_graph_modeling_AWS.ipynb)  \n",
    "4. [Network analysis of the English Wikipedia clickstream dataset](English_Wikipedia_network_analysis_AWS.ipynb)\n",
    "\n",
    "In the preceding notebook, we've used Louvain community detection on the English Wikipedia clickstream graph to find clusters of Wikipedia article that tend to be close to each over by user clickstream traffic. Inspecting the article titles in the reulting communities, it appears that the commuities tend to cluster on certain topics. Since these communities appear to reflect topics of interest, it would be useful to know what these topics are without having to browse through the individual article titles in the communities. We can use NLP to extract the community topics.  \n",
    "\n",
    "In this notebook, we'll use natural language processing techniques to derive topic descriptions for the article communities we've found in the English Wikipedia clickstream graph.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Notebook contents:  \n",
    "\n",
    "1. [Introduction](#1.-Introduction)  \n",
    "2. [Notebook setup](#2.-Notebook-setup)  \n",
    "3. [Analysis](#3.-Analysis)  \n",
    "   3.1 [Pre-process article titles](#3.1-Pre-process-article-titles)  \n",
    "   3.2 [Named entity recognition](#3.2-Named-entity-recognition)  \n",
    "      = [Named entity type descriptions](#Named-entity-type-descriptions)  \n",
    "   3.3 [Lemmatizing and stemming](#3.3-Lemmatizing-and-stemming)  \n",
    "   3.4 [Scaled traffic weights](#3.4-Scaled-traffic-weights)  \n",
    "   3.5 [Bags of words](#3.5-Bags-of-words)  \n",
    "      = [What are the topics of the top 20 communities?](#What-are-the-topics-of-the-top-20-communities?)\n",
    "4. [Summary](#4.-Summary)\n",
    "5. [Next steps](#5.-Next-steps)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Notebook setup  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import re\n",
    "\n",
    "from py2neo import authenticate, Graph, Node, Relationship\n",
    "\n",
    "\n",
    "import os\n",
    "import csv\n",
    "import pickle\n",
    "\n",
    "from time import sleep\n",
    "from timeit import default_timer as timer\n",
    "from datetime import datetime\n",
    "\n",
    "from IPython.display import display, HTML\n",
    "\n",
    "# custom general helper functions for this project\n",
    "import custom_utils as cu\n",
    "import importlib\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from collections import defaultdict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Note: not sure why, but on running this gensim import the kernel kept dying.\n",
    "# Running the following in command line fixed it:\n",
    "# conda install -f numpy\n",
    "import gensim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from gensim.utils import simple_preprocess\n",
    "from gensim.parsing.preprocessing import STOPWORDS\n",
    "from nltk.stem import WordNetLemmatizer, SnowballStemmer\n",
    "from nltk.stem.porter import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package wordnet to /home/arinai/nltk_data...\n",
      "[nltk_data]   Package wordnet is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "nltk.download('wordnet')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# reload imports as needed\n",
    "importlib.reload(cu);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# set up Pandas options\n",
    "pd.set_option('display.max_columns', 25)\n",
    "pd.set_option('display.max_rows', 100)\n",
    "pd.set_option('display.precision', 3)\n",
    "pd.options.display.float_format = '{:.2f}'.format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pd.options.display.max_colwidth = 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Connect to neo4j"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n4j_cred = cu.read_n4jpass()\n",
    "\n",
    "# set up authentication parameters\n",
    "authenticate(\"localhost:7474\", n4j_cred[\"user\"], n4j_cred[\"password\"])\n",
    "\n",
    "# connect to authenticated graph database\n",
    "graph = Graph(\"http://localhost:7474/db/data/\")\n",
    "\n",
    "# test query\n",
    "r = graph.data('CALL db.indexes;')\n",
    "pd.DataFrame(r)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Pull the input data from neo4j"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get a listing of all Louvain communities and their stats\n",
    "start_time = timer()\n",
    "q = \"\"\"\n",
    "    MATCH (a:Article)\n",
    "    WHERE exists(a.louvain_community)\n",
    "    RETURN \n",
    "        a.title as title,\n",
    "        a.louvain_community as louvain_community, \n",
    "        a.external_search_traffic as external_search_traffic,\n",
    "        a.link_in_traffic as link_in_traffic,\n",
    "        a.search_in_traffic as search_in_traffic\n",
    "    ORDER BY external_search_traffic IS NOT NULL desc, external_search_traffic desc;\n",
    "    \"\"\"\n",
    "r = graph.data(q)\n",
    "louvain_communities_for_NLP = pd.DataFrame(r)\n",
    "\n",
    "print(\"Number of articles:\",len(louvain_communities_for_NLP))\n",
    "\n",
    "cu.printRunTime(start_time)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Output from the above:  \n",
    "Number of articles: 2729767  \n",
    " Runtime: 3.84 min"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pickle the output\n",
    "myoutfile = \"pickles/en_1218_louvain_communities_for_NLP.pkl\"\n",
    "with open(myoutfile, 'wb') as picklefile:\n",
    "    pickle.dump(louvain_communities_for_NLP, picklefile)\n",
    "\n",
    "print(\"Pickle created: \" + myoutfile)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Output from the above:  \n",
    "\n",
    "Pickle created: pickles/en_1218_louvain_communities_for_NLP.pkl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>external_search_traffic</th>\n",
       "      <th>link_in_traffic</th>\n",
       "      <th>louvain_community</th>\n",
       "      <th>search_in_traffic</th>\n",
       "      <th>title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4576854.00</td>\n",
       "      <td>1108189.00</td>\n",
       "      <td>3</td>\n",
       "      <td>5630.00</td>\n",
       "      <td>George_H._W._Bush</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3538068.00</td>\n",
       "      <td>639353.00</td>\n",
       "      <td>4</td>\n",
       "      <td>6451.00</td>\n",
       "      <td>Jason_Momoa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3475113.00</td>\n",
       "      <td>223635.00</td>\n",
       "      <td>9</td>\n",
       "      <td>23563.00</td>\n",
       "      <td>2.0_(film)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3251996.00</td>\n",
       "      <td>682992.00</td>\n",
       "      <td>4</td>\n",
       "      <td>10416.00</td>\n",
       "      <td>Bird_Box_(film)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3020671.00</td>\n",
       "      <td>31170.00</td>\n",
       "      <td>1</td>\n",
       "      <td>nan</td>\n",
       "      <td>Main_Page</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2634665.00</td>\n",
       "      <td>408421.00</td>\n",
       "      <td>4</td>\n",
       "      <td>34309.00</td>\n",
       "      <td>Aquaman_(film)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2328884.00</td>\n",
       "      <td>200893.00</td>\n",
       "      <td>4</td>\n",
       "      <td>192.00</td>\n",
       "      <td>Bird_Box</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2231176.00</td>\n",
       "      <td>575481.00</td>\n",
       "      <td>3</td>\n",
       "      <td>3945.00</td>\n",
       "      <td>Priyanka_Chopra</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2226602.00</td>\n",
       "      <td>117115.00</td>\n",
       "      <td>5</td>\n",
       "      <td>958.00</td>\n",
       "      <td>List_of_most-disliked_YouTube_videos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2050628.00</td>\n",
       "      <td>336621.00</td>\n",
       "      <td>5</td>\n",
       "      <td>4161.00</td>\n",
       "      <td>Freddie_Mercury</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>1923035.00</td>\n",
       "      <td>109379.00</td>\n",
       "      <td>3</td>\n",
       "      <td>12636.00</td>\n",
       "      <td>Zero_(2018_film)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>1674952.00</td>\n",
       "      <td>907853.00</td>\n",
       "      <td>3</td>\n",
       "      <td>1273.00</td>\n",
       "      <td>George_W._Bush</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>1645911.00</td>\n",
       "      <td>137326.00</td>\n",
       "      <td>9</td>\n",
       "      <td>6821.00</td>\n",
       "      <td>K.G.F:_Chapter_1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>1568511.00</td>\n",
       "      <td>776311.00</td>\n",
       "      <td>5</td>\n",
       "      <td>1457.00</td>\n",
       "      <td>Nick_Jonas</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>1553836.00</td>\n",
       "      <td>260062.00</td>\n",
       "      <td>4</td>\n",
       "      <td>26665.00</td>\n",
       "      <td>Spider-Man:_Into_the_Spider-Verse</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>1457685.00</td>\n",
       "      <td>98173.00</td>\n",
       "      <td>4</td>\n",
       "      <td>3488.00</td>\n",
       "      <td>Penny_Marshall</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>1370107.00</td>\n",
       "      <td>442623.00</td>\n",
       "      <td>3</td>\n",
       "      <td>3236.00</td>\n",
       "      <td>Jimmy_Carter</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>1237938.00</td>\n",
       "      <td>277013.00</td>\n",
       "      <td>4</td>\n",
       "      <td>3417.00</td>\n",
       "      <td>Macaulay_Culkin</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>1198082.00</td>\n",
       "      <td>381422.00</td>\n",
       "      <td>3</td>\n",
       "      <td>941.00</td>\n",
       "      <td>List_of_highest-grossing_Indian_films</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>1175878.00</td>\n",
       "      <td>186277.00</td>\n",
       "      <td>3</td>\n",
       "      <td>2082.00</td>\n",
       "      <td>Mary,_Queen_of_Scots</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    external_search_traffic  link_in_traffic  louvain_community  \\\n",
       "0                4576854.00       1108189.00                  3   \n",
       "1                3538068.00        639353.00                  4   \n",
       "2                3475113.00        223635.00                  9   \n",
       "3                3251996.00        682992.00                  4   \n",
       "4                3020671.00         31170.00                  1   \n",
       "5                2634665.00        408421.00                  4   \n",
       "6                2328884.00        200893.00                  4   \n",
       "7                2231176.00        575481.00                  3   \n",
       "8                2226602.00        117115.00                  5   \n",
       "9                2050628.00        336621.00                  5   \n",
       "10               1923035.00        109379.00                  3   \n",
       "11               1674952.00        907853.00                  3   \n",
       "12               1645911.00        137326.00                  9   \n",
       "13               1568511.00        776311.00                  5   \n",
       "14               1553836.00        260062.00                  4   \n",
       "15               1457685.00         98173.00                  4   \n",
       "16               1370107.00        442623.00                  3   \n",
       "17               1237938.00        277013.00                  4   \n",
       "18               1198082.00        381422.00                  3   \n",
       "19               1175878.00        186277.00                  3   \n",
       "\n",
       "    search_in_traffic                                  title  \n",
       "0             5630.00                      George_H._W._Bush  \n",
       "1             6451.00                            Jason_Momoa  \n",
       "2            23563.00                             2.0_(film)  \n",
       "3            10416.00                        Bird_Box_(film)  \n",
       "4                 nan                              Main_Page  \n",
       "5            34309.00                         Aquaman_(film)  \n",
       "6              192.00                               Bird_Box  \n",
       "7             3945.00                        Priyanka_Chopra  \n",
       "8              958.00   List_of_most-disliked_YouTube_videos  \n",
       "9             4161.00                        Freddie_Mercury  \n",
       "10           12636.00                       Zero_(2018_film)  \n",
       "11            1273.00                         George_W._Bush  \n",
       "12            6821.00                       K.G.F:_Chapter_1  \n",
       "13            1457.00                             Nick_Jonas  \n",
       "14           26665.00      Spider-Man:_Into_the_Spider-Verse  \n",
       "15            3488.00                         Penny_Marshall  \n",
       "16            3236.00                           Jimmy_Carter  \n",
       "17            3417.00                        Macaulay_Culkin  \n",
       "18             941.00  List_of_highest-grossing_Indian_films  \n",
       "19            2082.00                   Mary,_Queen_of_Scots  "
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# unpickle\n",
    "with open(\"pickles/en_1218_louvain_communities_for_NLP.pkl\", 'rb') as picklefile: \n",
    "    louvain_communities_for_NLP = pickle.load(picklefile)\n",
    "\n",
    "louvain_communities_for_NLP.head(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2729767"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(louvain_communities_for_NLP)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.1 Pre-process article titles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>external_search_traffic</th>\n",
       "      <th>link_in_traffic</th>\n",
       "      <th>louvain_community</th>\n",
       "      <th>search_in_traffic</th>\n",
       "      <th>title_raw</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4576854.00</td>\n",
       "      <td>1108189.00</td>\n",
       "      <td>3</td>\n",
       "      <td>5630.00</td>\n",
       "      <td>George_H._W._Bush</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3538068.00</td>\n",
       "      <td>639353.00</td>\n",
       "      <td>4</td>\n",
       "      <td>6451.00</td>\n",
       "      <td>Jason_Momoa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3475113.00</td>\n",
       "      <td>223635.00</td>\n",
       "      <td>9</td>\n",
       "      <td>23563.00</td>\n",
       "      <td>2.0_(film)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3251996.00</td>\n",
       "      <td>682992.00</td>\n",
       "      <td>4</td>\n",
       "      <td>10416.00</td>\n",
       "      <td>Bird_Box_(film)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3020671.00</td>\n",
       "      <td>31170.00</td>\n",
       "      <td>1</td>\n",
       "      <td>nan</td>\n",
       "      <td>Main_Page</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   external_search_traffic  link_in_traffic  louvain_community  \\\n",
       "0               4576854.00       1108189.00                  3   \n",
       "1               3538068.00        639353.00                  4   \n",
       "2               3475113.00        223635.00                  9   \n",
       "3               3251996.00        682992.00                  4   \n",
       "4               3020671.00         31170.00                  1   \n",
       "\n",
       "   search_in_traffic          title_raw  \n",
       "0            5630.00  George_H._W._Bush  \n",
       "1            6451.00        Jason_Momoa  \n",
       "2           23563.00         2.0_(film)  \n",
       "3           10416.00    Bird_Box_(film)  \n",
       "4                nan          Main_Page  "
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "louvain_communities_for_NLP_proc =louvain_communities_for_NLP.copy()\n",
    "louvain_communities_for_NLP_proc.rename(index=str,columns={'title':'title_raw'}, inplace=True)\n",
    "louvain_communities_for_NLP_proc.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "# the importance of words will be weighted by search and link traffic\n",
    "louvain_communities_for_NLP_proc[\"weight\"] = louvain_communities_for_NLP_proc[[\n",
    "        \"external_search_traffic\", \"link_in_traffic\", \"search_in_traffic\"]].sum(axis=1).astype('int64').fillna(0)\n",
    "\n",
    "louvain_communities_for_NLP_proc.drop([\"external_search_traffic\", \"link_in_traffic\", \"search_in_traffic\"], \n",
    "                                      axis = 1,\n",
    "                                      inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>louvain_community</th>\n",
       "      <th>title_raw</th>\n",
       "      <th>weight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3</td>\n",
       "      <td>George_H._W._Bush</td>\n",
       "      <td>5690673</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4</td>\n",
       "      <td>Jason_Momoa</td>\n",
       "      <td>4183872</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9</td>\n",
       "      <td>2.0_(film)</td>\n",
       "      <td>3722311</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Bird_Box_(film)</td>\n",
       "      <td>3945404</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>Main_Page</td>\n",
       "      <td>3051841</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   louvain_community          title_raw   weight\n",
       "0                  3  George_H._W._Bush  5690673\n",
       "1                  4        Jason_Momoa  4183872\n",
       "2                  9         2.0_(film)  3722311\n",
       "3                  4    Bird_Box_(film)  3945404\n",
       "4                  1          Main_Page  3051841"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "louvain_communities_for_NLP_proc.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>louvain_community</th>\n",
       "      <th>title_raw</th>\n",
       "      <th>weight</th>\n",
       "      <th>title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3</td>\n",
       "      <td>George_H._W._Bush</td>\n",
       "      <td>5690673</td>\n",
       "      <td>George H. W. Bush</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4</td>\n",
       "      <td>Jason_Momoa</td>\n",
       "      <td>4183872</td>\n",
       "      <td>Jason Momoa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9</td>\n",
       "      <td>2.0_(film)</td>\n",
       "      <td>3722311</td>\n",
       "      <td>2.0 (film)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Bird_Box_(film)</td>\n",
       "      <td>3945404</td>\n",
       "      <td>Bird Box (film)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>Main_Page</td>\n",
       "      <td>3051841</td>\n",
       "      <td>Main Page</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   louvain_community          title_raw   weight              title\n",
       "0                  3  George_H._W._Bush  5690673  George H. W. Bush\n",
       "1                  4        Jason_Momoa  4183872        Jason Momoa\n",
       "2                  9         2.0_(film)  3722311         2.0 (film)\n",
       "3                  4    Bird_Box_(film)  3945404    Bird Box (film)\n",
       "4                  1          Main_Page  3051841          Main Page"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# clean up the title\n",
    "louvain_communities_for_NLP_proc[\"title\"] = \\\n",
    "    louvain_communities_for_NLP_proc.title_raw.str.replace('_', ' ')\n",
    "    \n",
    "louvain_communities_for_NLP_proc.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>louvain_community</th>\n",
       "      <th>weight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>2729767.00</td>\n",
       "      <td>2729767.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>11.48</td>\n",
       "      <td>1731.93</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>64.77</td>\n",
       "      <td>13116.59</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>3.00</td>\n",
       "      <td>66.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>5.00</td>\n",
       "      <td>200.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>10.00</td>\n",
       "      <td>735.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1697.00</td>\n",
       "      <td>5690673.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       louvain_community     weight\n",
       "count         2729767.00 2729767.00\n",
       "mean               11.48    1731.93\n",
       "std                64.77   13116.59\n",
       "min                 0.00       0.00\n",
       "25%                 3.00      66.00\n",
       "50%                 5.00     200.00\n",
       "75%                10.00     735.00\n",
       "max              1697.00 5690673.00"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "louvain_communities_for_NLP_proc.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.2 Named entity recognition  \n",
    "\n",
    "We can use the [spaCy](https://spacy.io/) NLP library to identify named entities in article titles.  \n",
    "SpaCy's installation instructions can be found [here](https://spacy.io/usage/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Editing the pipeline to cut down on processing time (dataset too large to use default pipeline)\n",
    "nlp = spacy.load('en', disable=['parser', 'tagger'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('ner', <spacy.pipeline.EntityRecognizer at 0x7ff34717b780>)]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nlp.pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Started running at 2019-03-01 19:57:26.153032 UTC\n",
      "Total number of rows to process: 2729767 \n",
      "\n",
      "Rows processed: 0.0 %, count =  0\n",
      "Elapsed time: 0.0001 min\n",
      "\n",
      "Last row's processing vars:\n",
      "i= 0 txt= George H. W. Bush ents_arr= ['PERSON'] \n",
      " updated data row:\n",
      " louvain_community                    3\n",
      "title_raw            George_H._W._Bush\n",
      "weight                         5690673\n",
      "title                George H. W. Bush\n",
      "named_entities                  PERSON\n",
      "Name: 0, dtype: object \n",
      "\n",
      "Rows processed: 3.6633 %, count =  100000\n",
      "Elapsed time: 6.8751 min\n",
      "\n",
      "Last row's processing vars:\n",
      "i= 100000 txt= List of cities and boroughs in Pennsylvania by population ents_arr= ['GPE'] \n",
      " updated data row:\n",
      " louvain_community                                                           10\n",
      "title_raw            List_of_cities_and_boroughs_in_Pennsylvania_by_population\n",
      "weight                                                                    5628\n",
      "title                List of cities and boroughs in Pennsylvania by population\n",
      "named_entities                                                             GPE\n",
      "Name: 100000, dtype: object \n",
      "\n",
      "Rows processed: 7.3266 %, count =  200000\n",
      "Elapsed time: 13.783 min\n",
      "\n",
      "Last row's processing vars:\n",
      "i= 200000 txt= Frank Vogel ents_arr= ['PERSON'] \n",
      " updated data row:\n",
      " louvain_community             11\n",
      "title_raw            Frank_Vogel\n",
      "weight                      3341\n",
      "title                Frank Vogel\n",
      "named_entities            PERSON\n",
      "Name: 200000, dtype: object \n",
      "\n",
      "Rows processed: 10.9899 %, count =  300000\n",
      "Elapsed time: 20.7096 min\n",
      "\n",
      "Last row's processing vars:\n",
      "i= 300000 txt= Steven Berghuis ents_arr= ['PERSON'] \n",
      " updated data row:\n",
      " louvain_community                  3\n",
      "title_raw            Steven_Berghuis\n",
      "weight                          2758\n",
      "title                Steven Berghuis\n",
      "named_entities                PERSON\n",
      "Name: 300000, dtype: object \n",
      "\n",
      "Rows processed: 14.6533 %, count =  400000\n",
      "Elapsed time: 27.6406 min\n",
      "\n",
      "Last row's processing vars:\n",
      "i= 400000 txt= Jean Rouch ents_arr= ['PERSON'] \n",
      " updated data row:\n",
      " louvain_community             7\n",
      "title_raw            Jean_Rouch\n",
      "weight                     1047\n",
      "title                Jean Rouch\n",
      "named_entities           PERSON\n",
      "Name: 400000, dtype: object \n",
      "\n",
      "Rows processed: 18.3166 %, count =  500000\n",
      "Elapsed time: 34.5862 min\n",
      "\n",
      "Last row's processing vars:\n",
      "i= 500000 txt= Chozhia Vellalar ents_arr= ['PERSON'] \n",
      " updated data row:\n",
      " louvain_community                   9\n",
      "title_raw            Chozhia_Vellalar\n",
      "weight                           1115\n",
      "title                Chozhia Vellalar\n",
      "named_entities                 PERSON\n",
      "Name: 500000, dtype: object \n",
      "\n",
      "Rows processed: 21.9799 %, count =  600000\n",
      "Elapsed time: 41.5412 min\n",
      "\n",
      "Last row's processing vars:\n",
      "i= 600000 txt= Greatest (Duran Duran album) ents_arr= [] \n",
      " updated data row:\n",
      " louvain_community                               5\n",
      "title_raw            Greatest_(Duran_Duran_album)\n",
      "weight                                       1567\n",
      "title                Greatest (Duran Duran album)\n",
      "named_entities                                   \n",
      "Name: 600000, dtype: object \n",
      "\n",
      "Rows processed: 25.6432 %, count =  700000\n",
      "Elapsed time: 48.5261 min\n",
      "\n",
      "Last row's processing vars:\n",
      "i= 700000 txt= Russian Prussia ents_arr= ['NORP'] \n",
      " updated data row:\n",
      " louvain_community                  3\n",
      "title_raw            Russian_Prussia\n",
      "weight                           657\n",
      "title                Russian Prussia\n",
      "named_entities                  NORP\n",
      "Name: 700000, dtype: object \n",
      "\n",
      "Rows processed: 29.3065 %, count =  800000\n",
      "Elapsed time: 55.4965 min\n",
      "\n",
      "Last row's processing vars:\n",
      "i= 800000 txt= Bus Stop (band) ents_arr= [] \n",
      " updated data row:\n",
      " louvain_community                  5\n",
      "title_raw            Bus_Stop_(band)\n",
      "weight                           457\n",
      "title                Bus Stop (band)\n",
      "named_entities                      \n",
      "Name: 800000, dtype: object \n",
      "\n",
      "Rows processed: 32.9698 %, count =  900000\n",
      "Elapsed time: 62.4856 min\n",
      "\n",
      "Last row's processing vars:\n",
      "i= 900000 txt= Fijian Americans ents_arr= ['NORP'] \n",
      " updated data row:\n",
      " louvain_community                  15\n",
      "title_raw            Fijian_Americans\n",
      "weight                            327\n",
      "title                Fijian Americans\n",
      "named_entities                   NORP\n",
      "Name: 900000, dtype: object \n",
      "\n",
      "Rows processed: 36.6332 %, count =  1000000\n",
      "Elapsed time: 69.4687 min\n",
      "\n",
      "Last row's processing vars:\n",
      "i= 1000000 txt= Federico Ricca ents_arr= ['PERSON'] \n",
      " updated data row:\n",
      " louvain_community                 3\n",
      "title_raw            Federico_Ricca\n",
      "weight                          350\n",
      "title                Federico Ricca\n",
      "named_entities               PERSON\n",
      "Name: 1000000, dtype: object \n",
      "\n",
      "Rows processed: 40.2965 %, count =  1100000\n",
      "Elapsed time: 76.4521 min\n",
      "\n",
      "Last row's processing vars:\n",
      "i= 1100000 txt= Willie Tuitama ents_arr= ['GPE'] \n",
      " updated data row:\n",
      " louvain_community                11\n",
      "title_raw            Willie_Tuitama\n",
      "weight                          677\n",
      "title                Willie Tuitama\n",
      "named_entities                  GPE\n",
      "Name: 1100000, dtype: object \n",
      "\n",
      "Rows processed: 43.9598 %, count =  1200000\n",
      "Elapsed time: 83.458 min\n",
      "\n",
      "Last row's processing vars:\n",
      "i= 1200000 txt= Jazz (perfume) ents_arr= [] \n",
      " updated data row:\n",
      " louvain_community                24\n",
      "title_raw            Jazz_(perfume)\n",
      "weight                          127\n",
      "title                Jazz (perfume)\n",
      "named_entities                     \n",
      "Name: 1200000, dtype: object \n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"Started running at\", datetime.now(), \"UTC\")\n",
    "\n",
    "start_time = timer()\n",
    "\n",
    "nrows = len(louvain_communities_for_NLP_proc)\n",
    "print(\"Total number of rows to process:\", nrows, \"\\n\")\n",
    "\n",
    "for i in range(nrows):\n",
    "    txt = louvain_communities_for_NLP_proc.iloc[i].title\n",
    "\n",
    "    doc = nlp(txt)\n",
    "\n",
    "    ents_arr = []\n",
    "    for ent in doc.ents:\n",
    "        ents_arr.append(ent.label_)\n",
    "    \n",
    "    louvain_communities_for_NLP_proc.at[str(i), \"named_entities\"] = \" \".join(ents_arr)\n",
    "    \n",
    "    if (i % 100000 == 0):\n",
    "        print(\"Rows processed:\", round(i * 100/len(louvain_communities_for_NLP_proc), 4), \"%,\", \"count = \", i )\n",
    "        print(\"Elapsed time:\", round((timer() - start_time)/60, 4), \"min\\n\")\n",
    "        print(\"Last row's processing vars:\")\n",
    "        print(\"i=\", i, \"txt=\", txt, \"ents_arr=\", ents_arr, \"\\n\",\n",
    "              \"updated data row:\\n\", louvain_communities_for_NLP_proc.iloc[i], \"\\n\")\n",
    "    \n",
    "\n",
    "cu.printRunTime(start_time)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code above took a few hours to run, and finished successfully, but the browser tunnel got interrupted at some point, so the print statements above are incomplete, but the dataset has been successfully populated with `named_entities`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Double-checking that all of the data got NER-processed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2729767"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nrows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2729766"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "i"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Christgau (disambiguation)'"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "louvain_community                             5\n",
       "title_raw            Christgau_(disambiguation)\n",
       "weight                                       18\n",
       "title                Christgau (disambiguation)\n",
       "named_entities                              GPE\n",
       "Name: 2729766, dtype: object"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "louvain_communities_for_NLP_proc.iloc[i]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>louvain_community</th>\n",
       "      <th>title_raw</th>\n",
       "      <th>weight</th>\n",
       "      <th>title</th>\n",
       "      <th>named_entities</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3</td>\n",
       "      <td>George_H._W._Bush</td>\n",
       "      <td>5690673</td>\n",
       "      <td>George H. W. Bush</td>\n",
       "      <td>PERSON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4</td>\n",
       "      <td>Jason_Momoa</td>\n",
       "      <td>4183872</td>\n",
       "      <td>Jason Momoa</td>\n",
       "      <td>PERSON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9</td>\n",
       "      <td>2.0_(film)</td>\n",
       "      <td>3722311</td>\n",
       "      <td>2.0 (film)</td>\n",
       "      <td>CARDINAL</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Bird_Box_(film)</td>\n",
       "      <td>3945404</td>\n",
       "      <td>Bird Box (film)</td>\n",
       "      <td>PERSON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>Main_Page</td>\n",
       "      <td>3051841</td>\n",
       "      <td>Main Page</td>\n",
       "      <td>PERSON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>4</td>\n",
       "      <td>Aquaman_(film)</td>\n",
       "      <td>3077395</td>\n",
       "      <td>Aquaman (film)</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>4</td>\n",
       "      <td>Bird_Box</td>\n",
       "      <td>2529969</td>\n",
       "      <td>Bird Box</td>\n",
       "      <td>PERSON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>3</td>\n",
       "      <td>Priyanka_Chopra</td>\n",
       "      <td>2810602</td>\n",
       "      <td>Priyanka Chopra</td>\n",
       "      <td>PERSON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>5</td>\n",
       "      <td>List_of_most-disliked_YouTube_videos</td>\n",
       "      <td>2344675</td>\n",
       "      <td>List of most-disliked YouTube videos</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>5</td>\n",
       "      <td>Freddie_Mercury</td>\n",
       "      <td>2391410</td>\n",
       "      <td>Freddie Mercury</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>3</td>\n",
       "      <td>Zero_(2018_film)</td>\n",
       "      <td>2045050</td>\n",
       "      <td>Zero (2018 film)</td>\n",
       "      <td>CARDINAL DATE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>3</td>\n",
       "      <td>George_W._Bush</td>\n",
       "      <td>2584078</td>\n",
       "      <td>George W. Bush</td>\n",
       "      <td>PERSON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>9</td>\n",
       "      <td>K.G.F:_Chapter_1</td>\n",
       "      <td>1790058</td>\n",
       "      <td>K.G.F: Chapter 1</td>\n",
       "      <td>LAW</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>5</td>\n",
       "      <td>Nick_Jonas</td>\n",
       "      <td>2346279</td>\n",
       "      <td>Nick Jonas</td>\n",
       "      <td>PERSON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>4</td>\n",
       "      <td>Spider-Man:_Into_the_Spider-Verse</td>\n",
       "      <td>1840563</td>\n",
       "      <td>Spider-Man: Into the Spider-Verse</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>4</td>\n",
       "      <td>Penny_Marshall</td>\n",
       "      <td>1559346</td>\n",
       "      <td>Penny Marshall</td>\n",
       "      <td>PERSON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>3</td>\n",
       "      <td>Jimmy_Carter</td>\n",
       "      <td>1815966</td>\n",
       "      <td>Jimmy Carter</td>\n",
       "      <td>PERSON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>4</td>\n",
       "      <td>Macaulay_Culkin</td>\n",
       "      <td>1518368</td>\n",
       "      <td>Macaulay Culkin</td>\n",
       "      <td>PERSON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>3</td>\n",
       "      <td>List_of_highest-grossing_Indian_films</td>\n",
       "      <td>1580445</td>\n",
       "      <td>List of highest-grossing Indian films</td>\n",
       "      <td>NORP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>3</td>\n",
       "      <td>Mary,_Queen_of_Scots</td>\n",
       "      <td>1364237</td>\n",
       "      <td>Mary, Queen of Scots</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    louvain_community                              title_raw   weight  \\\n",
       "0                   3                      George_H._W._Bush  5690673   \n",
       "1                   4                            Jason_Momoa  4183872   \n",
       "2                   9                             2.0_(film)  3722311   \n",
       "3                   4                        Bird_Box_(film)  3945404   \n",
       "4                   1                              Main_Page  3051841   \n",
       "5                   4                         Aquaman_(film)  3077395   \n",
       "6                   4                               Bird_Box  2529969   \n",
       "7                   3                        Priyanka_Chopra  2810602   \n",
       "8                   5   List_of_most-disliked_YouTube_videos  2344675   \n",
       "9                   5                        Freddie_Mercury  2391410   \n",
       "10                  3                       Zero_(2018_film)  2045050   \n",
       "11                  3                         George_W._Bush  2584078   \n",
       "12                  9                       K.G.F:_Chapter_1  1790058   \n",
       "13                  5                             Nick_Jonas  2346279   \n",
       "14                  4      Spider-Man:_Into_the_Spider-Verse  1840563   \n",
       "15                  4                         Penny_Marshall  1559346   \n",
       "16                  3                           Jimmy_Carter  1815966   \n",
       "17                  4                        Macaulay_Culkin  1518368   \n",
       "18                  3  List_of_highest-grossing_Indian_films  1580445   \n",
       "19                  3                   Mary,_Queen_of_Scots  1364237   \n",
       "\n",
       "                                    title named_entities  \n",
       "0                       George H. W. Bush         PERSON  \n",
       "1                             Jason Momoa         PERSON  \n",
       "2                              2.0 (film)       CARDINAL  \n",
       "3                         Bird Box (film)         PERSON  \n",
       "4                               Main Page         PERSON  \n",
       "5                          Aquaman (film)                 \n",
       "6                                Bird Box         PERSON  \n",
       "7                         Priyanka Chopra         PERSON  \n",
       "8    List of most-disliked YouTube videos            ORG  \n",
       "9                         Freddie Mercury            ORG  \n",
       "10                       Zero (2018 film)  CARDINAL DATE  \n",
       "11                         George W. Bush         PERSON  \n",
       "12                       K.G.F: Chapter 1            LAW  \n",
       "13                             Nick Jonas         PERSON  \n",
       "14      Spider-Man: Into the Spider-Verse                 \n",
       "15                         Penny Marshall         PERSON  \n",
       "16                           Jimmy Carter         PERSON  \n",
       "17                        Macaulay Culkin         PERSON  \n",
       "18  List of highest-grossing Indian films           NORP  \n",
       "19                   Mary, Queen of Scots                 "
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "louvain_communities_for_NLP_proc.head(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>louvain_community</th>\n",
       "      <th>title_raw</th>\n",
       "      <th>weight</th>\n",
       "      <th>title</th>\n",
       "      <th>named_entities</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2729747</th>\n",
       "      <td>16</td>\n",
       "      <td>Mantle_of_Luís_I</td>\n",
       "      <td>15</td>\n",
       "      <td>Mantle of Luís I</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2729748</th>\n",
       "      <td>13</td>\n",
       "      <td>List_of_Zero:_Black_Blood_episodes</td>\n",
       "      <td>14</td>\n",
       "      <td>List of Zero: Black Blood episodes</td>\n",
       "      <td>CARDINAL EVENT</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2729749</th>\n",
       "      <td>22</td>\n",
       "      <td>Moskvitch_404_Sport</td>\n",
       "      <td>15</td>\n",
       "      <td>Moskvitch 404 Sport</td>\n",
       "      <td>CARDINAL</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2729750</th>\n",
       "      <td>3</td>\n",
       "      <td>Caridina_loehae</td>\n",
       "      <td>11</td>\n",
       "      <td>Caridina loehae</td>\n",
       "      <td>GPE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2729751</th>\n",
       "      <td>3</td>\n",
       "      <td>Tatu_Miettunen</td>\n",
       "      <td>11</td>\n",
       "      <td>Tatu Miettunen</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2729752</th>\n",
       "      <td>2</td>\n",
       "      <td>HIST2H3C</td>\n",
       "      <td>10</td>\n",
       "      <td>HIST2H3C</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2729753</th>\n",
       "      <td>6</td>\n",
       "      <td>Muk,_Iran</td>\n",
       "      <td>17</td>\n",
       "      <td>Muk, Iran</td>\n",
       "      <td>GPE GPE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2729754</th>\n",
       "      <td>10</td>\n",
       "      <td>Wila,_Missouri</td>\n",
       "      <td>67</td>\n",
       "      <td>Wila, Missouri</td>\n",
       "      <td>PERSON GPE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2729755</th>\n",
       "      <td>3</td>\n",
       "      <td>Isaac_Rochussen</td>\n",
       "      <td>11</td>\n",
       "      <td>Isaac Rochussen</td>\n",
       "      <td>PERSON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2729756</th>\n",
       "      <td>7</td>\n",
       "      <td>Raffaello_Bertieri</td>\n",
       "      <td>26</td>\n",
       "      <td>Raffaello Bertieri</td>\n",
       "      <td>PERSON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2729757</th>\n",
       "      <td>5</td>\n",
       "      <td>Project_Blues:_Saturday_Suffering_Fools</td>\n",
       "      <td>10</td>\n",
       "      <td>Project Blues: Saturday Suffering Fools</td>\n",
       "      <td>WORK_OF_ART DATE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2729758</th>\n",
       "      <td>12</td>\n",
       "      <td>Swimming_at_the_2015_African_Games_–_Men's_4×100_metre_medley_relay</td>\n",
       "      <td>11</td>\n",
       "      <td>Swimming at the 2015 African Games – Men's 4×100 metre medley relay</td>\n",
       "      <td>DATE EVENT QUANTITY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2729759</th>\n",
       "      <td>5</td>\n",
       "      <td>The_Best_of_Jenny_Morris:_The_Story_So_Far</td>\n",
       "      <td>10</td>\n",
       "      <td>The Best of Jenny Morris: The Story So Far</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2729760</th>\n",
       "      <td>7</td>\n",
       "      <td>Die_Räuberbraut_(opera)</td>\n",
       "      <td>13</td>\n",
       "      <td>Die Räuberbraut (opera)</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2729761</th>\n",
       "      <td>1</td>\n",
       "      <td>Tiefwarensee_Ice_Age_Trail</td>\n",
       "      <td>10</td>\n",
       "      <td>Tiefwarensee Ice Age Trail</td>\n",
       "      <td>DATE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2729762</th>\n",
       "      <td>3</td>\n",
       "      <td>Joel_Rajalakso</td>\n",
       "      <td>13</td>\n",
       "      <td>Joel Rajalakso</td>\n",
       "      <td>PERSON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2729763</th>\n",
       "      <td>8</td>\n",
       "      <td>Rocket_landing_pad</td>\n",
       "      <td>14</td>\n",
       "      <td>Rocket landing pad</td>\n",
       "      <td>GPE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2729764</th>\n",
       "      <td>12</td>\n",
       "      <td>Athletics_at_the_Friendship_Games_–_Women's_100_metres</td>\n",
       "      <td>11</td>\n",
       "      <td>Athletics at the Friendship Games – Women's 100 metres</td>\n",
       "      <td>EVENT</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2729765</th>\n",
       "      <td>8</td>\n",
       "      <td>Bristol_Classic_Boat_Company</td>\n",
       "      <td>10</td>\n",
       "      <td>Bristol Classic Boat Company</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2729766</th>\n",
       "      <td>5</td>\n",
       "      <td>Christgau_(disambiguation)</td>\n",
       "      <td>18</td>\n",
       "      <td>Christgau (disambiguation)</td>\n",
       "      <td>GPE</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         louvain_community  \\\n",
       "2729747                 16   \n",
       "2729748                 13   \n",
       "2729749                 22   \n",
       "2729750                  3   \n",
       "2729751                  3   \n",
       "2729752                  2   \n",
       "2729753                  6   \n",
       "2729754                 10   \n",
       "2729755                  3   \n",
       "2729756                  7   \n",
       "2729757                  5   \n",
       "2729758                 12   \n",
       "2729759                  5   \n",
       "2729760                  7   \n",
       "2729761                  1   \n",
       "2729762                  3   \n",
       "2729763                  8   \n",
       "2729764                 12   \n",
       "2729765                  8   \n",
       "2729766                  5   \n",
       "\n",
       "                                                                   title_raw  \\\n",
       "2729747                                                     Mantle_of_Luís_I   \n",
       "2729748                                   List_of_Zero:_Black_Blood_episodes   \n",
       "2729749                                                  Moskvitch_404_Sport   \n",
       "2729750                                                      Caridina_loehae   \n",
       "2729751                                                       Tatu_Miettunen   \n",
       "2729752                                                             HIST2H3C   \n",
       "2729753                                                            Muk,_Iran   \n",
       "2729754                                                       Wila,_Missouri   \n",
       "2729755                                                      Isaac_Rochussen   \n",
       "2729756                                                   Raffaello_Bertieri   \n",
       "2729757                              Project_Blues:_Saturday_Suffering_Fools   \n",
       "2729758  Swimming_at_the_2015_African_Games_–_Men's_4×100_metre_medley_relay   \n",
       "2729759                           The_Best_of_Jenny_Morris:_The_Story_So_Far   \n",
       "2729760                                              Die_Räuberbraut_(opera)   \n",
       "2729761                                           Tiefwarensee_Ice_Age_Trail   \n",
       "2729762                                                       Joel_Rajalakso   \n",
       "2729763                                                   Rocket_landing_pad   \n",
       "2729764               Athletics_at_the_Friendship_Games_–_Women's_100_metres   \n",
       "2729765                                         Bristol_Classic_Boat_Company   \n",
       "2729766                                           Christgau_(disambiguation)   \n",
       "\n",
       "         weight  \\\n",
       "2729747      15   \n",
       "2729748      14   \n",
       "2729749      15   \n",
       "2729750      11   \n",
       "2729751      11   \n",
       "2729752      10   \n",
       "2729753      17   \n",
       "2729754      67   \n",
       "2729755      11   \n",
       "2729756      26   \n",
       "2729757      10   \n",
       "2729758      11   \n",
       "2729759      10   \n",
       "2729760      13   \n",
       "2729761      10   \n",
       "2729762      13   \n",
       "2729763      14   \n",
       "2729764      11   \n",
       "2729765      10   \n",
       "2729766      18   \n",
       "\n",
       "                                                                       title  \\\n",
       "2729747                                                     Mantle of Luís I   \n",
       "2729748                                   List of Zero: Black Blood episodes   \n",
       "2729749                                                  Moskvitch 404 Sport   \n",
       "2729750                                                      Caridina loehae   \n",
       "2729751                                                       Tatu Miettunen   \n",
       "2729752                                                             HIST2H3C   \n",
       "2729753                                                            Muk, Iran   \n",
       "2729754                                                       Wila, Missouri   \n",
       "2729755                                                      Isaac Rochussen   \n",
       "2729756                                                   Raffaello Bertieri   \n",
       "2729757                              Project Blues: Saturday Suffering Fools   \n",
       "2729758  Swimming at the 2015 African Games – Men's 4×100 metre medley relay   \n",
       "2729759                           The Best of Jenny Morris: The Story So Far   \n",
       "2729760                                              Die Räuberbraut (opera)   \n",
       "2729761                                           Tiefwarensee Ice Age Trail   \n",
       "2729762                                                       Joel Rajalakso   \n",
       "2729763                                                   Rocket landing pad   \n",
       "2729764               Athletics at the Friendship Games – Women's 100 metres   \n",
       "2729765                                         Bristol Classic Boat Company   \n",
       "2729766                                           Christgau (disambiguation)   \n",
       "\n",
       "              named_entities  \n",
       "2729747                       \n",
       "2729748       CARDINAL EVENT  \n",
       "2729749             CARDINAL  \n",
       "2729750                  GPE  \n",
       "2729751                       \n",
       "2729752                       \n",
       "2729753              GPE GPE  \n",
       "2729754           PERSON GPE  \n",
       "2729755               PERSON  \n",
       "2729756               PERSON  \n",
       "2729757     WORK_OF_ART DATE  \n",
       "2729758  DATE EVENT QUANTITY  \n",
       "2729759                       \n",
       "2729760                       \n",
       "2729761                 DATE  \n",
       "2729762               PERSON  \n",
       "2729763                  GPE  \n",
       "2729764                EVENT  \n",
       "2729765                  ORG  \n",
       "2729766                  GPE  "
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "louvain_communities_for_NLP_proc.tail(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The named entities assignment is not perfect, for example, both \"Main Page\" and \"Bird Box\" articles were labeled \"PERSON\", but in many cases it did quite well.  \n",
    "\n",
    "##### Named entity type descriptions  \n",
    "SpaCy's named entity type descriptions can be found [here](https://spacy.io/api/annotation#named-entities).  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The \"Main Page\" article seems to be an important node in the network, so let's fix its named_entities value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "louvain_community            1\n",
       "title_raw            Main_Page\n",
       "weight                 3051841\n",
       "title                Main Page\n",
       "named_entities          PERSON\n",
       "Name: 4, dtype: object"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "louvain_communities_for_NLP_proc.iloc[4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "louvain_communities_for_NLP_proc.at[str(4), \"named_entities\"] = ''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "louvain_community            1\n",
       "title_raw            Main_Page\n",
       "weight                 3051841\n",
       "title                Main Page\n",
       "named_entities                \n",
       "Name: 4, dtype: object"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "louvain_communities_for_NLP_proc.iloc[4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>louvain_community</th>\n",
       "      <th>title_raw</th>\n",
       "      <th>weight</th>\n",
       "      <th>title</th>\n",
       "      <th>named_entities</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3</td>\n",
       "      <td>George_H._W._Bush</td>\n",
       "      <td>5690673</td>\n",
       "      <td>George H. W. Bush</td>\n",
       "      <td>PERSON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4</td>\n",
       "      <td>Jason_Momoa</td>\n",
       "      <td>4183872</td>\n",
       "      <td>Jason Momoa</td>\n",
       "      <td>PERSON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9</td>\n",
       "      <td>2.0_(film)</td>\n",
       "      <td>3722311</td>\n",
       "      <td>2.0 (film)</td>\n",
       "      <td>CARDINAL</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Bird_Box_(film)</td>\n",
       "      <td>3945404</td>\n",
       "      <td>Bird Box (film)</td>\n",
       "      <td>PERSON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>Main_Page</td>\n",
       "      <td>3051841</td>\n",
       "      <td>Main Page</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>4</td>\n",
       "      <td>Aquaman_(film)</td>\n",
       "      <td>3077395</td>\n",
       "      <td>Aquaman (film)</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>4</td>\n",
       "      <td>Bird_Box</td>\n",
       "      <td>2529969</td>\n",
       "      <td>Bird Box</td>\n",
       "      <td>PERSON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>3</td>\n",
       "      <td>Priyanka_Chopra</td>\n",
       "      <td>2810602</td>\n",
       "      <td>Priyanka Chopra</td>\n",
       "      <td>PERSON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>5</td>\n",
       "      <td>List_of_most-disliked_YouTube_videos</td>\n",
       "      <td>2344675</td>\n",
       "      <td>List of most-disliked YouTube videos</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>5</td>\n",
       "      <td>Freddie_Mercury</td>\n",
       "      <td>2391410</td>\n",
       "      <td>Freddie Mercury</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   louvain_community                             title_raw   weight  \\\n",
       "0                  3                     George_H._W._Bush  5690673   \n",
       "1                  4                           Jason_Momoa  4183872   \n",
       "2                  9                            2.0_(film)  3722311   \n",
       "3                  4                       Bird_Box_(film)  3945404   \n",
       "4                  1                             Main_Page  3051841   \n",
       "5                  4                        Aquaman_(film)  3077395   \n",
       "6                  4                              Bird_Box  2529969   \n",
       "7                  3                       Priyanka_Chopra  2810602   \n",
       "8                  5  List_of_most-disliked_YouTube_videos  2344675   \n",
       "9                  5                       Freddie_Mercury  2391410   \n",
       "\n",
       "                                  title named_entities  \n",
       "0                     George H. W. Bush         PERSON  \n",
       "1                           Jason Momoa         PERSON  \n",
       "2                            2.0 (film)       CARDINAL  \n",
       "3                       Bird Box (film)         PERSON  \n",
       "4                             Main Page                 \n",
       "5                        Aquaman (film)                 \n",
       "6                              Bird Box         PERSON  \n",
       "7                       Priyanka Chopra         PERSON  \n",
       "8  List of most-disliked YouTube videos            ORG  \n",
       "9                       Freddie Mercury            ORG  "
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "louvain_communities_for_NLP_proc.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pickle created: pickles/en_1218_louvain_communities_for_NLP_proc.pkl\n"
     ]
    }
   ],
   "source": [
    "# pickle the output\n",
    "myoutfile = \"pickles/en_1218_louvain_communities_for_NLP_proc.pkl\"\n",
    "with open(myoutfile, 'wb') as picklefile:\n",
    "    pickle.dump(louvain_communities_for_NLP_proc, picklefile)\n",
    "\n",
    "print(\"Pickle created: \" + myoutfile)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>louvain_community</th>\n",
       "      <th>title_raw</th>\n",
       "      <th>weight</th>\n",
       "      <th>title</th>\n",
       "      <th>named_entities</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3</td>\n",
       "      <td>George_H._W._Bush</td>\n",
       "      <td>5690673</td>\n",
       "      <td>George H. W. Bush</td>\n",
       "      <td>PERSON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4</td>\n",
       "      <td>Jason_Momoa</td>\n",
       "      <td>4183872</td>\n",
       "      <td>Jason Momoa</td>\n",
       "      <td>PERSON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9</td>\n",
       "      <td>2.0_(film)</td>\n",
       "      <td>3722311</td>\n",
       "      <td>2.0 (film)</td>\n",
       "      <td>CARDINAL</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Bird_Box_(film)</td>\n",
       "      <td>3945404</td>\n",
       "      <td>Bird Box (film)</td>\n",
       "      <td>PERSON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>Main_Page</td>\n",
       "      <td>3051841</td>\n",
       "      <td>Main Page</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>4</td>\n",
       "      <td>Aquaman_(film)</td>\n",
       "      <td>3077395</td>\n",
       "      <td>Aquaman (film)</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>4</td>\n",
       "      <td>Bird_Box</td>\n",
       "      <td>2529969</td>\n",
       "      <td>Bird Box</td>\n",
       "      <td>PERSON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>3</td>\n",
       "      <td>Priyanka_Chopra</td>\n",
       "      <td>2810602</td>\n",
       "      <td>Priyanka Chopra</td>\n",
       "      <td>PERSON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>5</td>\n",
       "      <td>List_of_most-disliked_YouTube_videos</td>\n",
       "      <td>2344675</td>\n",
       "      <td>List of most-disliked YouTube videos</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>5</td>\n",
       "      <td>Freddie_Mercury</td>\n",
       "      <td>2391410</td>\n",
       "      <td>Freddie Mercury</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>3</td>\n",
       "      <td>Zero_(2018_film)</td>\n",
       "      <td>2045050</td>\n",
       "      <td>Zero (2018 film)</td>\n",
       "      <td>CARDINAL DATE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>3</td>\n",
       "      <td>George_W._Bush</td>\n",
       "      <td>2584078</td>\n",
       "      <td>George W. Bush</td>\n",
       "      <td>PERSON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>9</td>\n",
       "      <td>K.G.F:_Chapter_1</td>\n",
       "      <td>1790058</td>\n",
       "      <td>K.G.F: Chapter 1</td>\n",
       "      <td>LAW</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>5</td>\n",
       "      <td>Nick_Jonas</td>\n",
       "      <td>2346279</td>\n",
       "      <td>Nick Jonas</td>\n",
       "      <td>PERSON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>4</td>\n",
       "      <td>Spider-Man:_Into_the_Spider-Verse</td>\n",
       "      <td>1840563</td>\n",
       "      <td>Spider-Man: Into the Spider-Verse</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>4</td>\n",
       "      <td>Penny_Marshall</td>\n",
       "      <td>1559346</td>\n",
       "      <td>Penny Marshall</td>\n",
       "      <td>PERSON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>3</td>\n",
       "      <td>Jimmy_Carter</td>\n",
       "      <td>1815966</td>\n",
       "      <td>Jimmy Carter</td>\n",
       "      <td>PERSON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>4</td>\n",
       "      <td>Macaulay_Culkin</td>\n",
       "      <td>1518368</td>\n",
       "      <td>Macaulay Culkin</td>\n",
       "      <td>PERSON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>3</td>\n",
       "      <td>List_of_highest-grossing_Indian_films</td>\n",
       "      <td>1580445</td>\n",
       "      <td>List of highest-grossing Indian films</td>\n",
       "      <td>NORP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>3</td>\n",
       "      <td>Mary,_Queen_of_Scots</td>\n",
       "      <td>1364237</td>\n",
       "      <td>Mary, Queen of Scots</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    louvain_community                              title_raw   weight  \\\n",
       "0                   3                      George_H._W._Bush  5690673   \n",
       "1                   4                            Jason_Momoa  4183872   \n",
       "2                   9                             2.0_(film)  3722311   \n",
       "3                   4                        Bird_Box_(film)  3945404   \n",
       "4                   1                              Main_Page  3051841   \n",
       "5                   4                         Aquaman_(film)  3077395   \n",
       "6                   4                               Bird_Box  2529969   \n",
       "7                   3                        Priyanka_Chopra  2810602   \n",
       "8                   5   List_of_most-disliked_YouTube_videos  2344675   \n",
       "9                   5                        Freddie_Mercury  2391410   \n",
       "10                  3                       Zero_(2018_film)  2045050   \n",
       "11                  3                         George_W._Bush  2584078   \n",
       "12                  9                       K.G.F:_Chapter_1  1790058   \n",
       "13                  5                             Nick_Jonas  2346279   \n",
       "14                  4      Spider-Man:_Into_the_Spider-Verse  1840563   \n",
       "15                  4                         Penny_Marshall  1559346   \n",
       "16                  3                           Jimmy_Carter  1815966   \n",
       "17                  4                        Macaulay_Culkin  1518368   \n",
       "18                  3  List_of_highest-grossing_Indian_films  1580445   \n",
       "19                  3                   Mary,_Queen_of_Scots  1364237   \n",
       "\n",
       "                                    title named_entities  \n",
       "0                       George H. W. Bush         PERSON  \n",
       "1                             Jason Momoa         PERSON  \n",
       "2                              2.0 (film)       CARDINAL  \n",
       "3                         Bird Box (film)         PERSON  \n",
       "4                               Main Page                 \n",
       "5                          Aquaman (film)                 \n",
       "6                                Bird Box         PERSON  \n",
       "7                         Priyanka Chopra         PERSON  \n",
       "8    List of most-disliked YouTube videos            ORG  \n",
       "9                         Freddie Mercury            ORG  \n",
       "10                       Zero (2018 film)  CARDINAL DATE  \n",
       "11                         George W. Bush         PERSON  \n",
       "12                       K.G.F: Chapter 1            LAW  \n",
       "13                             Nick Jonas         PERSON  \n",
       "14      Spider-Man: Into the Spider-Verse                 \n",
       "15                         Penny Marshall         PERSON  \n",
       "16                           Jimmy Carter         PERSON  \n",
       "17                        Macaulay Culkin         PERSON  \n",
       "18  List of highest-grossing Indian films           NORP  \n",
       "19                   Mary, Queen of Scots                 "
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# unpickle\n",
    "with open(\"pickles/en_1218_louvain_communities_for_NLP_proc.pkl\", 'rb') as picklefile: \n",
    "    louvain_communities_for_NLP_proc = pickle.load(picklefile)\n",
    "\n",
    "louvain_communities_for_NLP_proc.head(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PERSON                                830645\n",
       "                                      814109\n",
       "ORG                                   390291\n",
       "GPE                                   170318\n",
       "DATE                                   46268\n",
       "NORP                                   38910\n",
       "GPE GPE                                38162\n",
       "CARDINAL                               29662\n",
       "ORG GPE                                28779\n",
       "PERSON GPE                             21082\n",
       "LOC                                    16358\n",
       "DATE ORG                               15102\n",
       "FAC                                    14702\n",
       "PERSON PERSON                          14446\n",
       "DATE EVENT                             14300\n",
       "PERSON DATE                            14223\n",
       "PERSON ORG                             11964\n",
       "ORG ORG                                11261\n",
       "EVENT                                  10618\n",
       "DATE GPE                               10552\n",
       "ORG DATE                                9295\n",
       "WORK_OF_ART                             8606\n",
       "ORG PERSON                              7630\n",
       "PERSON NORP                             7156\n",
       "GPE ORG                                 6698\n",
       "DATE NORP                               6308\n",
       "GPE DATE                                5328\n",
       "ORG CARDINAL                            5208\n",
       "NORP GPE                                5200\n",
       "PRODUCT                                 4824\n",
       "GPE PERSON                              4740\n",
       "CARDINAL ORG                            4632\n",
       "ORDINAL                                 4160\n",
       "DATE PERSON                             4122\n",
       "ORG GPE GPE                             3111\n",
       "ORDINAL ORG                             2688\n",
       "NORP DATE                               2473\n",
       "NORP ORG                                2403\n",
       "LOC GPE                                 2289\n",
       "CARDINAL GPE                            2275\n",
       "NORP PERSON                             2182\n",
       "CARDINAL PERSON                         2152\n",
       "DATE EVENT QUANTITY                     2142\n",
       "GPE CARDINAL                            2085\n",
       "PERSON CARDINAL                         2004\n",
       "GPE GPE GPE                             1850\n",
       "ORG NORP                                1699\n",
       "FAC GPE                                 1491\n",
       "DATE GPE GPE                            1353\n",
       "GPE DATE EVENT                          1284\n",
       "                                       ...  \n",
       "ORG ORG DATE FAC                           1\n",
       "GPE ORG CARDINAL PERSON                    1\n",
       "NORP NORP DATE EVENT                       1\n",
       "TIME GPE PERSON                            1\n",
       "CARDINAL NORP TIME                         1\n",
       "CARDINAL ORG GPE CARDINAL                  1\n",
       "DATE GPE ORG EVENT                         1\n",
       "LOC PERSON EVENT                           1\n",
       "LAW PERSON ORG                             1\n",
       "PERSON GPE LOC NORP                        1\n",
       "NORP GPE LOC DATE                          1\n",
       "ORDINAL LAW GPE                            1\n",
       "PERSON PERSON ORDINAL WORK_OF_ART          1\n",
       "NORP PRODUCT NORP                          1\n",
       "GPE GPE ORG CARDINAL                       1\n",
       "CARDINAL PERSON GPE PERSON                 1\n",
       "PERSON PERSON DATE PERSON                  1\n",
       "PERCENT GPE GPE                            1\n",
       "PERSON GPE QUANTITY                        1\n",
       "CARDINAL NORP ORG CARDINAL PERSON          1\n",
       "PERSON PERSON DATE DATE                    1\n",
       "PRODUCT LOC PERSON                         1\n",
       "NORP GPE GPE GPE PERSON                    1\n",
       "MONEY PERSON ORG                           1\n",
       "GPE ORG CARDINAL ORG                       1\n",
       "PRODUCT ORG WORK_OF_ART                    1\n",
       "GPE CARDINAL CARDINAL ORG                  1\n",
       "WORK_OF_ART CARDINAL NORP                  1\n",
       "NORP PERSON CARDINAL PERSON                1\n",
       "CARDINAL NORP PERSON ORG PRODUCT           1\n",
       "CARDINAL ORDINAL DATE                      1\n",
       "PERSON PERSON PERSON GPE GPE               1\n",
       "DATE EVENT ORG DATE                        1\n",
       "ORG QUANTITY GPE                           1\n",
       "NORP CARDINAL NORP                         1\n",
       "FAC ORG WORK_OF_ART                        1\n",
       "PERSON PERSON GPE CARDINAL                 1\n",
       "TIME TIME PERSON                           1\n",
       "LOC LOC ORG NORP                           1\n",
       "GPE TIME ORG                               1\n",
       "NORP WORK_OF_ART CARDINAL                  1\n",
       "ORG NORP NORP GPE EVENT                    1\n",
       "GPE LOC PRODUCT                            1\n",
       "PERSON PERSON QUANTITY                     1\n",
       "ORG ORG LOC NORP                           1\n",
       "CARDINAL DATE PERSON PERSON PERSON         1\n",
       "LANGUAGE PERSON PERSON                     1\n",
       "GPE QUANTITY PERSON                        1\n",
       "PERSON PRODUCT WORK_OF_ART                 1\n",
       "ORG GPE ORDINAL FAC                        1\n",
       "Name: named_entities, dtype: int64"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "louvain_communities_for_NLP_proc.named_entities.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>louvain_community</th>\n",
       "      <th>title_raw</th>\n",
       "      <th>weight</th>\n",
       "      <th>title</th>\n",
       "      <th>named_entities</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2175</th>\n",
       "      <td>14</td>\n",
       "      <td>List_of_NHL_statistical_leaders</td>\n",
       "      <td>102869</td>\n",
       "      <td>List of NHL statistical leaders</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3019</th>\n",
       "      <td>14</td>\n",
       "      <td>2019_World_Junior_Ice_Hockey_Championships</td>\n",
       "      <td>117350</td>\n",
       "      <td>2019 World Junior Ice Hockey Championships</td>\n",
       "      <td>DATE ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3411</th>\n",
       "      <td>14</td>\n",
       "      <td>Wayne_Gretzky</td>\n",
       "      <td>97377</td>\n",
       "      <td>Wayne Gretzky</td>\n",
       "      <td>PERSON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3480</th>\n",
       "      <td>14</td>\n",
       "      <td>24Hours</td>\n",
       "      <td>74156</td>\n",
       "      <td>24Hours</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4259</th>\n",
       "      <td>14</td>\n",
       "      <td>IIHF_World_U20_Championship</td>\n",
       "      <td>78838</td>\n",
       "      <td>IIHF World U20 Championship</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4548</th>\n",
       "      <td>14</td>\n",
       "      <td>List_of_Stanley_Cup_champions</td>\n",
       "      <td>73879</td>\n",
       "      <td>List of Stanley Cup champions</td>\n",
       "      <td>EVENT</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5399</th>\n",
       "      <td>14</td>\n",
       "      <td>Alexander_Ovechkin</td>\n",
       "      <td>69756</td>\n",
       "      <td>Alexander Ovechkin</td>\n",
       "      <td>PERSON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5408</th>\n",
       "      <td>14</td>\n",
       "      <td>2018_World_Junior_Ice_Hockey_Championships</td>\n",
       "      <td>76122</td>\n",
       "      <td>2018 World Junior Ice Hockey Championships</td>\n",
       "      <td>DATE EVENT</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5525</th>\n",
       "      <td>14</td>\n",
       "      <td>Sidney_Crosby</td>\n",
       "      <td>62898</td>\n",
       "      <td>Sidney Crosby</td>\n",
       "      <td>PERSON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5554</th>\n",
       "      <td>14</td>\n",
       "      <td>Spengler_Cup</td>\n",
       "      <td>56905</td>\n",
       "      <td>Spengler Cup</td>\n",
       "      <td>EVENT</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6472</th>\n",
       "      <td>14</td>\n",
       "      <td>William_Nylander</td>\n",
       "      <td>53988</td>\n",
       "      <td>William Nylander</td>\n",
       "      <td>PERSON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7369</th>\n",
       "      <td>14</td>\n",
       "      <td>Connor_McDavid</td>\n",
       "      <td>53172</td>\n",
       "      <td>Connor McDavid</td>\n",
       "      <td>PERSON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7413</th>\n",
       "      <td>14</td>\n",
       "      <td>National_Hockey_League</td>\n",
       "      <td>72391</td>\n",
       "      <td>National Hockey League</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8110</th>\n",
       "      <td>14</td>\n",
       "      <td>Miracle_on_Ice</td>\n",
       "      <td>53144</td>\n",
       "      <td>Miracle on Ice</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8285</th>\n",
       "      <td>14</td>\n",
       "      <td>Tyler_Bertuzzi</td>\n",
       "      <td>45429</td>\n",
       "      <td>Tyler Bertuzzi</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9571</th>\n",
       "      <td>14</td>\n",
       "      <td>Carter_Hart</td>\n",
       "      <td>39548</td>\n",
       "      <td>Carter Hart</td>\n",
       "      <td>PERSON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10121</th>\n",
       "      <td>14</td>\n",
       "      <td>Auston_Matthews</td>\n",
       "      <td>42376</td>\n",
       "      <td>Auston Matthews</td>\n",
       "      <td>PERSON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10375</th>\n",
       "      <td>14</td>\n",
       "      <td>2019_NHL_Winter_Classic</td>\n",
       "      <td>41086</td>\n",
       "      <td>2019 NHL Winter Classic</td>\n",
       "      <td>DATE EVENT</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10790</th>\n",
       "      <td>14</td>\n",
       "      <td>Don_Cherry</td>\n",
       "      <td>37994</td>\n",
       "      <td>Don Cherry</td>\n",
       "      <td>PERSON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11512</th>\n",
       "      <td>14</td>\n",
       "      <td>Hartford_Whalers</td>\n",
       "      <td>42674</td>\n",
       "      <td>Hartford Whalers</td>\n",
       "      <td>PERSON</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       louvain_community                                   title_raw  weight  \\\n",
       "2175                  14             List_of_NHL_statistical_leaders  102869   \n",
       "3019                  14  2019_World_Junior_Ice_Hockey_Championships  117350   \n",
       "3411                  14                               Wayne_Gretzky   97377   \n",
       "3480                  14                                     24Hours   74156   \n",
       "4259                  14                 IIHF_World_U20_Championship   78838   \n",
       "4548                  14               List_of_Stanley_Cup_champions   73879   \n",
       "5399                  14                          Alexander_Ovechkin   69756   \n",
       "5408                  14  2018_World_Junior_Ice_Hockey_Championships   76122   \n",
       "5525                  14                               Sidney_Crosby   62898   \n",
       "5554                  14                                Spengler_Cup   56905   \n",
       "6472                  14                            William_Nylander   53988   \n",
       "7369                  14                              Connor_McDavid   53172   \n",
       "7413                  14                      National_Hockey_League   72391   \n",
       "8110                  14                              Miracle_on_Ice   53144   \n",
       "8285                  14                              Tyler_Bertuzzi   45429   \n",
       "9571                  14                                 Carter_Hart   39548   \n",
       "10121                 14                             Auston_Matthews   42376   \n",
       "10375                 14                     2019_NHL_Winter_Classic   41086   \n",
       "10790                 14                                  Don_Cherry   37994   \n",
       "11512                 14                            Hartford_Whalers   42674   \n",
       "\n",
       "                                            title named_entities  \n",
       "2175              List of NHL statistical leaders            ORG  \n",
       "3019   2019 World Junior Ice Hockey Championships       DATE ORG  \n",
       "3411                                Wayne Gretzky         PERSON  \n",
       "3480                                      24Hours                 \n",
       "4259                  IIHF World U20 Championship            ORG  \n",
       "4548                List of Stanley Cup champions          EVENT  \n",
       "5399                           Alexander Ovechkin         PERSON  \n",
       "5408   2018 World Junior Ice Hockey Championships     DATE EVENT  \n",
       "5525                                Sidney Crosby         PERSON  \n",
       "5554                                 Spengler Cup          EVENT  \n",
       "6472                             William Nylander         PERSON  \n",
       "7369                               Connor McDavid         PERSON  \n",
       "7413                       National Hockey League            ORG  \n",
       "8110                               Miracle on Ice                 \n",
       "8285                               Tyler Bertuzzi            ORG  \n",
       "9571                                  Carter Hart         PERSON  \n",
       "10121                             Auston Matthews         PERSON  \n",
       "10375                     2019 NHL Winter Classic     DATE EVENT  \n",
       "10790                                  Don Cherry         PERSON  \n",
       "11512                            Hartford Whalers         PERSON  "
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "louvain_communities_for_NLP_proc[louvain_communities_for_NLP_proc.louvain_community == 14][:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.3 Lemmatizing and stemming"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "stemmer = SnowballStemmer(\"english\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def parse_title(title):\n",
    "    # words = re.sub('[-():]', \" \", title).split()\n",
    "    words = gensim.utils.simple_preprocess(title)\n",
    "    stopwords = gensim.parsing.preprocessing.STOPWORDS\n",
    "    \n",
    "    parsed=[]\n",
    "    for word in words:\n",
    "        if word not in stopwords:\n",
    "            lemmatized = stemmer.stem(WordNetLemmatizer().lemmatize(word, pos='v'))\n",
    "            parsed.append(lemmatized)\n",
    "            \n",
    "    return parsed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Started running at 2019-03-03 03:58:15.950026 UTC\n",
      "Num of rows: 2729767\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "                <audio controls=\"controls\" autoplay=\"autoplay\">\n",
       "                    <source src=\"data:audio/wav;base64,UklGRpgiAABXQVZFZm10IBAAAAABAAEARKwAAIhYAQACABAAZGF0YXQiAAAAAF8QeSALMNI+kkwPWRZkeG0Odbd6XH7uf2Z/xnwZeHRx8Wi1XupSw0V3N0EoYhgcCLX3cOeT11/IFrryrCyh9JZ3jteHMIOWgBOAq4FWhQWLoJIHnBOnlLNXwSHQtN/Q7y4AjRCmIDYw+z63TDFZM2SQbSB1xHpkfu9/YX+8fAl4XnHWaJVex1KcRUw3FCg0GO0Hh/dD52fXNcjvuc+sDaHalmGOxoclg5GAFYCzgWSFGIu5kiScNKe5s3/BTNDi3/7vXQC7ENMgYTAjP9xMUllQZKhtM3XRemt+8X9df7F8+XdIcbtodl6jUnVFIjfoJwYYvwdY9xXnOtcLyMi5q6ztoL+WTI62hxuDjYAXgLqBcYUri9GSQpxWp9+zqMF30A/gLPCMAOkQACGMMEw/Ak1zWW1kwG1Gdd56cn7yf1h/pnzpdzNxoGhWXn9STUX4Nrwn2BeQByr35+YO1+HHobmIrM6gpZY2jqeHEYOJgBiAwoF/hT+L6ZJfnHenBLTRwaPQPOBb8LoAGBEtIbcwdT8nTZVZimTYbVl163p5fvN/U3+cfNh3HXGFaDdeXFImRc42jyeqF2IH+/a55uLWt8d6uWSsr6CKliGOl4cHg4SAGoDKgYyFUosCk3ycmacqtPrBztBp4Inw6QBGEVoh4zCdP0xNtlmnZPBta3X4eoF+9X9Pf5F8yHcHcWpoF144Uv9EpDZjJ3wXMwfM9ozmttaNx1O5QayQoHCWDI6Hh/2CgIAcgNKBmoVlixqTmpy7p1C0I8L60Jbgt/AYAXQRiCEOMcY/cU3XWcRkCG5+dQV7iH72f0p/hny3d/FwT2j3XRRS2ER6NjYnThcEB572XuaJ1mTHLLkerHGgVZb2jXeH84J8gB6A2oGohXiLM5O3nN2ndbRLwiXRxODm8EYBoxG1ITkx7j+XTflZ4GQgbpB1EnuPfvd/RX98fKd323A0aNhd8FGwRE82CicgF9YGb/Yw5l3WOscFufurUqA7luGNZ4fpgniAIIDhgbWFjItLk9Wc/6ebtHTCUdHx4BTxdQHREeIhZDEXQLxNGlr9ZDhuo3Ufe5Z++H9Af3F8lnfFcBlouF3MUYlEJTbdJvMWpwZB9gLmMdYQx96416szoCGWzI1Yh9+CdIAigOmBw4Wfi2ST8pwhqMG0ncJ80R7hQ/GkAf8RDyKPMT9A4U07WhplT261dSx7nH75fzt/ZnyGd69w/meYXahRYUT7NbEmxRZ4BhL21eUF1ubGt7i0qxSgB5a3jUiH1YJwgCSA8oHRhbKLfZMQnUOo57TGwqjRTOFx8dIBLRI8IroxZ0AGTlxaNmVnbsd1OHujfvp/Nn9bfHV3mXDjZ3hdhFE6RNA1hCaXFkoG5PWn5dnVvMaQuJGr9Z/slaKNOYfLgm2AJoD6gd+FxouWky2dZagMte/C09F54Z/xAQJbEmki5TGQQCtOfVpTZX9u2nVFe6p++n8wf1B8ZHeCcMdnWF1gURJEpjVYJmkWGwa19XnlrdWTxmq4bqvWn9KVjY0ph8KCaYApgAKC7YXai6+TS52HqDK1GMP/0abhzvEwAooSliIQMrhAUE6eWm9llm7sdVJ7sX77fyt/RHxTd2xwrGc4XTxR60N7NSsmOxbtBYf1S+WB1WnGQ7hLq7efuJV4jRqHuIJlgCuACoL7he2Lx5NpnamoWLVBwyrS1OH88V8CuBLDIjsy4EB1Tr9ajGWubv51Xnu3fvx/Jn85fEJ3VnCRZxhdGFHDQ1E1/yUNFr4FWPUe5VXVP8YcuCirmJ+elWSNC4evgmKALYASggmGAYzgk4edy6h+tWvDVtIB4ivyjQLmEu8iZjIJQZpO4FqoZcVuEHZqe75+/H8gfy58MXc/cHVn+Fz0UJtDJzXSJd8VjwUp9fDkKdUWxva3Bat6n4SVT437hqWCXoAwgBuCF4YVjPmTpJ3tqKS1lMOB0i/iWfK8AhQTHCORMjFBvk4BW8Vl3G4idnd7xH79fxt/I3wgdylwWmfYXNBQdEP8NKYlsRVhBfv0w+T91OzFz7fiqlufapU6jeyGnIJbgDKAI4ImhiiMEpTCnQ+pyrW9w63SXOKH8usCQhNJI7wyWUHjTiJb4WX0bjR2g3vLfv1/FX8XfA93EnA+Z7hcrFBMQ9I0eSWDFTIFzPSV5NHUwsWpt7+qPZ9RlSWN3YaSgleANYAsgjSGPIwslOCdMqnwtebD2dKJ4rbyGQNxE3Yj5zKBQQhPQlv9ZQtvRnaPe9F+/n8Qfwx8/nb8byJnmFyHUCRDpzRMJVUVAwWe9GfkpdSZxYK3naoenzeVEY3OhomCVIA3gDSCQoZQjEWU/p1UqRa2D8QF07fi5PJIA58ToyMRM6lBLU9jWxlmIm9Ydpt7137+fwp/AHztduVvB2d3XGNQ/UJ8NCAlJxXVBG/0OuR51G/FXLd6qv+eHZX8jL+GgIJRgDqAPYJRhmSMXpQcnnapPLY5xDDT5OIT83cDzRPQIzwz0UFRT4RbNmY5b2l2qHvefv5/BH/1e9t2zm/rZldcP1DVQlI08yT5FKYEQfQM5E3URsU1t1eq4Z4DleiMsIZ3gk2APYBGgl+GeIx3lDuemaljtmLEXNMS40HzpQP7E/0jZzP5QXZPpFtSZlBve3a0e+R+/n//ful7yna4b89mN1waUK1CJzTGJMoUdwQS9N/jIdQcxQ+3NKrDnuqU04yhhm6CSoBAgE6CboaNjJGUWZ67qYm2i8SI0z/jcPPUAykUKiSSMyFCm0/FW25mZ2+NdsB76n7+f/l+3Xu5dqFvs2YWXPZPhUL9M5kknBRJBOTzseP10/PE6bYSqqSe0JS/jJOGZYJHgEOAV4J9hqGMqpR3nt6pr7a1xLTTbeOe8wMEVxRWJL0zSUK/T+ZbimZ+b552y3vwfv9/837Re6d2im+YZvZb0U9dQtIzbSRuFBoEtfOE48nTycTCtu+php63lKuMhIZcgkSARoBggouGtYzElJWeAKrVtt7E39Oa483zMQSFFIMk5zNxQuRPBlymZpVvsHbXe/Z+/n/tfsV7lnZzb3xm1VutTzVCpzNAJEAU6wOH81bjntOgxJy2zalonp2Ul4x1hlOCQYBJgGmCmobJjN2Us54jqvy2B8UL1Mjj+/NgBLMUsCQSNJlCCFAmXMFmrG/BduN7/H7+f+d+unuEdlxvYGa1W4hPDUJ8MxMkEhS9A1jzKeNy03fEdraqqUqehJSCjGeGSoI+gEyAcoKpht6M95TSnkaqIrcxxTfU9eMq9I8E4RTdJD00wUIsUEdc3WbDb9N273sCf/5/4X6ue3J2RW9EZpRbZE/lQVIz5iPkE44DKvP74kbTTcRQtoipK55rlG6MWIZBgjuAT4B7griG8owQlfCeaKpJt1rFY9Qj5Fj0vQQQFQklZzTpQlFQZ1z5Ztpv5Hb6ewd//n/bfqF7YXYubydmc1s/T71BJzO5I7YTXwP88s7iGtMkxCm2ZakNnlGUWoxKhjmCOYBSgIWCx4YHjSqVD5+Lqm+3hMWP1FHkh/TsBD4VNiWSNBBDdVCHXBVn8G/1dgZ8DX/+f9R+lXtPdhdvC2ZTWxpPlUH8Mo0jiBMxA83yoOLv0vvDA7ZDqe+dOJRGjDuGMII2gFWAjoLWhhuNRJUtn66qlbetxbvUfuS19BsFbBVjJbw0OEOZUKhcMGcHcAZ3EXwTf/1/zn6Jez12/27vZTJb9k5tQdEyYCNaEwIDn/Jz4sPS0sPdtSGp0Z0flDKMLYYngjOAWYCXguWGMI1elUyf0aq8t9fF59Ss5OT0SQWaFY8l5zRgQ75QyFxMZx5wGHcdfBh//X/Ifn17K3bobtNlEVvRTkVBpjIzIysT0wJw8kXil9Kow7e1/qiznQaUH4wehh+CMYBcgKCC9IZEjXeVap/0quO3AcYT1dnkEvV4BcgVvCURNYhD4lDoXGdnNHApdyh8Hn/9f8F+cXsZdtFutmXwWqxOHUF7MgYj/RKlAkLyGOJs0n/DkbXcqJad7ZMLjBCGF4IugGCAqoIDh1mNkZWJnxerCbgqxj/VB+VB9acF9hXoJTw1r0MGUQhdg2dKcDp3NHwjf/x/u35kewd2uW6aZdBah070QFEy2SLPEnYCE/Lq4UDSVsNrtbqoeJ3Uk/eLAoYOgiyAY4CzghKHbo2rlaifOqswuFTGa9U15W/11QUkFhUmZjXXQypRKF2eZ2FwS3c/fCh/+3+0flh79XWibn5lr1piTsxAJjKsIqESRwLl8b3hFNItw0W1mKhanbuT44v0hQaCKoBngL2CIoeDjcWVx59dq1a4fsaX1WLlnvUEBlIWQiaRNf5DTlFIXbpnd3Bcd0p8Ln/7f61+S3vjdYpuYWWOWj1OpED7MX8icxIYArfxkOHp0QTDH7V2qDydopPQi+aF/oEngGuAx4Ixh5iN35Xln4CrfbinxsPVkOXM9TMGgBZuJrs1JkRyUWhd1WeNcGx3VXwzf/p/p34/e9F1c25FZW1aGE58QNAxUiJEEuoBiPFi4b3R28L5tFSoH52Jk7yL2IX2gSWAboDQgkGHrY35lQSgo6ukuNHG79W+5fv1YQauFpsm5TVORJZRiF3wZ6RwfXdgfDh/+X+gfjJ7vnVbbihlTFrzTVNApTElIhYSuwFa8TXhktGywtS0MqgBnXGTqYvKhe2BI4BygNqCUIfCjRSWI6DGq8u4+8Yb1uvlKfaQBtwWxyYQNnVEulGoXQxounCOd2t8PX/4f5l+JXusdURuC2UqWs5NK0B6Mfgh6BGMASvxCOFm0YnCrrQQqOOcWJOVi7yF5YEhgHaA5IJgh9eNLpZCoOmr8bglx0fWGeZY9r4GChf0Jjo2nETeUchdJ2jQcJ93dnxCf/d/kn4Ze5l1LG7vZAlaqU0CQE4xyyG6EV4B/fDa4DvRYMKItO6nxpw/k4KLroXdgR+AeoDugm+H7I1IlmGgDKwYuU/Hc9ZH5of27QY3FyAnZDbERAJS511CaOZwr3eBfEd/9n+Lfgx7h3UUbtJk6FmETdo/IzGeIYsRLwHP8K3gD9E3wmK0zKeonCeTb4uhhdaBHYB+gPiCf4cBjmOWgKAvrD+5eceg1nXmtfYcB2UXTSePNutEJlIHXl1o/HDAd4x8TH/1f4R+/3p1dfxttWTHWV9NsT/4MHEhXREAAaDwgODk0A7CPbSqp4ucDpNbi5OFzoEbgIKAAoOPhxaOfZafoFOsZrmix8zWoubk9koHkxd5J7k2E0VKUideeGgScdB3lnxRf/R/fX7yemJ15G2YZKVZOk2JP80wRCEvEdIAcvBT4LjQ5cEXtIinbpz2kkiLhYXGgRmAh4AMg5+HLI6Xlr6gdqyNuczH+NbQ5hL3eQfBF6Un4zY6RW1SRl6TaChx4HehfFZ/8392fuV6T3XMbXtkhFkUTWA/ojAXIQERowBE8CXgjdC8wfKzZ6dQnN2SNYt4hb6BGICLgBaDr4dBjrKW3qCarLS59sck1/7mQfeoB+8X0icNN2FFkVJmXq5oPXHxd6x8Wn/yf29+2Ho9dbRtX2RjWe9MOD93MOog0hB0ABXw+N9i0JTBzLNFpzOcxZIii2uFt4EWgI+AIIO+h1aOzZb9oL2s27kgyFDXLOdv99YHHRj+Jzc3iEW1UoVeyWhTcQF4tnxff/B/Z37Leip1nG1CZEFZykwPP0swvSCkEEYA5+/L3zbQa8GnsyOnFpyskg+LXYWvgRSAk4Aqg86HbI7nlhyh4KwCukrIfdda5573BQhLGCsoYTevRdhSpV7jaGlxEXjBfGR/739gfr56F3WEbSRkIFmkTOc+IDCPIHYQFwC5757fC9BCwYGzAqf5m5SS/IpQhaiBE4CYgDWD34eCjgKXPKEErSm6dMip14fnzfczCHkYVyiMN9dF/FLEXv5ofnEheMt8aH/tf1h+sHoEdWxtB2T+WH9Mvj71L2IgRxDp/4rvcd/gzxnBXLPgptybfJLpikKFoIERgJyAP4Pvh5eOHZdboSitUbqfyNXXtef792IIphiDKLY3/kUgU+ReGWmUcTJ41nxtf+x/UX6jevF0VG3qY91YWUyVPsovNSAZELr/XO9D37XP8cA2s7+mvptkktaKNYWZgRCAoYBKg/+HrY43l3uhS614usnIAtjj5yr4kQjUGLAo4DclRkNTA18zaapxQnjgfHF/6n9JfpV63nQ7bc1ju1g0TGw+ni8IIOsPjP8u7xbfic/IwBGznaahm0ySw4oohZGBDoCmgFSDD4jDjlKXmqFvrZ+688gu2BHoWPi/CAIZ3CgKOExGZlMiX05pv3FReOp8dX/of0J+iHrLdCNtsGOZWA5MRD5zL9sfvA9d///u6d5ez6DA7LJ8poWbNJKxihuFioENgKqAX4MgiNiObZe6oZOtxrodyVvYP+iH+O4IMBkIKTQ4c0aKU0JfaWnUcWF49Hx5f+d/On57erh0Cm2SY3hY6UsbPkgvrR+ODy7/0e683jPPd8DGslumaJsckp6KDoWDgQyAr4BqgzCI7o6Il9mhtq3tukfJh9ht6Lb4HAleGTQpXjiaRq1TYV+DaepxcXj+fH5/5X8yfm16pXTybHVjVljDS/I9HC+AH2APAP+j7o/eCM9PwKGyOaZLmwSSi4oBhXyBC4C0gHSDQIgEj6OX+aHarRW7ccmz2Jvo5PhLCYsZYCmHOMFG0VOAX51p/3GBeAh9gn/jfyp+X3qRdNlsWGM0WJ5LyT3xLlMfMQ/R/nXuYt7dzibAfLIYpi6b7JF5ivSEdYEKgLmAf4NRiBqPvpcZov6tPLucyeDYyegT+XkJuRmNKbE46Eb0U59fuGkUcpF4En2Gf+F/I35Sen50wWw6YxJYeEugPcUuJh8DD6L+Ru413rLO/r9XsvelEZvUkWeK54RugQmAvoCKg2GIMI/ZlziiIq5ku8bJDNn26EL5qAnnGbkp2zgPRxdUvl/SaSlyoHgcfYp/338bfkR6a3SobB1j8FdSS3c9mi74HtUOdP4Y7gjehs7VvzKy1qX1mryRVIrbhGeBCIDDgJWDcohGj/SXWKJGrou78Mk52STpcPnXCRUa5SkFOTVHOlTdX+xpPnKweCZ9jn/dfxN+NnpXdI9s/2LOVyxLTj1uLssepg5F/urt291bzq2/DbK0pdiapZFCis6EYIEHgMiAoIODiFyPEJh4omqusrsbymXZUumf+QUKQhoRKi85XEddVPxfB2pTcr94MH2Sf9t/Cn4oekR0d2zhYqxXB0slPUMunh54Dhb+vO2u3TDOhL/osZOlu5qNkS+KwYRZgQaAzYCrg5SIc48rmJiijq7au0XKktmA6c35NApwGj0qWTmDR4BUG2Ahamhyz3g5fZV/2X8Cfhp6MHRebMRiilfhSvw8Fy5wHkkO6P2N7YHdBc5cv8OxcqWfmnaRHYq1hFOBBYDSgLaDpIiJj0aYuKKyrgK8b8q+2a7p/PliCp4aaSqCOapHo1Q5YDtqfXLeeEN9mX/Wf/p9DHoddEVspmJoV7tK0zzsLUMeGw65/V/tVN3azTS/nrFRpYKaXpELiqiETIEFgNiAwYO1iJ+PYpjYotauKbyayuvZ3Okr+pEKyxqVKqw50EfGVFhgVWqScu54TX2df9R/8n3+eQl0LGyIYkZXlUqqPMAtFh7tDYr9Me0n3a/NDL95sTClZppHkfmJnIRFgQSA3YDMg8aIto99mPii+q5RvMTKGNoK6ln6vwr5GsEq1jn3R+lUd2Bvaqdy/XhWfaB/0n/pffB59XMTbGpiJFdvSoE8lC3oHb4NW/0D7frchc3jvlSxEKVKmi+R54mPhD+BA4DigNiD14jMj5mYGKMer3i878pE2jjqiPruCicb7Sr/OR1IDFWWYIlqvHIMeWB9pH/Pf+F94nnhc/prTWICV0lKWDxpLbsdkA0t/dXszdxazbu+L7HvpC2aGJHViYOEOIEDgOiA44PoiOKPtJg4o0KvoLwZy3HaZuq3+hwLVBsZKyk6REgvVbRgomrQcht5aX2nf81/2X3Tec5z4WsvYt9WI0ouPD0tjR1hDf78puyg3C/Nk74Ksc6kEZoBkcOJd4QygQOA7YDvg/qI+Y/QmFijZ6/IvETLndqU6uX6SwuCG0UrUzprSFJV02C8auVyKnlyfat/yn/QfcV5unPIaxFivVb9SQU8ES1gHTMNz/x47HPcBM1rvuawraT1memQsYlrhCyBAoDzgPqDC4kQkOuYeaOLr/C8bsvK2sLqFPt5C68bcSt8OpFIdVXxYNZq+XI5eXt9rn/Hf8d9tnmmc69r82GbVtdJ3DvmLDIdBA2h/ErsR9zZzEO+wbCNpNmZ0pCfiV+EJYECgPmABoQciSaQB5mZo6+vF72Zy/fa8OpD+6gL3RudK6Y6t0iYVRBh8GoOc0h5hX2xf8V/v32oeZJzlWvVYXhWsEmzO7osBR3WDHL8HOwa3K7MG76csGykvJm7kI6JUoQfgQKA/oARhC2JPZAjmbmj1K8/vcPLI9sf63H71gsLHMkrzzreSLpVLmEJayJzV3mOfbR/wn+2fZl5fnN8a7ZhVlaKSYk7jizXHKgMQ/zu6+3bhMzzvXiwS6SgmaSQfIlGhBmBAoAEgR2EP4lUkD+Z2qP4r2e97stQ203roPsFDDgc9Sv5OgRJ3VVNYSNrN3NmeZd9t3+/f619i3lpc2NrmGEzVmRJYDtiLKoceQwV/MDrwNtZzMu9U7ArpISZjZBqiTuEE4ECgAqBKYRQiWuQWpn6oxywj70ZzH3be+vP+zMMZhwhLCI7K0kAVmthPGtLc3V5oH26f7x/pH18eVVzSWt6YRFWPkk3OzcsfBxLDOb7kuuT2y7Mo70vsAqkaJl2kFmJL4QNgQGAEIE1hGKJgpB2mRqkQbC3vUPMqtup6/37YgyTHEwsSztRSSJWiWFWa19zg3mpfb1/uX+bfW15QXMwa1xh7lUXSQ07CyxPHBwMt/tk62fbA8x7vQqw6qNNmV+QR4kjhAeBAoAWgUCEc4mZkJKZO6RlsN+9bszW29frLPyQDMEceCx1O3dJRVanYW9rc3OSebJ9wH+2f5J9X3ktcxZrPWHMVfFI5DrfKyEc7guJ+zbrOtvZy1O95q/JozGZSJA2iReEAYECgByBTISFibCQrplcpIqwB76ZzAPcBexb/L8M7hykLJ47nUlnVsVhiWuIc6F5un3Df7N/iX1QeRhz/WofYalVy0i6OrMr9Bu/C1r7B+sN267LK73Br6mjFZkykCWJC4T8gAKAIoFYhJeJx5DKmXykr7AvvsTMMNwz7In87QwcHdAsxzvESYpW5GGia5xzr3nDfcZ/r3+AfUF5BHPjagFhhlWkSJE6hyvGG5ELK/vZ6uDahMsDvZ2viaP5mBuQE4kAhPaAAoApgWWEqInekOeZnaTTsFe+78xd3GHsuPwcDUkd+yzxO+pJrFYCYrtrsHO+ecx9yX+sf3d9Mnnvcslq4mBjVX5IZzpbK5kbYgv9+qvqtNpZy9y8ea9oo96YBJACifSD8IACgC+BcYS6ifWQA5q+pPiwf74ZzYrcj+zn/EoNdx0nLRo8EErOViBi1GvEc8x51H3Lf6l/bn0jedtyr2rDYEFVV0g+Oi8raxs0C876feqH2i7LtLxUr0ijwpjuj/GI6YPrgAOANYF9hMyJDJEfmt6kHbGnvkTNt9y+7BX9eQ2kHVMtQzw2SvFWPmLua9hz2nndfc5/pX9kfRR5xnKWaqVgHlUxSBQ6Ays9GwULn/pP6lraBMuMvDCvKKOmmNeP4Ijdg+WAA4A8gYmE3okkkTua/6RCsc++b83k3OzsRP2nDdEdfy1sPFxKE1dcYgds63PpeeV90H+if1t9BXmxcnxqhmD7VApI6jnXKhAb1wpx+iHqLtrZymW8DK8Io4uYwY/PiNKD4IAEgEKBloTwiTuRWJogpWax976azRHdGu1z/dUN/x2qLZU8gko1V3liIGz/c/d57n3Tf55/UX31eJxyYmpoYNhU5EfBOasq4hqoCkL68+kB2q/KPbzoruiib5iqj76Ix4PagASASYGihAKKUpF0mkGli7Egv8XNPd1I7aH9BA4sHtYtvzyoSldXl2I5bBN0BXr2fdV/m39IfeZ4iHJIaklgtVS9R5c5fyq1GnkKE/rF6dXZhcoVvMSuyKJUmJSPrYi8g9WABYBPga6EFIpqkZGaYqWwsUi/8M1q3Xbt0P0yDloeAS7oPM5KeVe1YlFsJnQTev5913+Xfz5913hzci5qKmCSVJZHbTlTKocaSwrl+ZfpqNlayu67oK6oojmYfo+ciLCD0IAGgFaBu4QmioGRrZqDpdWxcL8bzpfdpe3//WEOhx4tLhE99EqbV9Niamw6dCF6Bn7af5N/NX3HeF5yFGoLYG9UcEdEOScqWRocCrb5ael82TDKxrt8roiiHZhnj4uIpYPKgAaAXYHIhDmKmZHKmqSl+rGZv0bOxN3T7S7+jw60HlguOj0ZS71X8GKDbE50L3oOftx/kH8rfbh4SXL5aexfTFRJRxo5+ykrGu4JiPk76U/ZBcqfu1iuaKICmFGPeoiag8WAB4BkgdSES4qxkeaaxaUfssG/cc7x3QHuXP69DuIehC5jPT9L31cOY5xsYXQ9ehd+3n+MfyF9qHg0ct9pzV8pVCJH8DjPKf4ZvwlZ+Q3pI9nbyXe7NK5IoueXO49qiI+DwIAIgGqB4YRdisiRA5vmpUSy6b+czh7eL+6L/uwODx+vLow9ZUsBWCtjtWx0dEt6H37gf4h/F32ZeB9yxWmuXwVU+0bGOKMp0BmRCSr54Oj22LHJULsQriiizJclj1mIhIO7gAmAcYHuhHCK4JEgmwemabISwMfOS95d7rr+Gg88H9sutT2LSyNYSWPNbIh0WHomfuJ/hH8NfYl4CnKraY9f4lPURpw4dymiGWIJ/Piy6MrYhskou+ytCaKxlw+PSYh6g7aACoB4gfuEgor4kTybKaaPsjrA8s543ozu6P5JD2ofBi/dPbBLRVhmY+Zsm3Rmei5+5H+AfwN9eXj0cZBpcF+/U61GczhKKXQZNAnN+ITondhcyQG7yK3poZaX+Y44iG+DsYALgH+BCIWVihCSWZtKprSyY8Adz6beuu4X/3cPlx8yLwY+1ktnWIRj/myudHR6Nn7mf3x/+XxpeN9xdmlRX5xThkZJOB4pRxkFCZ74Vuhx2DLJ2rqkrcmhe5fjjiiIZIOtgA2Ah4EVhaeKKJJ2m2um2bKLwEnP097o7kb/pQ/EH10vLz78S4lYoWMXbcF0gXo+fuh/d3/vfFl4ynFbaTJfeFNfRh848igZGdYIcPgo6ETYCMmzuoGtqqFgl82OF4hag6iADoCOgSKFuopAkpObjab+srTAdM8A3xfvdP/UD/EfiS9YPiFMqli+Yy9t1XSPekZ+6X9zf+V8Sni0cUFpE19VUzhG9TfGKOsYqAhB+PrnGNjeyIu6Xa2KoUWXuI4HiE+Do4APgJWBL4XNiliSsJuupiSz3cCfzy3fRe+j/wIQHiC0L4E+R0zMWNxjR23odJx6TX7rf29/23w6eJ9xJmnzXjFTEUbLN5kovRh5CBP4zOfs17TIZLo5rWuhKpeijveHRIOfgBGAnIE8heCKcJLNm8+mSbMFwcrPWt9z79L/MBBMIN8vqT5sTO1Y+WNgbft0qnpVfu1/an/QfCl4iXEMadReDlPqRaE3bSiQGEsI5Pee57/Xicg9uhatS6EPl4yO54c6g5qAEoCkgUmF8oqIkuqb8aZusy7B9c+H36HvAAA=\" type=\"audio/wav\" />\n",
       "                    Your browser does not support the audio element.\n",
       "                </audio>\n",
       "              "
      ],
      "text/plain": [
       "<IPython.lib.display.Audio object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Runtime: 12.51 min\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>louvain_community</th>\n",
       "      <th>title_raw</th>\n",
       "      <th>weight</th>\n",
       "      <th>title</th>\n",
       "      <th>named_entities</th>\n",
       "      <th>title_parsed</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3</td>\n",
       "      <td>George_H._W._Bush</td>\n",
       "      <td>5690673</td>\n",
       "      <td>George H. W. Bush</td>\n",
       "      <td>PERSON</td>\n",
       "      <td>georg bush</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4</td>\n",
       "      <td>Jason_Momoa</td>\n",
       "      <td>4183872</td>\n",
       "      <td>Jason Momoa</td>\n",
       "      <td>PERSON</td>\n",
       "      <td>jason momoa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9</td>\n",
       "      <td>2.0_(film)</td>\n",
       "      <td>3722311</td>\n",
       "      <td>2.0 (film)</td>\n",
       "      <td>CARDINAL</td>\n",
       "      <td>film</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Bird_Box_(film)</td>\n",
       "      <td>3945404</td>\n",
       "      <td>Bird Box (film)</td>\n",
       "      <td>PERSON</td>\n",
       "      <td>bird box film</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>Main_Page</td>\n",
       "      <td>3051841</td>\n",
       "      <td>Main Page</td>\n",
       "      <td></td>\n",
       "      <td>main page</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>4</td>\n",
       "      <td>Aquaman_(film)</td>\n",
       "      <td>3077395</td>\n",
       "      <td>Aquaman (film)</td>\n",
       "      <td></td>\n",
       "      <td>aquaman film</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>4</td>\n",
       "      <td>Bird_Box</td>\n",
       "      <td>2529969</td>\n",
       "      <td>Bird Box</td>\n",
       "      <td>PERSON</td>\n",
       "      <td>bird box</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>3</td>\n",
       "      <td>Priyanka_Chopra</td>\n",
       "      <td>2810602</td>\n",
       "      <td>Priyanka Chopra</td>\n",
       "      <td>PERSON</td>\n",
       "      <td>priyanka chopra</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>5</td>\n",
       "      <td>List_of_most-disliked_YouTube_videos</td>\n",
       "      <td>2344675</td>\n",
       "      <td>List of most-disliked YouTube videos</td>\n",
       "      <td>ORG</td>\n",
       "      <td>list dislik youtub video</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>5</td>\n",
       "      <td>Freddie_Mercury</td>\n",
       "      <td>2391410</td>\n",
       "      <td>Freddie Mercury</td>\n",
       "      <td>ORG</td>\n",
       "      <td>freddi mercuri</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>3</td>\n",
       "      <td>Zero_(2018_film)</td>\n",
       "      <td>2045050</td>\n",
       "      <td>Zero (2018 film)</td>\n",
       "      <td>CARDINAL DATE</td>\n",
       "      <td>zero film</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>3</td>\n",
       "      <td>George_W._Bush</td>\n",
       "      <td>2584078</td>\n",
       "      <td>George W. Bush</td>\n",
       "      <td>PERSON</td>\n",
       "      <td>georg bush</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>9</td>\n",
       "      <td>K.G.F:_Chapter_1</td>\n",
       "      <td>1790058</td>\n",
       "      <td>K.G.F: Chapter 1</td>\n",
       "      <td>LAW</td>\n",
       "      <td>chapter</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>5</td>\n",
       "      <td>Nick_Jonas</td>\n",
       "      <td>2346279</td>\n",
       "      <td>Nick Jonas</td>\n",
       "      <td>PERSON</td>\n",
       "      <td>nick jona</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>4</td>\n",
       "      <td>Spider-Man:_Into_the_Spider-Verse</td>\n",
       "      <td>1840563</td>\n",
       "      <td>Spider-Man: Into the Spider-Verse</td>\n",
       "      <td></td>\n",
       "      <td>spider man spider vers</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>4</td>\n",
       "      <td>Penny_Marshall</td>\n",
       "      <td>1559346</td>\n",
       "      <td>Penny Marshall</td>\n",
       "      <td>PERSON</td>\n",
       "      <td>penni marshal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>3</td>\n",
       "      <td>Jimmy_Carter</td>\n",
       "      <td>1815966</td>\n",
       "      <td>Jimmy Carter</td>\n",
       "      <td>PERSON</td>\n",
       "      <td>jimmi carter</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>4</td>\n",
       "      <td>Macaulay_Culkin</td>\n",
       "      <td>1518368</td>\n",
       "      <td>Macaulay Culkin</td>\n",
       "      <td>PERSON</td>\n",
       "      <td>macaulay culkin</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>3</td>\n",
       "      <td>List_of_highest-grossing_Indian_films</td>\n",
       "      <td>1580445</td>\n",
       "      <td>List of highest-grossing Indian films</td>\n",
       "      <td>NORP</td>\n",
       "      <td>list highest gross indian film</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>3</td>\n",
       "      <td>Mary,_Queen_of_Scots</td>\n",
       "      <td>1364237</td>\n",
       "      <td>Mary, Queen of Scots</td>\n",
       "      <td></td>\n",
       "      <td>mari queen scot</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    louvain_community                              title_raw   weight  \\\n",
       "0                   3                      George_H._W._Bush  5690673   \n",
       "1                   4                            Jason_Momoa  4183872   \n",
       "2                   9                             2.0_(film)  3722311   \n",
       "3                   4                        Bird_Box_(film)  3945404   \n",
       "4                   1                              Main_Page  3051841   \n",
       "5                   4                         Aquaman_(film)  3077395   \n",
       "6                   4                               Bird_Box  2529969   \n",
       "7                   3                        Priyanka_Chopra  2810602   \n",
       "8                   5   List_of_most-disliked_YouTube_videos  2344675   \n",
       "9                   5                        Freddie_Mercury  2391410   \n",
       "10                  3                       Zero_(2018_film)  2045050   \n",
       "11                  3                         George_W._Bush  2584078   \n",
       "12                  9                       K.G.F:_Chapter_1  1790058   \n",
       "13                  5                             Nick_Jonas  2346279   \n",
       "14                  4      Spider-Man:_Into_the_Spider-Verse  1840563   \n",
       "15                  4                         Penny_Marshall  1559346   \n",
       "16                  3                           Jimmy_Carter  1815966   \n",
       "17                  4                        Macaulay_Culkin  1518368   \n",
       "18                  3  List_of_highest-grossing_Indian_films  1580445   \n",
       "19                  3                   Mary,_Queen_of_Scots  1364237   \n",
       "\n",
       "                                    title named_entities  \\\n",
       "0                       George H. W. Bush         PERSON   \n",
       "1                             Jason Momoa         PERSON   \n",
       "2                              2.0 (film)       CARDINAL   \n",
       "3                         Bird Box (film)         PERSON   \n",
       "4                               Main Page                  \n",
       "5                          Aquaman (film)                  \n",
       "6                                Bird Box         PERSON   \n",
       "7                         Priyanka Chopra         PERSON   \n",
       "8    List of most-disliked YouTube videos            ORG   \n",
       "9                         Freddie Mercury            ORG   \n",
       "10                       Zero (2018 film)  CARDINAL DATE   \n",
       "11                         George W. Bush         PERSON   \n",
       "12                       K.G.F: Chapter 1            LAW   \n",
       "13                             Nick Jonas         PERSON   \n",
       "14      Spider-Man: Into the Spider-Verse                  \n",
       "15                         Penny Marshall         PERSON   \n",
       "16                           Jimmy Carter         PERSON   \n",
       "17                        Macaulay Culkin         PERSON   \n",
       "18  List of highest-grossing Indian films           NORP   \n",
       "19                   Mary, Queen of Scots                  \n",
       "\n",
       "                      title_parsed  \n",
       "0                       georg bush  \n",
       "1                      jason momoa  \n",
       "2                             film  \n",
       "3                    bird box film  \n",
       "4                        main page  \n",
       "5                     aquaman film  \n",
       "6                         bird box  \n",
       "7                  priyanka chopra  \n",
       "8         list dislik youtub video  \n",
       "9                   freddi mercuri  \n",
       "10                       zero film  \n",
       "11                      georg bush  \n",
       "12                         chapter  \n",
       "13                       nick jona  \n",
       "14          spider man spider vers  \n",
       "15                   penni marshal  \n",
       "16                    jimmi carter  \n",
       "17                 macaulay culkin  \n",
       "18  list highest gross indian film  \n",
       "19                 mari queen scot  "
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"Started running at\", datetime.now(), \"UTC\")\n",
    "\n",
    "start_time = timer()\n",
    "\n",
    "nrows = len(louvain_communities_for_NLP_proc)\n",
    "print(\"Num of rows:\", nrows)\n",
    "\n",
    "for i in range(nrows):\n",
    "    txt = louvain_communities_for_NLP_proc.iloc[i].title\n",
    "    parsed = parse_title(txt)\n",
    "    \n",
    "    louvain_communities_for_NLP_proc.at[str(i), \"title_parsed\"] = \" \".join(parsed)\n",
    "    \n",
    "cu.printRunTime(start_time)\n",
    "\n",
    "louvain_communities_for_NLP_proc.head(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pickle created: pickles/en_1218_louvain_communities_for_NLP_proc_2.pkl\n"
     ]
    }
   ],
   "source": [
    "# pickle the output\n",
    "myoutfile = \"pickles/en_1218_louvain_communities_for_NLP_proc_2.pkl\"\n",
    "with open(myoutfile, 'wb') as picklefile:\n",
    "    pickle.dump(louvain_communities_for_NLP_proc, picklefile)\n",
    "\n",
    "print(\"Pickle created: \" + myoutfile)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>louvain_community</th>\n",
       "      <th>title_raw</th>\n",
       "      <th>weight</th>\n",
       "      <th>title</th>\n",
       "      <th>named_entities</th>\n",
       "      <th>title_parsed</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3</td>\n",
       "      <td>George_H._W._Bush</td>\n",
       "      <td>5690673</td>\n",
       "      <td>George H. W. Bush</td>\n",
       "      <td>PERSON</td>\n",
       "      <td>georg bush</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4</td>\n",
       "      <td>Jason_Momoa</td>\n",
       "      <td>4183872</td>\n",
       "      <td>Jason Momoa</td>\n",
       "      <td>PERSON</td>\n",
       "      <td>jason momoa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9</td>\n",
       "      <td>2.0_(film)</td>\n",
       "      <td>3722311</td>\n",
       "      <td>2.0 (film)</td>\n",
       "      <td>CARDINAL</td>\n",
       "      <td>film</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Bird_Box_(film)</td>\n",
       "      <td>3945404</td>\n",
       "      <td>Bird Box (film)</td>\n",
       "      <td>PERSON</td>\n",
       "      <td>bird box film</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>Main_Page</td>\n",
       "      <td>3051841</td>\n",
       "      <td>Main Page</td>\n",
       "      <td></td>\n",
       "      <td>main page</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>4</td>\n",
       "      <td>Aquaman_(film)</td>\n",
       "      <td>3077395</td>\n",
       "      <td>Aquaman (film)</td>\n",
       "      <td></td>\n",
       "      <td>aquaman film</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>4</td>\n",
       "      <td>Bird_Box</td>\n",
       "      <td>2529969</td>\n",
       "      <td>Bird Box</td>\n",
       "      <td>PERSON</td>\n",
       "      <td>bird box</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>3</td>\n",
       "      <td>Priyanka_Chopra</td>\n",
       "      <td>2810602</td>\n",
       "      <td>Priyanka Chopra</td>\n",
       "      <td>PERSON</td>\n",
       "      <td>priyanka chopra</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>5</td>\n",
       "      <td>List_of_most-disliked_YouTube_videos</td>\n",
       "      <td>2344675</td>\n",
       "      <td>List of most-disliked YouTube videos</td>\n",
       "      <td>ORG</td>\n",
       "      <td>list dislik youtub video</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>5</td>\n",
       "      <td>Freddie_Mercury</td>\n",
       "      <td>2391410</td>\n",
       "      <td>Freddie Mercury</td>\n",
       "      <td>ORG</td>\n",
       "      <td>freddi mercuri</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>3</td>\n",
       "      <td>Zero_(2018_film)</td>\n",
       "      <td>2045050</td>\n",
       "      <td>Zero (2018 film)</td>\n",
       "      <td>CARDINAL DATE</td>\n",
       "      <td>zero film</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>3</td>\n",
       "      <td>George_W._Bush</td>\n",
       "      <td>2584078</td>\n",
       "      <td>George W. Bush</td>\n",
       "      <td>PERSON</td>\n",
       "      <td>georg bush</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>9</td>\n",
       "      <td>K.G.F:_Chapter_1</td>\n",
       "      <td>1790058</td>\n",
       "      <td>K.G.F: Chapter 1</td>\n",
       "      <td>LAW</td>\n",
       "      <td>chapter</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>5</td>\n",
       "      <td>Nick_Jonas</td>\n",
       "      <td>2346279</td>\n",
       "      <td>Nick Jonas</td>\n",
       "      <td>PERSON</td>\n",
       "      <td>nick jona</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>4</td>\n",
       "      <td>Spider-Man:_Into_the_Spider-Verse</td>\n",
       "      <td>1840563</td>\n",
       "      <td>Spider-Man: Into the Spider-Verse</td>\n",
       "      <td></td>\n",
       "      <td>spider man spider vers</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>4</td>\n",
       "      <td>Penny_Marshall</td>\n",
       "      <td>1559346</td>\n",
       "      <td>Penny Marshall</td>\n",
       "      <td>PERSON</td>\n",
       "      <td>penni marshal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>3</td>\n",
       "      <td>Jimmy_Carter</td>\n",
       "      <td>1815966</td>\n",
       "      <td>Jimmy Carter</td>\n",
       "      <td>PERSON</td>\n",
       "      <td>jimmi carter</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>4</td>\n",
       "      <td>Macaulay_Culkin</td>\n",
       "      <td>1518368</td>\n",
       "      <td>Macaulay Culkin</td>\n",
       "      <td>PERSON</td>\n",
       "      <td>macaulay culkin</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>3</td>\n",
       "      <td>List_of_highest-grossing_Indian_films</td>\n",
       "      <td>1580445</td>\n",
       "      <td>List of highest-grossing Indian films</td>\n",
       "      <td>NORP</td>\n",
       "      <td>list highest gross indian film</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>3</td>\n",
       "      <td>Mary,_Queen_of_Scots</td>\n",
       "      <td>1364237</td>\n",
       "      <td>Mary, Queen of Scots</td>\n",
       "      <td></td>\n",
       "      <td>mari queen scot</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    louvain_community                              title_raw   weight  \\\n",
       "0                   3                      George_H._W._Bush  5690673   \n",
       "1                   4                            Jason_Momoa  4183872   \n",
       "2                   9                             2.0_(film)  3722311   \n",
       "3                   4                        Bird_Box_(film)  3945404   \n",
       "4                   1                              Main_Page  3051841   \n",
       "5                   4                         Aquaman_(film)  3077395   \n",
       "6                   4                               Bird_Box  2529969   \n",
       "7                   3                        Priyanka_Chopra  2810602   \n",
       "8                   5   List_of_most-disliked_YouTube_videos  2344675   \n",
       "9                   5                        Freddie_Mercury  2391410   \n",
       "10                  3                       Zero_(2018_film)  2045050   \n",
       "11                  3                         George_W._Bush  2584078   \n",
       "12                  9                       K.G.F:_Chapter_1  1790058   \n",
       "13                  5                             Nick_Jonas  2346279   \n",
       "14                  4      Spider-Man:_Into_the_Spider-Verse  1840563   \n",
       "15                  4                         Penny_Marshall  1559346   \n",
       "16                  3                           Jimmy_Carter  1815966   \n",
       "17                  4                        Macaulay_Culkin  1518368   \n",
       "18                  3  List_of_highest-grossing_Indian_films  1580445   \n",
       "19                  3                   Mary,_Queen_of_Scots  1364237   \n",
       "\n",
       "                                    title named_entities  \\\n",
       "0                       George H. W. Bush         PERSON   \n",
       "1                             Jason Momoa         PERSON   \n",
       "2                              2.0 (film)       CARDINAL   \n",
       "3                         Bird Box (film)         PERSON   \n",
       "4                               Main Page                  \n",
       "5                          Aquaman (film)                  \n",
       "6                                Bird Box         PERSON   \n",
       "7                         Priyanka Chopra         PERSON   \n",
       "8    List of most-disliked YouTube videos            ORG   \n",
       "9                         Freddie Mercury            ORG   \n",
       "10                       Zero (2018 film)  CARDINAL DATE   \n",
       "11                         George W. Bush         PERSON   \n",
       "12                       K.G.F: Chapter 1            LAW   \n",
       "13                             Nick Jonas         PERSON   \n",
       "14      Spider-Man: Into the Spider-Verse                  \n",
       "15                         Penny Marshall         PERSON   \n",
       "16                           Jimmy Carter         PERSON   \n",
       "17                        Macaulay Culkin         PERSON   \n",
       "18  List of highest-grossing Indian films           NORP   \n",
       "19                   Mary, Queen of Scots                  \n",
       "\n",
       "                      title_parsed  \n",
       "0                       georg bush  \n",
       "1                      jason momoa  \n",
       "2                             film  \n",
       "3                    bird box film  \n",
       "4                        main page  \n",
       "5                     aquaman film  \n",
       "6                         bird box  \n",
       "7                  priyanka chopra  \n",
       "8         list dislik youtub video  \n",
       "9                   freddi mercuri  \n",
       "10                       zero film  \n",
       "11                      georg bush  \n",
       "12                         chapter  \n",
       "13                       nick jona  \n",
       "14          spider man spider vers  \n",
       "15                   penni marshal  \n",
       "16                    jimmi carter  \n",
       "17                 macaulay culkin  \n",
       "18  list highest gross indian film  \n",
       "19                 mari queen scot  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# unpickle\n",
    "with open(\"pickles/en_1218_louvain_communities_for_NLP_proc_2.pkl\", 'rb') as picklefile: \n",
    "    louvain_communities_for_NLP_proc = pickle.load(picklefile)\n",
    "\n",
    "louvain_communities_for_NLP_proc.head(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.4 Scaled traffic weights\n",
    "\n",
    "In the current dataset, each article shows up once. If we treat each article as a document for further bag-of-words parsing, it will reflect the words in the network of unique article titles, but it will not capture the differences in traffic to different articles.  \n",
    "We've identified our article communities using community detection with edges weighted by traffic, so that how much traffic goes to the articles is an important factor in the definition of article communities. To capture the differences in traffic, we'll use a weight based on the traffic. Ideally, we could repeat each article title in the dataset the number of times it was viewed, and then continue to bag-of-words parsing. But the dataset is too large to do that, and for the purposes of this project we only need the relative order of the word frequencies. So instead, we can take the log of the traffic weights to get smaller and more gradual scaled weights that are feasible to use here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "louvain_communities_for_NLP_proc[\"scaled_weight\"] = np.log(louvain_communities_for_NLP_proc.weight)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>louvain_community</th>\n",
       "      <th>title_raw</th>\n",
       "      <th>weight</th>\n",
       "      <th>title</th>\n",
       "      <th>named_entities</th>\n",
       "      <th>title_parsed</th>\n",
       "      <th>scaled_weight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3</td>\n",
       "      <td>George_H._W._Bush</td>\n",
       "      <td>5690673</td>\n",
       "      <td>George H. W. Bush</td>\n",
       "      <td>PERSON</td>\n",
       "      <td>georg bush</td>\n",
       "      <td>15.55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4</td>\n",
       "      <td>Jason_Momoa</td>\n",
       "      <td>4183872</td>\n",
       "      <td>Jason Momoa</td>\n",
       "      <td>PERSON</td>\n",
       "      <td>jason momoa</td>\n",
       "      <td>15.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9</td>\n",
       "      <td>2.0_(film)</td>\n",
       "      <td>3722311</td>\n",
       "      <td>2.0 (film)</td>\n",
       "      <td>CARDINAL</td>\n",
       "      <td>film</td>\n",
       "      <td>15.13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Bird_Box_(film)</td>\n",
       "      <td>3945404</td>\n",
       "      <td>Bird Box (film)</td>\n",
       "      <td>PERSON</td>\n",
       "      <td>bird box film</td>\n",
       "      <td>15.19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>Main_Page</td>\n",
       "      <td>3051841</td>\n",
       "      <td>Main Page</td>\n",
       "      <td></td>\n",
       "      <td>main page</td>\n",
       "      <td>14.93</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   louvain_community          title_raw   weight              title  \\\n",
       "0                  3  George_H._W._Bush  5690673  George H. W. Bush   \n",
       "1                  4        Jason_Momoa  4183872        Jason Momoa   \n",
       "2                  9         2.0_(film)  3722311         2.0 (film)   \n",
       "3                  4    Bird_Box_(film)  3945404    Bird Box (film)   \n",
       "4                  1          Main_Page  3051841          Main Page   \n",
       "\n",
       "  named_entities   title_parsed  scaled_weight  \n",
       "0         PERSON     georg bush          15.55  \n",
       "1         PERSON    jason momoa          15.25  \n",
       "2       CARDINAL           film          15.13  \n",
       "3         PERSON  bird box film          15.19  \n",
       "4                     main page          14.93  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "louvain_communities_for_NLP_proc.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "#### 3.5 Bags of words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "To get a sense of what each Wikipedia articles community is about, we can make simple bags of words and named entities for each community (weighted by traffic), and then take the top N most frequent words and named entities as a descriptor for each community."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Make community descriptors for all communities:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[3, 4, 9, 1, 5]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "louvain_community_ids = louvain_communities_for_NLP_proc.louvain_community.unique().tolist()\n",
    "louvain_community_ids[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1698"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(louvain_community_ids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Started running at 2019-03-03 23:47:18.127757 UTC\n",
      "Communities processed: 0.0 %, count =  0\n",
      "Elapsed time: 4.5866 min\n",
      "\n",
      "Last community's processing vars:\n",
      "cid= 3 \n",
      "community_topics_dict entry:\n",
      " {'topic_words': ['list', 'unit', 'state', 'footbal', 'elect', 'nation', 'th', 'constitu', 'parti', 'battl', 'john', 'film', 'st', 'district', 'war', 'cup', 'disambigu', 'india', 'leagu', 'world'], 'topic_words_weights': [86275, 62493, 55433, 49587, 41775, 39424, 30185, 29297, 26282, 23894, 23292, 22708, 22624, 20653, 19766, 17757, 16889, 16388, 15638, 15583], 'topic_ner': ['PERSON', 'ORG', 'GPE', 'DATE', 'NORP', 'CARDINAL', 'EVENT', 'ORDINAL', 'LOC', 'FAC', 'PRODUCT', 'LAW', 'WORK_OF_ART', 'LANGUAGE', 'QUANTITY', 'MONEY', 'TIME', 'PERCENT'], 'topic_ner_weights': [1088529, 559783, 426692, 171238, 112900, 43759, 37994, 30215, 28239, 14446, 5942, 3857, 3657, 2923, 1980, 811, 496, 112]}\n",
      "Communities processed: 5.8893 %, count =  100\n",
      "Elapsed time: 24.535 min\n",
      "\n",
      "Last community's processing vars:\n",
      "cid= 400 \n",
      "community_topics_dict entry:\n",
      " {'topic_words': ['innoc', 'mission', 'album', 'squar', 'sun', 'cathol', 'high', 'lancast', 'school', 'discographi', 'bird', 'neighborhood', 'umbrella', 'glow', 'plan', 'small', 'befriend', 'canada', 'lake', 'purvi'], 'topic_words_weights': [29, 29, 16, 6, 6, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 4, 4, 4], 'topic_ner': ['GPE', 'ORG', 'PERSON'], 'topic_ner_weights': [18, 15, 9]}\n",
      "Communities processed: 11.7786 %, count =  200\n",
      "Elapsed time: 24.5613 min\n",
      "\n",
      "Last community's processing vars:\n",
      "cid= 1411 \n",
      "community_topics_dict entry:\n",
      " {'topic_words': ['loudermil', 'hear', 'board', 'cleveland', 'educ', 'letter'], 'topic_words_weights': [18, 7, 6, 6, 6, 5], 'topic_ner': ['GPE', 'ORG'], 'topic_ner_weights': [18, 6]}\n",
      "Communities processed: 17.6678 %, count =  300\n",
      "Elapsed time: 24.5798 min\n",
      "\n",
      "Last community's processing vars:\n",
      "cid= 1280 \n",
      "community_topics_dict entry:\n",
      " {'topic_words': ['comic', 'doctor', 'stori', 'list', 'tenth', 'eleventh', 'ninth', 'twelfth', 'eighth', 'seventh', 'fifth', 'second', 'sixth', 'fourth', 'strip'], 'topic_words_weights': [59, 59, 55, 7, 5, 5, 5, 5, 4, 4, 4, 4, 4, 4, 4], 'topic_ner': ['ORG', 'ORDINAL'], 'topic_ner_weights': [33, 19]}\n",
      "Communities processed: 23.5571 %, count =  400\n",
      "Elapsed time: 24.5962 min\n",
      "\n",
      "Last community's processing vars:\n",
      "cid= 663 \n",
      "community_topics_dict entry:\n",
      " {'topic_words': ['charlott', 'church', 'ep', 'album', 'song', 'dream', 'discographi', 'darren', 'tate', 'enchant', 'scratch', 'issu', 'tissu', 'chick', 'crazi', 'brave', 'new', 'opera', 'world', 'best'], 'topic_words_weights': [49, 49, 20, 12, 10, 7, 7, 6, 6, 6, 6, 6, 6, 5, 5, 5, 5, 5, 5, 5], 'topic_ner': ['ORG', 'PERSON', 'CARDINAL'], 'topic_ner_weights': [49, 21, 20]}\n",
      "Communities processed: 29.4464 %, count =  500\n",
      "Elapsed time: 24.6117 min\n",
      "\n",
      "Last community's processing vars:\n",
      "cid= 567 \n",
      "community_topics_dict entry:\n",
      " {'topic_words': ['constitu', 'sabha', 'vidhan', 'karimganj', 'patharkandi', 'das', 'mallah', 'lok', 'laskar', 'ratabari', 'biswa', 'radheshyam', 'katlicherra', 'hailakandi', 'south', 'assam', 'badarpur', 'north', 'kripanath', 'nihar'], 'topic_words_weights': [44, 44, 37, 17, 11, 10, 8, 7, 7, 6, 6, 6, 5, 5, 5, 5, 5, 5, 5, 4], 'topic_ner': ['PERSON', 'ORG', 'GPE'], 'topic_ner_weights': [53, 17, 5]}\n",
      "Communities processed: 35.3357 %, count =  600\n",
      "Elapsed time: 24.6276 min\n",
      "\n",
      "Last community's processing vars:\n",
      "cid= 381 \n",
      "community_topics_dict entry:\n",
      " {'topic_words': ['keshoraipatan', 'keshav', 'rai', 'templ', 'baboo', 'lal', 'verma'], 'topic_words_weights': [6, 5, 5, 5, 5, 5, 5], 'topic_ner': ['PERSON'], 'topic_ner_weights': [10]}\n",
      "Communities processed: 41.225 %, count =  700\n",
      "Elapsed time: 24.6407 min\n",
      "\n",
      "Last community's processing vars:\n",
      "cid= 362 \n",
      "community_topics_dict entry:\n",
      " {'topic_words': ['album', 'bonapart', 'band', 'hors', 'like', 'open', 'sorri'], 'topic_words_weights': [10, 9, 6, 3, 3, 3, 3], 'topic_ner': [], 'topic_ner_weights': []}\n",
      "Communities processed: 47.1143 %, count =  800\n",
      "Elapsed time: 24.6557 min\n",
      "\n",
      "Last community's processing vars:\n",
      "cid= 1460 \n",
      "community_topics_dict entry:\n",
      " {'topic_words': ['nilkanth', 'vinodini', 'neelkanth', 'ramanbhai', 'vidyagauri', 'mahipatram', 'rupram'], 'topic_words_weights': [18, 6, 6, 6, 6, 6, 6], 'topic_ner': ['PERSON', 'ORG'], 'topic_ner_weights': [18, 6]}\n",
      "Communities processed: 53.0035 %, count =  900\n",
      "Elapsed time: 24.67 min\n",
      "\n",
      "Last community's processing vars:\n",
      "cid= 1056 \n",
      "community_topics_dict entry:\n",
      " {'topic_words': ['los', 'ciro', 'piojo', 'persa', 'andré', 'martínez', 'gustavo', 'kupinski', 'album', 'ay'], 'topic_words_weights': [10, 10, 5, 5, 5, 5, 4, 4, 3, 3], 'topic_ner': ['PERSON', 'ORG', 'GPE'], 'topic_ner_weights': [9, 5, 5]}\n",
      "Communities processed: 58.8928 %, count =  1000\n",
      "Elapsed time: 24.6828 min\n",
      "\n",
      "Last community's processing vars:\n",
      "cid= 820 \n",
      "community_topics_dict entry:\n",
      " {'topic_words': ['costin', 'miron', 'grigor', 'urech', 'ion', 'neculc'], 'topic_words_weights': [5, 5, 5, 5, 5, 5], 'topic_ner': ['PERSON'], 'topic_ner_weights': [15]}\n",
      "Communities processed: 64.7821 %, count =  1100\n",
      "Elapsed time: 24.6951 min\n",
      "\n",
      "Last community's processing vars:\n",
      "cid= 845 \n",
      "community_topics_dict entry:\n",
      " {'topic_words': ['music', 'slovak', 'slovakia', 'popular', 'folk', 'bella', 'ján', 'levoslav'], 'topic_words_weights': [14, 9, 5, 5, 4, 4, 4, 4], 'topic_ner': ['GPE', 'PERSON'], 'topic_ner_weights': [5, 4]}\n",
      "Communities processed: 70.6714 %, count =  1200\n",
      "Elapsed time: 24.707 min\n",
      "\n",
      "Last community's processing vars:\n",
      "cid= 86 \n",
      "community_topics_dict entry:\n",
      " {'topic_words': ['big', 'flunk', 'stanford', 'time', 'wong', 'genius', 'girl', 'millic', 'min', 'lisa', 'yee', 'eber', 'emili', 'total'], 'topic_words_weights': [5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 4, 4, 4], 'topic_ner': ['PERSON'], 'topic_ner_weights': [15]}\n",
      "Communities processed: 76.5607 %, count =  1300\n",
      "Elapsed time: 24.7197 min\n",
      "\n",
      "Last community's processing vars:\n",
      "cid= 436 \n",
      "community_topics_dict entry:\n",
      " {'topic_words': ['alamo', 'juan', 'pharr', 'san', 'school', 'high', 'district', 'independ', 'north', 'memori', 'southwest'], 'topic_words_weights': [20, 20, 20, 20, 20, 16, 4, 4, 4, 4, 4], 'topic_ner': [], 'topic_ner_weights': []}\n",
      "Communities processed: 82.4499 %, count =  1400\n",
      "Elapsed time: 24.7341 min\n",
      "\n",
      "Last community's processing vars:\n",
      "cid= 1342 \n",
      "community_topics_dict entry:\n",
      " {'topic_words': ['elect', 'general', 'puerto', 'rican'], 'topic_words_weights': [55, 55, 55, 55], 'topic_ner': ['NORP', 'DATE'], 'topic_ner_weights': [55, 50]}\n",
      "Communities processed: 88.3392 %, count =  1500\n",
      "Elapsed time: 24.7483 min\n",
      "\n",
      "Last community's processing vars:\n",
      "cid= 608 \n",
      "community_topics_dict entry:\n",
      " {'topic_words': ['footbal', 'hurrican', 'miami', 'team'], 'topic_words_weights': [37, 37, 37, 37], 'topic_ner': ['ORG', 'DATE', 'CARDINAL'], 'topic_ner_weights': [37, 33, 4]}\n",
      "Communities processed: 94.2285 %, count =  1600\n",
      "Elapsed time: 24.7643 min\n",
      "\n",
      "Last community's processing vars:\n",
      "cid= 332 \n",
      "community_topics_dict entry:\n",
      " {'topic_words': ['badger', 'footbal', 'team', 'wisconsin'], 'topic_words_weights': [71, 71, 71, 71], 'topic_ner': ['ORG', 'DATE', 'CARDINAL'], 'topic_ner_weights': [71, 49, 10]}\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "                <audio controls=\"controls\" autoplay=\"autoplay\">\n",
       "                    <source src=\"data:audio/wav;base64,UklGRpgiAABXQVZFZm10IBAAAAABAAEARKwAAIhYAQACABAAZGF0YXQiAAAAAF8QeSALMNI+kkwPWRZkeG0Odbd6XH7uf2Z/xnwZeHRx8Wi1XupSw0V3N0EoYhgcCLX3cOeT11/IFrryrCyh9JZ3jteHMIOWgBOAq4FWhQWLoJIHnBOnlLNXwSHQtN/Q7y4AjRCmIDYw+z63TDFZM2SQbSB1xHpkfu9/YX+8fAl4XnHWaJVex1KcRUw3FCg0GO0Hh/dD52fXNcjvuc+sDaHalmGOxoclg5GAFYCzgWSFGIu5kiScNKe5s3/BTNDi3/7vXQC7ENMgYTAjP9xMUllQZKhtM3XRemt+8X9df7F8+XdIcbtodl6jUnVFIjfoJwYYvwdY9xXnOtcLyMi5q6ztoL+WTI62hxuDjYAXgLqBcYUri9GSQpxWp9+zqMF30A/gLPCMAOkQACGMMEw/Ak1zWW1kwG1Gdd56cn7yf1h/pnzpdzNxoGhWXn9STUX4Nrwn2BeQByr35+YO1+HHobmIrM6gpZY2jqeHEYOJgBiAwoF/hT+L6ZJfnHenBLTRwaPQPOBb8LoAGBEtIbcwdT8nTZVZimTYbVl163p5fvN/U3+cfNh3HXGFaDdeXFImRc42jyeqF2IH+/a55uLWt8d6uWSsr6CKliGOl4cHg4SAGoDKgYyFUosCk3ycmacqtPrBztBp4Inw6QBGEVoh4zCdP0xNtlmnZPBta3X4eoF+9X9Pf5F8yHcHcWpoF144Uv9EpDZjJ3wXMwfM9ozmttaNx1O5QayQoHCWDI6Hh/2CgIAcgNKBmoVlixqTmpy7p1C0I8L60Jbgt/AYAXQRiCEOMcY/cU3XWcRkCG5+dQV7iH72f0p/hny3d/FwT2j3XRRS2ER6NjYnThcEB572XuaJ1mTHLLkerHGgVZb2jXeH84J8gB6A2oGohXiLM5O3nN2ndbRLwiXRxODm8EYBoxG1ITkx7j+XTflZ4GQgbpB1EnuPfvd/RX98fKd323A0aNhd8FGwRE82CicgF9YGb/Yw5l3WOscFufurUqA7luGNZ4fpgniAIIDhgbWFjItLk9Wc/6ebtHTCUdHx4BTxdQHREeIhZDEXQLxNGlr9ZDhuo3Ufe5Z++H9Af3F8lnfFcBlouF3MUYlEJTbdJvMWpwZB9gLmMdYQx96416szoCGWzI1Yh9+CdIAigOmBw4Wfi2ST8pwhqMG0ncJ80R7hQ/GkAf8RDyKPMT9A4U07WhplT261dSx7nH75fzt/ZnyGd69w/meYXahRYUT7NbEmxRZ4BhL21eUF1ubGt7i0qxSgB5a3jUiH1YJwgCSA8oHRhbKLfZMQnUOo57TGwqjRTOFx8dIBLRI8IroxZ0AGTlxaNmVnbsd1OHujfvp/Nn9bfHV3mXDjZ3hdhFE6RNA1hCaXFkoG5PWn5dnVvMaQuJGr9Z/slaKNOYfLgm2AJoD6gd+FxouWky2dZagMte/C09F54Z/xAQJbEmki5TGQQCtOfVpTZX9u2nVFe6p++n8wf1B8ZHeCcMdnWF1gURJEpjVYJmkWGwa19XnlrdWTxmq4bqvWn9KVjY0ph8KCaYApgAKC7YXai6+TS52HqDK1GMP/0abhzvEwAooSliIQMrhAUE6eWm9llm7sdVJ7sX77fyt/RHxTd2xwrGc4XTxR60N7NSsmOxbtBYf1S+WB1WnGQ7hLq7efuJV4jRqHuIJlgCuACoL7he2Lx5NpnamoWLVBwyrS1OH88V8CuBLDIjsy4EB1Tr9ajGWubv51Xnu3fvx/Jn85fEJ3VnCRZxhdGFHDQ1E1/yUNFr4FWPUe5VXVP8YcuCirmJ+elWSNC4evgmKALYASggmGAYzgk4edy6h+tWvDVtIB4ivyjQLmEu8iZjIJQZpO4FqoZcVuEHZqe75+/H8gfy58MXc/cHVn+Fz0UJtDJzXSJd8VjwUp9fDkKdUWxva3Bat6n4SVT437hqWCXoAwgBuCF4YVjPmTpJ3tqKS1lMOB0i/iWfK8AhQTHCORMjFBvk4BW8Vl3G4idnd7xH79fxt/I3wgdylwWmfYXNBQdEP8NKYlsRVhBfv0w+T91OzFz7fiqlufapU6jeyGnIJbgDKAI4ImhiiMEpTCnQ+pyrW9w63SXOKH8usCQhNJI7wyWUHjTiJb4WX0bjR2g3vLfv1/FX8XfA93EnA+Z7hcrFBMQ9I0eSWDFTIFzPSV5NHUwsWpt7+qPZ9RlSWN3YaSgleANYAsgjSGPIwslOCdMqnwtebD2dKJ4rbyGQNxE3Yj5zKBQQhPQlv9ZQtvRnaPe9F+/n8Qfwx8/nb8byJnmFyHUCRDpzRMJVUVAwWe9GfkpdSZxYK3naoenzeVEY3OhomCVIA3gDSCQoZQjEWU/p1UqRa2D8QF07fi5PJIA58ToyMRM6lBLU9jWxlmIm9Ydpt7137+fwp/AHztduVvB2d3XGNQ/UJ8NCAlJxXVBG/0OuR51G/FXLd6qv+eHZX8jL+GgIJRgDqAPYJRhmSMXpQcnnapPLY5xDDT5OIT83cDzRPQIzwz0UFRT4RbNmY5b2l2qHvefv5/BH/1e9t2zm/rZldcP1DVQlI08yT5FKYEQfQM5E3URsU1t1eq4Z4DleiMsIZ3gk2APYBGgl+GeIx3lDuemaljtmLEXNMS40HzpQP7E/0jZzP5QXZPpFtSZlBve3a0e+R+/n//ful7yna4b89mN1waUK1CJzTGJMoUdwQS9N/jIdQcxQ+3NKrDnuqU04yhhm6CSoBAgE6CboaNjJGUWZ67qYm2i8SI0z/jcPPUAykUKiSSMyFCm0/FW25mZ2+NdsB76n7+f/l+3Xu5dqFvs2YWXPZPhUL9M5kknBRJBOTzseP10/PE6bYSqqSe0JS/jJOGZYJHgEOAV4J9hqGMqpR3nt6pr7a1xLTTbeOe8wMEVxRWJL0zSUK/T+ZbimZ+b552y3vwfv9/837Re6d2im+YZvZb0U9dQtIzbSRuFBoEtfOE48nTycTCtu+php63lKuMhIZcgkSARoBggouGtYzElJWeAKrVtt7E39Oa483zMQSFFIMk5zNxQuRPBlymZpVvsHbXe/Z+/n/tfsV7lnZzb3xm1VutTzVCpzNAJEAU6wOH81bjntOgxJy2zalonp2Ul4x1hlOCQYBJgGmCmobJjN2Us54jqvy2B8UL1Mjj+/NgBLMUsCQSNJlCCFAmXMFmrG/BduN7/H7+f+d+unuEdlxvYGa1W4hPDUJ8MxMkEhS9A1jzKeNy03fEdraqqUqehJSCjGeGSoI+gEyAcoKpht6M95TSnkaqIrcxxTfU9eMq9I8E4RTdJD00wUIsUEdc3WbDb9N273sCf/5/4X6ue3J2RW9EZpRbZE/lQVIz5iPkE44DKvP74kbTTcRQtoipK55rlG6MWIZBgjuAT4B7griG8owQlfCeaKpJt1rFY9Qj5Fj0vQQQFQklZzTpQlFQZ1z5Ztpv5Hb6ewd//n/bfqF7YXYubydmc1s/T71BJzO5I7YTXwP88s7iGtMkxCm2ZakNnlGUWoxKhjmCOYBSgIWCx4YHjSqVD5+Lqm+3hMWP1FHkh/TsBD4VNiWSNBBDdVCHXBVn8G/1dgZ8DX/+f9R+lXtPdhdvC2ZTWxpPlUH8Mo0jiBMxA83yoOLv0vvDA7ZDqe+dOJRGjDuGMII2gFWAjoLWhhuNRJUtn66qlbetxbvUfuS19BsFbBVjJbw0OEOZUKhcMGcHcAZ3EXwTf/1/zn6Jez12/27vZTJb9k5tQdEyYCNaEwIDn/Jz4sPS0sPdtSGp0Z0flDKMLYYngjOAWYCXguWGMI1elUyf0aq8t9fF59Ss5OT0SQWaFY8l5zRgQ75QyFxMZx5wGHcdfBh//X/Ifn17K3bobtNlEVvRTkVBpjIzIysT0wJw8kXil9Kow7e1/qiznQaUH4wehh+CMYBcgKCC9IZEjXeVap/0quO3AcYT1dnkEvV4BcgVvCURNYhD4lDoXGdnNHApdyh8Hn/9f8F+cXsZdtFutmXwWqxOHUF7MgYj/RKlAkLyGOJs0n/DkbXcqJad7ZMLjBCGF4IugGCAqoIDh1mNkZWJnxerCbgqxj/VB+VB9acF9hXoJTw1r0MGUQhdg2dKcDp3NHwjf/x/u35kewd2uW6aZdBah070QFEy2SLPEnYCE/Lq4UDSVsNrtbqoeJ3Uk/eLAoYOgiyAY4CzghKHbo2rlaifOqswuFTGa9U15W/11QUkFhUmZjXXQypRKF2eZ2FwS3c/fCh/+3+0flh79XWibn5lr1piTsxAJjKsIqESRwLl8b3hFNItw0W1mKhanbuT44v0hQaCKoBngL2CIoeDjcWVx59dq1a4fsaX1WLlnvUEBlIWQiaRNf5DTlFIXbpnd3Bcd0p8Ln/7f61+S3vjdYpuYWWOWj1OpED7MX8icxIYArfxkOHp0QTDH7V2qDydopPQi+aF/oEngGuAx4Ixh5iN35Xln4CrfbinxsPVkOXM9TMGgBZuJrs1JkRyUWhd1WeNcGx3VXwzf/p/p34/e9F1c25FZW1aGE58QNAxUiJEEuoBiPFi4b3R28L5tFSoH52Jk7yL2IX2gSWAboDQgkGHrY35lQSgo6ukuNHG79W+5fv1YQauFpsm5TVORJZRiF3wZ6RwfXdgfDh/+X+gfjJ7vnVbbihlTFrzTVNApTElIhYSuwFa8TXhktGywtS0MqgBnXGTqYvKhe2BI4BygNqCUIfCjRSWI6DGq8u4+8Yb1uvlKfaQBtwWxyYQNnVEulGoXQxounCOd2t8PX/4f5l+JXusdURuC2UqWs5NK0B6Mfgh6BGMASvxCOFm0YnCrrQQqOOcWJOVi7yF5YEhgHaA5IJgh9eNLpZCoOmr8bglx0fWGeZY9r4GChf0Jjo2nETeUchdJ2jQcJ93dnxCf/d/kn4Ze5l1LG7vZAlaqU0CQE4xyyG6EV4B/fDa4DvRYMKItO6nxpw/k4KLroXdgR+AeoDugm+H7I1IlmGgDKwYuU/Hc9ZH5of27QY3FyAnZDbERAJS511CaOZwr3eBfEd/9n+Lfgx7h3UUbtJk6FmETdo/IzGeIYsRLwHP8K3gD9E3wmK0zKeonCeTb4uhhdaBHYB+gPiCf4cBjmOWgKAvrD+5eceg1nXmtfYcB2UXTSePNutEJlIHXl1o/HDAd4x8TH/1f4R+/3p1dfxttWTHWV9NsT/4MHEhXREAAaDwgODk0A7CPbSqp4ucDpNbi5OFzoEbgIKAAoOPhxaOfZafoFOsZrmix8zWoubk9koHkxd5J7k2E0VKUideeGgScdB3lnxRf/R/fX7yemJ15G2YZKVZOk2JP80wRCEvEdIAcvBT4LjQ5cEXtIinbpz2kkiLhYXGgRmAh4AMg5+HLI6Xlr6gdqyNuczH+NbQ5hL3eQfBF6Un4zY6RW1SRl6TaChx4HehfFZ/8392fuV6T3XMbXtkhFkUTWA/ojAXIQERowBE8CXgjdC8wfKzZ6dQnN2SNYt4hb6BGICLgBaDr4dBjrKW3qCarLS59sck1/7mQfeoB+8X0icNN2FFkVJmXq5oPXHxd6x8Wn/yf29+2Ho9dbRtX2RjWe9MOD93MOog0hB0ABXw+N9i0JTBzLNFpzOcxZIii2uFt4EWgI+AIIO+h1aOzZb9oL2s27kgyFDXLOdv99YHHRj+Jzc3iEW1UoVeyWhTcQF4tnxff/B/Z37Leip1nG1CZEFZykwPP0swvSCkEEYA5+/L3zbQa8GnsyOnFpyskg+LXYWvgRSAk4Aqg86HbI7nlhyh4KwCukrIfdda5573BQhLGCsoYTevRdhSpV7jaGlxEXjBfGR/739gfr56F3WEbSRkIFmkTOc+IDCPIHYQFwC5757fC9BCwYGzAqf5m5SS/IpQhaiBE4CYgDWD34eCjgKXPKEErSm6dMip14fnzfczCHkYVyiMN9dF/FLEXv5ofnEheMt8aH/tf1h+sHoEdWxtB2T+WH9Mvj71L2IgRxDp/4rvcd/gzxnBXLPgptybfJLpikKFoIERgJyAP4Pvh5eOHZdboSitUbqfyNXXtef792IIphiDKLY3/kUgU+ReGWmUcTJ41nxtf+x/UX6jevF0VG3qY91YWUyVPsovNSAZELr/XO9D37XP8cA2s7+mvptkktaKNYWZgRCAoYBKg/+HrY43l3uhS614usnIAtjj5yr4kQjUGLAo4DclRkNTA18zaapxQnjgfHF/6n9JfpV63nQ7bc1ju1g0TGw+ni8IIOsPjP8u7xbfic/IwBGznaahm0ySw4oohZGBDoCmgFSDD4jDjlKXmqFvrZ+688gu2BHoWPi/CAIZ3CgKOExGZlMiX05pv3FReOp8dX/of0J+iHrLdCNtsGOZWA5MRD5zL9sfvA9d///u6d5ez6DA7LJ8poWbNJKxihuFioENgKqAX4MgiNiObZe6oZOtxrodyVvYP+iH+O4IMBkIKTQ4c0aKU0JfaWnUcWF49Hx5f+d/On57erh0Cm2SY3hY6UsbPkgvrR+ODy7/0e683jPPd8DGslumaJsckp6KDoWDgQyAr4BqgzCI7o6Il9mhtq3tukfJh9ht6Lb4HAleGTQpXjiaRq1TYV+DaepxcXj+fH5/5X8yfm16pXTybHVjVljDS/I9HC+AH2APAP+j7o/eCM9PwKGyOaZLmwSSi4oBhXyBC4C0gHSDQIgEj6OX+aHarRW7ccmz2Jvo5PhLCYsZYCmHOMFG0VOAX51p/3GBeAh9gn/jfyp+X3qRdNlsWGM0WJ5LyT3xLlMfMQ/R/nXuYt7dzibAfLIYpi6b7JF5ivSEdYEKgLmAf4NRiBqPvpcZov6tPLucyeDYyegT+XkJuRmNKbE46Eb0U59fuGkUcpF4En2Gf+F/I35Sen50wWw6YxJYeEugPcUuJh8DD6L+Ru413rLO/r9XsvelEZvUkWeK54RugQmAvoCKg2GIMI/ZlziiIq5ku8bJDNn26EL5qAnnGbkp2zgPRxdUvl/SaSlyoHgcfYp/338bfkR6a3SobB1j8FdSS3c9mi74HtUOdP4Y7gjehs7VvzKy1qX1mryRVIrbhGeBCIDDgJWDcohGj/SXWKJGrou78Mk52STpcPnXCRUa5SkFOTVHOlTdX+xpPnKweCZ9jn/dfxN+NnpXdI9s/2LOVyxLTj1uLssepg5F/urt291bzq2/DbK0pdiapZFCis6EYIEHgMiAoIODiFyPEJh4omqusrsbymXZUumf+QUKQhoRKi85XEddVPxfB2pTcr94MH2Sf9t/Cn4oekR0d2zhYqxXB0slPUMunh54Dhb+vO2u3TDOhL/osZOlu5qNkS+KwYRZgQaAzYCrg5SIc48rmJiijq7au0XKktmA6c35NApwGj0qWTmDR4BUG2Ahamhyz3g5fZV/2X8Cfhp6MHRebMRiilfhSvw8Fy5wHkkO6P2N7YHdBc5cv8OxcqWfmnaRHYq1hFOBBYDSgLaDpIiJj0aYuKKyrgK8b8q+2a7p/PliCp4aaSqCOapHo1Q5YDtqfXLeeEN9mX/Wf/p9DHoddEVspmJoV7tK0zzsLUMeGw65/V/tVN3azTS/nrFRpYKaXpELiqiETIEFgNiAwYO1iJ+PYpjYotauKbyayuvZ3Okr+pEKyxqVKqw50EfGVFhgVWqScu54TX2df9R/8n3+eQl0LGyIYkZXlUqqPMAtFh7tDYr9Me0n3a/NDL95sTClZppHkfmJnIRFgQSA3YDMg8aIto99mPii+q5RvMTKGNoK6ln6vwr5GsEq1jn3R+lUd2Bvaqdy/XhWfaB/0n/pffB59XMTbGpiJFdvSoE8lC3oHb4NW/0D7frchc3jvlSxEKVKmi+R54mPhD+BA4DigNiD14jMj5mYGKMer3i878pE2jjqiPruCicb7Sr/OR1IDFWWYIlqvHIMeWB9pH/Pf+F94nnhc/prTWICV0lKWDxpLbsdkA0t/dXszdxazbu+L7HvpC2aGJHViYOEOIEDgOiA44PoiOKPtJg4o0KvoLwZy3HaZuq3+hwLVBsZKyk6REgvVbRgomrQcht5aX2nf81/2X3Tec5z4WsvYt9WI0ouPD0tjR1hDf78puyg3C/Nk74Ksc6kEZoBkcOJd4QygQOA7YDvg/qI+Y/QmFijZ6/IvETLndqU6uX6SwuCG0UrUzprSFJV02C8auVyKnlyfat/yn/QfcV5unPIaxFivVb9SQU8ES1gHTMNz/x47HPcBM1rvuawraT1memQsYlrhCyBAoDzgPqDC4kQkOuYeaOLr/C8bsvK2sLqFPt5C68bcSt8OpFIdVXxYNZq+XI5eXt9rn/Hf8d9tnmmc69r82GbVtdJ3DvmLDIdBA2h/ErsR9zZzEO+wbCNpNmZ0pCfiV+EJYECgPmABoQciSaQB5mZo6+vF72Zy/fa8OpD+6gL3RudK6Y6t0iYVRBh8GoOc0h5hX2xf8V/v32oeZJzlWvVYXhWsEmzO7osBR3WDHL8HOwa3K7MG76csGykvJm7kI6JUoQfgQKA/oARhC2JPZAjmbmj1K8/vcPLI9sf63H71gsLHMkrzzreSLpVLmEJayJzV3mOfbR/wn+2fZl5fnN8a7ZhVlaKSYk7jizXHKgMQ/zu6+3bhMzzvXiwS6SgmaSQfIlGhBmBAoAEgR2EP4lUkD+Z2qP4r2e97stQ203roPsFDDgc9Sv5OgRJ3VVNYSNrN3NmeZd9t3+/f619i3lpc2NrmGEzVmRJYDtiLKoceQwV/MDrwNtZzMu9U7ArpISZjZBqiTuEE4ECgAqBKYRQiWuQWpn6oxywj70ZzH3be+vP+zMMZhwhLCI7K0kAVmthPGtLc3V5oH26f7x/pH18eVVzSWt6YRFWPkk3OzcsfBxLDOb7kuuT2y7Mo70vsAqkaJl2kFmJL4QNgQGAEIE1hGKJgpB2mRqkQbC3vUPMqtup6/37YgyTHEwsSztRSSJWiWFWa19zg3mpfb1/uX+bfW15QXMwa1xh7lUXSQ07CyxPHBwMt/tk62fbA8x7vQqw6qNNmV+QR4kjhAeBAoAWgUCEc4mZkJKZO6RlsN+9bszW29frLPyQDMEceCx1O3dJRVanYW9rc3OSebJ9wH+2f5J9X3ktcxZrPWHMVfFI5DrfKyEc7guJ+zbrOtvZy1O95q/JozGZSJA2iReEAYECgByBTISFibCQrplcpIqwB76ZzAPcBexb/L8M7hykLJ47nUlnVsVhiWuIc6F5un3Df7N/iX1QeRhz/WofYalVy0i6OrMr9Bu/C1r7B+sN267LK73Br6mjFZkykCWJC4T8gAKAIoFYhJeJx5DKmXykr7AvvsTMMNwz7In87QwcHdAsxzvESYpW5GGia5xzr3nDfcZ/r3+AfUF5BHPjagFhhlWkSJE6hyvGG5ELK/vZ6uDahMsDvZ2viaP5mBuQE4kAhPaAAoApgWWEqInekOeZnaTTsFe+78xd3GHsuPwcDUkd+yzxO+pJrFYCYrtrsHO+ecx9yX+sf3d9Mnnvcslq4mBjVX5IZzpbK5kbYgv9+qvqtNpZy9y8ea9oo96YBJACifSD8IACgC+BcYS6ifWQA5q+pPiwf74ZzYrcj+zn/EoNdx0nLRo8EErOViBi1GvEc8x51H3Lf6l/bn0jedtyr2rDYEFVV0g+Oi8raxs0C876feqH2i7LtLxUr0ijwpjuj/GI6YPrgAOANYF9hMyJDJEfmt6kHbGnvkTNt9y+7BX9eQ2kHVMtQzw2SvFWPmLua9hz2nndfc5/pX9kfRR5xnKWaqVgHlUxSBQ6Ays9GwULn/pP6lraBMuMvDCvKKOmmNeP4Ijdg+WAA4A8gYmE3okkkTua/6RCsc++b83k3OzsRP2nDdEdfy1sPFxKE1dcYgds63PpeeV90H+if1t9BXmxcnxqhmD7VApI6jnXKhAb1wpx+iHqLtrZymW8DK8Io4uYwY/PiNKD4IAEgEKBloTwiTuRWJogpWax976azRHdGu1z/dUN/x2qLZU8gko1V3liIGz/c/d57n3Tf55/UX31eJxyYmpoYNhU5EfBOasq4hqoCkL68+kB2q/KPbzoruiib5iqj76Ix4PagASASYGihAKKUpF0mkGli7Egv8XNPd1I7aH9BA4sHtYtvzyoSldXl2I5bBN0BXr2fdV/m39IfeZ4iHJIaklgtVS9R5c5fyq1GnkKE/rF6dXZhcoVvMSuyKJUmJSPrYi8g9WABYBPga6EFIpqkZGaYqWwsUi/8M1q3Xbt0P0yDloeAS7oPM5KeVe1YlFsJnQTev5913+Xfz5913hzci5qKmCSVJZHbTlTKocaSwrl+ZfpqNlayu67oK6oojmYfo+ciLCD0IAGgFaBu4QmioGRrZqDpdWxcL8bzpfdpe3//WEOhx4tLhE99EqbV9Niamw6dCF6Bn7af5N/NX3HeF5yFGoLYG9UcEdEOScqWRocCrb5ael82TDKxrt8roiiHZhnj4uIpYPKgAaAXYHIhDmKmZHKmqSl+rGZv0bOxN3T7S7+jw60HlguOj0ZS71X8GKDbE50L3oOftx/kH8rfbh4SXL5aexfTFRJRxo5+ykrGu4JiPk76U/ZBcqfu1iuaKICmFGPeoiag8WAB4BkgdSES4qxkeaaxaUfssG/cc7x3QHuXP69DuIehC5jPT9L31cOY5xsYXQ9ehd+3n+MfyF9qHg0ct9pzV8pVCJH8DjPKf4ZvwlZ+Q3pI9nbyXe7NK5IoueXO49qiI+DwIAIgGqB4YRdisiRA5vmpUSy6b+czh7eL+6L/uwODx+vLow9ZUsBWCtjtWx0dEt6H37gf4h/F32ZeB9yxWmuXwVU+0bGOKMp0BmRCSr54Oj22LHJULsQriiizJclj1mIhIO7gAmAcYHuhHCK4JEgmwemabISwMfOS95d7rr+Gg88H9sutT2LSyNYSWPNbIh0WHomfuJ/hH8NfYl4CnKraY9f4lPURpw4dymiGWIJ/Piy6MrYhskou+ytCaKxlw+PSYh6g7aACoB4gfuEgor4kTybKaaPsjrA8s543ozu6P5JD2ofBi/dPbBLRVhmY+Zsm3Rmei5+5H+AfwN9eXj0cZBpcF+/U61GczhKKXQZNAnN+ITondhcyQG7yK3poZaX+Y44iG+DsYALgH+BCIWVihCSWZtKprSyY8Adz6beuu4X/3cPlx8yLwY+1ktnWIRj/myudHR6Nn7mf3x/+XxpeN9xdmlRX5xThkZJOB4pRxkFCZ74Vuhx2DLJ2rqkrcmhe5fjjiiIZIOtgA2Ah4EVhaeKKJJ2m2um2bKLwEnP097o7kb/pQ/EH10vLz78S4lYoWMXbcF0gXo+fuh/d3/vfFl4ynFbaTJfeFNfRh848igZGdYIcPgo6ETYCMmzuoGtqqFgl82OF4hag6iADoCOgSKFuopAkpObjab+srTAdM8A3xfvdP/UD/EfiS9YPiFMqli+Yy9t1XSPekZ+6X9zf+V8Sni0cUFpE19VUzhG9TfGKOsYqAhB+PrnGNjeyIu6Xa2KoUWXuI4HiE+Do4APgJWBL4XNiliSsJuupiSz3cCfzy3fRe+j/wIQHiC0L4E+R0zMWNxjR23odJx6TX7rf29/23w6eJ9xJmnzXjFTEUbLN5kovRh5CBP4zOfs17TIZLo5rWuhKpeijveHRIOfgBGAnIE8heCKcJLNm8+mSbMFwcrPWt9z79L/MBBMIN8vqT5sTO1Y+WNgbft0qnpVfu1/an/QfCl4iXEMadReDlPqRaE3bSiQGEsI5Pee57/Xicg9uhatS6EPl4yO54c6g5qAEoCkgUmF8oqIkuqb8aZusy7B9c+H36HvAAA=\" type=\"audio/wav\" />\n",
       "                    Your browser does not support the audio element.\n",
       "                </audio>\n",
       "              "
      ],
      "text/plain": [
       "<IPython.lib.display.Audio object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Runtime: 24.78 min\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"Started running at\", datetime.now(), \"UTC\")\n",
    "start_time = timer()\n",
    "\n",
    "community_topics_dict = {}\n",
    "count = 0\n",
    "\n",
    "for cid in louvain_community_ids:\n",
    "    # get a community\n",
    "    comm = louvain_communities_for_NLP_proc[louvain_communities_for_NLP_proc.louvain_community == cid]\n",
    "    \n",
    "    # put together title and NER doc lists per community\n",
    "    nrows = len(comm)\n",
    "    title_docs = []\n",
    "    ner_docs = []\n",
    "\n",
    "    for i in range(nrows):\n",
    "        title_parsed_arr = comm.iloc[i].title_parsed.split()\n",
    "        named_entities_arr = comm.iloc[i].named_entities.split()\n",
    "        weight = max(round(comm.iloc[i].scaled_weight).astype(\"int64\"), 1)\n",
    "\n",
    "        for j in range(weight):\n",
    "            title_docs.append(title_parsed_arr)\n",
    "            ner_docs.append(named_entities_arr)\n",
    "    \n",
    "    # Make a gensim dictionary from title docs\n",
    "    titles_dictionary = gensim.corpora.Dictionary(title_docs)\n",
    "    # Get top 20 most frequent words used in the article titles within the given community\n",
    "    topic_word_ids = sorted(titles_dictionary.dfs, key=titles_dictionary.dfs.__getitem__, reverse=True)[:20]\n",
    "\n",
    "    # store the top 20 title words and their weighted frequencies \n",
    "    topic_words_arr = []\n",
    "    topic_words_weights = []\n",
    "    for wid in topic_word_ids:\n",
    "        topic_words_arr.append(titles_dictionary[wid])\n",
    "        topic_words_weights.append(titles_dictionary.dfs[wid])\n",
    "    \n",
    "    # Make a gensim dictionary from NER docs\n",
    "    ner_dictionary = gensim.corpora.Dictionary(ner_docs)\n",
    "    # Get top 20 most frequent NERs used in the article titles within the given community\n",
    "    topic_ner_ids = sorted(ner_dictionary.dfs, key=ner_dictionary.dfs.__getitem__, reverse=True)[:20]\n",
    "\n",
    "    # store the top 20 title named entities and their weighted frequencies \n",
    "    topic_ner_arr = []\n",
    "    topic_ner_weights = []\n",
    "    for nid in topic_ner_ids:\n",
    "        topic_ner_arr.append(ner_dictionary[nid])\n",
    "        topic_ner_weights.append(ner_dictionary.dfs[nid])\n",
    "    \n",
    "    # store the community topic arrays in a dict\n",
    "    community_topics_dict[cid] = {}\n",
    "    community_topics_dict[cid][\"topic_words\"] = topic_words_arr\n",
    "    community_topics_dict[cid][\"topic_words_weights\"] = topic_words_weights\n",
    "    community_topics_dict[cid][\"topic_ner\"] = topic_ner_arr\n",
    "    community_topics_dict[cid][\"topic_ner_weights\"] = topic_ner_weights\n",
    "    \n",
    "    \n",
    "    if (count % 100 == 0):\n",
    "        print(\"Communities processed:\", round(count * 100/len(louvain_community_ids), 4), \"%,\", \"count = \", count )\n",
    "        print(\"Elapsed time:\", round((timer() - start_time)/60, 4), \"min\\n\")\n",
    "        print(\"Last community's processing vars:\")\n",
    "        print(\"cid=\", cid, \"\\ncommunity_topics_dict entry:\\n\", community_topics_dict[cid])\n",
    "    count += 1\n",
    "    \n",
    "cu.printRunTime(start_time)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'topic_ner': ['ORG',\n",
       "  'PERSON',\n",
       "  'DATE',\n",
       "  'GPE',\n",
       "  'CARDINAL',\n",
       "  'EVENT',\n",
       "  'PRODUCT',\n",
       "  'NORP',\n",
       "  'TIME',\n",
       "  'ORDINAL',\n",
       "  'FAC',\n",
       "  'LOC',\n",
       "  'LAW',\n",
       "  'WORK_OF_ART',\n",
       "  'QUANTITY',\n",
       "  'LANGUAGE',\n",
       "  'PERCENT',\n",
       "  'MONEY'],\n",
       " 'topic_ner_weights': [65251,\n",
       "  64819,\n",
       "  23494,\n",
       "  14602,\n",
       "  11542,\n",
       "  11079,\n",
       "  5620,\n",
       "  5000,\n",
       "  1148,\n",
       "  953,\n",
       "  663,\n",
       "  585,\n",
       "  293,\n",
       "  229,\n",
       "  218,\n",
       "  55,\n",
       "  28,\n",
       "  4],\n",
       " 'topic_words': ['grand',\n",
       "  'prix',\n",
       "  'championship',\n",
       "  'seri',\n",
       "  'engin',\n",
       "  'race',\n",
       "  'car',\n",
       "  'formula',\n",
       "  'list',\n",
       "  'ford',\n",
       "  'honda',\n",
       "  'motorcycl',\n",
       "  'motor',\n",
       "  'world',\n",
       "  'automobil',\n",
       "  'toyota',\n",
       "  'driver',\n",
       "  'merced',\n",
       "  'bmw',\n",
       "  'benz'],\n",
       " 'topic_words_weights': [10329,\n",
       "  9949,\n",
       "  6733,\n",
       "  6195,\n",
       "  6132,\n",
       "  6073,\n",
       "  5593,\n",
       "  4185,\n",
       "  4161,\n",
       "  4154,\n",
       "  3680,\n",
       "  3466,\n",
       "  3441,\n",
       "  3335,\n",
       "  2688,\n",
       "  2684,\n",
       "  2427,\n",
       "  2388,\n",
       "  2303,\n",
       "  2291]}"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "community_topics_dict[22]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pickle created: pickles/en_1218_louvain_community_topics_dict.pkl\n"
     ]
    }
   ],
   "source": [
    "# pickle the output\n",
    "myoutfile = \"pickles/en_1218_louvain_community_topics_dict.pkl\"\n",
    "with open(myoutfile, 'wb') as picklefile:\n",
    "    pickle.dump(community_topics_dict, picklefile)\n",
    "\n",
    "print(\"Pickle created: \" + myoutfile)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'topic_ner': ['ORG',\n",
       "  'PERSON',\n",
       "  'DATE',\n",
       "  'GPE',\n",
       "  'CARDINAL',\n",
       "  'EVENT',\n",
       "  'PRODUCT',\n",
       "  'NORP',\n",
       "  'TIME',\n",
       "  'ORDINAL',\n",
       "  'FAC',\n",
       "  'LOC',\n",
       "  'LAW',\n",
       "  'WORK_OF_ART',\n",
       "  'QUANTITY',\n",
       "  'LANGUAGE',\n",
       "  'PERCENT',\n",
       "  'MONEY'],\n",
       " 'topic_ner_weights': [65251,\n",
       "  64819,\n",
       "  23494,\n",
       "  14602,\n",
       "  11542,\n",
       "  11079,\n",
       "  5620,\n",
       "  5000,\n",
       "  1148,\n",
       "  953,\n",
       "  663,\n",
       "  585,\n",
       "  293,\n",
       "  229,\n",
       "  218,\n",
       "  55,\n",
       "  28,\n",
       "  4],\n",
       " 'topic_words': ['grand',\n",
       "  'prix',\n",
       "  'championship',\n",
       "  'seri',\n",
       "  'engin',\n",
       "  'race',\n",
       "  'car',\n",
       "  'formula',\n",
       "  'list',\n",
       "  'ford',\n",
       "  'honda',\n",
       "  'motorcycl',\n",
       "  'motor',\n",
       "  'world',\n",
       "  'automobil',\n",
       "  'toyota',\n",
       "  'driver',\n",
       "  'merced',\n",
       "  'bmw',\n",
       "  'benz'],\n",
       " 'topic_words_weights': [10329,\n",
       "  9949,\n",
       "  6733,\n",
       "  6195,\n",
       "  6132,\n",
       "  6073,\n",
       "  5593,\n",
       "  4185,\n",
       "  4161,\n",
       "  4154,\n",
       "  3680,\n",
       "  3466,\n",
       "  3441,\n",
       "  3335,\n",
       "  2688,\n",
       "  2684,\n",
       "  2427,\n",
       "  2388,\n",
       "  2303,\n",
       "  2291]}"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# unpickle\n",
    "with open(\"pickles/en_1218_louvain_community_topics_dict.pkl\", 'rb') as picklefile: \n",
    "    community_topics_dict = pickle.load(picklefile)\n",
    "\n",
    "community_topics_dict[22]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### What are the topics of the top 20 communities?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 220,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>articles_count</th>\n",
       "      <th>external_search_traffic</th>\n",
       "      <th>link_edges_count</th>\n",
       "      <th>link_traffic</th>\n",
       "      <th>louvain_community</th>\n",
       "      <th>search_edges_count</th>\n",
       "      <th>search_traffic</th>\n",
       "      <th>total_visits</th>\n",
       "      <th>avg_external_search_traffic</th>\n",
       "      <th>avg_link_traffic_per_edge</th>\n",
       "      <th>avg_visits_per_article</th>\n",
       "      <th>link_network_density</th>\n",
       "      <th>link_network_density_delta</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>507642</td>\n",
       "      <td>535227964</td>\n",
       "      <td>3385462</td>\n",
       "      <td>329625267</td>\n",
       "      <td>3</td>\n",
       "      <td>102124</td>\n",
       "      <td>4188154</td>\n",
       "      <td>1124655652</td>\n",
       "      <td>1054.34</td>\n",
       "      <td>97.36</td>\n",
       "      <td>2215.45</td>\n",
       "      <td>0.00</td>\n",
       "      <td>5.67</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>302120</td>\n",
       "      <td>288453340</td>\n",
       "      <td>2037310</td>\n",
       "      <td>140416238</td>\n",
       "      <td>7</td>\n",
       "      <td>41383</td>\n",
       "      <td>1257742</td>\n",
       "      <td>592415339</td>\n",
       "      <td>954.76</td>\n",
       "      <td>68.92</td>\n",
       "      <td>1960.86</td>\n",
       "      <td>0.00</td>\n",
       "      <td>5.74</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>267541</td>\n",
       "      <td>626988532</td>\n",
       "      <td>2540296</td>\n",
       "      <td>461475849</td>\n",
       "      <td>4</td>\n",
       "      <td>98857</td>\n",
       "      <td>3938984</td>\n",
       "      <td>1350876078</td>\n",
       "      <td>2343.52</td>\n",
       "      <td>181.66</td>\n",
       "      <td>5049.23</td>\n",
       "      <td>0.00</td>\n",
       "      <td>8.50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>233396</td>\n",
       "      <td>150495954</td>\n",
       "      <td>1209989</td>\n",
       "      <td>84128861</td>\n",
       "      <td>10</td>\n",
       "      <td>25894</td>\n",
       "      <td>932688</td>\n",
       "      <td>322074785</td>\n",
       "      <td>644.81</td>\n",
       "      <td>69.53</td>\n",
       "      <td>1379.95</td>\n",
       "      <td>0.00</td>\n",
       "      <td>4.18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>223864</td>\n",
       "      <td>219874886</td>\n",
       "      <td>1327054</td>\n",
       "      <td>151652674</td>\n",
       "      <td>5</td>\n",
       "      <td>32332</td>\n",
       "      <td>930331</td>\n",
       "      <td>468611692</td>\n",
       "      <td>982.18</td>\n",
       "      <td>114.28</td>\n",
       "      <td>2093.29</td>\n",
       "      <td>0.00</td>\n",
       "      <td>4.93</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>218001</td>\n",
       "      <td>279571288</td>\n",
       "      <td>1593048</td>\n",
       "      <td>95420595</td>\n",
       "      <td>1</td>\n",
       "      <td>39273</td>\n",
       "      <td>1056534</td>\n",
       "      <td>1095599823</td>\n",
       "      <td>1282.43</td>\n",
       "      <td>59.90</td>\n",
       "      <td>5025.66</td>\n",
       "      <td>0.00</td>\n",
       "      <td>6.31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>209065</td>\n",
       "      <td>259036837</td>\n",
       "      <td>1333634</td>\n",
       "      <td>96546076</td>\n",
       "      <td>2</td>\n",
       "      <td>45927</td>\n",
       "      <td>1568306</td>\n",
       "      <td>503087845</td>\n",
       "      <td>1239.03</td>\n",
       "      <td>72.39</td>\n",
       "      <td>2406.37</td>\n",
       "      <td>0.00</td>\n",
       "      <td>5.38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>110397</td>\n",
       "      <td>117206803</td>\n",
       "      <td>700021</td>\n",
       "      <td>56716133</td>\n",
       "      <td>11</td>\n",
       "      <td>23456</td>\n",
       "      <td>972333</td>\n",
       "      <td>220630202</td>\n",
       "      <td>1061.68</td>\n",
       "      <td>81.02</td>\n",
       "      <td>1998.52</td>\n",
       "      <td>0.00</td>\n",
       "      <td>5.34</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>81807</td>\n",
       "      <td>72472648</td>\n",
       "      <td>420037</td>\n",
       "      <td>28116882</td>\n",
       "      <td>6</td>\n",
       "      <td>10319</td>\n",
       "      <td>320477</td>\n",
       "      <td>139408430</td>\n",
       "      <td>885.90</td>\n",
       "      <td>66.94</td>\n",
       "      <td>1704.11</td>\n",
       "      <td>0.00</td>\n",
       "      <td>4.13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>73172</td>\n",
       "      <td>37184350</td>\n",
       "      <td>542438</td>\n",
       "      <td>36767267</td>\n",
       "      <td>8</td>\n",
       "      <td>10775</td>\n",
       "      <td>286442</td>\n",
       "      <td>100143853</td>\n",
       "      <td>508.18</td>\n",
       "      <td>67.78</td>\n",
       "      <td>1368.61</td>\n",
       "      <td>0.00</td>\n",
       "      <td>6.41</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>68036</td>\n",
       "      <td>81907298</td>\n",
       "      <td>479675</td>\n",
       "      <td>44943883</td>\n",
       "      <td>13</td>\n",
       "      <td>11474</td>\n",
       "      <td>394931</td>\n",
       "      <td>163908794</td>\n",
       "      <td>1203.88</td>\n",
       "      <td>93.70</td>\n",
       "      <td>2409.15</td>\n",
       "      <td>0.00</td>\n",
       "      <td>6.05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>57734</td>\n",
       "      <td>49520060</td>\n",
       "      <td>366834</td>\n",
       "      <td>36611085</td>\n",
       "      <td>17</td>\n",
       "      <td>8766</td>\n",
       "      <td>306433</td>\n",
       "      <td>112475075</td>\n",
       "      <td>857.73</td>\n",
       "      <td>99.80</td>\n",
       "      <td>1948.16</td>\n",
       "      <td>0.00</td>\n",
       "      <td>5.35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>49144</td>\n",
       "      <td>19912183</td>\n",
       "      <td>154235</td>\n",
       "      <td>8664354</td>\n",
       "      <td>12</td>\n",
       "      <td>2870</td>\n",
       "      <td>81325</td>\n",
       "      <td>39492159</td>\n",
       "      <td>405.18</td>\n",
       "      <td>56.18</td>\n",
       "      <td>803.60</td>\n",
       "      <td>0.00</td>\n",
       "      <td>2.14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>44664</td>\n",
       "      <td>23130719</td>\n",
       "      <td>165676</td>\n",
       "      <td>8931956</td>\n",
       "      <td>21</td>\n",
       "      <td>5059</td>\n",
       "      <td>209524</td>\n",
       "      <td>43132483</td>\n",
       "      <td>517.88</td>\n",
       "      <td>53.91</td>\n",
       "      <td>965.71</td>\n",
       "      <td>0.00</td>\n",
       "      <td>2.71</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>42420</td>\n",
       "      <td>61816953</td>\n",
       "      <td>305945</td>\n",
       "      <td>34593538</td>\n",
       "      <td>9</td>\n",
       "      <td>14884</td>\n",
       "      <td>768763</td>\n",
       "      <td>116819777</td>\n",
       "      <td>1457.26</td>\n",
       "      <td>113.07</td>\n",
       "      <td>2753.88</td>\n",
       "      <td>0.00</td>\n",
       "      <td>6.21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>37983</td>\n",
       "      <td>40009679</td>\n",
       "      <td>286443</td>\n",
       "      <td>20695881</td>\n",
       "      <td>22</td>\n",
       "      <td>7970</td>\n",
       "      <td>229685</td>\n",
       "      <td>79225111</td>\n",
       "      <td>1053.36</td>\n",
       "      <td>72.25</td>\n",
       "      <td>2085.80</td>\n",
       "      <td>0.00</td>\n",
       "      <td>6.54</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>29397</td>\n",
       "      <td>8464822</td>\n",
       "      <td>127957</td>\n",
       "      <td>5057107</td>\n",
       "      <td>0</td>\n",
       "      <td>1350</td>\n",
       "      <td>29778</td>\n",
       "      <td>19688503</td>\n",
       "      <td>287.95</td>\n",
       "      <td>39.52</td>\n",
       "      <td>669.75</td>\n",
       "      <td>0.00</td>\n",
       "      <td>3.35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>24444</td>\n",
       "      <td>39844645</td>\n",
       "      <td>244682</td>\n",
       "      <td>34602434</td>\n",
       "      <td>15</td>\n",
       "      <td>7582</td>\n",
       "      <td>295427</td>\n",
       "      <td>90688876</td>\n",
       "      <td>1630.04</td>\n",
       "      <td>141.42</td>\n",
       "      <td>3710.07</td>\n",
       "      <td>0.00</td>\n",
       "      <td>9.01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>22194</td>\n",
       "      <td>6207271</td>\n",
       "      <td>77818</td>\n",
       "      <td>5019260</td>\n",
       "      <td>19</td>\n",
       "      <td>2126</td>\n",
       "      <td>105863</td>\n",
       "      <td>14822923</td>\n",
       "      <td>279.68</td>\n",
       "      <td>64.50</td>\n",
       "      <td>667.88</td>\n",
       "      <td>0.00</td>\n",
       "      <td>2.51</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>17954</td>\n",
       "      <td>10772752</td>\n",
       "      <td>85442</td>\n",
       "      <td>5217063</td>\n",
       "      <td>14</td>\n",
       "      <td>2613</td>\n",
       "      <td>69270</td>\n",
       "      <td>20881742</td>\n",
       "      <td>600.02</td>\n",
       "      <td>61.06</td>\n",
       "      <td>1163.07</td>\n",
       "      <td>0.00</td>\n",
       "      <td>3.76</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    articles_count  external_search_traffic  link_edges_count  link_traffic  \\\n",
       "0           507642                535227964           3385462     329625267   \n",
       "1           302120                288453340           2037310     140416238   \n",
       "2           267541                626988532           2540296     461475849   \n",
       "3           233396                150495954           1209989      84128861   \n",
       "4           223864                219874886           1327054     151652674   \n",
       "5           218001                279571288           1593048      95420595   \n",
       "6           209065                259036837           1333634      96546076   \n",
       "7           110397                117206803            700021      56716133   \n",
       "8            81807                 72472648            420037      28116882   \n",
       "9            73172                 37184350            542438      36767267   \n",
       "10           68036                 81907298            479675      44943883   \n",
       "11           57734                 49520060            366834      36611085   \n",
       "12           49144                 19912183            154235       8664354   \n",
       "13           44664                 23130719            165676       8931956   \n",
       "14           42420                 61816953            305945      34593538   \n",
       "15           37983                 40009679            286443      20695881   \n",
       "16           29397                  8464822            127957       5057107   \n",
       "17           24444                 39844645            244682      34602434   \n",
       "18           22194                  6207271             77818       5019260   \n",
       "19           17954                 10772752             85442       5217063   \n",
       "\n",
       "    louvain_community  search_edges_count  search_traffic  total_visits  \\\n",
       "0                   3              102124         4188154    1124655652   \n",
       "1                   7               41383         1257742     592415339   \n",
       "2                   4               98857         3938984    1350876078   \n",
       "3                  10               25894          932688     322074785   \n",
       "4                   5               32332          930331     468611692   \n",
       "5                   1               39273         1056534    1095599823   \n",
       "6                   2               45927         1568306     503087845   \n",
       "7                  11               23456          972333     220630202   \n",
       "8                   6               10319          320477     139408430   \n",
       "9                   8               10775          286442     100143853   \n",
       "10                 13               11474          394931     163908794   \n",
       "11                 17                8766          306433     112475075   \n",
       "12                 12                2870           81325      39492159   \n",
       "13                 21                5059          209524      43132483   \n",
       "14                  9               14884          768763     116819777   \n",
       "15                 22                7970          229685      79225111   \n",
       "16                  0                1350           29778      19688503   \n",
       "17                 15                7582          295427      90688876   \n",
       "18                 19                2126          105863      14822923   \n",
       "19                 14                2613           69270      20881742   \n",
       "\n",
       "    avg_external_search_traffic  avg_link_traffic_per_edge  \\\n",
       "0                       1054.34                      97.36   \n",
       "1                        954.76                      68.92   \n",
       "2                       2343.52                     181.66   \n",
       "3                        644.81                      69.53   \n",
       "4                        982.18                     114.28   \n",
       "5                       1282.43                      59.90   \n",
       "6                       1239.03                      72.39   \n",
       "7                       1061.68                      81.02   \n",
       "8                        885.90                      66.94   \n",
       "9                        508.18                      67.78   \n",
       "10                      1203.88                      93.70   \n",
       "11                       857.73                      99.80   \n",
       "12                       405.18                      56.18   \n",
       "13                       517.88                      53.91   \n",
       "14                      1457.26                     113.07   \n",
       "15                      1053.36                      72.25   \n",
       "16                       287.95                      39.52   \n",
       "17                      1630.04                     141.42   \n",
       "18                       279.68                      64.50   \n",
       "19                       600.02                      61.06   \n",
       "\n",
       "    avg_visits_per_article  link_network_density  link_network_density_delta  \n",
       "0                  2215.45                  0.00                        5.67  \n",
       "1                  1960.86                  0.00                        5.74  \n",
       "2                  5049.23                  0.00                        8.50  \n",
       "3                  1379.95                  0.00                        4.18  \n",
       "4                  2093.29                  0.00                        4.93  \n",
       "5                  5025.66                  0.00                        6.31  \n",
       "6                  2406.37                  0.00                        5.38  \n",
       "7                  1998.52                  0.00                        5.34  \n",
       "8                  1704.11                  0.00                        4.13  \n",
       "9                  1368.61                  0.00                        6.41  \n",
       "10                 2409.15                  0.00                        6.05  \n",
       "11                 1948.16                  0.00                        5.35  \n",
       "12                  803.60                  0.00                        2.14  \n",
       "13                  965.71                  0.00                        2.71  \n",
       "14                 2753.88                  0.00                        6.21  \n",
       "15                 2085.80                  0.00                        6.54  \n",
       "16                  669.75                  0.00                        3.35  \n",
       "17                 3710.07                  0.00                        9.01  \n",
       "18                  667.88                  0.00                        2.51  \n",
       "19                 1163.07                  0.00                        3.76  "
      ]
     },
     "execution_count": 220,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# unpickle\n",
    "with open(\"pickles/en_1218_louvain_communities.pkl\", 'rb') as picklefile: \n",
    "    louvain_communities = pickle.load(picklefile)\n",
    "\n",
    "louvain_communities.head(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 221,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0      3\n",
       "1      7\n",
       "2      4\n",
       "3     10\n",
       "4      5\n",
       "5      1\n",
       "6      2\n",
       "7     11\n",
       "8      6\n",
       "9      8\n",
       "10    13\n",
       "11    17\n",
       "12    12\n",
       "13    21\n",
       "14     9\n",
       "15    22\n",
       "16     0\n",
       "17    15\n",
       "18    19\n",
       "19    14\n",
       "Name: louvain_community, dtype: int64"
      ]
     },
     "execution_count": 221,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top_20_louvain_communities_ids = louvain_communities[:20].louvain_community\n",
    "top_20_louvain_communities_ids"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 222,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Community id = 3\n",
      "Community topic words:\n",
      " list unit state footbal elect nation th constitu parti battl john film st district war cup disambigu india leagu world\n",
      "Community topic NER:\n",
      " PERSON ORG GPE DATE NORP CARDINAL EVENT ORDINAL LOC FAC PRODUCT LAW WORK_OF_ART LANGUAGE QUANTITY MONEY TIME PERCENT\n",
      "\n",
      "Community id = 7\n",
      "Community topic words:\n",
      " list disambigu church al john novel saint languag histori st film battl ii cathol roman book william art la christian\n",
      "Community topic NER:\n",
      " PERSON ORG GPE NORP DATE CARDINAL LOC WORK_OF_ART FAC EVENT ORDINAL LANGUAGE PRODUCT LAW TIME MONEY QUANTITY PERCENT\n",
      "\n",
      "Community id = 4\n",
      "Community topic words:\n",
      " film list seri tv actor comic season award disambigu episod school john charact novel man star love song album david\n",
      "Community topic NER:\n",
      " PERSON ORG DATE GPE CARDINAL NORP WORK_OF_ART EVENT LOC ORDINAL FAC PRODUCT TIME LAW LANGUAGE QUANTITY MONEY PERCENT\n",
      "\n",
      "Community id = 10\n",
      "Community topic words:\n",
      " state list new counti school unit california york station park district elect john citi texa high nation hous north river\n",
      "Community topic NER:\n",
      " PERSON GPE ORG DATE LOC FAC CARDINAL NORP ORDINAL EVENT WORK_OF_ART PRODUCT LANGUAGE LAW TIME MONEY QUANTITY PERCENT\n",
      "\n",
      "Community id = 5\n",
      "Community topic words:\n",
      " album song band list discographi love music live musician record tour disambigu ep best world award rock time new singl\n",
      "Community topic NER:\n",
      " PERSON ORG GPE DATE CARDINAL NORP WORK_OF_ART LOC EVENT FAC ORDINAL PRODUCT TIME LAW LANGUAGE QUANTITY PERCENT MONEY\n",
      "\n",
      "Community id = 1\n",
      "Community topic words:\n",
      " list disambigu softwar power station number engin space unit program network model theorem group theori data compani energi function manag\n",
      "Community topic NER:\n",
      " ORG PERSON GPE CARDINAL DATE NORP LOC PRODUCT ORDINAL FAC EVENT WORK_OF_ART LAW QUANTITY TIME LANGUAGE MONEY PERCENT\n",
      "\n",
      "Community id = 2\n",
      "Community topic words:\n",
      " list disambigu syndrom nation park diseas food black acid hospit state red white cell medic dog unit bird health american\n",
      "Community topic NER:\n",
      " ORG PERSON GPE NORP CARDINAL DATE LOC PRODUCT FAC EVENT WORK_OF_ART ORDINAL LANGUAGE LAW QUANTITY TIME MONEY PERCENT\n",
      "\n",
      "Community id = 11\n",
      "Community topic words:\n",
      " footbal team season school basketbal high list american basebal state leagu men bowl new colleg nation univers game john stadium\n",
      "Community topic NER:\n",
      " PERSON ORG DATE GPE NORP EVENT CARDINAL FAC LOC PRODUCT ORDINAL WORK_OF_ART LAW TIME LANGUAGE QUANTITY MONEY\n",
      "\n",
      "Community id = 6\n",
      "Community topic words:\n",
      " univers school list colleg bank institut compani state intern educ group law technolog scienc unit nation john mall new busi\n",
      "Community topic NER:\n",
      " ORG PERSON GPE NORP DATE CARDINAL FAC LOC ORDINAL EVENT WORK_OF_ART LANGUAGE PRODUCT LAW TIME MONEY QUANTITY PERCENT\n",
      "\n",
      "Community id = 8\n",
      "Community topic words:\n",
      " airport air uss list class hms airlin intern squadron flight submarin forc aircraft ship th aviat destroy unit raf base\n",
      "Community topic NER:\n",
      " PERSON ORG GPE FAC DATE NORP CARDINAL PRODUCT EVENT ORDINAL LOC WORK_OF_ART QUANTITY LAW LANGUAGE TIME MONEY\n",
      "\n",
      "Community id = 13\n",
      "Community topic words:\n",
      " game list video episod charact seri disambigu film dragon world war ii super manga album adventur star japan battl anim\n",
      "Community topic NER:\n",
      " PERSON ORG GPE CARDINAL DATE NORP EVENT WORK_OF_ART PRODUCT LOC ORDINAL FAC LAW TIME MONEY LANGUAGE PERCENT QUANTITY\n",
      "\n",
      "Community id = 17\n",
      "Community topic words:\n",
      " film list china seri tv chines hong kong district lee song station kim korean love univers peopl album languag award\n",
      "Community topic NER:\n",
      " PERSON ORG GPE DATE NORP CARDINAL LOC FAC WORK_OF_ART EVENT ORDINAL PRODUCT LANGUAGE LAW TIME QUANTITY PERCENT MONEY\n",
      "\n",
      "Community id = 12\n",
      "Community topic words:\n",
      " olymp championship summer men world women game athlet metr open winter cup singl list swim tenni asian european ski team\n",
      "Community topic NER:\n",
      " PERSON DATE EVENT ORG GPE QUANTITY NORP CARDINAL FAC LOC PRODUCT LANGUAGE WORK_OF_ART LAW ORDINAL TIME MONEY PERCENT\n",
      "\n",
      "Community id = 21\n",
      "Community topic words:\n",
      " cricket rugbi list new south australia australian union leagu team wale nation cup footbal world victoria zealand season queensland club\n",
      "Community topic NER:\n",
      " PERSON ORG GPE DATE NORP CARDINAL EVENT LOC FAC ORDINAL PRODUCT WORK_OF_ART LAW LANGUAGE TIME QUANTITY PERCENT\n",
      "\n",
      "Community id = 9\n",
      "Community topic words:\n",
      " film list templ award sri tamil actor actress tv colleg disambigu best lanka seri station krishna district raja kannada oru\n",
      "Community topic NER:\n",
      " PERSON ORG GPE DATE NORP CARDINAL LOC EVENT ORDINAL FAC WORK_OF_ART PRODUCT LAW LANGUAGE PERCENT TIME QUANTITY\n",
      "\n",
      "Community id = 22\n",
      "Community topic words:\n",
      " grand prix championship seri engin race car formula list ford honda motorcycl motor world automobil toyota driver merced bmw benz\n",
      "Community topic NER:\n",
      " ORG PERSON DATE GPE CARDINAL EVENT PRODUCT NORP TIME ORDINAL FAC LOC LAW WORK_OF_ART QUANTITY LANGUAGE PERCENT MONEY\n",
      "\n",
      "Community id = 0\n",
      "Community topic words:\n",
      " station railway class line rail metro list park train transport british london south road tram coaster tube seri locomot new\n",
      "Community topic NER:\n",
      " ORG PERSON GPE CARDINAL NORP FAC LOC DATE PRODUCT EVENT ORDINAL WORK_OF_ART QUANTITY LAW TIME MONEY LANGUAGE\n",
      "\n",
      "Community id = 15\n",
      "Community topic words:\n",
      " wrestl championship world vs ufc wrestler list boxer box fighter fight heavyweight wwe profession olymp summer champion night team tag\n",
      "Community topic NER:\n",
      " PERSON ORG DATE GPE EVENT CARDINAL NORP WORK_OF_ART QUANTITY ORDINAL PRODUCT LOC FAC TIME LAW LANGUAGE PERCENT\n",
      "\n",
      "Community id = 19\n",
      "Community topic words:\n",
      " footbal leagu cup nation malaysia district stadium championship fc list team asian afc station futsal al feder season constitu game\n",
      "Community topic NER:\n",
      " PERSON ORG GPE DATE EVENT NORP CARDINAL FAC PRODUCT LOC ORDINAL WORK_OF_ART LAW LANGUAGE QUANTITY TIME PERCENT\n",
      "\n",
      "Community id = 14\n",
      "Community topic words:\n",
      " hockey ice season list nhl cup world leagu championship nation school junior men fm team arena stanley draft iihf mike\n",
      "Community topic NER:\n",
      " PERSON ORG GPE DATE EVENT CARDINAL NORP LOC FAC ORDINAL PRODUCT WORK_OF_ART TIME LAW LANGUAGE\n"
     ]
    }
   ],
   "source": [
    "# print out the topic descriptors\n",
    "for cid in top_20_louvain_communities_ids:\n",
    "    print(\"\\nCommunity id =\", cid)\n",
    "    print(\"Community topic words:\\n\", \" \".join(community_topics_dict[cid][\"topic_words\"]))\n",
    "    print(\"Community topic NER:\\n\", \" \".join(community_topics_dict[cid][\"topic_ner\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Grouping communities by common terms  \n",
    "\n",
    "We can search the community descriptor words for specific terms to find groups of communities on adjacent topics.  \n",
    "For example, we can find all communities about books, or music, or United States."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 216,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['unit', 'state']\n"
     ]
    }
   ],
   "source": [
    "# choose some words to search for\n",
    "words = \"novel book read\"\n",
    "words = \"album discography music song\"\n",
    "words = \"united states\"\n",
    "\n",
    "# process the selected words\n",
    "preprocessed = gensim.utils.simple_preprocess(words)\n",
    "    \n",
    "lemmatized = []\n",
    "for w in preprocessed:\n",
    "    stem = stemmer.stem(WordNetLemmatizer().lemmatize(w, pos='v'))\n",
    "    lemmatized.append(stem)\n",
    "    \n",
    "print(lemmatized)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 218,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Community id = 3\n",
      "Community topic words:\n",
      " list unit state footbal elect nation th constitu parti battl john film st district war cup disambigu india leagu world\n",
      "Community topic NER:\n",
      " PERSON ORG GPE DATE NORP CARDINAL EVENT ORDINAL LOC FAC PRODUCT LAW WORK_OF_ART LANGUAGE QUANTITY MONEY TIME PERCENT\n",
      "\n",
      "Community id = 6\n",
      "Community topic words:\n",
      " univers school list colleg bank institut compani state intern educ group law technolog scienc unit nation john mall new busi\n",
      "Community topic NER:\n",
      " ORG PERSON GPE NORP DATE CARDINAL FAC LOC ORDINAL EVENT WORK_OF_ART LANGUAGE PRODUCT LAW TIME MONEY QUANTITY PERCENT\n",
      "\n",
      "Community id = 10\n",
      "Community topic words:\n",
      " state list new counti school unit california york station park district elect john citi texa high nation hous north river\n",
      "Community topic NER:\n",
      " PERSON GPE ORG DATE LOC FAC CARDINAL NORP ORDINAL EVENT WORK_OF_ART PRODUCT LANGUAGE LAW TIME MONEY QUANTITY PERCENT\n",
      "\n",
      "Community id = 2\n",
      "Community topic words:\n",
      " list disambigu syndrom nation park diseas food black acid hospit state red white cell medic dog unit bird health american\n",
      "Community topic NER:\n",
      " ORG PERSON GPE NORP CARDINAL DATE LOC PRODUCT FAC EVENT WORK_OF_ART ORDINAL LANGUAGE LAW QUANTITY TIME MONEY PERCENT\n",
      "\n",
      "Community id = 84\n",
      "Community topic words:\n",
      " orchestra symphoni philharmon state youth dispatch new chamber san album fiedler pop unit sanderl boston nation north conductor franci valley\n",
      "Community topic NER:\n",
      " PERSON ORG GPE NORP DATE CARDINAL LOC EVENT\n",
      "\n",
      "Community id = 554\n",
      "Community topic words:\n",
      " traina wilsey nagel canin state unit link nick trevor dede ethan todd ambassador austria list montandon patricia sean nagl boy\n",
      "Community topic NER:\n",
      " PERSON ORG GPE CARDINAL\n",
      "\n",
      "Community id = 104\n",
      "Community topic words:\n",
      " villag kf counti unit albanian divis state tirana fk ambassador list fc second albania peqin rie charl tiranë ks stadium\n",
      "Community topic NER:\n",
      " GPE PERSON ORG DATE\n",
      "\n",
      "Community id = 1018\n",
      "Community topic words:\n",
      " act workforc invest innov opportun peyser wagner employ servic state unit board\n",
      "Community topic NER:\n",
      " ORG DATE EVENT LAW\n",
      "\n",
      "Community id = 271\n",
      "Community topic words:\n",
      " relat zambia china foreign india state unit\n",
      "Community topic NER:\n",
      " GPE\n",
      "\n",
      "Community id = 770\n",
      "Community topic words:\n",
      " hous list live member repres state unit denton winfield\n",
      "Community topic NER:\n",
      " ORG PERSON\n",
      "\n",
      "Community id = 976\n",
      "Community topic words:\n",
      " budget feder state unit\n",
      "Community topic NER:\n",
      " GPE DATE CARDINAL\n",
      "\n",
      "Community id = 1158\n",
      "Community topic words:\n",
      " madagascar relat foreign india affair minist franc state unit china alphons dovo eloi maxim\n",
      "Community topic NER:\n",
      " GPE\n",
      "\n",
      "Community id = 1139\n",
      "Community topic words:\n",
      " congo republ relat foreign state unit diplomat list mission china\n",
      "Community topic NER:\n",
      " GPE\n",
      "\n",
      "Community id = 1605\n",
      "Community topic words:\n",
      " djibouti relat china foreign state unit franc diplomat list mission\n",
      "Community topic NER:\n",
      " GPE\n",
      "\n",
      "Community id = 575\n",
      "Community topic words:\n",
      " surinam relat diplomat list mission india indonesia foreign guyana state unit brazil brazilian china\n",
      "Community topic NER:\n",
      " GPE PERSON NORP\n",
      "\n",
      "Community id = 786\n",
      "Community topic words:\n",
      " center list natur state new carolina florida pennsylvania unit michigan minnesota york north connecticut texa jersey ohio california illinoi south\n",
      "Community topic NER:\n",
      " GPE ORG\n",
      "\n",
      "Community id = 700\n",
      "Community topic words:\n",
      " list monument mnarani memori kenya sit state unit ruin\n",
      "Community topic NER:\n",
      " GPE\n",
      "\n",
      "Community id = 1675\n",
      "Community topic words:\n",
      " burkina faso relat foreign soviet union china state unit korea north\n",
      "Community topic NER:\n",
      " ORG\n",
      "\n",
      "Community id = 588\n",
      "Community topic words:\n",
      " elect state unit senat minnesota vermont special\n",
      "Community topic NER:\n",
      " ORG GPE DATE CARDINAL ORDINAL\n",
      "\n",
      "Community id = 856\n",
      "Community topic words:\n",
      " elect gubernatori state unit\n",
      "Community topic NER:\n",
      " GPE DATE\n",
      "\n",
      "Community id = 58\n",
      "Community topic words:\n",
      " california elect hous repres state unit\n",
      "Community topic NER:\n",
      " GPE ORG DATE CARDINAL\n",
      "\n",
      "Community id = 356\n",
      "Community topic words:\n",
      " carolina elect senat south state unit special\n",
      "Community topic NER:\n",
      " DATE GPE ORG\n",
      "\n",
      "Community id = 373\n",
      "Community topic words:\n",
      " elect pennsylvania senat state unit blatt geneviev\n",
      "Community topic NER:\n",
      " GPE ORG DATE CARDINAL PERSON ORDINAL\n",
      "\n",
      "Community id = 538\n",
      "Community topic words:\n",
      " elect pennsylvania senat state unit special\n",
      "Community topic NER:\n",
      " GPE ORG DATE ORDINAL\n",
      "\n",
      "Community id = 29\n",
      "Community topic words:\n",
      " elect hous repres state unit vermont\n",
      "Community topic NER:\n",
      " GPE ORG DATE CARDINAL\n",
      "\n",
      "Community id = 1088\n",
      "Community topic words:\n",
      " columbia district elect hous repres state unit\n",
      "Community topic NER:\n",
      " GPE ORG DATE CARDINAL\n",
      "\n",
      "Community id = 328\n",
      "Community topic words:\n",
      " elect minnesota senat state unit special\n",
      "Community topic NER:\n",
      " DATE GPE ORG\n",
      "\n",
      "Community id = 1493\n",
      "Community topic words:\n",
      " elect senat state unit virginia special\n",
      "Community topic NER:\n",
      " GPE ORG DATE CARDINAL ORDINAL\n",
      "\n",
      "Community id = 132\n",
      "Community topic words:\n",
      " arizona elect senat state unit\n",
      "Community topic NER:\n",
      " GPE ORG DATE CARDINAL ORDINAL\n",
      "\n",
      "Community id = 621\n",
      "Community topic words:\n",
      " elect senat state unit vermont special\n",
      "Community topic NER:\n",
      " GPE ORG DATE ORDINAL\n",
      "\n",
      "Community id = 707\n",
      "Community topic words:\n",
      " elect massachusett senat state unit special\n",
      "Community topic NER:\n",
      " GPE ORG DATE ORDINAL CARDINAL\n",
      "\n",
      "Community id = 915\n",
      "Community topic words:\n",
      " delawar elect presidenti state unit\n",
      "Community topic NER:\n",
      " GPE DATE\n",
      "\n",
      "Community id = 969\n",
      "Community topic words:\n",
      " dakota elect north senat state unit special\n",
      "Community topic NER:\n",
      " DATE GPE ORG\n"
     ]
    }
   ],
   "source": [
    "# find communities that have the selected words\n",
    "\n",
    "for cid in louvain_community_ids:\n",
    "    #if (lemmatized in community_topics_dict[cid][\"topic_words\"]):\n",
    "    #if not set(lemmatized).isdisjoint(set(community_topics_dict[cid][\"topic_words\"])):\n",
    "    if set(lemmatized).issubset(set(community_topics_dict[cid][\"topic_words\"])):\n",
    "        print(\"\\nCommunity id =\", cid)\n",
    "        print(\"Community topic words:\\n\", \" \".join(community_topics_dict[cid][\"topic_words\"]))\n",
    "        print(\"Community topic NER:\\n\", \" \".join(community_topics_dict[cid][\"topic_ner\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### 4. Summary  \n",
    "\n",
    "In this notebook, we've extracted named entities from article titles in order to generalize articles about specific people, places, etc. We then created bags of words for each community, and extracted the top words and named entities as descriptors for each community. These community descriptor words can also be used to group communities based on common terms."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Next steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have defined Wikipedia article communities, their stats, and a way to describe their topics, we can build a D3 visualization to summarize and illustrate these findings.  \n",
    "\n",
    "But since we're looking at data that represents browsing behaviors on Wikipedia, it would be interesting to see what was the more obscure browsing on Wikipedia in December 2018. The [next notebook](English_Wikipedia_deepWiki.ipynb) in this project explores that."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}