assignments/06_NLP_intro_lab.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1 id=\"Introduction-to-Python-and-Natural-Language-Technologies\">Introduction to Python and Natural Language Technologies</h1>\n",
    "<h2 id=\"Laboratory-06,-NLP-Introduction\">Laboratory 06, NLP Introduction</h2>\n",
    "<p><strong>March 18, 2020</strong></p>\n",
    "<p><strong>&Aacute;d&aacute;m Kov&aacute;cs</strong></p>\n",
    "<p>During this laboratory we are going to use a classification dataset of SemEval 2019 - Task 6. This is called Identifying and Categorizing Offensive Language in Social Media.</p>\n",
    "<h2 id=\"Preparation\">Preparation</h2>\n",
    "<p style=\"padding-left: 40px;\"><a href=\"http://sandbox.hlt.bme.hu/~adaamko/glove.6B.100d.txt\" target=\"_blank\" rel=\"noopener\">Download GLOVE</a>(and place it into this directory)</p>\n",
    "<p style=\"padding-left: 40px;\">Download the dataset (with python code)</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "if not os.path.isdir('./data'):\n",
    "    os.mkdir('./data')\n",
    "\n",
    "import urllib\n",
    "u = urllib.request.URLopener()\n",
    "u.retrieve(\"http://sandbox.hlt.bme.hu/~adaamko/offenseval.tsv\", \"data/offenseval.tsv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Train a Logistic Regression on the dataset\n",
    "\n",
    "Use a CountVectorizer for featurizing your data. You can reuse the code presented during the lecture"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.1 Read in the dataset into a Pandas DataFrame\n",
    "Use `pd.read_csv` with the correct parameters to read in the dataset. If done correctly, `DataFrame` should have 3 columns, \n",
    "`id`, `tweet`, `subtask_a`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "245ae3153d23ed632c3f49c1f0478d3e",
     "grade": false,
     "grade_id": "cell-28244ef9e290dace",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "def read_dataset():\n",
    "    # YOUR CODE HERE\n",
    "    raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "dac5b55254d3e1bc940aabb8b98b3f11",
     "grade": true,
     "grade_id": "cell-8cdaaf32c6a9bcd4",
     "locked": true,
     "points": 0,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "train_data_unprocessed = read_dataset()\n",
    "\n",
    "assert type(train_data_unprocessed) == pd.core.frame.DataFrame\n",
    "assert len(train_data_unprocessed.columns) == 3\n",
    "assert (train_data_unprocessed.columns == ['id', 'tweet', 'subtask_a']).all()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2 Convert `subtask_a` into a binary label\n",
    "The task is to classify the given tweets into two category: _offensive(OFF)_ , _not offensive (NOT)_. For machine learning algorithms you will need integer labels instead of strings. Add a new column to the dataframe called `label`, and transform the `subtask_a` column into a binary integer label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "f0a97018f8bd5fdc07b40efdb873d503",
     "grade": false,
     "grade_id": "cell-731e83e7c0331c22",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "def transform(train_data):\n",
    "    # YOUR CODE HERE\n",
    "    raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "eff0405739147a8bb4b0584bca759c50",
     "grade": true,
     "grade_id": "cell-9162edb7c3cbea87",
     "locked": true,
     "points": 0,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "from pandas.api.types import is_numeric_dtype\n",
    "\n",
    "train_data = transform(train_data_unprocessed)\n",
    "\n",
    "assert \"label\" in train_data\n",
    "assert is_numeric_dtype(train_data.label)\n",
    "assert (train_data.label.isin([0,1])).all()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data.groupby(\"label\").size()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3 Initialize CountVectorizer and _train_ it on the _tweet_ column of the dataset\n",
    "The _training_ will prepare the vocabulary for us so we will be able to use it for training a LogisticRegression algorithm later. Set the number of `max_features` to 5000 so vocabulary won't be too big for training. Also filter out english `stop_words`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "dc3d77a0b35ffc7e7a1bf4117acf76a8",
     "grade": false,
     "grade_id": "cell-c14091eabbbdefb8",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# We will need to use a random seed for our methods so they will be reproducible\n",
    "SEED = 1234"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "e11dcb24d6bd42ab3f2bbc0d516b6a25",
     "grade": false,
     "grade_id": "cell-d66cb77a6cdd86cb",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "def prepare_vectorizer(train_data):\n",
    "    # YOUR CODE HERE\n",
    "    raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "652a63730eb7cb7cc7e0bfba7e726265",
     "grade": true,
     "grade_id": "cell-edef3ff41f4c1bde",
     "locked": true,
     "points": 0,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "vectorizer = prepare_vectorizer(train_data)\n",
    "\n",
    "transformed = vectorizer.transform([\"hello this is the intro to nlp\"])\n",
    "assert transformed.dtype == np.dtype('int64')\n",
    "assert transformed.shape == (1, 5000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.4 Featurize the dataset with the prepared CountVectorizer, and split it into _train_ and _test_ dataset\n",
    "You should use the random seed when you are splitting the dataset. The scale of the training and the test dataset should be 70% to 30%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "a55b9f4dcb26e14c42f9ab6e7822b2f9",
     "grade": false,
     "grade_id": "cell-2f4b7f4e32e2bc36",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "import gensim\n",
    "from tqdm import tqdm\n",
    "from sklearn.model_selection import train_test_split as split\n",
    "\n",
    "def vectorize_to_bow(tr_data, tst_data, vectorizer):   \n",
    "    # YOUR CODE HERE\n",
    "    raise NotImplementedError()\n",
    "\n",
    "def get_features_and_labels(data, labels, vectorizer):\n",
    "    # tr_data,tst_data,tr_labels,tst_labels = split...\n",
    "    # ...\n",
    "    # tr_vecs, tst_vecs = vectorize_to_bow(...\n",
    "    # YOUR CODE HERE\n",
    "    raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "ff6f2048306f4c58336f5e2b88065068",
     "grade": true,
     "grade_id": "cell-8cb9baeaccf044d7",
     "locked": true,
     "points": 0,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "tr_vecs, tr_labels, tst_vecs, tst_labels = get_features_and_labels(train_data.tweet, train_data.label, vectorizer)\n",
    "assert tr_vecs.shape == (9268, 5000)\n",
    "assert tr_labels.shape == (9268,)\n",
    "assert tst_vecs.shape == (3972, 5000)\n",
    "assert tst_labels.shape == (3972,)\n",
    "assert tr_vecs[0].toarray().shape == (1, 5000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import a bunch of stuff from sklearn\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "# We will train a LogisticRegression algorithm for the classification\n",
    "lr  = LogisticRegression(n_jobs=-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.5 Train and evaluate your method!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "0a225537e2b31495a027d806dcd7d6a8",
     "grade": false,
     "grade_id": "cell-58727c0fba0e7d13",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# Training on the train dataset\n",
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "55ab78ccc35209c5f1716b20a88d7bc4",
     "grade": true,
     "grade_id": "cell-a6e0028e6cfeaa33",
     "locked": true,
     "points": 0,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.utils.validation import check_is_fitted\n",
    "from sklearn.exceptions import NotFittedError\n",
    "\n",
    "try:\n",
    "    check_is_fitted(lr)\n",
    "except NotFittedError as e:\n",
    "    assert None, repr(e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "96755e97d86dd7ceb2261886a1f95430",
     "grade": false,
     "grade_id": "cell-eef0add921511581",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "# Evaluation on the test dataset\n",
    "def preds(lr, tst_vecs):\n",
    "    # YOUR CODE HERE\n",
    "    raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "f91da323e048cc77e1f69e2f96933057",
     "grade": true,
     "grade_id": "cell-bc6195f27f0f93b0",
     "locked": true,
     "points": 0,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# If you have done everything right, the accuracy should be around 75%\n",
    "lr_pred = preds(lr, tst_vecs)\n",
    "assert lr_pred.shape == (3972,)\n",
    "print(\"Logistic Regression Test accuracy : {}\".format(\n",
    "    accuracy_score(tst_labels, lr_pred)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.1 Change to TfidfVectorizer, and also change the configuration\n",
    "\n",
    "Look up the documentation of [TfidfVectorizer](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html). It has a lot of parameters to play with. \n",
    "\n",
    "This time, change the parameters to include _maximum_ of __10000__ features. Also include filtering of _stopwords_ and _lowercasing_ the features. (hint: look at the parameter names in the documentation)\n",
    "\n",
    "Also [_ngram_](https://en.wikipedia.org/wiki/N-gram) features can improve the performance of the model. A bigram is an n-gram for n=2, trigram is when n=3, etc..\n",
    "\n",
    "\n",
    "Bigram features include not only single words in the vocabulary, but the frequency of every occuring bigram in the text (e.g. it will include not only the words _brown_ and _dog_ but __brown dog__ also)\n",
    "\n",
    "Change the configuration of the _TfidfVectorizer_ to also include the _bigrams_ and _trigrams_ in the vocabulary.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "c811e892327a2cba08939276e67c09c8",
     "grade": false,
     "grade_id": "cell-4e74fe87b4b17324",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "def prepare_tfidf_vectorizer(train_data):\n",
    "    # YOUR CODE HERE\n",
    "    raise NotImplementedError()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "f2f6d06c5b75dcd3459f23ec673797e8",
     "grade": false,
     "grade_id": "cell-470b0bb87098c483",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "tr_vecs, tr_labels, tst_vecs, tst_labels = get_features_and_labels(\n",
    "    train_data.tweet, train_data.label, prepare_tfidf_vectorizer(train_data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "abfa8f70571a8ec882f6830492c16906",
     "grade": false,
     "grade_id": "cell-2a89733013d98c1d",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# Train and evaluate! \n",
    "lr  = LogisticRegression(n_jobs=-1)\n",
    "\n",
    "#lr.fit...\n",
    "\n",
    "#lr_pred = ..\n",
    "\n",
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "ddb3dc0fd6562f838a7ccf31bb32c106",
     "grade": true,
     "grade_id": "cell-58df3ea2b9f1335b",
     "locked": true,
     "points": 0,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.utils.validation import check_is_fitted\n",
    "from sklearn.exceptions import NotFittedError\n",
    "\n",
    "try:\n",
    "    check_is_fitted(lr)\n",
    "except NotFittedError as e:\n",
    "    assert None, repr(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2 Write a custom tokenizer for TfidfVectorizer\n",
    "\n",
    "Right now, the vectorizer uses it's own tokenizer for creating the vocabulary. You can also create a custom function and tell the vectorizer to use that when tokenizing the text.\n",
    "\n",
    "Use [spacy](https://spacy.io/) for tokenization. write your own function.\n",
    "\n",
    "Your function should:\n",
    "- get a sentence as an input\n",
    "- run spacy on the input text\n",
    "- return a token list that includes:\n",
    "    - filtering of stop words\n",
    "    - filtering of punctuation\n",
    "    - lemmatizing the text\n",
    "    - lowercasing the text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "9458296f3b9931a28cecc11924990e56",
     "grade": false,
     "grade_id": "cell-31035c3e0e78d8f6",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "import spacy\n",
    "\n",
    "nlp = spacy.load(\"en\")\n",
    "\n",
    "\n",
    "def spacy_tokenizer(sentence):\n",
    "    # YOUR CODE HERE\n",
    "    raise NotImplementedError()\n",
    "\n",
    "\n",
    "vectorizer_with_spacy = TfidfVectorizer(\n",
    "    max_features=10000, tokenizer=spacy_tokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "b5097da78652b20a0ca2b09ddf4ed454",
     "grade": true,
     "grade_id": "cell-70b92faae3867608",
     "locked": true,
     "points": 0,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "assert (spacy_tokenizer(\"This is the NLP lab, this text should not contain any punctuations and stopwords, and the text should be lowercased.\") == [\n",
    "        'nlp', 'lab', 'text', 'contain', 'punctuation', 'stopword', 'text', 'lowercase'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "6cd0b906365a12ac9816fb4b716552e4",
     "grade": false,
     "grade_id": "cell-932a31d7a48415ce",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "X = vectorizer_with_spacy.fit(train_data.tweet)\n",
    "\n",
    "tr_vecs, tr_labels, tst_vecs, tst_labels = get_features_and_labels(train_data.tweet, train_data.label, X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "7c9ce3e1c9db008b94c8e2a518ea1c9e",
     "grade": false,
     "grade_id": "cell-28cbd9926b7630ca",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# Train and evaluate! \n",
    "# If you have done everything right you should get the same or a little better performance than the standard\n",
    "# TfidfVectorizer and CountVectorizer\n",
    "lr  = LogisticRegression(n_jobs=-1)\n",
    "\n",
    "#lr.fit...\n",
    "\n",
    "#lr_pred = ..\n",
    "\n",
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Word embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1 Transform word vectors to sentence vector taking the average of the word vectors\n",
    "Word vectors transform words to a vector space where similar words have similar vectors.\n",
    "These vectors can be used as features for ML algorithms. But to feature a sentence first you need to create a _sentence vector_ from the vectors of the words. The easiest way of transforming word vectors to sentence vector is to take the average of all the word vectors.\n",
    "\n",
    "![ww](https://www.researchgate.net/profile/Md-Shajalal/publication/329394770/figure/fig1/AS:701809937088513@1544335936936/A-framework-for-learning-word-vectors-7_W640.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Load the embedding file\n",
    "embedding_file = \"glove.6B.100d.txt\"\n",
    "\n",
    "model = gensim.models.KeyedVectors.load_word2vec_format(embedding_file, binary=False)\n",
    "vectorizer = model.wv\n",
    "vocab_length = len(model.wv.vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Your transform function should:**\n",
    "- tokenize the sentence with the spacy tokenizer\n",
    "- get the embedding vector:\n",
    "    - get the embedding vector from the model if the word is in the vocabulary\n",
    "    - initialize a vector with zeros with the same dimension if the word is not in the vocabulary\n",
    "- take the mean of the word vectors to return a sentence vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "075d84b4a8325a3baaa488afc30d05a3",
     "grade": false,
     "grade_id": "cell-f6d0bba39013a876",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "def transform(words):\n",
    "    # YOUR CODE HERE\n",
    "    raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "0533f8b61bf05e8fbb8c0f11261f423c",
     "grade": true,
     "grade_id": "cell-c015012d531b7c3a",
     "locked": true,
     "points": 0,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "assert transform(\"this is a nlp lab\").shape == (100,)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**We can calculate similarities between sentences now the same way that we did between words! For this we need to use the cosine_similarity function!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "\n",
    "print(cosine_similarity(transform(\"hello my name is adam\").reshape(\n",
    "    1, -1), transform(\"hello my name is andrea\").reshape(1, -1))[0][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "5947c3827d4c3def144392cac336c108",
     "grade": true,
     "grade_id": "cell-6a71e2b32b6277e8",
     "locked": true,
     "points": 0,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "assert cosine_similarity(transform(\"hello my name is adam\").reshape(\n",
    "    1, -1), transform(\"hello my name is andrea\").reshape(1, -1)).shape == (1, 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.4 Finding Analogies\n",
    "Word vectors have been shown to sometimes have the ability to solve analogies.\n",
    "\n",
    "We discussed this during the lecture that for the analogy \"man : king :: woman : x\" (read: man is to king as woman is to x), x is _queen_\n",
    "\n",
    "Find more examples of analogies that holds according to these vectors (i.e. the intended word is ranked top)!\n",
    "\n",
    "Also find an example of analogy that does not hold according to these vectors!\n",
    "\n",
    "Summarize your findings in a few sentences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "7dae55ed9ff89f94f896945692ad39f4",
     "grade": true,
     "grade_id": "cell-a32ae4b69f94e14b",
     "locked": false,
     "points": 0,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.5 Bias in word vectors\n",
    "\n",
    "It's important to be cognizant of the biases (gender, race, sexual orientation etc.) implicit in our word embeddings. Bias  in word vectors can be dangerous because it can incorporate stereotypes through applications that employ these models.\n",
    "\n",
    "Run the cell below, to examine a sample of gender bias present in the data. Try to come up with another examples that can reflect biases in datasets (gender, race, sexual orientation etc.)\n",
    "\n",
    "Summarize your findings in a few sentences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(model.most_similar(positive=['woman', 'doctor'], negative=['man']))\n",
    "\n",
    "print(model.most_similar(positive=['man', 'doctor'], negative=['woman']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "0247a46741127cfff3b9de15b4f9a72f",
     "grade": true,
     "grade_id": "cell-7d3945be6a14bd81",
     "locked": false,
     "points": 0,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ================ PASSING LEVEL ===================="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Logistic regression using word vectors\n",
    "\n",
    "These sentence vectors can be used as feature vectors for classifiers. Rewrite the featurizing process and transform each sentence into a sentence vector using the embedding model!\n",
    "\n",
    "__Note: it is OK if your model is not better than the other classifiers__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "b2d7bd95147f36144b514fcc987d4f01",
     "grade": false,
     "grade_id": "cell-c3d782b99a73d946",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "def vectorize_to_embedding(tr_data, tst_data):    \n",
    "    # YOUR CODE HERE\n",
    "    raise NotImplementedError()\n",
    "\n",
    "def get_features_and_labels(data, labels):\n",
    "    # YOUR CODE HERE\n",
    "    raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "d38ff4466f18e04f6eee8435f83b79fe",
     "grade": true,
     "grade_id": "cell-c87f042243566944",
     "locked": true,
     "points": 0,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "tr_vecs, tr_labels, tst_vecs, tst_labels = get_features_and_labels(train_data.tweet, train_data.label)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "cee08cb7dd74cb5111fba7942873eb3a",
     "grade": true,
     "grade_id": "cell-11a9e1ed6def8a1e",
     "locked": true,
     "points": 0,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "assert tr_vecs[0].shape == (100,)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "2fe1f890e3007c6ca2458d4d8bcd2612",
     "grade": false,
     "grade_id": "cell-5189b74cd4f4372f",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# Train and evaluate! \n",
    "lr  = LogisticRegression(n_jobs=-1)\n",
    "\n",
    "#lr.fit...\n",
    "\n",
    "#lr_pred = ..\n",
    "\n",
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1 Ensemble model\n",
    "\n",
    "Try out other classifiers from: [sklearn](https://scikit-learn.org/stable/supervised_learning.html). Choose three and build a [VotingClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.VotingClassifier.html) with the choosen classifiers. If the _voting_ strategy is set to _hard_ it will do a majority voting among the classifiers and choose the class with the most votes.\n",
    "\n",
    "Make a [Pipeline](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html) with a TFIdfVectorizer and with your Ensemble model. Pipeline objects make it easy to assemble several steps together and makes your machine learning pipeline executable in just one step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "d0bbd431b65ed0c72a51b79f53660b3b",
     "grade": false,
     "grade_id": "cell-496b5cf9c757f418",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.ensemble import VotingClassifier\n",
    "\n",
    "\n",
    "def make_pipeline_ensemble(tweet, label):\n",
    "    \n",
    "    # YOUR CODE HERE\n",
    "    raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "a2aa00a09ac0821d059b76920daf77e5",
     "grade": false,
     "grade_id": "cell-91e45b470dd2fdc2",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "pipeline = make_pipeline_ensemble(train_data.tweet, train_data.label)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "1883ed3bcab25408ae1c3570c80ad5d4",
     "grade": true,
     "grade_id": "cell-ac56617dcdd3842a",
     "locked": true,
     "points": 0,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "assert type(pipeline) == Pipeline\n",
    "assert type(pipeline.steps[0][1]) == TfidfVectorizer\n",
    "assert type(pipeline.steps[1][1]) == VotingClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "6abb4535a19cf53912cda707a07c97ea",
     "grade": false,
     "grade_id": "cell-4d271a643404ed6b",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# Train and evaluate! \n",
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2 __Also evaluate your classifiers separately as well. Summarize your results in a cell below. Did the ensemble model improved your performance?__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "9d4cfd5c8edfd5725cf3f460efb37440",
     "grade": true,
     "grade_id": "cell-5fedea728b48ee0a",
     "locked": false,
     "points": 0,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ================ EXTRA LEVEL ===================="
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}