assignments/08_Sequence_modeling_lab.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "f2c56c32a7010370e303d1df03fdc0ee",
     "grade": false,
     "grade_id": "cell-bf12d2c95f08882e",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "# Introduction to Python and Natural Language Technologies\n",
    "\n",
    "__Laboratory 08, Sequence modeling__\n",
    "\n",
    "__April 2, 2020__\n",
    "\n",
    "The starter code in this notebook is a copy of the sequence tagging experiment from [Lecture 8](https://github.com/bmeaut/python_nlp_2021_spring/blob/main/lectures/08_Sequence_modeling.ipynb) without the explanations.\n",
    "We suggest (re)reading the lecture notebook before doing the homework.\n",
    "\n",
    "The training data has been replaced with a more difficult classification task, mood for Hungarian verbs (indicative, conditional or subjunctive).\n",
    "\n",
    "Your task is to improve the model by editing the relevant parts of the code. You should rerun the full notebook for each experiment and document your findings in a few sentences.\n",
    "\n",
    "Passing level: Task 1, 2, 3\n",
    "\n",
    "Extra level: Task 4, 5\n",
    "\n",
    "Very extra: Task 6\n",
    "\n",
    "## Task 1 - LSTMClassifier options\n",
    "\n",
    "Add more configuration options to `LSTMClasifier`. Make the following options configurable:\n",
    "- embedding size\n",
    "- number of layers\n",
    "- uni- or bidirectional LSTM\n",
    "- dropout\n",
    "\n",
    "Try out a few options and summarize your findings in a few sentences in the following cell. Make sure that you try at least two values for each parameter. You can use Markdown tables to present your results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "8c47eb3776c42d76884cbc0fe0475c0e",
     "grade": true,
     "grade_id": "cell-a7aa12d0e3a76681",
     "locked": false,
     "points": 0,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "source": [
    "YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "73b2521df3cc1aabc1775928085b5576",
     "grade": false,
     "grade_id": "cell-004d4bf68ee613e9",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "## Task 2 - Early stopping\n",
    "\n",
    "Implement early stopping. Early stopping should halt the training process if the development loss and accuracy no longer improve. You can assume they no longer improve if there is no positive change for `patience` epochs where `patience` is usually set to 2 or 3. You can come up with a more sophisticated stopping criteria.\n",
    "\n",
    "When does the training process stop? \n",
    "What's the advantage of using early stopping?\n",
    "Summarize your findings in the following cell."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "1ca440ac010e8a9378bd40328202461d",
     "grade": true,
     "grade_id": "cell-096d202c82100b14",
     "locked": false,
     "points": 0,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "source": [
    "YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "63848f6c28832f48869a842589745a33",
     "grade": false,
     "grade_id": "cell-f252d069012d47aa",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "## Task 3 - Shuffling batches\n",
    "\n",
    "`BatchedIterator` currently yields each batch in the order they are in. Add a shuffling option to the class. If True, it should yield the batches in random order. Make sure that it is randomized for each epoch, not just once.\n",
    "\n",
    "Does it improve the training process?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "35d4244f07b1c9fe626321f996ec4d88",
     "grade": true,
     "grade_id": "cell-c78d7d326d949eee",
     "locked": false,
     "points": 0,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "source": [
    "YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "b43f4e76a74cf78cf42d38a82081cd1c",
     "grade": false,
     "grade_id": "cell-258a4fc7ddf91b6b",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "## Task 4 - Sorting by length\n",
    "\n",
    "The inputs of recurrent neural networks are often sorted in decreasing order by sequence length. Sort the train data by length.\n",
    "\n",
    "You should only do this on the train set.\n",
    "\n",
    "Summarize your findings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "331fcda7ebdceb246e6fbd4fad0a8222",
     "grade": true,
     "grade_id": "cell-58b2fdc68e865f3b",
     "locked": false,
     "points": 0,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "source": [
    "YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "2f982109d4d4fc74db47908a1660d941",
     "grade": false,
     "grade_id": "cell-0ddea6154dd81fa2",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "## Task 5 - Another dataset\n",
    "\n",
    "Find another reasonable classification task in [Unimorph](https://unimorph.github.io/).\n",
    "You can pick any of the 110 languages as long as there is enough data.\n",
    "\n",
    "The errors are listed at the end of the notebook.\n",
    "What kind of errors does the model make?\n",
    "If you understand the language, try to explain the source of the errors.\n",
    "\n",
    "Summarize your findings on the new dataset in the following cell."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "c68b86561dde7c3aecf285b9d1d412ef",
     "grade": true,
     "grade_id": "cell-743e0d5134f7c336",
     "locked": false,
     "points": 0,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "source": [
    "YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "ada11848f8d28260df76a3306a2c64bc",
     "grade": false,
     "grade_id": "cell-4dabdc5ff99757e0",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "## Task 6 - Padding at the batch level\n",
    "\n",
    "Now that the data is sorted by length, most batches have far too many PAD symbols. Change the padding process so that it works at the batch level and not the dataset level. You should write a new `PaddedBatchIterator` class for this.\n",
    "\n",
    "Summarize your findings in the following cell."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "1c1f73c504fd1586c1098056534667e7",
     "grade": true,
     "grade_id": "cell-c25374199154ace8",
     "locked": false,
     "points": 0,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "source": [
    "YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import os\n",
    "import subprocess\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "from collections import defaultdict\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cloning the data repository"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "language = \"hun\"\n",
    "unimorph_path = f\"data/unimorph_{language}/\"\n",
    "pipe = subprocess.Popen(f\"git clone https://github.com/unimorph/{language}.git {unimorph_path}\",\n",
    "                        shell=True, stderr=subprocess.PIPE, stdout=subprocess.PIPE)\n",
    "stdout, stderr = pipe.communicate()\n",
    "print(stdout.decode('utf8'))\n",
    "print(stderr.decode('utf8'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data loading"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.read_table(f\"{unimorph_path}/{language}\", names=['lemma', 'infl', 'tags'], skip_blank_lines=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Target extraction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_target(tags_str):\n",
    "    \"\"\"Extracts target if present, returns None otherwise.\"\"\"\n",
    "    tags = tags_str.split(\";\")\n",
    "    if tags[0] != 'V':\n",
    "        return None\n",
    "    if len(tags) < 6:\n",
    "        return None\n",
    "    return tags[1]\n",
    " \n",
    "data['target'] = data.tags.apply(extract_target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = data[data.target.notnull()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data['target'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train/dev/test set creation\n",
    "\n",
    "We avoid lemma overlaps between the splits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lemmas = data.lemma.unique()\n",
    "len(lemmas), type(lemmas)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(12)\n",
    "np.random.shuffle(lemmas)\n",
    "train_size = int(0.8 * len(lemmas))\n",
    "dev_size = int(0.1 * len(lemmas))\n",
    "train_lemmas = lemmas[:train_size]\n",
    "dev_lemmas = lemmas[train_size:train_size+dev_size]\n",
    "test_lemmas = lemmas[train_size+dev_size:]\n",
    "\n",
    "train_lemmas = set(train_lemmas)\n",
    "dev_lemmas = set(dev_lemmas)\n",
    "test_lemmas = set(test_lemmas)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df = data[data.lemma.isin(train_lemmas)]\n",
    "dev_df = data[data.lemma.isin(dev_lemmas)]\n",
    "test_df = data[data.lemma.isin(test_lemmas)]\n",
    "len(train_df), len(dev_df), len(test_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df = train_df.sample(1000, random_state=1).reset_index(drop=True)\n",
    "dev_df = dev_df.sample(200, random_state=1).reset_index(drop=True)\n",
    "test_df = test_df.sample(200, random_state=1).reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "alphabet = set()\n",
    "for token in train_df.infl:\n",
    "    alphabet |= set(token)\n",
    "len(alphabet)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "alphabet.add('<PAD>')\n",
    "alphabet.add('<BOS>')\n",
    "alphabet.add('<EOS>')\n",
    "alphabet.add('<UNK>')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab = {symbol: i for i, symbol in enumerate(alphabet)}\n",
    "len(vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def encode_token(token):\n",
    "    ids = []\n",
    "    ids.append(vocab['<BOS>'])\n",
    "    # dev and test might contain characters outside the alphabet\n",
    "    ids.extend(vocab.get(c, vocab['<UNK>']) for c in token)\n",
    "    ids.append(vocab['<EOS>'])\n",
    "    return ids\n",
    "\n",
    "print(f\"{encode_token('alma') = }\")\n",
    "print(f\"{vocab['<UNK>'] = }\")\n",
    "print(f\"{encode_token('ALMA') = }\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df['encoded'] = train_df.infl.apply(encode_token)\n",
    "dev_df['encoded'] = dev_df.infl.apply(encode_token)\n",
    "test_df['encoded'] = test_df.infl.apply(encode_token)\n",
    "train_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "maxlen = train_df.encoded.apply(len).max()\n",
    "print(maxlen)\n",
    "\n",
    "def pad_sequence(sequence):\n",
    "    if len(sequence) > maxlen:\n",
    "        return sequence[:maxlen]\n",
    "    return sequence + [vocab['<PAD>'] for _ in range(maxlen-len(sequence))]\n",
    "\n",
    "print(pad_sequence([1, 2, 3]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df['padded'] = train_df.encoded.apply(pad_sequence)\n",
    "dev_df['padded'] = dev_df.encoded.apply(pad_sequence)\n",
    "test_df['padded'] = test_df.encoded.apply(pad_sequence)\n",
    "\n",
    "train_df['padded'].apply(len).value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df['seqlen'] = train_df.encoded.apply(len)\n",
    "dev_df['seqlen'] = dev_df.encoded.apply(len)\n",
    "test_df['seqlen'] = test_df.encoded.apply(len)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "label_to_id = {label: i for i, label in enumerate(train_df.target.unique())}\n",
    "label_to_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df['label'] = train_df.target.apply(lambda c: label_to_id[c])\n",
    "dev_df['label'] = dev_df.target.apply(lambda c: label_to_id[c])\n",
    "test_df['label'] = test_df.target.apply(lambda c: label_to_id[c])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creating tensors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = torch.from_numpy(np.array(list(train_df.padded)))\n",
    "y_train = torch.LongTensor(train_df.label.values)\n",
    "seqlen_train = torch.LongTensor(train_df.seqlen.values)\n",
    "print(f\"{X_train.size() = },\\n{y_train.size() = }\\n{seqlen_train.size() = }\\n\")\n",
    "\n",
    "X_dev = torch.from_numpy(np.array(list(dev_df.padded)))\n",
    "y_dev = torch.LongTensor(dev_df.label.values)\n",
    "seqlen_dev = torch.LongTensor(dev_df.seqlen.values)\n",
    "print(f\"{X_dev.size() = },\\n{y_dev.size() = }\\n{seqlen_dev.size() = }\\n\")\n",
    "\n",
    "X_test = torch.from_numpy(np.array(list(test_df.padded)))\n",
    "y_test = torch.LongTensor(test_df.label.values)\n",
    "seqlen_test = torch.LongTensor(test_df.seqlen.values)\n",
    "print(f\"{X_test.size() = },\\n{y_test.size() = }\\n{seqlen_test.size() = }\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class LSTMClassifier(nn.Module):\n",
    "    def __init__(self, input_size, embedding_size, hidden_size, output_size):\n",
    "        super().__init__()\n",
    "        self.embedding = nn.Embedding(input_size, embedding_size)\n",
    "        self.lstm = nn.LSTM(\n",
    "            input_size=embedding_size,\n",
    "            hidden_size=hidden_size,\n",
    "            num_layers=1,\n",
    "            bidirectional=True,\n",
    "            batch_first=True,\n",
    "        )\n",
    "        self.dense = nn.Linear(hidden_size * 2, output_size)\n",
    "        \n",
    "    # the input signature of forward changes\n",
    "    def forward(self, sequences, sequence_lens):\n",
    "        embedded = self.embedding(sequences)\n",
    "        \n",
    "        # THIS IS THE MODIFIED PART\n",
    "        # returns a PackedSequence object\n",
    "        packed = nn.utils.rnn.pack_padded_sequence(\n",
    "            embedded,\n",
    "            sequence_lens,\n",
    "            enforce_sorted=False,\n",
    "            batch_first=True)\n",
    "        packed_outputs, (h, c) = self.lstm(packed)\n",
    "        # extract LSTM outputs (not used here)\n",
    "        lstm_outputs, lens = nn.utils.rnn.pad_packed_sequence(packed_outputs)\n",
    "        \n",
    "        h = torch.cat((h[0], h[1]), dim=-1)\n",
    "        output = self.dense(h)\n",
    "        return output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_size = len(vocab)\n",
    "embedding_size = 30\n",
    "hidden_size = 64\n",
    "output_size = train_df.label.nunique()\n",
    "\n",
    "model = LSTMClassifier(input_size, embedding_size, hidden_size, output_size)\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Batching"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class BatchedIterator:\n",
    "    def __init__(self, *tensors, batch_size):\n",
    "        # all tensors must have the same first dimension\n",
    "        assert len(set(len(tensor) for tensor in tensors)) == 1\n",
    "        self.tensors = tensors\n",
    "        self.batch_size = batch_size\n",
    "    \n",
    "    def iterate_once(self):\n",
    "        num_data = len(self.tensors[0])\n",
    "        for start in range(0, num_data, self.batch_size):\n",
    "            end = start + self.batch_size\n",
    "            yield tuple(tensor[start:end] for tensor in self.tensors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = optim.Adam(model.parameters())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_epochs = 20\n",
    "batch_size = 128\n",
    "\n",
    "metrics = defaultdict(list)\n",
    "train_iter = BatchedIterator(X_train, seqlen_train, y_train, batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for epoch in range(num_epochs):\n",
    "    model.train()\n",
    "    # Training loop\n",
    "    for X_batch, seqlen_batch, y_batch in train_iter.iterate_once():\n",
    "        y_out = model(X_batch, seqlen_batch)\n",
    "        loss = criterion(y_out, y_batch)\n",
    "        optimizer.zero_grad()\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        \n",
    "    model.eval()  # or model.train(False)\n",
    "    # Train and dev loss at the end of the epoch\n",
    "    y_out = model(X_train, seqlen_train)\n",
    "    train_loss = criterion(y_out, y_train).item()\n",
    "    metrics['train_loss'].append(train_loss)\n",
    "    labels = y_out.argmax(axis=1)\n",
    "    train_accuracy = (torch.eq(y_train, labels).sum() / labels.size(0)).item()\n",
    "    metrics['train_accuracy'].append(train_accuracy)\n",
    "    \n",
    "    y_out = model(X_dev, seqlen_dev)\n",
    "    dev_loss = criterion(y_out, y_dev).item()\n",
    "    metrics['dev_loss'].append(dev_loss)\n",
    "    labels = y_out.argmax(axis=1)\n",
    "    dev_accuracy = (torch.eq(y_dev, labels).sum() / labels.size(0)).item()\n",
    "    metrics['dev_accuracy'].append(dev_accuracy)\n",
    "    \n",
    "    print(f\"{epoch=} -- {train_loss=:.3f} - {train_accuracy=:.1%} - {dev_loss=:.3f} - {dev_accuracy=:.1%}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluation\n",
    "\n",
    "## Plot the results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(1, 2, figsize=(16, 4))\n",
    "\n",
    "sns.lineplot(data=metrics['train_loss'], ax=ax[0], label='train loss')\n",
    "sns.lineplot(data=metrics['dev_loss'], ax=ax[0], label='dev loss')\n",
    "\n",
    "sns.lineplot(data=metrics['train_accuracy'], ax=ax[1], label='train acc')\n",
    "sns.lineplot(data=metrics['dev_accuracy'], ax=ax[1], label='dev acc')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "logits = model(X_test, seqlen_test)\n",
    "test_prediction = logits.argmax(axis=1)\n",
    "test_accuracy = torch.sum(torch.eq(test_prediction, y_test)) / test_prediction.size(0)\n",
    "print(f\"Test accuracy: {test_accuracy:.1%}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Error analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_df['prediction'] = test_prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "id_to_label = {i: l for l, i in label_to_id.items()}\n",
    "test_df['predicted_target'] = test_df['prediction'].apply(lambda id_: id_to_label[id_])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "incorrect = test_df[test_df.prediction != test_df.label][['infl', 'target', 'predicted_target']]\n",
    "incorrect.sample(min(len(incorrect), 20))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}