Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ada62d9a-445b-412e-97e0-332901d59370",
   "metadata": {},
   "source": [
    "# Post-Hoc Experimentation for the [Clickbait Challenge at SemEval 2023](https://pan.webis.de/semeval23/pan23-web/clickbait-challenge.html)\n",
    "\n",
    "This notebook serves as tutorial on post-hoc experiments on the [Clickbait Challenge at SemEval 2023](https://pan.webis.de/semeval23/pan23-web/clickbait-challenge.html).\n",
    "The notebook covers:\n",
    "\n",
    "- 1.) Diving into artifacts submitted to the shared task\n",
    "- 2.) Re-evaluation of submitted approaches (e.g., a different subset of the data or different measures)\n",
    "- 3.) Execution of submitted approaches on new or manipulated data (e.g., for ablation studies or ensembles)\n",
    "\n",
    "All software submissions and evaluators come as docker images.\n",
    "Hence, you a minimal environment is sufficient: You need `Python3` and `Docker`.\n",
    "\n",
    "The installation of the environment is simplified with a virtual environment and executing `make jupyterlab` installs the virtual environment (if not already done) and starts the jupyter notebook ready to run all parts of the tutorial.\n",
    "\n",
    "**This tutorial currently includes only the baselines as the shared task is currently running (to prevent side effects to participants). After completion of the shared task, all submitted approaches can be executed and evaluated (as they are all available in Docker images) as showcased with the baselines.***"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47e11ece-1b08-4149-963c-f80380c8be46",
   "metadata": {},
   "source": [
    "### Import Requirements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "53aba3b7-6d2b-4a27-88f1-d7c40b6978a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "pd.set_option('display.max_colwidth', 0)\n",
    "import tira\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c76a8c64-a42e-4f8a-9316-d092e0e3552f",
   "metadata": {},
   "source": [
    "# 1.) Overview of Artifacts submitted to the Shared Task"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "8ccfb9a2-c4b4-4791-bae4-7ace344f4b90",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>approach</th>\n",
       "      <th>dataset</th>\n",
       "      <th>result-size</th>\n",
       "      <th>bleu-score</th>\n",
       "      <th>missing-predictions</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>clickbait-spoiling/princess-knight/express-pitch</td>\n",
       "      <td>task-2-spoiler-generation-validation-20220924-training</td>\n",
       "      <td>800</td>\n",
       "      <td>0.02111</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>clickbait-spoiling/princess-knight/express-pitch</td>\n",
       "      <td>task-2-spoiler-generation-validation-20220924-training</td>\n",
       "      <td>800</td>\n",
       "      <td>0.02111</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>clickbait-spoiling/princess-knight/express-pitch</td>\n",
       "      <td>task-2-spoiler-generation-validation-20220924-training</td>\n",
       "      <td>800</td>\n",
       "      <td>0.02111</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           approach  \\\n",
       "0  clickbait-spoiling/princess-knight/express-pitch   \n",
       "1  clickbait-spoiling/princess-knight/express-pitch   \n",
       "2  clickbait-spoiling/princess-knight/express-pitch   \n",
       "\n",
       "                                                  dataset  result-size  \\\n",
       "0  task-2-spoiler-generation-validation-20220924-training  800           \n",
       "1  task-2-spoiler-generation-validation-20220924-training  800           \n",
       "2  task-2-spoiler-generation-validation-20220924-training  800           \n",
       "\n",
       "   bleu-score  missing-predictions  \n",
       "0  0.02111     0                    \n",
       "1  0.02111     0                    \n",
       "2  0.02111     0                    "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Overview of evaluated approaches (Only baselines at the moment)\n",
    "tira.all_evaluated_appraoches()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "c52bb8ce-71ca-4210-9f10-568e479debd2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>dataset</th>\n",
       "      <th>image</th>\n",
       "      <th>command</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>task-1-type-classification-validation-20220924-training</td>\n",
       "      <td>webis/pan-clickbait-spoiling-evaluator:0.0.10</td>\n",
       "      <td>bash -c '/clickbait-spoiling-eval.py --task 1 --ground_truth_classes $inputDataset --input_run $inputRun --output_prototext ${outputDir}/evaluation.prototext'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>task-2-spoiler-generation-validation-20220924-training</td>\n",
       "      <td>webis/pan-clickbait-spoiling-evaluator:0.0.10</td>\n",
       "      <td>bash -c '/clickbait-spoiling-eval.py --task 2 --ground_truth_spoiler $inputDataset --input_run $inputRun --output_prototext ${outputDir}/evaluation.prototext'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>del-me-20220813-training</td>\n",
       "      <td>webis/pan-clickbait-spoiling-evaluator:0.0.3</td>\n",
       "      <td>bash -c '/clickbait-spoiling-eval.py --task 1 --ground_truth_classes $inputDataset --input_run $inputRun --output_prototext ${outputDir}/evaluation.prototext'</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   dataset  \\\n",
       "0  task-1-type-classification-validation-20220924-training   \n",
       "1  task-2-spoiler-generation-validation-20220924-training    \n",
       "2  del-me-20220813-training                                  \n",
       "\n",
       "                                           image  \\\n",
       "0  webis/pan-clickbait-spoiling-evaluator:0.0.10   \n",
       "1  webis/pan-clickbait-spoiling-evaluator:0.0.10   \n",
       "2  webis/pan-clickbait-spoiling-evaluator:0.0.3    \n",
       "\n",
       "                                                                                                                                                          command  \n",
       "0  bash -c '/clickbait-spoiling-eval.py --task 1 --ground_truth_classes $inputDataset --input_run $inputRun --output_prototext ${outputDir}/evaluation.prototext'  \n",
       "1  bash -c '/clickbait-spoiling-eval.py --task 2 --ground_truth_spoiler $inputDataset --input_run $inputRun --output_prototext ${outputDir}/evaluation.prototext'  \n",
       "2  bash -c '/clickbait-spoiling-eval.py --task 1 --ground_truth_classes $inputDataset --input_run $inputRun --output_prototext ${outputDir}/evaluation.prototext'  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tira.all_evaluators()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "f6aabd7e-f2f4-4b62-874c-b2eb69cb4a9e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>approach</th>\n",
       "      <th>team</th>\n",
       "      <th>image</th>\n",
       "      <th>command</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>clickbait-spoiling/bg40lecu/absolute-officer</td>\n",
       "      <td>bg40lecu</td>\n",
       "      <td>docker.io/webis/clickbait-spoiling-submissions:bg40lecu-my-software-0-0-1-tira-docker-software-id-ebony-groove</td>\n",
       "      <td>sleep 3m</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>clickbait-spoiling/princess-knight/immature-ramp</td>\n",
       "      <td>princess-knight</td>\n",
       "      <td>docker.io/webis/clickbait-spoiling-submissions:princess-knight-my-software-0-0-1-tira-docker-software-id-cloudy-duck</td>\n",
       "      <td>sleep 70m</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>clickbait-spoiling/princess-knight/brass-cheetah</td>\n",
       "      <td>princess-knight</td>\n",
       "      <td>docker.io/webis/clickbait-spoiling-submissions:princess-knight-my-software-0-0-1-tira-docker-software-id-isomorphic-burger</td>\n",
       "      <td>cat $inputDataset/input.jsonl &gt; $outputDir/run.jsonl</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>clickbait-spoiling/princess-knight/potential-borzoi</td>\n",
       "      <td>princess-knight</td>\n",
       "      <td>docker.io/webis/clickbait-spoiling-submissions:princess-knight-naive-baseline-task2-0-0-1-tira-docker-software-id-icy-flag</td>\n",
       "      <td>/naive-baseline-task-2.py --input $inputDataset/input.jsonl --output $outputDir/run.jsonl</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>clickbait-spoiling/bg40lecu/radiant-history</td>\n",
       "      <td>bg40lecu</td>\n",
       "      <td>docker.io/webis/clickbait-spoiling-submissions:bg40lecu-my-software-0-0-1-tira-docker-software-id-linear-headset</td>\n",
       "      <td>ls</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>clickbait-spoiling/princess-knight/muffled-exponential</td>\n",
       "      <td>princess-knight</td>\n",
       "      <td>docker.io/webis/clickbait-spoiling-submissions:princess-knight-my-software-0-0-1-tira-docker-software-id-nimble-groove</td>\n",
       "      <td>echo \"my final test number 4\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>clickbait-spoiling/princess-knight/crazy-ghoul</td>\n",
       "      <td>princess-knight</td>\n",
       "      <td>docker.io/webis/clickbait-spoiling-submissions:princess-knight-my-software-0-0-1-tira-docker-software-id-latent-decibel</td>\n",
       "      <td>cat /sys/fs/cgroup/cpu/cpuacct.usage; cat /sys/fs/cgroup/memory/memory.usage_in_bytes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>clickbait-spoiling/princess-knight/primary-anthology</td>\n",
       "      <td>princess-knight</td>\n",
       "      <td>docker.io/webis/clickbait-spoiling-submissions:princess-knight-naive-baseline-task1-0-0-1-tira-docker-software-id-augmenting-highway</td>\n",
       "      <td>/naive-baseline-task-1.py --input $inputDataset/input.jsonl --output $outputDir/run.jsonl</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>clickbait-spoiling/princess-knight/express-pitch</td>\n",
       "      <td>princess-knight</td>\n",
       "      <td>docker.io/webis/clickbait-spoiling-submissions:princess-knight-naive-baseline-task2-0-0-1-tira-docker-software-id-genteel-upstream</td>\n",
       "      <td>/naive-baseline-task-2.py --input $inputDataset/input.jsonl --output $outputDir/run.jsonl</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>clickbait-spoiling/princess-knight/many-process</td>\n",
       "      <td>princess-knight</td>\n",
       "      <td>docker.io/webis/clickbait-spoiling-submissions:princess-knight-my-software-0-0-1-tira-docker-software-id-senile-oregano</td>\n",
       "      <td>echo -e \"\\n\\ncpuset.cpu_exclusive\"; cat /sys/fs/cgroup/cpuset/cpuset.cpu_exclusive; echo -e \"\\n\\nmemory.max_usage_in_bytes\"; cat /sys/fs/cgroup/memory/memory.max_usage_in_bytes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>clickbait-spoiling/princess-knight/shortest-keyword</td>\n",
       "      <td>princess-knight</td>\n",
       "      <td>docker.io/webis/clickbait-spoiling-submissions:princess-knight-my-software-0-0-1-tira-docker-software-id-solid-align</td>\n",
       "      <td>echo \"hello world\"</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                  approach             team  \\\n",
       "0   clickbait-spoiling/bg40lecu/absolute-officer            bg40lecu          \n",
       "1   clickbait-spoiling/princess-knight/immature-ramp        princess-knight   \n",
       "2   clickbait-spoiling/princess-knight/brass-cheetah        princess-knight   \n",
       "3   clickbait-spoiling/princess-knight/potential-borzoi     princess-knight   \n",
       "4   clickbait-spoiling/bg40lecu/radiant-history             bg40lecu          \n",
       "5   clickbait-spoiling/princess-knight/muffled-exponential  princess-knight   \n",
       "6   clickbait-spoiling/princess-knight/crazy-ghoul          princess-knight   \n",
       "7   clickbait-spoiling/princess-knight/primary-anthology    princess-knight   \n",
       "8   clickbait-spoiling/princess-knight/express-pitch        princess-knight   \n",
       "9   clickbait-spoiling/princess-knight/many-process         princess-knight   \n",
       "10  clickbait-spoiling/princess-knight/shortest-keyword     princess-knight   \n",
       "\n",
       "                                                                                                                                   image  \\\n",
       "0   docker.io/webis/clickbait-spoiling-submissions:bg40lecu-my-software-0-0-1-tira-docker-software-id-ebony-groove                         \n",
       "1   docker.io/webis/clickbait-spoiling-submissions:princess-knight-my-software-0-0-1-tira-docker-software-id-cloudy-duck                   \n",
       "2   docker.io/webis/clickbait-spoiling-submissions:princess-knight-my-software-0-0-1-tira-docker-software-id-isomorphic-burger             \n",
       "3   docker.io/webis/clickbait-spoiling-submissions:princess-knight-naive-baseline-task2-0-0-1-tira-docker-software-id-icy-flag             \n",
       "4   docker.io/webis/clickbait-spoiling-submissions:bg40lecu-my-software-0-0-1-tira-docker-software-id-linear-headset                       \n",
       "5   docker.io/webis/clickbait-spoiling-submissions:princess-knight-my-software-0-0-1-tira-docker-software-id-nimble-groove                 \n",
       "6   docker.io/webis/clickbait-spoiling-submissions:princess-knight-my-software-0-0-1-tira-docker-software-id-latent-decibel                \n",
       "7   docker.io/webis/clickbait-spoiling-submissions:princess-knight-naive-baseline-task1-0-0-1-tira-docker-software-id-augmenting-highway   \n",
       "8   docker.io/webis/clickbait-spoiling-submissions:princess-knight-naive-baseline-task2-0-0-1-tira-docker-software-id-genteel-upstream     \n",
       "9   docker.io/webis/clickbait-spoiling-submissions:princess-knight-my-software-0-0-1-tira-docker-software-id-senile-oregano                \n",
       "10  docker.io/webis/clickbait-spoiling-submissions:princess-knight-my-software-0-0-1-tira-docker-software-id-solid-align                   \n",
       "\n",
       "                                                                                                                                                                             command  \n",
       "0   sleep 3m                                                                                                                                                                          \n",
       "1   sleep 70m                                                                                                                                                                         \n",
       "2   cat $inputDataset/input.jsonl > $outputDir/run.jsonl                                                                                                                              \n",
       "3   /naive-baseline-task-2.py --input $inputDataset/input.jsonl --output $outputDir/run.jsonl                                                                                         \n",
       "4   ls                                                                                                                                                                                \n",
       "5   echo \"my final test number 4\"                                                                                                                                                     \n",
       "6   cat /sys/fs/cgroup/cpu/cpuacct.usage; cat /sys/fs/cgroup/memory/memory.usage_in_bytes                                                                                             \n",
       "7   /naive-baseline-task-1.py --input $inputDataset/input.jsonl --output $outputDir/run.jsonl                                                                                         \n",
       "8   /naive-baseline-task-2.py --input $inputDataset/input.jsonl --output $outputDir/run.jsonl                                                                                         \n",
       "9   echo -e \"\\n\\ncpuset.cpu_exclusive\"; cat /sys/fs/cgroup/cpuset/cpuset.cpu_exclusive; echo -e \"\\n\\nmemory.max_usage_in_bytes\"; cat /sys/fs/cgroup/memory/memory.max_usage_in_bytes  \n",
       "10  echo \"hello world\"                                                                                                                                                                "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tira.all_softwares()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "4cd629b2-f73b-4a16-957e-531bfbb7ca93",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>dataset</th>\n",
       "      <th>records</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>task-1-type-classification-validation-20220924-training</td>\n",
       "      <td>800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>task-2-spoiler-generation-validation-20220924-training</td>\n",
       "      <td>800</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   dataset  records\n",
       "1  task-1-type-classification-validation-20220924-training  800    \n",
       "0  task-2-spoiler-generation-validation-20220924-training   800    "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tira.all_datasets()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef897e66-bac3-4407-b10d-1b9f2b19a2b5",
   "metadata": {},
   "source": [
    "# 2.) Re-evaluation of submitted approaches\n",
    "\n",
    "We run the approach `'clickbait-spoiling/princess-knight/express-pitch'` on the dataset `'task-2-spoiler-generation-validation-20220924-training'`.\n",
    "There, `clickbait-spoiling` is the task,  `princess-knight` is the group, and `express-pitch` is the approach of the group (`princess-knight` is the group that submitts all the baselines, and `express-pitch` is an baseline).\n",
    "\n",
    "**This tutorial currently includes only the baselines as the shared task is currently running (to prevent side effects to participants). After completion of the shared task, all submitted approaches can be executed and evaluated (as they are all available in Docker images) as showcased with the baselines.***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "5d8c1ad5-2be8-4ed7-8b22-546b75809c00",
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions, evaluation = tira.run(\n",
    "    'clickbait-spoiling/princess-knight/express-pitch',\n",
    "    data='task-2-spoiler-generation-validation-20220924-training',\n",
    "    evaluate=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "7db303a0-88e2-4ccf-990b-0663958df77e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>uuid</th>\n",
       "      <th>spoiler</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6dc7ddef-4e8e-4a6b-9296-526377518071</td>\n",
       "      <td>Five Nights at Freddy’s Sequel Delayed for Weird Reason</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>435b24de-56f6-4d4e-9c38-54b8e0630aac</td>\n",
       "      <td>Why Arizona Sheriff Joe Arpaio’s fate could hang on a single word</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8091ba93-6376-473a-9117-01d6cf0507e4</td>\n",
       "      <td>Here’s how much you should be tipping your hairdresser</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>8b713277-0e5b-4873-a216-b650f21f3b4c</td>\n",
       "      <td>Alan Rickman &amp; Rupert Grint On 'CBGB,' Reuniting Post-'Potter' And Favorite Halloween Costumes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>a2f91b65-c36c-481f-92b1-2fc77d6411fc</td>\n",
       "      <td>Man swallowed a microSD card and you won't believe what happened next!</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   uuid  \\\n",
       "0  6dc7ddef-4e8e-4a6b-9296-526377518071   \n",
       "1  435b24de-56f6-4d4e-9c38-54b8e0630aac   \n",
       "2  8091ba93-6376-473a-9117-01d6cf0507e4   \n",
       "3  8b713277-0e5b-4873-a216-b650f21f3b4c   \n",
       "4  a2f91b65-c36c-481f-92b1-2fc77d6411fc   \n",
       "\n",
       "                                                                                          spoiler  \n",
       "0  Five Nights at Freddy’s Sequel Delayed for Weird Reason                                         \n",
       "1  Why Arizona Sheriff Joe Arpaio’s fate could hang on a single word                               \n",
       "2  Here’s how much you should be tipping your hairdresser                                          \n",
       "3  Alan Rickman & Rupert Grint On 'CBGB,' Reuniting Post-'Potter' And Favorite Halloween Costumes  \n",
       "4  Man swallowed a microSD card and you won't believe what happened next!                          "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Show some predictions of the approach 'clickbait-spoiling/princess-knight/express-pitch'\n",
    "\n",
    "predictions.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "a4aa90db-be8d-4fd6-9002-6100c6ec05dd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>approach</th>\n",
       "      <th>evaluate</th>\n",
       "      <th>result-size</th>\n",
       "      <th>bleu-score</th>\n",
       "      <th>missing-predictions</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>clickbait-spoiling/princess-knight/express-pitch</td>\n",
       "      <td>task-2-spoiler-generation-validation-20220924-training</td>\n",
       "      <td>800</td>\n",
       "      <td>0.02111</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           approach  \\\n",
       "0  clickbait-spoiling/princess-knight/express-pitch   \n",
       "\n",
       "                                                 evaluate  result-size  \\\n",
       "0  task-2-spoiler-generation-validation-20220924-training  800           \n",
       "\n",
       "   bleu-score  missing-predictions  \n",
       "0  0.02111     0                    "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Show evaluation results for the approach 'clickbait-spoiling/princess-knight/express-pitch' on the full data 'task-2-spoiler-generation-validation-20220924-training'\n",
    "evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a85e6649-0ebc-4e30-8b61-aa1bdc85b404",
   "metadata": {},
   "source": [
    "# 3.) Execution of submitted approaches on new or manipulated data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea0c0c13-6fdc-4265-a7ab-6f492420f9ca",
   "metadata": {},
   "source": [
    "### 3.1.) Execution of submitted approaches on new data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "191ec2f6-3e74-42d3-ab6e-5cbd36ed56bf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>uuid</th>\n",
       "      <th>postText</th>\n",
       "      <th>targetParagraphs</th>\n",
       "      <th>targetTitle</th>\n",
       "      <th>spoiler</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>example-1</td>\n",
       "      <td>[This \"Star Wars\" theory about Rey will blow your mind]</td>\n",
       "      <td>[But a new theory has awakened: that Rey is in fact Emperor Palpatine's granddaughter.]</td>\n",
       "      <td>This \"Star Wars\" Theory About Rey Is Insane</td>\n",
       "      <td>[that Rey is in fact Emperor Palpatine's granddaughter]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>example-2</td>\n",
       "      <td>[J.K. Rowling has revealed a heartbreaking truth about Hagrid]</td>\n",
       "      <td>[Rowling's response may point to the fact that the spell is simply too difficult for Hagrid, who was expelled from Hogwarts as a teenager, to perform. But fans were also worried that the lovable half-giant, abandoned by his mother as a child, may just not have enough happy memories to produce a Patronus. Oh, Hagrid!]</td>\n",
       "      <td>J.K. Rowling Has Revealed A Heartbreaking Truth About Hagrid</td>\n",
       "      <td>not have enough happy memories to produce a Patronus</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        uuid                                                        postText  \\\n",
       "0  example-1  [This \"Star Wars\" theory about Rey will blow your mind]          \n",
       "1  example-2  [J.K. Rowling has revealed a heartbreaking truth about Hagrid]   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                 targetParagraphs  \\\n",
       "0  [But a new theory has awakened: that Rey is in fact Emperor Palpatine's granddaughter.]                                                                                                                                                                                                                                          \n",
       "1  [Rowling's response may point to the fact that the spell is simply too difficult for Hagrid, who was expelled from Hogwarts as a teenager, to perform. But fans were also worried that the lovable half-giant, abandoned by his mother as a child, may just not have enough happy memories to produce a Patronus. Oh, Hagrid!]   \n",
       "\n",
       "                                                    targetTitle  \\\n",
       "0  This \"Star Wars\" Theory About Rey Is Insane                    \n",
       "1  J.K. Rowling Has Revealed A Heartbreaking Truth About Hagrid   \n",
       "\n",
       "                                                   spoiler  \n",
       "0  [that Rey is in fact Emperor Palpatine's granddaughter]  \n",
       "1  not have enough happy memories to produce a Patronus     "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Here we create an new dataset consisting of two entries\n",
    "\n",
    "df = pd.DataFrame([\n",
    "    {\"uuid\": \"example-1\", \"postText\": [\"This \\\"Star Wars\\\" theory about Rey will blow your mind\"], \"targetParagraphs\": [\"But a new theory has awakened: that Rey is in fact Emperor Palpatine's granddaughter.\"], \"targetTitle\": \"This \\\"Star Wars\\\" Theory About Rey Is Insane\", \"spoiler\": [\"that Rey is in fact Emperor Palpatine's granddaughter\"]},\n",
    "    {\"uuid\": \"example-2\", \"postText\": [\"J.K. Rowling has revealed a heartbreaking truth about Hagrid\"], \"targetTitle\": \"J.K. Rowling Has Revealed A Heartbreaking Truth About Hagrid\",  \"targetParagraphs\": [\"Rowling's response may point to the fact that the spell is simply too difficult for Hagrid, who was expelled from Hogwarts as a teenager, to perform. But fans were also worried that the lovable half-giant, abandoned by his mother as a child, may just not have enough happy memories to produce a Patronus. Oh, Hagrid!\"], \"spoiler\": \"not have enough happy memories to produce a Patronus\"},\n",
    "])\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "f6e6ee7f-b178-4a58-bcc4-42e7e040ce6d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Apply some approach to the dataset to get the predictions and the evaluation on the new data\n",
    "predictions, evaluation = tira.run('clickbait-spoiling/princess-knight/express-pitch', data=df, evaluate='task-2-spoiler-generation')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "cd2d11e3-cbc3-4ca8-b8b6-34f7ea0dada9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>uuid</th>\n",
       "      <th>spoiler</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>example-1</td>\n",
       "      <td>This \"Star Wars\" Theory About Rey Is Insane</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>example-2</td>\n",
       "      <td>J.K. Rowling Has Revealed A Heartbreaking Truth About Hagrid</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        uuid                                                       spoiler\n",
       "0  example-1  This \"Star Wars\" Theory About Rey Is Insane                 \n",
       "1  example-2  J.K. Rowling Has Revealed A Heartbreaking Truth About Hagrid"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Show the predictions of the approach 'clickbait-spoiling/princess-knight/express-pitch' on the new dataset\n",
    "predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "95f2dd45-ce97-4db2-bb36-f3f1a1256d95",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>approach</th>\n",
       "      <th>evaluate</th>\n",
       "      <th>result-size</th>\n",
       "      <th>bleu-score</th>\n",
       "      <th>missing-predictions</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>clickbait-spoiling/princess-knight/express-pitch</td>\n",
       "      <td>task-2-spoiler-generation-validation-20220924-training</td>\n",
       "      <td>2</td>\n",
       "      <td>5.122457e-232</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           approach  \\\n",
       "0  clickbait-spoiling/princess-knight/express-pitch   \n",
       "\n",
       "                                                 evaluate  result-size  \\\n",
       "0  task-2-spoiler-generation-validation-20220924-training  2             \n",
       "\n",
       "      bleu-score  missing-predictions  \n",
       "0  5.122457e-232  0                    "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Show evaluation results for the approach 'clickbait-spoiling/princess-knight/express-pitch' on the new dataset\n",
    "evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a48297b4-57af-4d8b-b131-a3a762fff0ee",
   "metadata": {},
   "source": [
    "### 3.2.) Execution of submitted approaches on manipulated data\n",
    "\n",
    "tbd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "240f35d5-c9c3-4e4a-99c6-a30502a18951",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>uuid</th>\n",
       "      <th>postId</th>\n",
       "      <th>postText</th>\n",
       "      <th>postPlatform</th>\n",
       "      <th>targetParagraphs</th>\n",
       "      <th>targetTitle</th>\n",
       "      <th>targetDescription</th>\n",
       "      <th>targetKeywords</th>\n",
       "      <th>targetMedia</th>\n",
       "      <th>targetUrl</th>\n",
       "      <th>provenance</th>\n",
       "      <th>spoiler</th>\n",
       "      <th>spoilerPositions</th>\n",
       "      <th>tags</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6dc7ddef-4e8e-4a6b-9296-526377518071</td>\n",
       "      <td>800048986762423_885081814925806</td>\n",
       "      <td></td>\n",
       "      <td>Facebook</td>\n",
       "      <td>[Five Nights at Freddy’s creator Scott Cawthon...</td>\n",
       "      <td>Five Nights at Freddy’s Sequel Delayed for Wei...</td>\n",
       "      <td>Five Nights at Freddy's creator Scott Cawthon ...</td>\n",
       "      <td>None</td>\n",
       "      <td>[https://gamerant.com/wp-content/uploads/five-...</td>\n",
       "      <td>https://gamerant.com/five-nights-at-freddys-se...</td>\n",
       "      <td>{'source': 'anonymized', 'humanSpoiler': 'They...</td>\n",
       "      <td>[some of the plot elements are so disturbing t...</td>\n",
       "      <td>[[[2, 158], [2, 236]]]</td>\n",
       "      <td>[passage]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>435b24de-56f6-4d4e-9c38-54b8e0630aac</td>\n",
       "      <td>4jp20e</td>\n",
       "      <td></td>\n",
       "      <td>reddit</td>\n",
       "      <td>[© REUTERS/Laura Segall Maricopa County Sherif...</td>\n",
       "      <td>Why Arizona Sheriff Joe Arpaio’s fate could ha...</td>\n",
       "      <td>&lt;p&gt;A single word — \"intentionally\" — could tra...</td>\n",
       "      <td>None</td>\n",
       "      <td>[https://archive.is/0eGBT/bbf6d79b366fc7e75b8b...</td>\n",
       "      <td>https://archive.is/0eGBT</td>\n",
       "      <td>{'source': 'anonymized', 'humanSpoiler': '\"Int...</td>\n",
       "      <td>[\"intentionally\", could transform a court case...</td>\n",
       "      <td>[[[0, 197], [0, 212]], [[0, 215], [0, 328]]]</td>\n",
       "      <td>[multi]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   uuid                           postId  \\\n",
       "0  6dc7ddef-4e8e-4a6b-9296-526377518071  800048986762423_885081814925806   \n",
       "1  435b24de-56f6-4d4e-9c38-54b8e0630aac                           4jp20e   \n",
       "\n",
       "  postText postPlatform                                   targetParagraphs  \\\n",
       "0              Facebook  [Five Nights at Freddy’s creator Scott Cawthon...   \n",
       "1                reddit  [© REUTERS/Laura Segall Maricopa County Sherif...   \n",
       "\n",
       "                                         targetTitle  \\\n",
       "0  Five Nights at Freddy’s Sequel Delayed for Wei...   \n",
       "1  Why Arizona Sheriff Joe Arpaio’s fate could ha...   \n",
       "\n",
       "                                   targetDescription targetKeywords  \\\n",
       "0  Five Nights at Freddy's creator Scott Cawthon ...           None   \n",
       "1  <p>A single word — \"intentionally\" — could tra...           None   \n",
       "\n",
       "                                         targetMedia  \\\n",
       "0  [https://gamerant.com/wp-content/uploads/five-...   \n",
       "1  [https://archive.is/0eGBT/bbf6d79b366fc7e75b8b...   \n",
       "\n",
       "                                           targetUrl  \\\n",
       "0  https://gamerant.com/five-nights-at-freddys-se...   \n",
       "1                           https://archive.is/0eGBT   \n",
       "\n",
       "                                          provenance  \\\n",
       "0  {'source': 'anonymized', 'humanSpoiler': 'They...   \n",
       "1  {'source': 'anonymized', 'humanSpoiler': '\"Int...   \n",
       "\n",
       "                                             spoiler  \\\n",
       "0  [some of the plot elements are so disturbing t...   \n",
       "1  [\"intentionally\", could transform a court case...   \n",
       "\n",
       "                               spoilerPositions       tags  \n",
       "0                        [[[2, 158], [2, 236]]]  [passage]  \n",
       "1  [[[0, 197], [0, 212]], [[0, 215], [0, 328]]]    [multi]  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Manipulate the dataset for an ablation study: Has the postText an impact on the effectiveness? We remove the postText and then evaluate the approaches.\n",
    "\n",
    "df = tira.load_data('task-2-spoiler-generation-validation')\n",
    "\n",
    "pd.set_option('display.max_colwidth', 50)\n",
    "\n",
    "df['postText'] = ''\n",
    "df.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "d628ee41-9d5e-4060-85fc-360e00635d2f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Apply some approach to the manipulated dataset (postText removed) to get the predictions and the evaluation on the manipulated data\n",
    "predictions, evaluation = tira.run('clickbait-spoiling/princess-knight/express-pitch', data=df, evaluate='task-2-spoiler-generation')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "2f73eb6c-f5a0-456e-a602-1320ccd67e09",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>uuid</th>\n",
       "      <th>spoiler</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6dc7ddef-4e8e-4a6b-9296-526377518071</td>\n",
       "      <td>Five Nights at Freddy’s Sequel Delayed for Wei...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>435b24de-56f6-4d4e-9c38-54b8e0630aac</td>\n",
       "      <td>Why Arizona Sheriff Joe Arpaio’s fate could ha...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8091ba93-6376-473a-9117-01d6cf0507e4</td>\n",
       "      <td>Here’s how much you should be tipping your hai...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   uuid  \\\n",
       "0  6dc7ddef-4e8e-4a6b-9296-526377518071   \n",
       "1  435b24de-56f6-4d4e-9c38-54b8e0630aac   \n",
       "2  8091ba93-6376-473a-9117-01d6cf0507e4   \n",
       "\n",
       "                                             spoiler  \n",
       "0  Five Nights at Freddy’s Sequel Delayed for Wei...  \n",
       "1  Why Arizona Sheriff Joe Arpaio’s fate could ha...  \n",
       "2  Here’s how much you should be tipping your hai...  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Show the predictions of the approach 'clickbait-spoiling/princess-knight/express-pitch' on the manipulated dataset\n",
    "predictions.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "73b989db-4364-4156-b801-6c6385527da7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>approach</th>\n",
       "      <th>evaluate</th>\n",
       "      <th>result-size</th>\n",
       "      <th>bleu-score</th>\n",
       "      <th>missing-predictions</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>clickbait-spoiling/princess-knight/express-pitch</td>\n",
       "      <td>task-2-spoiler-generation-validation-20220924-...</td>\n",
       "      <td>800</td>\n",
       "      <td>0.02111</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           approach  \\\n",
       "0  clickbait-spoiling/princess-knight/express-pitch   \n",
       "\n",
       "                                            evaluate  result-size  bleu-score  \\\n",
       "0  task-2-spoiler-generation-validation-20220924-...          800     0.02111   \n",
       "\n",
       "   missing-predictions  \n",
       "0                    0  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Show evaluation results for the approach 'clickbait-spoiling/princess-knight/express-pitch' on the new dataset\n",
    "evaluation"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}