Train_Tokenizer_and_Prepare_Dataset.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5t_nVFwzYgFa"
      },
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github//jayralencar/pictoBERT/blob/main/Train_Tokenizer_and_Prepare_Dataset.ipynb \"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/jayralencar/pictoBERT/blob/main/Train_Tokenizer_and_Prepare_Dataset.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "frJfNrUOrJKG"
      },
      "source": [
        "# Train Tokenizer and Prepare Dataset\n",
        "\n",
        "This notebook presents the process for creating the PictoBERT tokenizer and preparing dataset.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-axoYjs4rX5O"
      },
      "source": [
        "## Dataset\n",
        "As our task is word-sense language modeling, we need a word-sense labeled dataset.  Besides, as the task consists of predicting word-senses in sequence, we need a dataset with all the nouns, verbs, adjectives, and adverbs labeled. The well-known and used dataset that comes closest to that is the SemCor 3.0 \\cite{miller1993semantic}, which is labeled with senses from WordNet 3.0 and counts with 20 thousand annotated sentences. However, it is too tiny for BERT pre-training, originally trained with a 3,300M words dataset. Also, SemCor has sentences in formal text rather than conversational, which we consider more significant for an also conversational task like pictogram prediction. \n",
        "\n",
        "The Child Language Data Exchange System (CHILDES) \\cite{macwhinney2014childes} is a ~2 million sentence multilingual corpus composed of transcribed children's speech. As it is from conversational data, we decide to use it as a training dataset. To make it possible, we labeled part of CHILDES with word-senses using SupWSD \\cite{papandreaetal:EMNLP2017Demos}. We choose sentences in North American English. The result is a 955 k sentence labeled corpus that we call SemCHILDES (Semantic CHILDES).\n",
        "\n",
        "This [Notebook](https://github.com/jayralencar/pictoBERT/blob/main/SemCHILDES.ipynb) present the procedure for building SemCHILDES.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "new0QpJYrtjD"
      },
      "source": [
        "### Download Dataset\n",
        "\n",
        "The dataset used in this nootebook can be downloaded [here](https://drive.google.com/file/d/18xuy-PmffJxTgG76x5nio9f18lCjE_kL/view?usp=sharing). Or running the following cell."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "z0q7JIR5q_vb",
        "outputId": "1490bd11-0e4a-48a8-e829-af9306c42f59"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Downloading...\n",
            "From: https://drive.google.com/uc?id=18xuy-PmffJxTgG76x5nio9f18lCjE_kL\n",
            "To: /content/all_mt_2.txt\n",
            "52.5MB [00:01, 47.0MB/s]\n"
          ]
        }
      ],
      "source": [
        "!wget http://jayr.clubedosgeeks.com.br/pictobert/all_mt_2.txt"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wvUCWs_dsZN4",
        "outputId": "68acd24a-d839-4705-dd74-949e6019bcd0"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "955489"
            ]
          },
          "execution_count": 3,
          "metadata": {
            "tags": []
          },
          "output_type": "execute_result"
        }
      ],
      "source": [
        "examples = open(\"./all_mt_2.txt\",'r').readlines()\n",
        "examples = [s.rstrip() for s in examples]\n",
        "len(examples)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WUKh4mNHsQTs"
      },
      "source": [
        "## Training Tokenizer\n",
        "\n",
        "To allow the usage of a different vocabulary on BERT, we have to train a new tokenizer. Before inputting data into a language model, it is necessary to tokenize it. Tokenization consists of splitting the words in a sentence according to some rules and then transform the split tokens into numbers. Those numbers are what the model will process. Initially, BERT uses a Word Piece tokenizer that split sentences into words or subwords (e.g., \\textquote{playing} into \\textquote{play##} and \\textquote{##ing}). To allow the use of word-senses, we trained a Word Level tokenizer, which split words in a sentence by whitespace. It enables the usage of sense keys.\n",
        "\n",
        "We use Hugging Face's tokenizers lib."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tRV6hzvDsMUn",
        "outputId": "29d8fe67-8cb0-479c-eec0-9642d2ea6862"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Collecting tokenizers\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/ae/04/5b870f26a858552025a62f1649c20d29d2672c02ff3c3fb4c688ca46467a/tokenizers-0.10.2-cp37-cp37m-manylinux2010_x86_64.whl (3.3MB)\n",
            "\u001b[K     |████████████████████████████████| 3.3MB 4.2MB/s \n",
            "\u001b[?25hInstalling collected packages: tokenizers\n",
            "Successfully installed tokenizers-0.10.2\n"
          ]
        }
      ],
      "source": [
        "!pip install tokenizers"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EUz8yH1_socD"
      },
      "source": [
        "### Create Tokenizer"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "b4wnm5PIsjTg"
      },
      "outputs": [],
      "source": [
        "from tokenizers import Tokenizer\n",
        "from tokenizers.models import WordLevel\n",
        "from tokenizers.pre_tokenizers import WhitespaceSplit\n",
        "from tokenizers.processors import BertProcessing\n",
        "\n",
        "sense_tokenizer = Tokenizer(WordLevel(unk_token=\"[UNK]\"\n",
        "  ))\n",
        "sense_tokenizer.add_special_tokens([\"[SEP]\", \"[CLS]\", \"[PAD]\", \"[MASK]\",\"[UNK]\"])\n",
        "sense_tokenizer.pre_tokenizer = WhitespaceSplit()\n",
        "\n",
        "sep_token = \"[SEP]\"\n",
        "cls_token = \"[CLS]\"\n",
        "pad_token = \"[PAD]\"\n",
        "unk_token = \"[UNK]\"\n",
        "sep_token_id = sense_tokenizer.token_to_id(str(sep_token))\n",
        "cls_token_id = sense_tokenizer.token_to_id(str(cls_token))\n",
        "pad_token_id = sense_tokenizer.token_to_id(str(pad_token))\n",
        "unk_token_id = sense_tokenizer.token_to_id(str(unk_token))\n",
        "\n",
        "\n",
        "sense_tokenizer.post_processor = BertProcessing(\n",
        "                (str(sep_token), sep_token_id), (str(cls_token), cls_token_id)\n",
        "            )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Frqymlejtnwb"
      },
      "source": [
        "### Train tokenizer"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DZwDolO9tyAx",
        "outputId": "0a38f79e-7183-49ec-f33d-3760bef97efa"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Vocab size:  13584\n"
          ]
        }
      ],
      "source": [
        "from tokenizers.trainers import WordLevelTrainer\n",
        "g = WordLevelTrainer(special_tokens=[\"[UNK]\"])\n",
        "sense_tokenizer.train_from_iterator(examples, trainer=g)\n",
        "print(\"Vocab size: \", sense_tokenizer.get_vocab_size())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OvGkqAa_tzV_"
      },
      "source": [
        "### Save tokenizer\n",
        "\n",
        "It is necessary to export the created tokenizer to enable its usage in the future. If you want to use a different tokenizer that we used for training PictoBERT, you have to download the JSON file and upload it in the next steps' notebooks (create model, train)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zRdq2h_Dt3in"
      },
      "outputs": [],
      "source": [
        "sense_tokenizer.save(\"./senses_tokenizer.json\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TfFF3NzwvIHB"
      },
      "source": [
        "## Dataset Preparation\n",
        "\n",
        "We load the trained tokenizer and the dataset and perform data encoding and spliting."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5ikkraCZyu-p"
      },
      "source": [
        "### Split Data\n",
        "\n",
        "We splited in 98/1/1 train, test and validation. To change this, alter TEST_SIZE below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lnmSVO4Qyua0"
      },
      "outputs": [],
      "source": [
        "TEST_SIZE = 0.02\n",
        "from sklearn.model_selection import train_test_split\n",
        "train_idx, val_idx = train_test_split(list(range(len(examples))), test_size=TEST_SIZE, random_state=32)\n",
        "test_idx, val_idx = train_test_split(val_idx, test_size=0.5, random_state=3)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Am-wd9aty-kw"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "train_examples = np.array(examples).take(train_idx)\n",
        "val_examples = np.array(examples).take(val_idx)\n",
        "test_examples = np.array(examples).take(test_idx)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gMLsAwyPvcMj"
      },
      "source": [
        "### Load tokenizer\n",
        "\n",
        "It is necessary to load the trained tokenizer using the `PreTrainedTokenizerFast` class from Hugging Face Transformers lib.\n",
        "\n",
        "To ensure the success of this demonstration, we download the [final tokenizer](https://drive.google.com/file/d/1-2g-GCxjBwESqDn3JByAJABU9Dkuqy0m/view?usp=sharing) used for training PictoBERT in the next cell.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Srj9W7R2wHca",
        "outputId": "3083a31f-8245-4d35-e868-1f909d6d374b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Downloading...\n",
            "From: https://drive.google.com/uc?id=1-2g-GCxjBwESqDn3JByAJABU9Dkuqy0m\n",
            "To: /content/childes_all_new.json\n",
            "\r  0% 0.00/332k [00:00<?, ?B/s]\r100% 332k/332k [00:00<00:00, 46.3MB/s]\n"
          ]
        }
      ],
      "source": [
        "!wget http://jayr.clubedosgeeks.com.br/pictobert/childes_all_new.json"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "b6xFgObGwe2O",
        "outputId": "53a9b8ec-a140-476c-fa29-f05769a41150"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Collecting transformers\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/d8/b2/57495b5309f09fa501866e225c84532d1fd89536ea62406b2181933fb418/transformers-4.5.1-py3-none-any.whl (2.1MB)\n",
            "\u001b[K     |████████████████████████████████| 2.1MB 5.6MB/s \n",
            "\u001b[?25hRequirement already satisfied: tokenizers<0.11,>=0.10.1 in /usr/local/lib/python3.7/dist-packages (from transformers) (0.10.2)\n",
            "Collecting sacremoses\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/75/ee/67241dc87f266093c533a2d4d3d69438e57d7a90abb216fa076e7d475d4a/sacremoses-0.0.45-py3-none-any.whl (895kB)\n",
            "\u001b[K     |████████████████████████████████| 901kB 26.3MB/s \n",
            "\u001b[?25hRequirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.7/dist-packages (from transformers) (4.41.1)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from transformers) (2.23.0)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from transformers) (3.0.12)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers) (2019.12.20)\n",
            "Requirement already satisfied: importlib-metadata; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from transformers) (3.10.1)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from transformers) (20.9)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.7/dist-packages (from transformers) (1.19.5)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers) (1.15.0)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers) (1.0.1)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers) (7.1.2)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (2.10)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (2020.12.5)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (1.24.3)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (3.0.4)\n",
            "Requirement already satisfied: typing-extensions>=3.6.4; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < \"3.8\"->transformers) (3.7.4.3)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < \"3.8\"->transformers) (3.4.1)\n",
            "Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging->transformers) (2.4.7)\n",
            "Installing collected packages: sacremoses, transformers\n",
            "Successfully installed sacremoses-0.0.45 transformers-4.5.1\n"
          ]
        }
      ],
      "source": [
        "!pip install transformers"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eydbCQ09v-T9"
      },
      "outputs": [],
      "source": [
        "TOKENIZER_PATH = \"./childes_all_new.json\" # you can change this path to use your custom tokenizer\n",
        "\n",
        "from transformers import PreTrainedTokenizerFast\n",
        "\n",
        "loaded_tokenizer = PreTrainedTokenizerFast(tokenizer_file=TOKENIZER_PATH)\n",
        "loaded_tokenizer.pad_token = \"[PAD]\"\n",
        "loaded_tokenizer.sep_token = \"[SEP]\"\n",
        "loaded_tokenizer.mask_token = \"[MASK]\"\n",
        "loaded_tokenizer.cls_token = \"[CLS]\"\n",
        "loaded_tokenizer.unk_token = \"[UNK]\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eCvK_29Fw0qw"
      },
      "source": [
        "### Tokenizer function\n",
        "\n",
        "This function encodes the examples using the tokenizer. Notice that we used a sequence length of 32, but you can change this value. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "q7SCW3n_w2-W"
      },
      "outputs": [],
      "source": [
        "max_len = 32\n",
        "\n",
        "def tokenize_function(tokenizer,examples):\n",
        "    # Remove empty lines\n",
        "    examples = [line for line in examples if len(line) > 0 and not line.isspace()]\n",
        "    bert = tokenizer(\n",
        "        examples,\n",
        "        padding=\"max_length\",\n",
        "        max_length=max_len,\n",
        "        return_special_tokens_mask=True,\n",
        "        truncation=True\n",
        "    )\n",
        "    ngram = tokenizer(examples,add_special_tokens=False).input_ids\n",
        "    return bert,ngram"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UBufKUexxIWu"
      },
      "outputs": [],
      "source": [
        "train_tokenized_examples, train_ngram = tokenize_function(loaded_tokenizer,train_examples)\n",
        "val_tokenized_examples, val_ngram = tokenize_function(loaded_tokenizer,val_examples)\n",
        "test_tokenized_examples, test_ngram = tokenize_function(loaded_tokenizer,test_examples)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RXd24eCVzWlo"
      },
      "source": [
        "### Save data\n",
        "\n",
        "We transform the data in dicts and save using pickle"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PwufgwOjzWBN"
      },
      "outputs": [],
      "source": [
        "from torch import tensor\n",
        "def make_dict(examples,ngrams):\n",
        "  return {\n",
        "      \"input_ids\": examples.input_ids,\n",
        "      \"attention_mask\":examples.attention_mask,\n",
        "      \"special_tokens_mask\":examples.special_tokens_mask,\n",
        "      \"ngrams\":ngrams\n",
        "  }"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gIqULblRzi1y"
      },
      "outputs": [],
      "source": [
        "import pickle\n",
        "\n",
        "TRAIN_DATA_PATH = \"./train_data.pt\"\n",
        "TEST_DATA_PATH = \"./test_data.pt\"\n",
        "VAL_DATA_PATH = \"./val_data.pt\"\n",
        "\n",
        "pickle.dump(make_dict(train_tokenized_examples, train_ngram),open(TRAIN_DATA_PATH,'wb'))\n",
        "pickle.dump(make_dict(val_tokenized_examples,val_ngram),open(TEST_DATA_PATH,'wb'))\n",
        "pickle.dump(make_dict(test_tokenized_examples, test_ngram),open(VAL_DATA_PATH ,'wb'))"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "Train Tokenizer and Prepare Dataset.ipynb",
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}