notebooks/openvino-tokenizers/openvino-tokenizers.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "fefeca4e-2808-4f33-a346-07f7e12bc59e",
   "metadata": {},
   "source": [
    "# OpenVINO Tokenizers: Incorporate Text Processing Into OpenVINO Pipelines\n",
    "\n",
    "<center><img src=\"https://github.com/openvinotoolkit/openvino_notebooks/assets/51917466/047f9167-a4ef-4d3d-a33b-d124541f9e2c\"></center>\n",
    "\n",
    "OpenVINO Tokenizers is an OpenVINO extension and a Python library designed to streamline tokenizer conversion for seamless integration into your projects. It supports Python and C++ environments and is compatible with all major platforms: Linux, Windows, and MacOS.\n",
    "\n",
    "\n",
    "#### Table of contents:\n",
    "- [Tokenization Basics](#Tokenization-Basics)\n",
    "- [Acquiring OpenVINO Tokenizers](#Acquiring-OpenVINO-Tokenizers)\n",
    "    - [Convert Tokenizer from HuggingFace Hub with CLI Tool](#Convert-Tokenizer-from_HuggingFace-Hub-with-CLI-Tool)\n",
    "    - [Convert Tokenizer from HuggingFace Hub with Python API](#Convert-Tokenizer-from-HuggingFace-Hub-with-Python-API)\n",
    "- [Text Generation Pipeline with OpenVINO Tokenizers](#Text-Generation-Pipeline-with-OpenVINO-Tokenizers)\n",
    "- [Merge Tokenizer into a Model](#Merge-Tokenizer-into-a-Model)\n",
    "- [Conclusion](#Conclusion)\n",
    "- [Links](#Links)\n",
    "\n",
    "\n",
    "## Tokenization Basics\n",
    "\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "One does not simply put text into a neural network, only numbers. The process of transforming text into a sequence of numbers is called **tokenization**. It usually contains several steps that transform the original string, splitting it into parts - tokens - with an associated number in a dictionary. You can check the [interactive GPT-4 tokenizer](https://platform.openai.com/tokenizer) to gain an intuitive understanding of the principles of tokenizer work.\n",
    "\n",
    "<center><img src=\"https://github.com/openvinotoolkit/openvino_notebooks/assets/51917466/55cc2420-da2a-4022-bae3-84d566a68e4c\"></center>\n",
    "\n",
    "There are two important points in the tokenizer-model relation:\n",
    "1. Every neural network with text input is paired with a tokenizer and _cannot be used without it_.\n",
    "2. To reproduce the model's accuracy on a specific task, it is essential to _utilize the same tokenizer employed during the model training_.\n",
    "\n",
    "That is why almost all model repositories on [HuggingFace Hub](https://HuggingFace.co/models) also contain tokenizer files (`tokenizer.json`, `vocab.txt`, `merges.txt`, etc.).\n",
    "\n",
    "The process of transforming a sequence of numbers into a string is called **detokenization**. Detokenizer can share the token dictionary with a tokenizer, like any LLM chat model, or operate with an entirely distinct dictionary. For instance, translation models dealing with different source and target languages often necessitate separate dictionaries.\n",
    "\n",
    "<center><img src=\"https://github.com/openvinotoolkit/openvino_notebooks/assets/51917466/942f1692-b7f2-4aca-93ef-32e9866c204f\"></center>\n",
    "\n",
    "Some tasks only need a tokenizer, like text classification, named entity recognition, question answering, and feature extraction. On the other hand, for tasks such as text generation, chat, translation, and abstractive summarization, both a tokenizer and a detokenizer are required.\n",
    "### Installation Instructions\n",
    "\n",
    "This is a self-contained example that relies solely on its own code.\n",
    "\n",
    "We recommend  running the notebook in a virtual environment. You only need a Jupyter server to start.\n",
    "For details, please refer to [Installation Guide](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/README.md#-installation-guide).\n",
    "\n",
    "<img referrerpolicy=\"no-referrer-when-downgrade\" src=\"https://static.scarf.sh/a.png?x-pxid=5b5a4db0-7875-4bfb-bdbd-01698b5b1a77&file=notebooks/openvino-tokenizers/openvino-tokenizers.ipynb\" />\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "8d502090-78eb-454c-a7a6-03b128f95ff6",
   "metadata": {},
   "source": [
    "## Acquiring OpenVINO Tokenizers\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "OpenVINO Tokenizers Python library allows you to convert HuggingFace tokenizers into OpenVINO models. To install all required dependencies use `pip install openvino-tokenizers[transformers]`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c698c94-b852-4b06-b699-7d417fb55e10",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -Uq pip\n",
    "%pip install --pre -Uq openvino-tokenizers[transformers] --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly\n",
    "%pip install \"numpy<2.0.0\" \"torch>=2.1\" --extra-index-url https://download.pytorch.org/whl/cpu"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "576b3d35-1282-4328-aa11-e871759a6dbb",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "\n",
    "tokenizer_dir = Path(\"tokenizer/\")\n",
    "model_id = \"TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "0da69135-265e-482f-a399-521e3980dec7",
   "metadata": {},
   "source": [
    "### Convert Tokenizer from HuggingFace Hub with CLI Tool\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "The first way is to use the CLI utility, bundled with OpenVINO Tokenizers. Use `--with-detokenizer` flag to add the detokenizer model to the output. By setting `--clean-up-tokenization-spaces=False` we ensure that the detokenizer correctly decodes a code-generation model output. `--trust-remote-code` flag works the same way as passing `trust_remote_code=True` to `AutoTokenizer.from_pretrained` constructor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "f8ff589d-2f44-4812-894a-ba25a826292e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading Huggingface Tokenizer...\n",
      "Converting Huggingface Tokenizer to OpenVINO...\n",
      "Saved OpenVINO Tokenizer: tokenizer/openvino_tokenizer.xml, tokenizer/openvino_tokenizer.bin\n",
      "Saved OpenVINO Detokenizer: tokenizer/openvino_detokenizer.xml, tokenizer/openvino_detokenizer.bin\n"
     ]
    }
   ],
   "source": [
    "!convert_tokenizer $model_id --with-detokenizer -o $tokenizer_dir"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e6cbb08b-f980-458e-8c5b-0d752d47aa26",
   "metadata": {},
   "source": [
    "> ⚠️ If you have any problems with the command above on MacOS, try to [install tbb](https://formulae.brew.sh/formula/tbb#default).\n",
    "\n",
    "The result is two OpenVINO models: `openvino_tokenizer` and `openvino_detokenizer`. Both can be interacted with using `read_model`, `compile_model` and `save_model`, similar to any other OpenVINO model.\n",
    "\n",
    "### Convert Tokenizer from HuggingFace Hub with Python API\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "The other method is to pass HuggingFace `hf_tokenizer` object to `convert_tokenizer` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a517c20c-b375-4ebc-b8a5-9eaae31bf62b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(<Model: 'tokenizer'\n",
       " inputs[\n",
       " <ConstOutput: names[string_input] shape[?] type: string>\n",
       " ]\n",
       " outputs[\n",
       " <ConstOutput: names[input_ids] shape[?,?] type: i64>,\n",
       " <ConstOutput: names[attention_mask] shape[?,?] type: i64>\n",
       " ]>,\n",
       " <Model: 'detokenizer'\n",
       " inputs[\n",
       " <ConstOutput: names[Parameter_21] shape[?,?] type: i64>\n",
       " ]\n",
       " outputs[\n",
       " <ConstOutput: names[string_output] shape[?] type: string>\n",
       " ]>)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from transformers import AutoTokenizer\n",
    "from openvino_tokenizers import convert_tokenizer\n",
    "\n",
    "\n",
    "hf_tokenizer = AutoTokenizer.from_pretrained(model_id)\n",
    "ov_tokenizer, ov_detokenizer = convert_tokenizer(hf_tokenizer, with_detokenizer=True)\n",
    "ov_tokenizer, ov_detokenizer"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "cd211b9f-37a9-4ae7-bc3e-4619643c08b8",
   "metadata": {},
   "source": [
    "That way you get OpenVINO model objects. Use `save_model` function from OpenVINO to reuse converted tokenizers later:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "eefda978-2668-4486-825e-30a1afbd240b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from openvino import save_model\n",
    "\n",
    "\n",
    "save_model(ov_tokenizer, tokenizer_dir / \"openvino_tokenizer.xml\")\n",
    "save_model(ov_detokenizer, tokenizer_dir / \"openvino_detokenizer.xml\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "0b16883a-2a69-4ae7-ab57-99d9b35784e1",
   "metadata": {},
   "source": [
    "To use the tokenizer, compile the converted model and input a list of strings.  It's essential to be aware that not all original tokenizers support multiple strings (also called batches) as input.  This limitation arises from the requirement for all resulting number sequences to maintain the same length. To address this, a padding token must be specified, which will be appended to shorter tokenized strings. In cases where no padding token is determined in the original tokenizer, OpenVINO Tokenizers defaults to using $0$ for padding.  Presently, _only right-side padding is supported_, typically used for classification tasks, but not suitable for text generation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "0fb96120-57f4-4f2d-b585-84ce0367cb51",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Token ids: [[   1 4321]\n",
      " [   1 6031]]\n",
      "Detokenized text: ['<s> Test' '<s> strings']\n"
     ]
    }
   ],
   "source": [
    "from openvino import compile_model\n",
    "\n",
    "\n",
    "tokenizer, detokenizer = compile_model(ov_tokenizer), compile_model(ov_detokenizer)\n",
    "test_strings = [\"Test\", \"strings\"]\n",
    "\n",
    "token_ids = tokenizer(test_strings)[\"input_ids\"]\n",
    "print(f\"Token ids: {token_ids}\")\n",
    "\n",
    "detokenized_text = detokenizer(token_ids)[\"string_output\"]\n",
    "print(f\"Detokenized text: {detokenized_text}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e602d7b3-b1cb-46eb-9738-4fbb41bb2b83",
   "metadata": {},
   "source": [
    "We can compare the result of converted (de)tokenizer with the original one:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "753727b4-0318-4806-90f3-a9c9c56a4297",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Token ids: [[1, 4321], [1, 6031]]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-04-02 18:45:50.238827: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.\n",
      "2024-04-02 18:45:50.275055: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n",
      "To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2024-04-02 18:45:50.909410: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Detokenized text: ['<s> Test', '<s> strings']\n"
     ]
    }
   ],
   "source": [
    "hf_token_ids = hf_tokenizer(test_strings).input_ids\n",
    "print(f\"Token ids: {hf_token_ids}\")\n",
    "\n",
    "hf_detokenized_text = hf_tokenizer.batch_decode(hf_token_ids)\n",
    "print(f\"Detokenized text: {hf_detokenized_text}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "9deb9c29-31e0-436c-9f52-4e396a37283f",
   "metadata": {},
   "source": [
    "## Text Generation Pipeline with OpenVINO Tokenizers\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Let's build a text generation pipeline with OpenVINO Tokenizers and minimal dependencies. To obtain an OpenVINO model we will use the Optimum library. The latest version allows you to get a so-called [stateful model](https://docs.openvino.ai/2024/openvino-workflow/running-inference/stateful-models.html).\n",
    "\n",
    "The original `TinyLlama-1.1B-intermediate-step-1431k-3T` model is 4.4Gb. To reduce network and disk usage we will load a converted model which has also been compressed to `int8`. The original conversion command is commented."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "a9870da3-477f-49c5-af80-ef081f92abeb",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_dir = Path(Path(model_id).name)\n",
    "\n",
    "if not model_dir.exists():\n",
    "    # converting the original model\n",
    "    # %pip install -U \"git+https://github.com/huggingface/optimum-intel.git\" \"nncf>=2.8.0\" onnx\n",
    "    # %optimum-cli export openvino -m $model_id --task text-generation-with-past $model_dir\n",
    "\n",
    "    # load already converted model\n",
    "    from huggingface_hub import hf_hub_download\n",
    "\n",
    "    hf_hub_download(\n",
    "        \"chgk13/TinyLlama-1.1B-intermediate-step-1431k-3T\",\n",
    "        filename=\"openvino_model.xml\",\n",
    "        local_dir=model_dir,\n",
    "    )\n",
    "    hf_hub_download(\n",
    "        \"chgk13/TinyLlama-1.1B-intermediate-step-1431k-3T\",\n",
    "        filename=\"openvino_model.bin\",\n",
    "        local_dir=model_dir,\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "fe7c0342-0e40-4f17-8340-a0ff0c99b6f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from tqdm.notebook import trange\n",
    "from pathlib import Path\n",
    "from openvino_tokenizers import add_greedy_decoding\n",
    "from openvino_tokenizers.constants import EOS_TOKEN_ID_NAME\n",
    "from openvino import Core\n",
    "\n",
    "\n",
    "core = Core()\n",
    "\n",
    "# add the greedy decoding subgraph on top of LLM to get the most probable token as an output\n",
    "ov_model = add_greedy_decoding(core.read_model(model_dir / \"openvino_model.xml\"))\n",
    "compiled_model = core.compile_model(ov_model)\n",
    "infer_request = compiled_model.create_infer_request()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "f6f53a35-446d-4a07-9e67-5794a53b12ba",
   "metadata": {},
   "source": [
    "The `infer_request` object provides control over the model's state - a  Key-Value cache that speeds up inference by reducing computations Multiple inference requests can be created, and each request maintains a distinct and separate state.."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "58a493cc-e29d-46b5-bb6c-d28e17651d85",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "af5dd23fe83c4fed8ce6b7b0a8ed41e9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/10 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prompt:\n",
      "Quick brown fox jumped\n",
      "Generated:\n",
      "over the fence.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "text_input = [\"Quick brown fox jumped\"]\n",
    "\n",
    "model_input = {name.any_name: output for name, output in tokenizer(text_input).items()}\n",
    "\n",
    "if \"position_ids\" in (input.any_name for input in infer_request.model_inputs):\n",
    "    model_input[\"position_ids\"] = np.arange(model_input[\"input_ids\"].shape[1], dtype=np.int64)[np.newaxis, :]\n",
    "\n",
    "# no beam search, set idx to 0\n",
    "model_input[\"beam_idx\"] = np.array([0], dtype=np.int32)\n",
    "# end of sentence token is that model signifies the end of text generation\n",
    "# read EOS token ID from rt_info of tokenizer/detokenizer ov.Model object\n",
    "eos_token = ov_tokenizer.get_rt_info(EOS_TOKEN_ID_NAME).value\n",
    "\n",
    "tokens_result = np.array([[]], dtype=np.int64)\n",
    "\n",
    "# reset KV cache inside the model before inference\n",
    "infer_request.reset_state()\n",
    "max_infer = 10\n",
    "\n",
    "for _ in trange(max_infer):\n",
    "    infer_request.start_async(model_input)\n",
    "    infer_request.wait()\n",
    "\n",
    "    # get a prediction for the last token on the first inference\n",
    "    output_token = infer_request.get_output_tensor().data[:, -1:]\n",
    "    tokens_result = np.hstack((tokens_result, output_token))\n",
    "    if output_token[0, 0] == eos_token:\n",
    "        break\n",
    "\n",
    "    # prepare input for new inference\n",
    "    model_input[\"input_ids\"] = output_token\n",
    "    model_input[\"attention_mask\"] = np.hstack((model_input[\"attention_mask\"].data, [[1]]))\n",
    "    model_input[\"position_ids\"] = np.hstack(\n",
    "        (\n",
    "            model_input[\"position_ids\"].data,\n",
    "            [[model_input[\"position_ids\"].data.shape[-1]]],\n",
    "        )\n",
    "    )\n",
    "\n",
    "text_result = detokenizer(tokens_result)[\"string_output\"]\n",
    "print(f\"Prompt:\\n{text_input[0]}\")\n",
    "print(f\"Generated:\\n{text_result[0]}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "7beb6ec6-2484-44ce-b61b-c7ae605dffee",
   "metadata": {},
   "source": [
    "## Merge Tokenizer into a Model\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Packages like `tensorflow-text` offer the convenience of integrating text processing directly into the model, streamlining both distribution and usage.  Similarly, with OpenVINO Tokenizers, you can create models that combine a converted tokenizer and a model. It's important to note that not all scenarios benefit from this merge. In cases where a tokenizer is used once and a model is inferred multiple times, as seen in the earlier text generation example, maintaining a separate (de)tokenizer and model is advisable to prevent unnecessary tokenization-detokenization cycles during inference. Conversely, if both a tokenizer and a model are used once in each pipeline inference, merging simplifies the workflow and aids in avoiding the creation of intermediate objects:\n",
    "\n",
    "<center><img src=\"https://github.com/openvinotoolkit/openvino_notebooks/assets/51917466/d4ece285-e445-4b76-a1ab-356427900860\"></center>\n",
    "\n",
    "The OpenVINO Python API allows you to avoid this by using the `share_inputs` option during inference, but it requires additional input from a developer every time the model is inferred. Combining the models and tokenizers simplifies memory management."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "c044b56b-dae0-4fdb-97df-2aa555285f35",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_id = \"mrm8488/bert-tiny-finetuned-sms-spam-detection\"\n",
    "model_dir = Path(Path(model_id).name)\n",
    "\n",
    "if not model_dir.exists():\n",
    "    %pip install -qU git+https://github.com/huggingface/optimum-intel.git onnx\n",
    "    !optimum-cli export openvino --model $model_id --task text-classification $model_dir\n",
    "    !convert_tokenizer $model_id -o $model_dir"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "c6a42d4c-1982-41b9-9612-aa19138518ac",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Logits: [[ 1.2007061 -1.4698029]]\n"
     ]
    }
   ],
   "source": [
    "from openvino import Core, save_model\n",
    "from openvino_tokenizers import connect_models\n",
    "\n",
    "\n",
    "core = Core()\n",
    "text_input = [\"Free money!!!\"]\n",
    "\n",
    "ov_tokenizer = core.read_model(model_dir / \"openvino_tokenizer.xml\")\n",
    "ov_model = core.read_model(model_dir / \"openvino_model.xml\")\n",
    "combined_model = connect_models(ov_tokenizer, ov_model)\n",
    "save_model(combined_model, model_dir / \"combined_openvino_model.xml\")\n",
    "\n",
    "compiled_combined_model = core.compile_model(combined_model)\n",
    "openvino_output = compiled_combined_model(text_input)\n",
    "\n",
    "print(f\"Logits: {openvino_output['logits']}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5e5e3084-e206-4b39-8ca0-2ee087af10ef",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "The OpenVINO Tokenizers integrate text processing operations into the OpenVINO ecosystem. Enabling the conversion of HuggingFace tokenizers into OpenVINO models, the library allows efficient deployment of deep learning pipelines across varied environments. The feature of combining tokenizers and models not only simplifies memory management but also helps to streamline model usage and deployment.\n",
    "\n",
    "\n",
    "## Links\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "- [Installation instructions for different environments](https://github.com/openvinotoolkit/openvino_tokenizers?tab=readme-ov-file#installation)\n",
    "- [Supported Tokenizer Types](https://github.com/openvinotoolkit/openvino_tokenizers?tab=readme-ov-file#supported-tokenizer-types)\n",
    "- [OpenVINO.GenAI repository with the C++ example of OpenVINO Tokenizers usage](https://github.com/openvinotoolkit/openvino.genai/tree/master/samples/cpp/greedy_causal_lm)\n",
    "- [HuggingFace Tokenizers Comparison Table](https://github.com/openvinotoolkit/openvino_tokenizers?tab=readme-ov-file#output-match-by-model)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "openvino_notebooks": {
   "imageUrl": "https://github.com/openvinotoolkit/openvino_notebooks/assets/51917466/047f9167-a4ef-4d3d-a33b-d124541f9e2c",
   "tags": {
    "categories": [
     "API Overview",
     "Convert",
     "Optimize"
    ],
    "libraries": [],
    "other": [],
    "tasks": [
     "Text Classification",
     "Text Generation"
    ]
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}