intro_2023_exercise_13_solution.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyOzjjuMFLbqlJprwAeVrcaN",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/TurkuNLP/intro-to-nlp/blob/master/intro_2023_exercise_13_solution.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Example solution to exercise task 13\n",
        "\n",
        "This notebook contains example solutions to exercise task 13, demonstrating zero-shot, one-shot, and two-shot prediction for a number of tasks."
      ],
      "metadata": {
        "id": "tIQ1s96UCcJW"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "---\n",
        "\n",
        "## Setup"
      ],
      "metadata": {
        "id": "FCZPjc-by7AJ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install --quiet transformers"
      ],
      "metadata": {
        "id": "4fUBJmXHCHw-"
      },
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Import the `AutoTokenizer`, `AutoModelForCausalLM`, and `pipeline` classes. The first two support loading tokenizers and generative models from the [Hugging Face repository](https://huggingface.co/models), and the last wraps a tokenizer and a model for convenience."
      ],
      "metadata": {
        "id": "5ZRNZgRJCt6Q"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "jwyK005xCFSF"
      },
      "outputs": [],
      "source": [
        "from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "---\n",
        "\n",
        "## English solution\n",
        "\n",
        "We'll provide an example solution for English. A Finnish solution would proceed similarly using a Finnish model, setting e.g.\n",
        "\n",
        "```MODEL_NAME = 'TurkuNLP/gpt3-finnish-large'```\n",
        "\n",
        "and using Finnish templates and examples."
      ],
      "metadata": {
        "id": "ANeibFsDzBKg"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Load a generative model and its tokenizer."
      ],
      "metadata": {
        "id": "6QJPDe3ZC_sL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "MODEL_NAME = 'gpt2-large'\n",
        "\n",
        "tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)\n",
        "model = AutoModelForCausalLM.from_pretrained(MODEL_NAME)"
      ],
      "metadata": {
        "id": "wqTxn_QaCNjZ"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Instantiate a text generation pipeline using the tokenizer and model."
      ],
      "metadata": {
        "id": "9ADWWb77e1sY"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pipe = pipeline(\n",
        "    'text-generation',\n",
        "    model=model,\n",
        "    tokenizer=tokenizer,\n",
        "    device=model.device\n",
        ")"
      ],
      "metadata": {
        "id": "0IIJzNrEe5qx"
      },
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Simple convenience function calling the pipeline with a prompt and returning one generated output:"
      ],
      "metadata": {
        "id": "eAohNr1ciwaU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def generate(prompt, max_new_tokens=5):\n",
        "    output = pipe(\n",
        "        prompt,\n",
        "        max_new_tokens=max_new_tokens,\n",
        "        do_sample=False,    # make repeatable\n",
        "        pad_token_id=tokenizer.eos_token_id    # suppresses an unnecessary warning\n",
        "    )\n",
        "    return output[0]['generated_text']"
      ],
      "metadata": {
        "id": "jWcOJkiKi5vr"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Test generation"
      ],
      "metadata": {
        "id": "SNRMsxXOjSo0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(generate(\"Hi, how's it going?\", max_new_tokens=10))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9Op7MJ6XjahG",
        "outputId": "5e1564f8-9b0c-47cc-9391-5777c7c1fc33"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py:1219: UserWarning: You have modified the pretrained model configuration to control generation. This is a deprecated strategy to control generation and will be removed soon, in a future version. Please use a generation configuration file (see https://huggingface.co/docs/transformers/main_classes/text_generation)\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Hi, how's it going?\n",
            "\n",
            "I'm doing great.\n",
            "\n",
            "I\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Binary sentiment classification\n",
        "\n",
        "We'll use the following test cases:"
      ],
      "metadata": {
        "id": "Pf2gUwb1zyRn"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Zero-shot**:\n",
        "\n",
        "We'll use the following template with a natural language instruction for our inputs."
      ],
      "metadata": {
        "id": "CzOBGbjB0I1I"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "sentences = [\n",
        "    \"This movie is absolutely wonderful, I love it!\",\n",
        "    \"That is certainly the worst laptop computer ever made.\",\n",
        "    \"I'm not sure what to think about it, but I guess on balance it's OK\",\n",
        "    \"This is certainly not particularly good.\",\n",
        "    \"I'm feeling quite upbeat today!\"\n",
        "]"
      ],
      "metadata": {
        "id": "QMqosUwD0MFe"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "template = \"\"\"Do the following texts express a positive or negative sentiment?\n",
        "\n",
        "Text: {}\n",
        "Answer:\"\"\""
      ],
      "metadata": {
        "id": "Pu8I95yaz8VR"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can fill the template like so:"
      ],
      "metadata": {
        "id": "yuU1TK4b0-p5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(template.format(sentences[0]))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yqgr1da7097I",
        "outputId": "dbb47791-41ee-4a89-8437-1dbe6df29e68"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Do the following texts express a positive or negative sentiment?\n",
            "\n",
            "Text: This movie is absolutely wonderful, I love it!\n",
            "Answer:\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can now generate for all of the sentences:"
      ],
      "metadata": {
        "id": "Grge0dSL2iXo"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "for s in sentences:\n",
        "    print(generate(template.format(s)))\n",
        "    print('---')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "h0CIx0U80Adb",
        "outputId": "a1f0df6c-83fe-4094-e6dd-e5b729243002"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Do the following texts express a positive or negative sentiment?\n",
            "\n",
            "Text: This movie is absolutely wonderful, I love it!\n",
            "Answer: I love it!\n",
            "\n",
            "---\n",
            "Do the following texts express a positive or negative sentiment?\n",
            "\n",
            "Text: That is certainly the worst laptop computer ever made.\n",
            "Answer: I am not sure.\n",
            "---\n",
            "Do the following texts express a positive or negative sentiment?\n",
            "\n",
            "Text: I'm not sure what to think about it, but I guess on balance it's OK\n",
            "Answer: I'm not sure what\n",
            "---\n",
            "Do the following texts express a positive or negative sentiment?\n",
            "\n",
            "Text: This is certainly not particularly good.\n",
            "Answer: This is not particularly good\n",
            "---\n",
            "Do the following texts express a positive or negative sentiment?\n",
            "\n",
            "Text: I'm feeling quite upbeat today!\n",
            "Answer: I'm feeling quite upbeat\n",
            "---\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "None of these show the desired output (\"positive\" or \"negative\"). Results like this are quite common for zero-shot settings with \"pure\" language models. While this could be improved on by working on the prompt, let's move on to few-shot settings."
      ],
      "metadata": {
        "id": "1L7yZ0Tq19aM"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**One-shot**\n",
        "\n",
        "Let's add one example into our template:"
      ],
      "metadata": {
        "id": "G9zJgwfo2ojJ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "template = \"\"\"Do the following texts express a positive or negative sentiment?\n",
        "\n",
        "Text: This is great!\n",
        "Answer: positive\n",
        "\n",
        "Text: {}\n",
        "Answer:\"\"\""
      ],
      "metadata": {
        "id": "IAW_ejwr1Gdk"
      },
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "And now just run the same generation:"
      ],
      "metadata": {
        "id": "nVAdjb8R20UO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "for s in sentences:\n",
        "    print(generate(template.format(s)))\n",
        "    print('---')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2uxEQSAH2xxj",
        "outputId": "35666933-3c21-48bb-d340-ea9d1fe6ed74"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Do the following texts express a positive or negative sentiment?\n",
            "\n",
            "Text: This is great!\n",
            "Answer: positive\n",
            "\n",
            "Text: This movie is absolutely wonderful, I love it!\n",
            "Answer: negative\n",
            "\n",
            "Text:\n",
            "---\n",
            "Do the following texts express a positive or negative sentiment?\n",
            "\n",
            "Text: This is great!\n",
            "Answer: positive\n",
            "\n",
            "Text: That is certainly the worst laptop computer ever made.\n",
            "Answer: negative\n",
            "\n",
            "Text:\n",
            "---\n",
            "Do the following texts express a positive or negative sentiment?\n",
            "\n",
            "Text: This is great!\n",
            "Answer: positive\n",
            "\n",
            "Text: I'm not sure what to think about it, but I guess on balance it's OK\n",
            "Answer: negative\n",
            "\n",
            "Text:\n",
            "---\n",
            "Do the following texts express a positive or negative sentiment?\n",
            "\n",
            "Text: This is great!\n",
            "Answer: positive\n",
            "\n",
            "Text: This is certainly not particularly good.\n",
            "Answer: negative\n",
            "\n",
            "Text:\n",
            "---\n",
            "Do the following texts express a positive or negative sentiment?\n",
            "\n",
            "Text: This is great!\n",
            "Answer: positive\n",
            "\n",
            "Text: I'm feeling quite upbeat today!\n",
            "Answer: positive\n",
            "\n",
            "Text:\n",
            "---\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Some of these are now correct. Note that the model generates extra text after the answer, predicting that the pattern of \"Text:\" and \"Answer:\" will continue: there is nothing telling the model it should stop. We could get rid of this by reducing the value of `max_new_tokens` when calling the pipeline. "
      ],
      "metadata": {
        "id": "XSs2yKo53iY3"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Two-shot**"
      ],
      "metadata": {
        "id": "mHhKaSpe4DxH"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Again the only thing we change is add another example to our template:"
      ],
      "metadata": {
        "id": "NMjHf6qA4JSY"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "template = \"\"\"Do the following texts express a positive or negative sentiment?\n",
        "\n",
        "Text: This is great!\n",
        "Answer: positive\n",
        "\n",
        "Text: This is terrible!\n",
        "Answer: negative\n",
        "\n",
        "Text: {}\n",
        "Answer:\"\"\""
      ],
      "metadata": {
        "id": "0-jeot0j22aB"
      },
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Rerun:"
      ],
      "metadata": {
        "id": "CXmki9Ub4VaM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "for s in sentences:\n",
        "    print(generate(template.format(s)))\n",
        "    print('---')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EaGdFvA84TYh",
        "outputId": "06e8c0f9-f3ff-42f0-9444-7dd83e28662c"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Do the following texts express a positive or negative sentiment?\n",
            "\n",
            "Text: This is great!\n",
            "Answer: positive\n",
            "\n",
            "Text: This is terrible!\n",
            "Answer: negative\n",
            "\n",
            "Text: This movie is absolutely wonderful, I love it!\n",
            "Answer: negative\n",
            "\n",
            "Text:\n",
            "---\n",
            "Do the following texts express a positive or negative sentiment?\n",
            "\n",
            "Text: This is great!\n",
            "Answer: positive\n",
            "\n",
            "Text: This is terrible!\n",
            "Answer: negative\n",
            "\n",
            "Text: That is certainly the worst laptop computer ever made.\n",
            "Answer: positive\n",
            "\n",
            "Text:\n",
            "---\n",
            "Do the following texts express a positive or negative sentiment?\n",
            "\n",
            "Text: This is great!\n",
            "Answer: positive\n",
            "\n",
            "Text: This is terrible!\n",
            "Answer: negative\n",
            "\n",
            "Text: I'm not sure what to think about it, but I guess on balance it's OK\n",
            "Answer: negative\n",
            "\n",
            "Text:\n",
            "---\n",
            "Do the following texts express a positive or negative sentiment?\n",
            "\n",
            "Text: This is great!\n",
            "Answer: positive\n",
            "\n",
            "Text: This is terrible!\n",
            "Answer: negative\n",
            "\n",
            "Text: This is certainly not particularly good.\n",
            "Answer: negative\n",
            "\n",
            "Text:\n",
            "---\n",
            "Do the following texts express a positive or negative sentiment?\n",
            "\n",
            "Text: This is great!\n",
            "Answer: positive\n",
            "\n",
            "Text: This is terrible!\n",
            "Answer: negative\n",
            "\n",
            "Text: I'm feeling quite upbeat today!\n",
            "Answer: positive\n",
            "\n",
            "Text:\n",
            "---\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Curiously, that appears to have gotten worse, although the model does maintain the correct pattern of responses. We could continue going with 3-shot, 4-shot etc. and would expect some improvement, but let's move on to the next task."
      ],
      "metadata": {
        "id": "0Us5Hjbg48X2"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "---\n",
        "\n",
        "## Person name recognition\n",
        "\n",
        "We'll follow the same pattern as above, providing a number of example sentences and separate templates for zero-, one-, and two-shot prediction."
      ],
      "metadata": {
        "id": "6QUiUvyk6rkM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "sentences = [\n",
        "    \"John went to the store.\",\n",
        "    \"Her name is Jane.\",\n",
        "    \"The president is Joe Biden.\",\n",
        "    \"Is she called Mary?\",\n",
        "    \"John and Jane went home.\",\n",
        "]"
      ],
      "metadata": {
        "id": "RkdcUG6u4WXy"
      },
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Zero-shot**"
      ],
      "metadata": {
        "id": "fAF7fa2B76Ii"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "template = \"\"\"List the person names occurring in the following texts.\n",
        "\n",
        "Text: {}\n",
        "Names:\"\"\"\n",
        "\n",
        "for s in sentences:\n",
        "    print(generate(template.format(s)))\n",
        "    print('---')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "70HbhOUb7e44",
        "outputId": "d7c2eee5-229a-4e36-f68d-2e07ceb14272"
      },
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "List the person names occurring in the following texts.\n",
            "\n",
            "Text: John went to the store.\n",
            "Names: John, John, John\n",
            "---\n",
            "List the person names occurring in the following texts.\n",
            "\n",
            "Text: Her name is Jane.\n",
            "Names: Jane, Jane, Jane\n",
            "---\n",
            "List the person names occurring in the following texts.\n",
            "\n",
            "Text: The president is Joe Biden.\n",
            "Names: Joe Biden, Joe Biden\n",
            "---\n",
            "List the person names occurring in the following texts.\n",
            "\n",
            "Text: Is she called Mary?\n",
            "Names: Mary, Mary, Mary\n",
            "---\n",
            "List the person names occurring in the following texts.\n",
            "\n",
            "Text: John and Jane went home.\n",
            "Names: John and Jane went home\n",
            "---\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "This has some indications of the right answers, but none are fully correct."
      ],
      "metadata": {
        "id": "IF3E7ydc8BMc"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**One-shot**"
      ],
      "metadata": {
        "id": "ZaUeY1Mq8PFx"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "template = \"\"\"List the person names occurring in the following texts.\n",
        "\n",
        "Text: Bob is buying groceries.\n",
        "Names: Bob\n",
        "\n",
        "Text: {}\n",
        "Names:\"\"\"\n",
        "\n",
        "for s in sentences:\n",
        "    print(generate(template.format(s)))\n",
        "    print('---')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "a6PMY8q-73gQ",
        "outputId": "663886dd-0427-48ff-e01e-5c2ffbc3bfae"
      },
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "List the person names occurring in the following texts.\n",
            "\n",
            "Text: Bob is buying groceries.\n",
            "Names: Bob\n",
            "\n",
            "Text: John went to the store.\n",
            "Names: John\n",
            "\n",
            "Text:\n",
            "---\n",
            "List the person names occurring in the following texts.\n",
            "\n",
            "Text: Bob is buying groceries.\n",
            "Names: Bob\n",
            "\n",
            "Text: Her name is Jane.\n",
            "Names: Jane\n",
            "\n",
            "Text:\n",
            "---\n",
            "List the person names occurring in the following texts.\n",
            "\n",
            "Text: Bob is buying groceries.\n",
            "Names: Bob\n",
            "\n",
            "Text: The president is Joe Biden.\n",
            "Names: Joe\n",
            "\n",
            "Text:\n",
            "---\n",
            "List the person names occurring in the following texts.\n",
            "\n",
            "Text: Bob is buying groceries.\n",
            "Names: Bob\n",
            "\n",
            "Text: Is she called Mary?\n",
            "Names: Mary\n",
            "\n",
            "Text:\n",
            "---\n",
            "List the person names occurring in the following texts.\n",
            "\n",
            "Text: Bob is buying groceries.\n",
            "Names: Bob\n",
            "\n",
            "Text: John and Jane went home.\n",
            "Names: Jane and John\n",
            "\n",
            "\n",
            "---\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "This has some correct answers, but it fails when there is more than one proper noun. Let's give an example of that."
      ],
      "metadata": {
        "id": "UqH361GV8hB6"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Two-shot**"
      ],
      "metadata": {
        "id": "zgbqEsvB8rgy"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "template = \"\"\"List the person names occurring in the following texts.\n",
        "\n",
        "Text: Bob is buying groceries.\n",
        "Names: Bob\n",
        "\n",
        "Text: Alice is a fan of the vice president, Kamala Harris.\n",
        "Names: Alice, Kamala Harris\n",
        "\n",
        "Text: {}\n",
        "Names:\"\"\"\n",
        "\n",
        "for s in sentences:\n",
        "    print(generate(template.format(s)))\n",
        "    print('---')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JOKo6F2e8pSB",
        "outputId": "a9ebc180-bac1-4882-d491-b259b2d4e968"
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "List the person names occurring in the following texts.\n",
            "\n",
            "Text: Bob is buying groceries.\n",
            "Names: Bob\n",
            "\n",
            "Text: Alice is a fan of the vice president, Kamala Harris.\n",
            "Names: Alice, Kamala Harris\n",
            "\n",
            "Text: John went to the store.\n",
            "Names: John\n",
            "\n",
            "Text:\n",
            "---\n",
            "List the person names occurring in the following texts.\n",
            "\n",
            "Text: Bob is buying groceries.\n",
            "Names: Bob\n",
            "\n",
            "Text: Alice is a fan of the vice president, Kamala Harris.\n",
            "Names: Alice, Kamala Harris\n",
            "\n",
            "Text: Her name is Jane.\n",
            "Names: Jane\n",
            "\n",
            "Text:\n",
            "---\n",
            "List the person names occurring in the following texts.\n",
            "\n",
            "Text: Bob is buying groceries.\n",
            "Names: Bob\n",
            "\n",
            "Text: Alice is a fan of the vice president, Kamala Harris.\n",
            "Names: Alice, Kamala Harris\n",
            "\n",
            "Text: The president is Joe Biden.\n",
            "Names: Joe Biden\n",
            "\n",
            "Text\n",
            "---\n",
            "List the person names occurring in the following texts.\n",
            "\n",
            "Text: Bob is buying groceries.\n",
            "Names: Bob\n",
            "\n",
            "Text: Alice is a fan of the vice president, Kamala Harris.\n",
            "Names: Alice, Kamala Harris\n",
            "\n",
            "Text: Is she called Mary?\n",
            "Names: Mary\n",
            "\n",
            "Text:\n",
            "---\n",
            "List the person names occurring in the following texts.\n",
            "\n",
            "Text: Bob is buying groceries.\n",
            "Names: Bob\n",
            "\n",
            "Text: Alice is a fan of the vice president, Kamala Harris.\n",
            "Names: Alice, Kamala Harris\n",
            "\n",
            "Text: John and Jane went home.\n",
            "Names: John and Jane\n",
            "\n",
            "\n",
            "---\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "All but one correct. Again we could continue providing more examples or refine the prompt to try to improve on this."
      ],
      "metadata": {
        "id": "wHRughgm9IZJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Two-digit addition\n",
        "\n",
        "We'll follow the same pattern as above with a simple patter `X + Y = ?`"
      ],
      "metadata": {
        "id": "wyUEWC6V9X6J"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "examples = [\n",
        "    \"10 + 10\",\n",
        "    \"15 + 25\",\n",
        "    \"31 + 42\",\n",
        "    \"78 + 47\",\n",
        "    \"99 + 98\",\n",
        "]"
      ],
      "metadata": {
        "id": "01BT6x5Y9ktL"
      },
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**One-shot**"
      ],
      "metadata": {
        "id": "jV6abxoQ9_Dt"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "template = \"\"\"This is a first grade math exam.\n",
        "\n",
        "{} =\"\"\"\n",
        "\n",
        "for e in examples:\n",
        "    print(generate(template.format(e)))\n",
        "    print('---')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "sSyUtu4984i-",
        "outputId": "0a616891-a3c6-4737-f450-e9f25c8a1e82"
      },
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "This is a first grade math exam.\n",
            "\n",
            "10 + 10 = 20\n",
            "\n",
            "20 +\n",
            "---\n",
            "This is a first grade math exam.\n",
            "\n",
            "15 + 25 = 50\n",
            "\n",
            "15 +\n",
            "---\n",
            "This is a first grade math exam.\n",
            "\n",
            "31 + 42 = 63\n",
            "\n",
            "The answer\n",
            "---\n",
            "This is a first grade math exam.\n",
            "\n",
            "78 + 47 = 100\n",
            "\n",
            "The answer\n",
            "---\n",
            "This is a first grade math exam.\n",
            "\n",
            "99 + 98 = 100\n",
            "\n",
            "The answer\n",
            "---\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "It manages 10 + 10, but not the others. Let's see if an example helps."
      ],
      "metadata": {
        "id": "9ICB7yHv-JPT"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**One-shot**"
      ],
      "metadata": {
        "id": "IobLlXPW-M-l"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "template = \"\"\"This is a first grade math exam.\n",
        "\n",
        "11 + 12 = 23\n",
        "\n",
        "{} =\"\"\"\n",
        "\n",
        "for e in examples:\n",
        "    print(generate(template.format(e)))\n",
        "    print('---')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-TOFC6Dp9-Gg",
        "outputId": "24202e11-461c-4c14-f0b1-55bb3c7b1698"
      },
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "This is a first grade math exam.\n",
            "\n",
            "11 + 12 = 23\n",
            "\n",
            "10 + 10 = 23\n",
            "\n",
            "9 +\n",
            "---\n",
            "This is a first grade math exam.\n",
            "\n",
            "11 + 12 = 23\n",
            "\n",
            "15 + 25 = 33\n",
            "\n",
            "15 +\n",
            "---\n",
            "This is a first grade math exam.\n",
            "\n",
            "11 + 12 = 23\n",
            "\n",
            "31 + 42 = 63\n",
            "\n",
            "31 +\n",
            "---\n",
            "This is a first grade math exam.\n",
            "\n",
            "11 + 12 = 23\n",
            "\n",
            "78 + 47 = 97\n",
            "\n",
            "The answer\n",
            "---\n",
            "This is a first grade math exam.\n",
            "\n",
            "11 + 12 = 23\n",
            "\n",
            "99 + 98 = 99\n",
            "\n",
            "99 +\n",
            "---\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Nope, definitely not; now it even got 10 + 10 wrong, copying the first answer instead. How about two shots?"
      ],
      "metadata": {
        "id": "mXh_IMiH-mwO"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Two-shot**"
      ],
      "metadata": {
        "id": "goGuiD55-1mk"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "template = \"\"\"This is a first grade math exam.\n",
        "\n",
        "11 + 12 = 23\n",
        "\n",
        "42 + 42 = 84\n",
        "\n",
        "{} =\"\"\"\n",
        "\n",
        "for e in examples:\n",
        "    print(generate(template.format(e)))\n",
        "    print('---')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xkMC7Bzk-RyP",
        "outputId": "c00def80-ba1a-4179-df53-0be73b936667"
      },
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "This is a first grade math exam.\n",
            "\n",
            "11 + 12 = 23\n",
            "\n",
            "42 + 42 = 84\n",
            "\n",
            "10 + 10 = 15\n",
            "\n",
            "10 +\n",
            "---\n",
            "This is a first grade math exam.\n",
            "\n",
            "11 + 12 = 23\n",
            "\n",
            "42 + 42 = 84\n",
            "\n",
            "15 + 25 = 33\n",
            "\n",
            "15 +\n",
            "---\n",
            "This is a first grade math exam.\n",
            "\n",
            "11 + 12 = 23\n",
            "\n",
            "42 + 42 = 84\n",
            "\n",
            "31 + 42 = 63\n",
            "\n",
            "31 +\n",
            "---\n",
            "This is a first grade math exam.\n",
            "\n",
            "11 + 12 = 23\n",
            "\n",
            "42 + 42 = 84\n",
            "\n",
            "78 + 47 = 100\n",
            "\n",
            "The answer\n",
            "---\n",
            "This is a first grade math exam.\n",
            "\n",
            "11 + 12 = 23\n",
            "\n",
            "42 + 42 = 84\n",
            "\n",
            "99 + 98 = 144\n",
            "\n",
            "So,\n",
            "---\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "This one seems quite hopeless. As surprising as it is that a model that shows some emergent ability to do natural language tasks such as sentiment classification and name recognition can't do a basic computer task like two-digit addition, this is actually a common result: even very large language models make arithmetic mistakes, although not in cases this simple. "
      ],
      "metadata": {
        "id": "lEddR9to-85g"
      }
    }
  ]
}