Fine_tuning_GPT_2_1_5B_using_TPUs.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Fine-tuning GPT-2 1.5B using TPUs.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "TPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/jonbaer/googlecolab/blob/master/Fine_tuning_GPT_2_1_5B_using_TPUs.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OjkEFt765Qio",
        "colab_type": "code",
        "outputId": "4cea1783-0966-4518-b9d7-bf6f2eadc1bb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 147
        }
      },
      "source": [
        "!git clone https://github.com/shawwn/gpt-2 /content/gpt-2\n",
        "%cd /content/gpt-2"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Cloning into '/content/gpt-2'...\n",
            "remote: Enumerating objects: 13, done.\u001b[K\n",
            "remote: Counting objects: 100% (13/13), done.\u001b[K\n",
            "remote: Compressing objects: 100% (9/9), done.\u001b[K\n",
            "remote: Total 723 (delta 7), reused 10 (delta 4), pack-reused 710\u001b[K\n",
            "Receiving objects: 100% (723/723), 4.45 MiB | 16.65 MiB/s, done.\n",
            "Resolving deltas: 100% (470/470), done.\n",
            "/content/gpt-2\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1CSTC1Jo5KI0",
        "colab_type": "code",
        "outputId": "4b1f4514-a1d8-4ad6-c6f6-8a6a3734696c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 131
        }
      },
      "source": [
        "!python3 download_model.py 1558M"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\rFetching checkpoint:   0%|                                              | 0.00/77.0 [00:00<?, ?it/s]\rFetching checkpoint: 1.00kit [00:00, 791kit/s]                                                      \n",
            "\rFetching encoder.json:   0%|                                           | 0.00/1.04M [00:00<?, ?it/s]\rFetching encoder.json: 1.04Mit [00:00, 55.3Mit/s]                                                   \n",
            "Fetching hparams.json: 1.00kit [00:00, 772kit/s]                                                    \n",
            "Fetching model.ckpt.data-00000-of-00001: 6.23Git [03:54, 26.6Mit/s]                                 \n",
            "Fetching model.ckpt.index: 21.0kit [00:00, 9.61Mit/s]                                               \n",
            "Fetching model.ckpt.meta: 1.84Mit [00:00, 56.5Mit/s]                                                \n",
            "Fetching vocab.bpe: 457kit [00:00, 50.5Mit/s]                                                       \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fKHtO6y65Smy",
        "colab_type": "code",
        "outputId": "d02a8294-6860-4381-dffd-b4a3b5277fb7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "!pip install regex toposort magic-wormhole"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting regex\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/e3/8e/cbf2295643d7265e7883326fb4654e643bfc93b3a8a8274d8010a39d8804/regex-2019.11.1-cp36-cp36m-manylinux1_x86_64.whl (643kB)\n",
            "\r\u001b[K     |▌                               | 10kB 18.4MB/s eta 0:00:01\r\u001b[K     |█                               | 20kB 2.2MB/s eta 0:00:01\r\u001b[K     |█▌                              | 30kB 3.1MB/s eta 0:00:01\r\u001b[K     |██                              | 40kB 2.1MB/s eta 0:00:01\r\u001b[K     |██▌                             | 51kB 2.6MB/s eta 0:00:01\r\u001b[K     |███                             | 61kB 3.1MB/s eta 0:00:01\r\u001b[K     |███▋                            | 71kB 3.5MB/s eta 0:00:01\r\u001b[K     |████                            | 81kB 4.0MB/s eta 0:00:01\r\u001b[K     |████▋                           | 92kB 4.5MB/s eta 0:00:01\r\u001b[K     |█████                           | 102kB 3.4MB/s eta 0:00:01\r\u001b[K     |█████▋                          | 112kB 3.4MB/s eta 0:00:01\r\u001b[K     |██████                          | 122kB 3.4MB/s eta 0:00:01\r\u001b[K     |██████▋                         | 133kB 3.4MB/s eta 0:00:01\r\u001b[K     |███████▏                        | 143kB 3.4MB/s eta 0:00:01\r\u001b[K     |███████▋                        | 153kB 3.4MB/s eta 0:00:01\r\u001b[K     |████████▏                       | 163kB 3.4MB/s eta 0:00:01\r\u001b[K     |████████▋                       | 174kB 3.4MB/s eta 0:00:01\r\u001b[K     |█████████▏                      | 184kB 3.4MB/s eta 0:00:01\r\u001b[K     |█████████▊                      | 194kB 3.4MB/s eta 0:00:01\r\u001b[K     |██████████▏                     | 204kB 3.4MB/s eta 0:00:01\r\u001b[K     |██████████▊                     | 215kB 3.4MB/s eta 0:00:01\r\u001b[K     |███████████▏                    | 225kB 3.4MB/s eta 0:00:01\r\u001b[K     |███████████▊                    | 235kB 3.4MB/s eta 0:00:01\r\u001b[K     |████████████▏                   | 245kB 3.4MB/s eta 0:00:01\r\u001b[K     |████████████▊                   | 256kB 3.4MB/s eta 0:00:01\r\u001b[K     |█████████████▎                  | 266kB 3.4MB/s eta 0:00:01\r\u001b[K     |█████████████▊                  | 276kB 3.4MB/s eta 0:00:01\r\u001b[K     |██████████████▎                 | 286kB 3.4MB/s eta 0:00:01\r\u001b[K     |██████████████▊                 | 296kB 3.4MB/s eta 0:00:01\r\u001b[K     |███████████████▎                | 307kB 3.4MB/s eta 0:00:01\r\u001b[K     |███████████████▉                | 317kB 3.4MB/s eta 0:00:01\r\u001b[K     |████████████████▎               | 327kB 3.4MB/s eta 0:00:01\r\u001b[K     |████████████████▉               | 337kB 3.4MB/s eta 0:00:01\r\u001b[K     |█████████████████▎              | 348kB 3.4MB/s eta 0:00:01\r\u001b[K     |█████████████████▉              | 358kB 3.4MB/s eta 0:00:01\r\u001b[K     |██████████████████▎             | 368kB 3.4MB/s eta 0:00:01\r\u001b[K     |██████████████████▉             | 378kB 3.4MB/s eta 0:00:01\r\u001b[K     |███████████████████▍            | 389kB 3.4MB/s eta 0:00:01\r\u001b[K     |███████████████████▉            | 399kB 3.4MB/s eta 0:00:01\r\u001b[K     |████████████████████▍           | 409kB 3.4MB/s eta 0:00:01\r\u001b[K     |████████████████████▉           | 419kB 3.4MB/s eta 0:00:01\r\u001b[K     |█████████████████████▍          | 430kB 3.4MB/s eta 0:00:01\r\u001b[K     |██████████████████████          | 440kB 3.4MB/s eta 0:00:01\r\u001b[K     |██████████████████████▍         | 450kB 3.4MB/s eta 0:00:01\r\u001b[K     |███████████████████████         | 460kB 3.4MB/s eta 0:00:01\r\u001b[K     |███████████████████████▍        | 471kB 3.4MB/s eta 0:00:01\r\u001b[K     |████████████████████████        | 481kB 3.4MB/s eta 0:00:01\r\u001b[K     |████████████████████████▍       | 491kB 3.4MB/s eta 0:00:01\r\u001b[K     |█████████████████████████       | 501kB 3.4MB/s eta 0:00:01\r\u001b[K     |█████████████████████████▌      | 512kB 3.4MB/s eta 0:00:01\r\u001b[K     |██████████████████████████      | 522kB 3.4MB/s eta 0:00:01\r\u001b[K     |██████████████████████████▌     | 532kB 3.4MB/s eta 0:00:01\r\u001b[K     |███████████████████████████     | 542kB 3.4MB/s eta 0:00:01\r\u001b[K     |███████████████████████████▌    | 552kB 3.4MB/s eta 0:00:01\r\u001b[K     |████████████████████████████    | 563kB 3.4MB/s eta 0:00:01\r\u001b[K     |████████████████████████████▌   | 573kB 3.4MB/s eta 0:00:01\r\u001b[K     |█████████████████████████████   | 583kB 3.4MB/s eta 0:00:01\r\u001b[K     |█████████████████████████████▌  | 593kB 3.4MB/s eta 0:00:01\r\u001b[K     |██████████████████████████████  | 604kB 3.4MB/s eta 0:00:01\r\u001b[K     |██████████████████████████████▌ | 614kB 3.4MB/s eta 0:00:01\r\u001b[K     |███████████████████████████████ | 624kB 3.4MB/s eta 0:00:01\r\u001b[K     |███████████████████████████████▋| 634kB 3.4MB/s eta 0:00:01\r\u001b[K     |████████████████████████████████| 645kB 3.4MB/s \n",
            "\u001b[?25hCollecting toposort\n",
            "  Downloading https://files.pythonhosted.org/packages/e9/8a/321cd8ea5f4a22a06e3ba30ef31ec33bea11a3443eeb1d89807640ee6ed4/toposort-1.5-py2.py3-none-any.whl\n",
            "Collecting magic-wormhole\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/82/98/3e8d12fdb90457e8f3e1f5b877ee27f5db58dbaf4a4fbe95f7287a568401/magic_wormhole-0.11.2-py2.py3-none-any.whl (130kB)\n",
            "\u001b[K     |████████████████████████████████| 133kB 50.4MB/s \n",
            "\u001b[?25hCollecting twisted[tls]>=17.5.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/14/49/eb654da38b15285d1f594933eefff36ce03106356197dba28ee8f5721a79/Twisted-19.7.0-cp36-cp36m-manylinux1_x86_64.whl (3.1MB)\n",
            "\u001b[K     |████████████████████████████████| 3.1MB 53.2MB/s \n",
            "\u001b[?25hCollecting automat\n",
            "  Downloading https://files.pythonhosted.org/packages/e5/11/756922e977bb296a79ccf38e8d45cafee446733157d59bcd751d3aee57f5/Automat-0.8.0-py2.py3-none-any.whl\n",
            "Requirement already satisfied: humanize in /usr/local/lib/python3.6/dist-packages (from magic-wormhole) (0.5.1)\n",
            "Collecting txtorcon>=18.0.2\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/4e/36/910a6991534847eda30e99a7c363cd2bbe6710fc05a2d54c9abe22507ffe/txtorcon-19.1.0-py2.py3-none-any.whl (341kB)\n",
            "\u001b[K     |████████████████████████████████| 348kB 51.1MB/s \n",
            "\u001b[?25hRequirement already satisfied: tqdm>=4.13.0 in /usr/local/lib/python3.6/dist-packages (from magic-wormhole) (4.28.1)\n",
            "Collecting pynacl\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/27/15/2cd0a203f318c2240b42cd9dd13c931ddd61067809fee3479f44f086103e/PyNaCl-1.3.0-cp34-abi3-manylinux1_x86_64.whl (759kB)\n",
            "\u001b[K     |████████████████████████████████| 768kB 53.3MB/s \n",
            "\u001b[?25hCollecting spake2==0.8\n",
            "  Downloading https://files.pythonhosted.org/packages/92/b8/e1bdd53d1a865f4769769fa2473d8f410622ba21b45d062acaa0cecaa937/spake2-0.8-py2.py3-none-any.whl\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.6/dist-packages (from magic-wormhole) (7.0)\n",
            "Requirement already satisfied: attrs>=16.3.0 in /usr/local/lib/python3.6/dist-packages (from magic-wormhole) (19.3.0)\n",
            "Collecting autobahn[twisted]>=0.14.1\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/4c/1e/619ab12c3cf4dfd251dcb6cbca9581fbb2e648bf80f68d7be4b4b6b99337/autobahn-19.10.1-py2.py3-none-any.whl (771kB)\n",
            "\u001b[K     |████████████████████████████████| 778kB 26.7MB/s \n",
            "\u001b[?25hCollecting hkdf\n",
            "  Downloading https://files.pythonhosted.org/packages/c3/be/327e072850db181ce56afd51e26ec7aa5659b18466c709fa5ea2548c935f/hkdf-0.0.3.tar.gz\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from magic-wormhole) (1.12.0)\n",
            "Collecting incremental>=16.10.1\n",
            "  Downloading https://files.pythonhosted.org/packages/f5/1d/c98a587dc06e107115cf4a58b49de20b19222c83d75335a192052af4c4b7/incremental-17.5.0-py2.py3-none-any.whl\n",
            "Collecting hyperlink>=17.1.1\n",
            "  Downloading https://files.pythonhosted.org/packages/7f/91/e916ca10a2de1cb7101a9b24da546fb90ee14629e23160086cf3361c4fb8/hyperlink-19.0.0-py2.py3-none-any.whl\n",
            "Collecting constantly>=15.1\n",
            "  Downloading https://files.pythonhosted.org/packages/b9/65/48c1909d0c0aeae6c10213340ce682db01b48ea900a7d9fce7a7910ff318/constantly-15.1.0-py2.py3-none-any.whl\n",
            "Collecting zope.interface>=4.4.2\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/19/17/1d198a6aaa9aa4590862fe3d3a2ed7dd808050cab4eebe8a2f2f813c1376/zope.interface-4.6.0-cp36-cp36m-manylinux1_x86_64.whl (167kB)\n",
            "\u001b[K     |████████████████████████████████| 174kB 51.2MB/s \n",
            "\u001b[?25hCollecting PyHamcrest>=1.9.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/9a/d5/d37fd731b7d0e91afcc84577edeccf4638b4f9b82f5ffe2f8b62e2ddc609/PyHamcrest-1.9.0-py2.py3-none-any.whl (52kB)\n",
            "\u001b[K     |████████████████████████████████| 61kB 8.2MB/s \n",
            "\u001b[?25hCollecting service-identity>=18.1.0; extra == \"tls\"\n",
            "  Downloading https://files.pythonhosted.org/packages/e9/7c/2195b890023e098f9618d43ebc337d83c8b38d414326685339eb024db2f6/service_identity-18.1.0-py2.py3-none-any.whl\n",
            "Collecting pyopenssl>=16.0.0; extra == \"tls\"\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/01/c8/ceb170d81bd3941cbeb9940fc6cc2ef2ca4288d0ca8929ea4db5905d904d/pyOpenSSL-19.0.0-py2.py3-none-any.whl (53kB)\n",
            "\u001b[K     |████████████████████████████████| 61kB 7.7MB/s \n",
            "\u001b[?25hRequirement already satisfied: idna!=2.3,>=0.6; extra == \"tls\" in /usr/local/lib/python3.6/dist-packages (from twisted[tls]>=17.5.0->magic-wormhole) (2.8)\n",
            "Collecting cryptography\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/ca/9a/7cece52c46546e214e10811b36b2da52ce1ea7fa203203a629b8dfadad53/cryptography-2.8-cp34-abi3-manylinux2010_x86_64.whl (2.3MB)\n",
            "\u001b[K     |████████████████████████████████| 2.3MB 36.1MB/s \n",
            "\u001b[?25hRequirement already satisfied: cffi>=1.4.1 in /usr/local/lib/python3.6/dist-packages (from pynacl->magic-wormhole) (1.13.1)\n",
            "Collecting txaio>=18.8.1\n",
            "  Downloading https://files.pythonhosted.org/packages/e9/6d/e1a6f7835cde86728e5bb1f577be9b2d7d273fdb33c286e70b087d418ded/txaio-18.8.1-py2.py3-none-any.whl\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from zope.interface>=4.4.2->twisted[tls]>=17.5.0->magic-wormhole) (41.4.0)\n",
            "Requirement already satisfied: pyasn1 in /usr/local/lib/python3.6/dist-packages (from service-identity>=18.1.0; extra == \"tls\"->twisted[tls]>=17.5.0->magic-wormhole) (0.4.7)\n",
            "Requirement already satisfied: pyasn1-modules in /usr/local/lib/python3.6/dist-packages (from service-identity>=18.1.0; extra == \"tls\"->twisted[tls]>=17.5.0->magic-wormhole) (0.2.7)\n",
            "Requirement already satisfied: pycparser in /usr/local/lib/python3.6/dist-packages (from cffi>=1.4.1->pynacl->magic-wormhole) (2.19)\n",
            "Building wheels for collected packages: hkdf\n",
            "  Building wheel for hkdf (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for hkdf: filename=hkdf-0.0.3-cp36-none-any.whl size=3736 sha256=7d94daa4305b5550e308311b1735f3c2d932f2c2fdd42197c7e598ba774fed5b\n",
            "  Stored in directory: /root/.cache/pip/wheels/6d/91/ba/2e93364022df91743d71cc6bd91414f29a4afd20a89a3077df\n",
            "Successfully built hkdf\n",
            "Installing collected packages: regex, toposort, incremental, automat, hyperlink, constantly, zope.interface, PyHamcrest, cryptography, service-identity, pyopenssl, twisted, txtorcon, pynacl, hkdf, spake2, txaio, autobahn, magic-wormhole\n",
            "Successfully installed PyHamcrest-1.9.0 autobahn-19.10.1 automat-0.8.0 constantly-15.1.0 cryptography-2.8 hkdf-0.0.3 hyperlink-19.0.0 incremental-17.5.0 magic-wormhole-0.11.2 pynacl-1.3.0 pyopenssl-19.0.0 regex-2019.11.1 service-identity-18.1.0 spake2-0.8 toposort-1.5 twisted-19.7.0 txaio-18.8.1 txtorcon-19.1.0 zope.interface-4.6.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eeewvXtT6Wyd",
        "colab_type": "code",
        "outputId": "81a9e8f3-f67e-43da-9477-b77359a445d2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 82
        }
      },
      "source": [
        "!wormhole receive 1-therapist-mural --accept-file"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Receiving file (18.4 MB) into: sessions-all-gpt2.txt.npz\n",
            "Receiving (->relay:tcp:transit.magic-wormhole.io:4001)..\n",
            "100% 18.4M/18.4M [00:45<00:00, 277kB/s]\n",
            "Received file written to sessions-all-gpt2.txt.npz\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c959127m6jBz",
        "colab_type": "code",
        "outputId": "225e3f7c-2429-4552-d5b4-7e28e0910bdb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "!PYTHONPATH=src python3 train.py  --model_name 1558M --dataset sessions-all-gpt2.txt.npz --batch_size 1 --save_every 50 --sample_every 20 --sample_num 1 --sample_length 192 --save_on_ctrlc"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /content/gpt-2/src/model.py:147: The name tf.AUTO_REUSE is deprecated. Please use tf.compat.v1.AUTO_REUSE instead.\n",
            "\n",
            "WARNING:tensorflow:\n",
            "The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
            "For more information, please see:\n",
            "  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
            "  * https://github.com/tensorflow/addons\n",
            "  * https://github.com/tensorflow/io (for I/O related ops)\n",
            "If you depend on functionality not listed there, please file an issue.\n",
            "\n",
            "WARNING:tensorflow:From /content/gpt-2/src/memory_saving_gradients.py:13: The name tf.GraphKeys is deprecated. Please use tf.compat.v1.GraphKeys instead.\n",
            "\n",
            "grpc://10.87.1.98:8470\n",
            "WARNING:tensorflow:From train.py:122: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.\n",
            "\n",
            "WARNING:tensorflow:From train.py:125: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.\n",
            "\n",
            "Using TPU grpc://10.87.1.98:8470\n",
            "WARNING:tensorflow:From train.py:131: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n",
            "\n",
            "WARNING:tensorflow:From /content/gpt-2/src/model.py:148: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.\n",
            "\n",
            "WARNING:tensorflow:From /content/gpt-2/src/model.py:152: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.\n",
            "\n",
            "WARNING:tensorflow:From /content/gpt-2/src/model.py:36: The name tf.rsqrt is deprecated. Please use tf.math.rsqrt instead.\n",
            "\n",
            "WARNING:tensorflow:From /content/gpt-2/src/model.py:166: The name tf.add_to_collection is deprecated. Please use tf.compat.v1.add_to_collection instead.\n",
            "\n",
            "WARNING:tensorflow:From /content/gpt-2/src/sample.py:65: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use `tf.cast` instead.\n",
            "WARNING:tensorflow:From /content/gpt-2/src/sample.py:16: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.where in 2.0, which has the same broadcast rule as np.where\n",
            "WARNING:tensorflow:From /content/gpt-2/src/sample.py:70: multinomial (from tensorflow.python.ops.random_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use `tf.random.categorical` instead.\n",
            "WARNING:tensorflow:From train.py:156: The name tf.trainable_variables is deprecated. Please use tf.compat.v1.trainable_variables instead.\n",
            "\n",
            "This model is using 1557611200 parameters (1485.45M)\n",
            "WARNING:tensorflow:From train.py:163: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.\n",
            "\n",
            "WARNING:tensorflow:From train.py:192: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.\n",
            "\n",
            "WARNING:tensorflow:From train.py:195: The name tf.summary.merge is deprecated. Please use tf.compat.v1.summary.merge instead.\n",
            "\n",
            "WARNING:tensorflow:From train.py:197: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.\n",
            "\n",
            "WARNING:tensorflow:From train.py:200: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.\n",
            "\n",
            "WARNING:tensorflow:From train.py:204: The name tf.global_variables_initializer is deprecated. Please use tf.compat.v1.global_variables_initializer instead.\n",
            "\n",
            "Loading checkpoint gs://sgappa-multi/gpt-2/models/1558M/model.ckpt\n",
            "2019-11-05 19:10:49.578113: W tensorflow/core/platform/cloud/google_auth_provider.cc:178] All attempts to get a Google authentication bearer token failed, returning an empty token. Retrieving token from files failed with \"Not found: Could not locate the credentials file.\". Retrieving token from GCE failed with \"Cancelled: GCE check skipped due to presence of $NO_GCE_CHECK environment variable.\".\n",
            "Loading dataset...\n",
            "100% 1/1 [00:01<00:00,  1.61s/it]\n",
            "dataset has 37818008 tokens\n",
            "Training...\n",
            "Generating samples...\n",
            "======== SAMPLE 1 ========\n",
            " A lot of it is because we still have a lot of work to do. But as we have said, we are very positive about India's future.\n",
            "\n",
            "The US President has also invited the Indian team here for the IOA G20 summit in Hamburg. The President himself has been in India over the years to attend numerous international summit.\n",
            "\n",
            "Indian people are very proud of their president. The American people are also very proud of him.\n",
            "\n",
            "President Barack Obama on his visit to India, in which he held talks with Prime Minister Narendra Modi, during which he also launched new technology, on 20 June, 2012. AFP\n",
            "\n",
            "I think the two leaders were able to meet on a very positive and a very cordial basis. It started as an amicable, good-natured meeting and eventually the atmosphere in the room changed, because obviously, the India-US relationship is in a very different phase from what it was. And that's why\n",
            "\n",
            "======== SAMPLE 2 ========\n",
            " A.C. A.J. Ackerly and L.F. Mazzola , A comparative analysis of genetic polymorphisms and phenotypic traits in a non-human primate population with a focus on the evolution of aggression , Journal of Zoology , 286 , 1 , (35) , .\n",
            "\n",
            "James A. Clark, Michael V. O'Callaghan and Steven C. Miller , A genetic model for human personality and emotional response: A meta-analysis , Developmental Psychobiology , 53 , 3 , (399-420) , .\n",
            "\n",
            "Kirsten M. Hultman and Michael D. Stoneking , Human Emotional and Cognitive Responses to Pain , Psychological Science in the Public Interest , 8 , 3 , (71) , .\n",
            "\n",
            "Hugh J. Miller, Richard J. Crisp and John K. Achtermeyer , Maternal genetic correlations and the behavioral genetics of maternal stress and depression\n",
            "\n",
            "======== SAMPLE 3 ========\n",
            " A-5's or A-1's and have all that armor protection. (And then again, a lot of the equipment in the A-5 is fairly specialized, meaning it's not very versatile.) I could imagine one pilot getting really into it and really wanting the protection (as if it was worth the time to be the only pilot in the A-5 without anti-air power), and all the other pilots in the A-5 being completely safe.\n",
            "\n",
            "The A-14 is probably my favorite of these jets, even if it's really hard to get into the cockpit; there are only four flight seats. (I am a huge advocate of flight seats, in general. I think they prevent so much pain that nobody should go without; they're actually the only viable means of flying these older, more archaic airplanes.) The A-14 is a very simple jet, and very easy to fly, except for the fact that it's extremely\n",
            "\n",
            "======== SAMPLE 4 ========\n",
            " A.J. is an \"artistic genius.\" His work, including the award-winning documentary \"Dirty Wars\" and \"The Assassination Complex,\" has been aired on PBS and broadcast on the PBS NewsHour. He is a frequent contributor to the media site Salon. He lives in Los Angeles.\n",
            "\n",
            "JONATHAN JOHNSON was on his way to a wedding in San Diego, Calif., in August 2010, when his motorcycle rear-ended a sport utility vehicle while speeding down a four-lane highway near the city of Imperial Beach. The impact caused Johnson to swerve out of his lane.\n",
            "\n",
            "When he pulled over, he noticed a man standing by the driver's side. The driver quickly approached and asked what happened. At first Johnson, who works as a security guard, thought the man had pulled a weapon and threatened him, but when he looked closer, he realized that he was holding a camera and started filming.\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}