tf_mnist_cnn_jupyter.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Save and Resume a Tensorflow MNIST ConvNet Model\n",
    "\n",
    "This jupyter notebook, show you how to save and resume a Tensorflow Model. In this example we will use the Deep Learning hello-world!: the MNIST classification task.\n",
    "Note: to run code cell you have to press **`Shift + Enter`**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import Packages\n",
    "First we need a single point with all the dependencies:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import absolute_import\n",
    "from __future__ import division\n",
    "from __future__ import print_function\n",
    "\n",
    "from tensorflow.contrib.learn.python.learn.datasets.mnist import read_data_sets\n",
    "\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import shutil, os\n",
    "\n",
    "tf.logging.set_verbosity(tf.logging.INFO)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyper Parameters and Variables\n",
    "\n",
    "Even for Hyper-Parameters and Variables is a good practice have a single point, this improve code readability and experiments interation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Where to save Checkpoint(In the /output folder)\n",
    "resumepath =\"/model/mnist_convnet_model\"\n",
    "filepath = \"/output/mnist_convnet_model\" \n",
    "\n",
    "# Hyper-parameters\n",
    "batch_size = 128\n",
    "num_classes = 10\n",
    "num_epochs = 12\n",
    "learning_rate = 1e-3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Resuming from Previuos Run\n",
    "\n",
    "If we have mounted a previuos run, copy the checkpoint to the `/output` folder so that the Model will continue from that and save everything in it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If exists an checkpoint model, move it into the /output folder\n",
    "if os.path.exists(resumepath):\n",
    "    shutil.copytree(resumepath, filepath)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Processing and Transformation\n",
    "Next, we process the dataset sample in tensor, ready to be feed into the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully downloaded train-images-idx3-ubyte.gz 9912422 bytes.\n",
      "Extracting /input/MNIST_data/train-images-idx3-ubyte.gz\n",
      "Successfully downloaded train-labels-idx1-ubyte.gz 28881 bytes.\n",
      "Extracting /input/MNIST_data/train-labels-idx1-ubyte.gz\n",
      "Successfully downloaded t10k-images-idx3-ubyte.gz 1648877 bytes.\n",
      "Extracting /input/MNIST_data/t10k-images-idx3-ubyte.gz\n",
      "Successfully downloaded t10k-labels-idx1-ubyte.gz 4542 bytes.\n",
      "Extracting /input/MNIST_data/t10k-labels-idx1-ubyte.gz\n",
      "(60000, 784)\n",
      "(10000, 784)\n"
     ]
    }
   ],
   "source": [
    "# Load training and eval data\n",
    "mnist = read_data_sets(train_dir='/input/MNIST_data', validation_size=0)\n",
    "train_data = mnist.train.images  # Returns np.array\n",
    "train_labels = np.asarray(mnist.train.labels, dtype=np.int32)\n",
    "eval_data = mnist.test.images  # Returns np.array\n",
    "eval_labels = np.asarray(mnist.test.labels, dtype=np.int32)\n",
    "\n",
    "print (train_data.shape)\n",
    "print (eval_data.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define the Model\n",
    "A ConvNet Model, state of the art for image classification task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cnn_model_fn(features, labels, mode):\n",
    "    \"\"\"Model function for CNN.\"\"\"\n",
    "    # Input Layer\n",
    "    # Reshape X to 4-D tensor: [batch_size, width, height, channels]\n",
    "    # MNIST images are 28x28 pixels, and have one color channel\n",
    "    input_layer = tf.reshape(features[\"x\"], [-1, 28, 28, 1])\n",
    "\n",
    "    # Convolutional Layer #1\n",
    "    # Computes 32 features using a 3x3 filter with ReLU activation.\n",
    "    # Input Tensor Shape: [batch_size, 28, 28, 1]\n",
    "    # Output Tensor Shape: [batch_size, 26, 26, 32]\n",
    "    conv1 = tf.layers.conv2d(\n",
    "      inputs=input_layer,\n",
    "      filters=32,\n",
    "      kernel_size=[3, 3],\n",
    "      activation=tf.nn.relu)\n",
    "\n",
    "    # Convolutional Layer #2\n",
    "    # Computes 64 features using a 3x3 filter.\n",
    "    # Input Tensor Shape: [batch_size, 26, 26 32]\n",
    "    # Output Tensor Shape: [batch_size, 24, 24, 64]\n",
    "    conv2 = tf.layers.conv2d(\n",
    "      inputs=conv1,\n",
    "      filters=64,\n",
    "      kernel_size=[3, 3],\n",
    "      activation=tf.nn.relu)\n",
    "\n",
    "    # Pooling Layer\n",
    "    # Max pooling layer with a 2x2 filter and stride of 2\n",
    "    # Input Tensor Shape: [batch_size, 24, 24, 64]\n",
    "    # Output Tensor Shape: [batch_size, 12, 12, 64]\n",
    "    pool = tf.layers.max_pooling2d(inputs=conv2, pool_size=[2, 2], strides=2)\n",
    "\n",
    "    # Dropout # 1\n",
    "    # Add dropout operation; 0.25 probability that element will be kept\n",
    "    dropout = tf.layers.dropout(\n",
    "      inputs=pool, rate=0.25, training=mode == tf.estimator.ModeKeys.TRAIN)\n",
    "\n",
    "    # Flatten tensor into a batch of vectors\n",
    "    # Input Tensor Shape: [batch_size, 12, 12, 64]\n",
    "    # Output Tensor Shape: [batch_size, 12 * 12 * 64]\n",
    "    flat = tf.reshape(dropout, [-1, 12 * 12 * 64])  # 9216\n",
    "\n",
    "    \n",
    "    # Dense Layer # 1\n",
    "    # Densely connected layer with 128 neurons\n",
    "    # Input Tensor Shape: [batch_size, 12 * 12 * 64] (batch_size, 9216)\n",
    "    # Output Tensor Shape: [batch_size, 128]\n",
    "    dense1 = tf.layers.dense(inputs=flat, units=128, activation=tf.nn.relu)\n",
    "    \n",
    "    # Dropout # 2\n",
    "    # Add dropout operation; 0.5 probability that element will be kept\n",
    "    dropout2 = tf.layers.dropout(\n",
    "      inputs=dense1, rate=0.5, training=mode == tf.estimator.ModeKeys.TRAIN)\n",
    "\n",
    "    # Logits layer\n",
    "    # Input Tensor Shape: [batch_size, 128]\n",
    "    # Output Tensor Shape: [batch_size, 10]\n",
    "    logits = tf.layers.dense(inputs=dropout2, units=num_classes)\n",
    "\n",
    "    predictions = {\n",
    "        # Generate predictions (for PREDICT and EVAL mode)\n",
    "        \"classes\": tf.argmax(input=logits, axis=1),\n",
    "        # Add `softmax_tensor` to the graph. It is used for PREDICT and by the\n",
    "        # `logging_hook`.\n",
    "        \"probabilities\": tf.nn.softmax(logits, name=\"softmax_tensor\")\n",
    "    }\n",
    "    # Inference (for TEST mode)\n",
    "    if mode == tf.estimator.ModeKeys.PREDICT:\n",
    "        return tf.estimator.EstimatorSpec(mode=mode, predictions=predictions)\n",
    "\n",
    "    # Calculate Loss (for both TRAIN and EVAL modes)\n",
    "    onehot_labels = tf.one_hot(indices=tf.cast(labels, tf.int32), depth=num_classes)\n",
    "    # Cross Entropy\n",
    "    loss = tf.losses.softmax_cross_entropy(\n",
    "      onehot_labels=onehot_labels, logits=logits)\n",
    "\n",
    "    # Configure the Training Op (for TRAIN mode)\n",
    "    if mode == tf.estimator.ModeKeys.TRAIN:\n",
    "        # AdamOptimizer\n",
    "        optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)\n",
    "        train_op = optimizer.minimize(\n",
    "            loss=loss,\n",
    "            global_step=tf.train.get_global_step())\n",
    "        return tf.estimator.EstimatorSpec(mode=mode, loss=loss, train_op=train_op)\n",
    "\n",
    "    # Add evaluation metrics (for EVAL mode)\n",
    "    eval_metric_ops = {\n",
    "      \"accuracy\": tf.metrics.accuracy(\n",
    "          labels=labels, predictions=predictions[\"classes\"])}\n",
    "    return tf.estimator.EstimatorSpec(\n",
    "      mode=mode, loss=loss, eval_metric_ops=eval_metric_ops)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Checkpoint Strategy\n",
    "\n",
    "The strategy we have adopted for the this example is the following:\n",
    "\n",
    "- Keep only one checkpoints\n",
    "- Trigger the strategy at the end of every epoch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Checkpoint Strategy configuration\n",
    "run_config = tf.contrib.learn.RunConfig(\n",
    "    model_dir=filepath,\n",
    "    keep_checkpoint_max=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using config: {'_task_type': None, '_task_id': 0, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fc252758d30>, '_master': '', '_num_ps_replicas': 0, '_num_worker_replicas': 0, '_environment': 'local', '_is_chief': True, '_evaluation_master': '', '_tf_config': gpu_options {\n",
      "  per_process_gpu_memory_fraction: 1.0\n",
      "}\n",
      ", '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_secs': 600, '_log_step_count_steps': 100, '_session_config': None, '_save_checkpoints_steps': None, '_keep_checkpoint_max': 1, '_keep_checkpoint_every_n_hours': 10000, '_model_dir': '/output/mnist_convnet_model'}\n"
     ]
    }
   ],
   "source": [
    "# Create the Estimator\n",
    "mnist_classifier = tf.estimator.Estimator(\n",
    "      model_fn=cnn_model_fn, config=run_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training\n",
    "Let's train the model and see our checkpoint strategy in action."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Begin Training - Epoch 1/12\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from /output/mnist_convnet_model/model.ckpt-3284\n",
      "INFO:tensorflow:Saving checkpoints for 3285 into /output/mnist_convnet_model/model.ckpt.\n",
      "INFO:tensorflow:loss = 0.0112094, step = 3285\n",
      "INFO:tensorflow:global_step/sec: 55.4695\n",
      "INFO:tensorflow:loss = 0.0575566, step = 3385 (1.804 sec)\n"
     ]
    }
   ],
   "source": [
    "# Keep track of the best accuracy\n",
    "best_acc = 0\n",
    "\n",
    "# Training for num_epochs\n",
    "for i in range(num_epochs):\n",
    "    print(\"Begin Training - Epoch {}/{}\".format(i+1, num_epochs))\n",
    "    # Train the model for 1 epoch\n",
    "    train_input_fn = tf.estimator.inputs.numpy_input_fn(\n",
    "        x={\"x\": train_data},\n",
    "        y=train_labels,\n",
    "        batch_size=batch_size,\n",
    "        num_epochs=1,\n",
    "        shuffle=True)\n",
    "\n",
    "    mnist_classifier.train(\n",
    "        input_fn=train_input_fn)\n",
    "\n",
    "    # Evaluate the model and print results\n",
    "    eval_input_fn = tf.estimator.inputs.numpy_input_fn(\n",
    "        x={\"x\": eval_data},\n",
    "        y=eval_labels,\n",
    "        num_epochs=1,\n",
    "        shuffle=False)\n",
    "    \n",
    "    eval_results = mnist_classifier.evaluate(input_fn=eval_input_fn)\n",
    "    \n",
    "    accuracy = eval_results[\"accuracy\"] * 100\n",
    "    # Set the best acc if we have a new best or if it is the first step \n",
    "    if accuracy > best_acc or i == 0:\n",
    "        best_acc = accuracy\n",
    "        print (\"=> New Best Accuracy {}\".format(accuracy))\n",
    "    else:\n",
    "        print(\"=> Validation Accuracy did not improve\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Resume the checkpoint after the training\n",
    "Let's take a look at the checkpoint just created. (you should see the `mnist_convnet_model` folder)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0m\u001b[01;34mMNIST-data\u001b[0m/     command.sh                     \u001b[01;34mmnist_convnet_model\u001b[0m/\r\n",
      "README.md       keras_mnist_cnn.py             pytorch_mnist_cnn.py\r\n",
      "Untitled.ipynb  keras_mnist_cnn_jupyter.ipynb  pytorch_mnist_cnn_jupyter.ipynb\r\n"
     ]
    }
   ],
   "source": [
    "% ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Jupyter Notebook run in the `/output folder`, so it's here. If you want to load it, you only need to restart the **Training** Cell Code, the Estimator will take care of everything."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}