ML0120EN-4.1-Review-RBMMNIST.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-DL0120EN-SkillsNetwork/images/IDSN-logo.png\" width=\"300\" alt=\"cognitiveclass.ai logo\">\n",
    "\n",
    "<h1 align=\"center\"><font size=\"5\">RESTRICTED BOLTZMANN MACHINES</font></h1>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Estimated time needed: **25** minutes\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Introduction</h3>\n",
    "<b>Restricted Boltzmann Machine (RBM):</b>  RBMs are shallow neural nets that learn to reconstruct data by themselves in an unsupervised fashion.  \n",
    "\n",
    "\n",
    "<h4>Why are RBMs important?</h4>\n",
    "An RBM are a basic form of autoencoder.  It can automatically extract <b>meaningful</b> features from a given input.\n",
    "\n",
    "\n",
    "<h4>How does it work?</h4>\n",
    "RBM is a 2 layer neural network. Simply, RBM takes the inputs and translates those into a set of binary values that represents them in the hidden layer. Then, these numbers can be translated back to reconstruct the inputs. Through several forward and backward passes, the RBM will be trained, and a trained RBM can reveal which features are the most important ones when detecting patterns.   \n",
    "\n",
    "\n",
    "<h4>What are the applications of an RBM?</h4>\n",
    "RBM is useful for <a href='http://www.cs.utoronto.ca/~hinton/absps/netflixICML.pdf'>  Collaborative Filtering</a>, dimensionality reduction, classification, regression, feature learning, topic modeling and even <b>Deep Belief Networks</b>.\n",
    "\n",
    "<h4>Is RBM a generative or Discriminative model?</h4>\n",
    "RBM is a generative model. Let me explain it by first, see what is different between discriminative and generative models: \n",
    "\n",
    "<b>Discriminative:</b> Consider a classification problem where we want to learn to distinguish between Sedan cars (y = 1) and SUV cars (y = 0), based on some features of cars. Given a training set, an algorithm like logistic regression tries to find a straight line, or <i>decision boundary</i>, that separates the suv and sedan.  \n",
    "\n",
    "<b>Generative:</b> looking at cars, we can build a model of what Sedan cars look like. Then, looking at SUVs, we can build a separate model of what SUV cars look like. Finally, to classify a new car, we can match the new car against the Sedan model, and match it against the SUV model, to see whether the new car looks more like the SUV or Sedan. \n",
    "\n",
    "Generative Models specify a probability distribution over a dataset of input vectors. We can carry out both supervised and unsupervised tasks with generative models:\n",
    "<ul>\n",
    "    <li>In an unsupervised task, we try to form a model for $P(x)$, where $P$ is the probability given $x$ as an input vector.</li>\n",
    "    <li>In the supervised task, we first form a model for $P(x|y)$, where $P$ is the probability of $x$ given $y$(the label for $x$). For example, if $y = 0$ indicates that a car is an SUV, and $y = 1$ indicates that a car is a sedan, then $p(x|y = 0)$ models the distribution of SUV features, and $p(x|y = 1)$ models the distribution of sedan features. If we manage to find $P(x|y)$ and $P(y)$, then we can use <b>Bayes rule</b> to estimate $P(y|x)$, because:   \n",
    "        $$p(y|x) = \\frac{p(x|y)p(y)}{p(x)}$$</li>\n",
    "</ul>\n",
    "Now the question is, can we build a generative model, and then use it to create synthetic data by directly sampling from the modeled probability distributions? Lets see. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Table of Contents</h2>\n",
    "<ol>\n",
    "    <li><a href=\"#ref1\">Initialization</a></li>\n",
    "    <li><a href=\"#ref2\">RBM layers</a></li>\n",
    "    <li><a href=\"#ref3\">What RBM can do after training?</a></li>\n",
    "    <li><a href=\"#ref4\">How to train the model?</a></li>\n",
    "    <li><a href=\"#ref5\">Learned features</a></li>\n",
    "</ol>\n",
    "<p></p>\n",
    "</div>\n",
    "<br>\n",
    "\n",
    "<hr>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ref1\"></a>\n",
    "<h3>Initialization</h3>\n",
    "\n",
    "First, we have to load the utility file which contains different utility functions that are not connected\n",
    "in any way to the networks presented in the tutorials, but rather help in\n",
    "processing the outputs into a more understandable way.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import urllib.request\n",
    "with urllib.request.urlopen(\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-DL0120EN-SkillsNetwork/labs/Week4/data/utils.py\") as url:\n",
    "    response = url.read()\n",
    "target = open('utils.py', 'w')\n",
    "target.write(response.decode('utf-8'))\n",
    "target.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Installing TensorFlow </h2>\n",
    "\n",
    "We will installing TensorFlow version 2.9.0 and its required prerequistes.  Also installing pillow...\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: grpcio==1.24.3 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (1.24.3)\n",
      "Requirement already satisfied: six>=1.5.2 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from grpcio==1.24.3) (1.16.0)\n",
      "Requirement already satisfied: tensorflow==2.9.0 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (2.9.0)\n",
      "Requirement already satisfied: absl-py>=1.0.0 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (1.4.0)\n",
      "Requirement already satisfied: astunparse>=1.6.0 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (1.6.3)\n",
      "Requirement already satisfied: flatbuffers<2,>=1.12 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (1.12)\n",
      "Requirement already satisfied: gast<=0.4.0,>=0.2.1 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (0.4.0)\n",
      "Requirement already satisfied: google-pasta>=0.1.1 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (0.2.0)\n",
      "Requirement already satisfied: grpcio<2.0,>=1.24.3 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (1.24.3)\n",
      "Requirement already satisfied: h5py>=2.9.0 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (3.8.0)\n",
      "Requirement already satisfied: keras<2.10.0,>=2.9.0rc0 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (2.9.0)\n",
      "Requirement already satisfied: keras-preprocessing>=1.1.1 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (1.1.2)\n",
      "Requirement already satisfied: libclang>=13.0.0 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (16.0.6)\n",
      "Requirement already satisfied: numpy>=1.20 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (1.21.4)\n",
      "Requirement already satisfied: opt-einsum>=2.3.2 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (3.3.0)\n",
      "Requirement already satisfied: packaging in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (23.1)\n",
      "Requirement already satisfied: protobuf>=3.9.2 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (3.19.6)\n",
      "Requirement already satisfied: setuptools in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (67.7.2)\n",
      "Requirement already satisfied: six>=1.12.0 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (1.16.0)\n",
      "Requirement already satisfied: tensorboard<2.10,>=2.9 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (2.9.1)\n",
      "Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (0.34.0)\n",
      "Requirement already satisfied: tensorflow-estimator<2.10.0,>=2.9.0rc0 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (2.9.0)\n",
      "Requirement already satisfied: termcolor>=1.1.0 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (2.3.0)\n",
      "Requirement already satisfied: typing-extensions>=3.6.6 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (4.5.0)\n",
      "Requirement already satisfied: wrapt>=1.11.0 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorflow==2.9.0) (1.14.1)\n",
      "Requirement already satisfied: wheel<1.0,>=0.23.0 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from astunparse>=1.6.0->tensorflow==2.9.0) (0.40.0)\n",
      "Requirement already satisfied: google-auth<3,>=1.6.3 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorboard<2.10,>=2.9->tensorflow==2.9.0) (2.23.4)\n",
      "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorboard<2.10,>=2.9->tensorflow==2.9.0) (0.4.6)\n",
      "Requirement already satisfied: markdown>=2.6.8 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorboard<2.10,>=2.9->tensorflow==2.9.0) (3.4.3)\n",
      "Requirement already satisfied: requests<3,>=2.21.0 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorboard<2.10,>=2.9->tensorflow==2.9.0) (2.29.0)\n",
      "Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorboard<2.10,>=2.9->tensorflow==2.9.0) (0.6.1)\n",
      "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorboard<2.10,>=2.9->tensorflow==2.9.0) (1.8.1)\n",
      "Requirement already satisfied: werkzeug>=1.0.1 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from tensorboard<2.10,>=2.9->tensorflow==2.9.0) (2.2.3)\n",
      "Requirement already satisfied: cachetools<6.0,>=2.0.0 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from google-auth<3,>=1.6.3->tensorboard<2.10,>=2.9->tensorflow==2.9.0) (5.3.2)\n",
      "Requirement already satisfied: pyasn1-modules>=0.2.1 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from google-auth<3,>=1.6.3->tensorboard<2.10,>=2.9->tensorflow==2.9.0) (0.3.0)\n",
      "Requirement already satisfied: rsa<5,>=3.1.4 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from google-auth<3,>=1.6.3->tensorboard<2.10,>=2.9->tensorflow==2.9.0) (4.9)\n",
      "Requirement already satisfied: requests-oauthlib>=0.7.0 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.10,>=2.9->tensorflow==2.9.0) (1.3.1)\n",
      "Requirement already satisfied: importlib-metadata>=4.4 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from markdown>=2.6.8->tensorboard<2.10,>=2.9->tensorflow==2.9.0) (4.11.4)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.10,>=2.9->tensorflow==2.9.0) (3.1.0)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.10,>=2.9->tensorflow==2.9.0) (3.4)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.10,>=2.9->tensorflow==2.9.0) (1.26.15)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.10,>=2.9->tensorflow==2.9.0) (2023.5.7)\n",
      "Requirement already satisfied: MarkupSafe>=2.1.1 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from werkzeug>=1.0.1->tensorboard<2.10,>=2.9->tensorflow==2.9.0) (2.1.1)\n",
      "Requirement already satisfied: zipp>=0.5 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard<2.10,>=2.9->tensorflow==2.9.0) (3.15.0)\n",
      "Requirement already satisfied: pyasn1<0.6.0,>=0.4.6 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard<2.10,>=2.9->tensorflow==2.9.0) (0.5.0)\n",
      "Requirement already satisfied: oauthlib>=3.0.0 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.10,>=2.9->tensorflow==2.9.0) (3.2.2)\n",
      "Requirement already satisfied: pillow==8.1.0 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (8.1.0)\n"
     ]
    }
   ],
   "source": [
    "!pip install grpcio==1.24.3\n",
    "!pip install tensorflow==2.9.0\n",
    "!pip install pillow==8.1.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Restart kernel for latest version of TensorFlow to be activated, then continue to the next section.**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Notice:</b> This notebook has been created with TensorFlow version 2.9.0, and might not work with other versions. Therefore we check:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-04 15:39:37.378670: I tensorflow/core/util/util.cc:169] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.\n",
      "2023-11-04 15:39:37.383988: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory\n",
      "2023-11-04 15:39:37.384017: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "from IPython.display import Markdown, display\n",
    "TF_ENABLE_ONEDNN_OPTS=0\n",
    "def printmd(string):\n",
    "    display(Markdown('# <span style=\"color:red\">'+string+'</span>'))\n",
    "\n",
    "\n",
    "if not tf.__version__ == '2.9.0':\n",
    "    printmd('<<<<<!!!!! ERROR !!!! please upgrade to TensorFlow 2.9.0, or restart your Kernel (Kernel->Restart & Clear Output)>>>>>')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we load in all the packages that we use to create the net including the TensorFlow package:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "\n",
    "from PIL import Image\n",
    "from utils import tile_raster_images\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ref2\"></a>\n",
    "<h3>RBM layers</h3>\n",
    "\n",
    "An RBM has two layers. The first layer of the RBM is called the <b>visible</b> (or input layer). Imagine that our toy example, has only vectors with 7 values, so the visible layer must have $V=7$ input nodes. \n",
    "The second layer is the <b>hidden</b> layer, which has $H$ neurons in our case. Each hidden node takes on values of either 0 or 1 (i.e., $h_i = 1$ or $h_i$ = 0), with a probability that is a logistic function of the inputs it receives from the other $V$ visible units, called for example, $p(h_i = 1)$. For our toy sample, we'll use 2 nodes in the hidden layer, so $H = 2$.\n",
    "\n",
    "<center><img src=\"https://ibm.box.com/shared/static/eu26opvcefgls6vnwuo29uwp0nudmokh.png\" alt=\"RBM Model\" style=\"width: 400px;\"></center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "     \n",
    "\n",
    "Each node in the first layer also has a <b>bias</b>. We will denote the bias as $v_{bias}$, and this single value is shared among the $V$ visible units.\n",
    "\n",
    "The <b>bias</b> of the second is defined similarly as $h_{bias}$, and this single value among the $H$ hidden units.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-04 15:39:45.000646: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory\n",
      "2023-11-04 15:39:45.000705: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)\n",
      "2023-11-04 15:39:45.000750: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (jupyterlab-yashmenaria2): /proc/driver/nvidia/version does not exist\n",
      "2023-11-04 15:39:45.001109: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F AVX512_VNNI FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n"
     ]
    }
   ],
   "source": [
    "v_bias = tf.Variable(tf.zeros([7]), tf.float32)\n",
    "h_bias = tf.Variable(tf.zeros([2]), tf.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have to define weights among the input layer and hidden layer nodes. In the weight matrix, the number of rows are equal to the input nodes, and the number of columns are equal to the output nodes. We define a tensor $\\mathbf{W}$ of shape = (7,2), where the number of visible neurons = 7, and the number of hidden neurons = 2. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "W = tf.constant(np.random.normal(loc=0.0, scale=1.0, size=(7, 2)).astype(np.float32))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ref3\"></a>\n",
    "<h3>What RBM can do after training?</h3>\n",
    "Think of RBM as a model that has been trained based on images of a dataset of many SUV and sedan cars. Also, imagine that the RBM network has only two hidden nodes, where one node encodes the weight and, and the other encodes the size.  \n",
    "In a sense, the different configurations represent different cars, where one is an SUV and the other is Sedan.  In a training process, through many forward and backward passes, the RBM adjust its weights to send a stronger signal to either the SUV node (0, 1) or the sedan node (1, 0) in the hidden layer, given the pixels of images. Now, given an SUV in hidden layer, which distribution of pixels should we expect? RBM can give you 2 things. First, it encodes your images in hidden layer. Second, it gives you the probability of observing a case, given some hidden values.\n",
    "\n",
    "\n",
    "<h3>The Inference Process</h3>\n",
    "\n",
    "RBM has two phases:\n",
    "<ul>\n",
    "    <li>Forward Pass</li>  \n",
    "    <li>Backward Pass or Reconstruction</li>\n",
    "</ul>\n",
    "\n",
    "<b>Phase 1) Forward pass:</b>  \n",
    "\n",
    "Input one training sample (one image) $\\mathbf{x}$ through all visible nodes, and pass it to all hidden nodes. Processing happens in each node in the hidden layer. This computation begins by making stochastic decisions about whether to transmit that input or not (i.e. to determine the state of each hidden layer).  First, the probability vector is computed using the input feature vector $\\mathbf{x}$, the weight matrix $\\mathbf{W}$, and the bias term $h_{bias}$, as \n",
    "\n",
    "\n",
    "$$p({h_j}|\\mathbf x)= \\sigma( \\sum_{i=1}^V W_{ij} x_i + h_{bias} )$$, \n",
    "\n",
    "where $\\sigma(z) = (1+e^{-z})^{-1}$ is the logistic function.\n",
    "\n",
    "\n",
    "So, what does $p({h_j})$ represent? It is the <b>probability distribution</b> of the hidden units. That is, RBM uses inputs $x_i$ to make predictions about hidden node activations. For example, imagine that the hidden node activation values are [0.51 0.84] for the first training item. It tells you that the conditional probability for each hidden neuron for Phase 1 is: \n",
    "\n",
    "$$p(h_{1} = 1|\\mathbf{v}) = 0.51$$\n",
    "$$p(h_{2} = 1|\\mathbf{v}) = 0.84$$\n",
    "\n",
    "As a result, for each row in the training set, vector of probabilities is generated.  In TensorFlow, this is referred to as a `tensor` with a shape of (1,2). \n",
    "\n",
    "We then turn unit $j$ with probability $p(h_{j}|\\mathbf{v})$, and turn it off with probability $1 - p(h_{j}|\\mathbf{v})$ by generating a uniform random number vector $\\mathbf{\\xi}$, and comparing it to the activation probability as \n",
    "\n",
    "<center>If $\\xi_j>p(h_{j}|\\mathbf{v})$, then $h_j=1$, else $h_j=0$.</center>\n",
    "\n",
    "\n",
    "Therefore, the conditional probability of a configuration of $\\mathbf{h}$ given $\\mathbf{v}$ (for a training sample) is:\n",
    "\n",
    "$$p(\\mathbf{h} \\mid \\mathbf{v}) = \\prod_{j=1}^H p(h_j \\mid \\mathbf{v})$$\n",
    "\n",
    "\n",
    "where $H$ is the number of hidden units.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we go further, let's look at a toy example for one case out of all input. Assume that we have a trained RBM, and a very simple input vector, such as [1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0].   \n",
    "Let's see what the output of forward pass would look like:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input:  tf.Tensor([[1. 0. 0. 1. 0. 0. 0.]], shape=(1, 7), dtype=float32)\n",
      "hb:  tf.Tensor([0.1 0.1], shape=(2,), dtype=float32)\n",
      "w:  tf.Tensor(\n",
      "[[ 1.2196002  -0.08672109]\n",
      " [-0.44729114 -0.7471718 ]\n",
      " [ 1.6227167   0.7451829 ]\n",
      " [-0.1504482   0.40499145]\n",
      " [ 0.28201276 -1.9111356 ]\n",
      " [ 0.33912623 -1.8325052 ]\n",
      " [-1.4747669   1.07278   ]], shape=(7, 2), dtype=float32)\n",
      "p(h|v):  tf.Tensor([[0.7629917  0.60306925]], shape=(1, 2), dtype=float32)\n",
      "h0 states: tf.Tensor([[1. 0.]], shape=(1, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "X = tf.constant([[1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0]], tf.float32)\n",
    "\n",
    "v_state = X\n",
    "print (\"Input: \", v_state)\n",
    "\n",
    "h_bias = tf.constant([0.1, 0.1])\n",
    "print (\"hb: \", h_bias)\n",
    "print (\"w: \", W)\n",
    "\n",
    "# Calculate the probabilities of turning the hidden units on:\n",
    "h_prob = tf.nn.sigmoid(tf.matmul(v_state, W) + h_bias)  #probabilities of the hidden units\n",
    "print (\"p(h|v): \", h_prob)\n",
    "\n",
    "# Draw samples from the distribution:\n",
    "h_state = tf.nn.relu(tf.sign(h_prob - tf.random.uniform(tf.shape(h_prob)))) #states\n",
    "print (\"h0 states:\", h_state)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Phase 2) Backward Pass (Reconstruction):</b>\n",
    "The RBM reconstructs data by making several forward and backward passes between the visible and hidden layers.\n",
    "\n",
    "So, in the second phase (i.e. reconstruction phase), the samples from the hidden layer (i.e. $\\mathbf h$) becomes the input in the backward pass. The same weight matrix and visible layer biases are used to passed to the sigmoid function. The reproduced output is a reconstruction which is an approximation of the original input.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b:  tf.Tensor([0.1 0.2 0.1 0.1 0.1 0.2 0.1], shape=(7,), dtype=float32)\n",
      "p(vi∣h):  tf.Tensor(\n",
      "[[0.7891152  0.43849033 0.84847844 0.4873906  0.59435844 0.63160914\n",
      "  0.20185077]], shape=(1, 7), dtype=float32)\n",
      "v probability states:  tf.Tensor([[1. 1. 1. 0. 1. 1. 0.]], shape=(1, 7), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "vb = tf.constant([0.1, 0.2, 0.1, 0.1, 0.1, 0.2, 0.1])\n",
    "print (\"b: \", vb)\n",
    "v_prob = tf.nn.sigmoid(tf.matmul(h_state, tf.transpose(W)) + vb)\n",
    "print (\"p(vi∣h): \", v_prob)\n",
    "v_state = tf.nn.relu(tf.sign(v_prob - tf.random.uniform(tf.shape(v_prob))))\n",
    "print (\"v probability states: \", v_state)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "RBM learns a probability distribution over the input, and then, after being trained, the RBM can generate new samples from the learned probability distribution. As you know, <b>probability distribution</b>, is a mathematical function that provides the probabilities of occurrence of different possible outcomes in an experiment.\n",
    "\n",
    "The (conditional) probability distribution over the visible units v is given by\n",
    "\n",
    "$$p(\\mathbf{v} \\mid \\mathbf{h}) = \\prod_{i=1}^V p(v_i \\mid \\mathbf{h}),$$\n",
    "\n",
    "\n",
    "where,\n",
    "\n",
    "$$p(v_i \\mid \\mathbf{h}) = \\sigma\\left(\\sum_{j=1}^H W_{ji} h_j + v_{bias} \\right)$$\n",
    "\n",
    "so, given current state of hidden units and weights, what is the probability of generating [1. 0. 0. 1. 0. 0. 0.] in reconstruction phase, based on the above <b>probability distribution</b> function?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input X: [[1. 0. 0. 1. 0. 0. 0.]]\n",
      "probablity vector: [0.7891152  0.43849033 0.84847844 0.4873906  0.59435844 0.63160914\n",
      " 0.20185077]\n",
      "probability of generating X:  0.0039028747\n"
     ]
    }
   ],
   "source": [
    "inp = X\n",
    "print(\"input X:\" , inp.numpy())\n",
    "\n",
    "print(\"probablity vector:\" , v_prob[0].numpy())\n",
    "v_probability = 1\n",
    "\n",
    "for elm, p in zip(inp[0],v_prob[0]) :\n",
    "    if elm ==1:\n",
    "        v_probability *= p\n",
    "    else:\n",
    "        v_probability *= (1-p)\n",
    "\n",
    "print(\"probability of generating X: \" , v_probability.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How similar are vectors $\\mathbf{x}$ and $\\mathbf{v}$? Of course, the reconstructed values most likely will not look anything like the input vector, because our network has not been trained yet. Our objective is to train the model in such a way that the input vector and reconstructed vector to be same. Therefore, based on how different the input values look to the ones that we just reconstructed, the weights are adjusted. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<h2>MNIST</h2>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will be using the MNIST dataset to practice the usage of RBMs. The following cell loads the MNIST dataset.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "#loading training and test data\n",
    "mnist = tf.keras.datasets.mnist\n",
    "(trX, trY), (teX, teY) = mnist.load_data()\n",
    "\n",
    "# showing an example of the Flatten class and operation\n",
    "from tensorflow.keras.layers import Flatten\n",
    "flatten = Flatten(dtype='float32')\n",
    "trX = flatten(trX/255.0)\n",
    "trY = flatten(trY/255.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets look at the dimension of the images.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MNIST images have 784 pixels, so the visible layer must have 784 input nodes.  For our case, we'll use 50 nodes in the hidden layer, so i = 50.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "vb = tf.Variable(tf.zeros([784]), tf.float32)\n",
    "hb = tf.Variable(tf.zeros([50]), tf.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let $\\mathbf W$ be the Tensor of 784x50 (784 - number of visible neurons, 50 - number of hidden neurons) that represents weights between the neurons. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "W = tf.Variable(tf.zeros([784,50]), tf.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets define the visible layer:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(1, 50), dtype=float32, numpy=\n",
       "array([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0.]], dtype=float32)>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v0_state = tf.Variable(tf.zeros([784]), tf.float32)\n",
    "\n",
    "#testing to see if the matrix product works\n",
    "tf.matmul( [v0_state], W)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can define hidden layer:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "h0_state shape:  tf.Tensor([ 1 50], shape=(2,), dtype=int32)\n",
      "first 15 hidden states:  tf.Tensor([1. 0. 0. 0. 0. 1. 0. 1. 0. 0. 1. 0. 1. 0. 1.], shape=(15,), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "#computing the hidden nodes probability vector and checking shape\n",
    "h0_prob = tf.nn.sigmoid(tf.matmul([v0_state], W) + hb)  #probabilities of the hidden units\n",
    "print(\"h0_state shape: \" , tf.shape(h0_prob))\n",
    "\n",
    "#defining a function to return only the generated hidden states \n",
    "def hidden_layer(v0_state, W, hb):\n",
    "    h0_prob = tf.nn.sigmoid(tf.matmul([v0_state], W) + hb)  #probabilities of the hidden units\n",
    "    h0_state = tf.nn.relu(tf.sign(h0_prob - tf.random.uniform(tf.shape(h0_prob)))) #sample_h_given_X\n",
    "    return h0_state\n",
    "\n",
    "\n",
    "h0_state = hidden_layer(v0_state, W, hb)\n",
    "print(\"first 15 hidden states: \", h0_state[0][0:15])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we define reconstruction part:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hidden state shape:  (1, 50)\n",
      "v0 state shape:   (784,)\n",
      "v1 state shape:   (784,)\n"
     ]
    }
   ],
   "source": [
    "def reconstructed_output(h0_state, W, vb):\n",
    "    v1_prob = tf.nn.sigmoid(tf.matmul(h0_state, tf.transpose(W)) + vb) \n",
    "    v1_state = tf.nn.relu(tf.sign(v1_prob - tf.random.uniform(tf.shape(v1_prob)))) #sample_v_given_h\n",
    "    return v1_state[0]\n",
    "\n",
    "v1_state = reconstructed_output(h0_state, W, vb)\n",
    "print(\"hidden state shape: \", h0_state.shape)\n",
    "print(\"v0 state shape:  \", v0_state.shape)\n",
    "print(\"v1 state shape:  \", v1_state.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>What is the objective function?</h3>\n",
    "\n",
    "<b>Goal</b>: Maximize the likelihood of our data being drawn from that distribution\n",
    "\n",
    "<b>Calculate error:</b>  \n",
    "In each epoch, we compute the \"error\" as a sum of the squared difference between step 1 and step n,\n",
    "e.g the error shows the difference between the data and its reconstruction.\n",
    "\n",
    "<b>Note:</b> tf.reduce_mean computes the mean of elements across dimensions of a tensor.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "error 0.49362245\n"
     ]
    }
   ],
   "source": [
    "def error(v0_state, v1_state):\n",
    "    return tf.reduce_mean(tf.square(v0_state - v1_state))\n",
    "\n",
    "err = tf.reduce_mean(tf.square(v0_state - v1_state))\n",
    "print(\"error\" , err.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ref4\"></a>\n",
    "<h3>Training the Model</h3>\n",
    "<b>Warning...</b> The following part is math-heavy, but you can skip it if you just want to run the cells in the next section.\n",
    "\n",
    "As mentioned, we want to give a high probability to the input data we train on. So, in order to train an RBM, we have to maximize the product of probabilities assigned to all rows $\\mathbf{v}$ (images) in the training set $\\mathbf{V}$ (a matrix, where each row of it is treated as a visible vector $\\mathbf{v}$)\n",
    "\n",
    "$$\\arg \\max_W \\prod_{\\mathbf{v}\\in\\mathbf{V}_T} p(\\mathbf{v})$$\n",
    "\n",
    "\n",
    "which is equivalent to maximizing the expectation of the log probability, given as\n",
    "\n",
    "$$\\arg\\max_W\\left[ \\mathbb{E} \\left(\\prod_{\\mathbf v\\in \\mathbf V}\\text{log} \\left(p(\\mathbf v)\\right) \\right) \\right].$$\n",
    "\n",
    "\n",
    "So, we have to update the weights $W_{ij}$  to increase $p(\\mathbf{v})$ for all $\\mathbf{v}$ in our training data during training. So we have to calculate the derivative:\n",
    "\n",
    "\n",
    "$$\\frac{\\partial \\log p(\\mathbf v)}{\\partial W_{ij}}$$\n",
    "\n",
    "This cannot be easily done by typical <b>gradient descent (SGD)</b>, so we can use another approach, which has 2 steps:\n",
    "<ol>\n",
    "    <li>Gibbs Sampling</li>\n",
    "    <li>Contrastive Divergence</li>\n",
    "</ol>    \n",
    "    \n",
    "<h3>Gibbs Sampling</h3>   \n",
    "\n",
    "<h4>Gibbs Sampling Step 1</h4> \n",
    "Given an input vector $\\mathbf{v}$, we are using $p(\\mathbf{h}|\\mathbf{v})$ to predict the hidden values $\\mathbf{h}$. \n",
    "  $$p({h_j}|\\mathbf v)= \\sigma\\left(\\sum_{i=1}^V W_{ij} v_i + h_{bias} \\right)$$\n",
    "The samples are generated from this distribution by generating the uniform random variate vector $\\mathbf{\\xi} \\sim U[0,1]$ of length $H$ and comparing to the computed probabilities as\n",
    "\n",
    "\n",
    "<center>If $\\xi_j>p(h_{j}|\\mathbf{v})$, then $h_j=1$, else $h_j=0$.</center>\n",
    "\n",
    "\n",
    "<h4>Gibbs Sampling Step 2</h4> \n",
    "Then, knowing the hidden values, we use $p(\\mathbf v| \\mathbf h)$ for reconstructing of new input values v. \n",
    "\n",
    "   $$p({v_i}|\\mathbf h)= \\sigma\\left(\\sum_{j=1}^H W^{T}_{ij} h_j + v_{bias} \\right)$$\n",
    "\n",
    "\n",
    "The samples are generated from this distribution by generating a uniform random variate vector $\\mathbf{\\xi} \\sim U[0,1]$ of length $V$ and comparing to the computed probabilities as\n",
    "\n",
    "<center>If $\\xi_i>p(v_{i}|\\mathbf{h})$, then $v_i=1$, else $v_i=0$.</center>\n",
    "\n",
    "Let vectors $\\mathbf v_k$ and $\\mathbf h_k$ be for the $k$th iteration.  In general, the $kth$ state is generrated as: \n",
    "\n",
    "\n",
    "    \n",
    "<b>Iteration</b> $k$: \n",
    "\n",
    "$$\\mathbf v_{k-1} \\Rightarrow p(\\mathbf h_{k-1}|\\mathbf v_{k-1})\\Rightarrow \\mathbf h_{k-1}\\Rightarrow p(\\mathbf v_{k}|\\mathbf h_{k-1})\\Rightarrow \\mathbf v_k$$       \n",
    "    \n",
    "<h3>Contrastive Divergence (CD-k)</h3>\n",
    "The update of the weight matrix is done during the Contrastive Divergence step. \n",
    "\n",
    "Vectors v0 and vk are used to calculate the activation probabilities for hidden values h0 and hk. The difference between the outer products of those probabilities with input vectors v0 and vk results in the update matrix:\n",
    "\n",
    "\n",
    "$$\\Delta \\mathbf W_k =\\mathbf v_k \\otimes \\mathbf h_k - \\mathbf v_{k-1} \\otimes \\mathbf h_{k-1}$$\n",
    "\n",
    "Contrastive Divergence is actually matrix of values that is computed and used to adjust values of the $\\mathbf W$ matrix. Changing $\\mathbf W$ incrementally leads to training of the  $\\mathbf W$ values. Then, on each step (epoch), $\\mathbf W$ is updated using the following:\n",
    "\n",
    "$$\\mathbf W_k = \\mathbf W_{k-1} + \\alpha * \\Delta \\mathbf W_k$$\n",
    "\n",
    "\n",
    "\n",
    "Reconstruction steps:\n",
    "<ul>\n",
    "    <li> Get one data point from data set, like <i>x</i>, and pass it through the following steps:</li>\n",
    "    \n",
    "<b>Iteration</b> $k=1$: \n",
    "    \n",
    "Sampling (starting with input image)\n",
    "    $$\\mathbf x = \\mathbf v_0 \\Rightarrow p(\\mathbf h_0|\\mathbf v_0)\\Rightarrow \\mathbf h_0 \\Rightarrow p(\\mathbf v_1|\\mathbf h_0)\\Rightarrow \\mathbf v_1$$   \n",
    "    followed by the CD-k step\n",
    "$$\\Delta \\mathbf W_1 =\\mathbf v_1 \\otimes \\mathbf h_1 - \\mathbf v_{0} \\otimes \\mathbf h_{0}$$     \n",
    "$$\\mathbf W_1 = \\mathbf W_{0} + \\alpha * \\Delta \\mathbf W_1$$ \n",
    " \n",
    "<li> $\\mathbf v_1$ is the reconstruction of $\\mathbf x$ sent to the next iteration).</li>\n",
    "\n",
    "<b>Iteration</b> $k=2$: \n",
    "\n",
    "Sampling (starting with $\\mathbf v_1$)\n",
    "\n",
    "$$\\mathbf v_1 \\Rightarrow p(\\mathbf h_1|\\mathbf v_1)\\Rightarrow \\mathbf h_1\\Rightarrow p(\\mathbf v_2|\\mathbf h_1)\\Rightarrow \\mathbf v_2$$   \n",
    "\n",
    "followed by the CD-k step\n",
    "$$\\Delta \\mathbf W_2 =\\mathbf v_2 \\otimes \\mathbf h_2 - \\mathbf v_{1} \\otimes \\mathbf h_{1}$$     \n",
    "$$\\mathbf W_2 = \\mathbf W_{1} + \\alpha * \\Delta \\mathbf W_2$$ \n",
    "\n",
    "<li> $\\mathbf v_2$ is the reconstruction of $\\mathbf v_1$ sent to the next iteration).</li>    \n",
    "      \n",
    "<b>Iteration</b> $k=K$:\n",
    "    \n",
    "Sampling (starting with $\\mathbf v_{K-1}$)\n",
    "\n",
    "$$\\mathbf v_{K-1} \\Rightarrow p(\\mathbf h_{K-1}|\\mathbf v_{K-1})\\Rightarrow \\mathbf h_{K-1}\\Rightarrow p(\\mathbf v_K|\\mathbf h_{K-1})\\Rightarrow \\mathbf v_K$$   \n",
    "\n",
    "followed by the CD-k step\n",
    "$$\\Delta \\mathbf W_K =\\mathbf v_K \\otimes \\mathbf h_K - \\mathbf v_{K-1} \\otimes \\mathbf h_{K-1}$$     \n",
    "$$\\mathbf W_K = \\mathbf W_{K-1} + \\alpha * \\Delta \\mathbf W_K$$ \n",
    "    \n",
    "<b>What is $\\alpha$?</b>  \n",
    "Here, alpha is some small step size, and is also known as the \"learning rate\".\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$K$ is adjustable, and good performance can be achieved with $K=1$, so that we just take one set of sampling steps per image.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "h1_prob = tf.nn.sigmoid(tf.matmul([v1_state], W) + hb)\n",
    "h1_state = tf.nn.relu(tf.sign(h1_prob - tf.random.uniform(tf.shape(h1_prob)))) #sample_h_given_X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets look at the error of the first run:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "error:  tf.Tensor(0.49362245, shape=(), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "print(\"error: \", error(v0_state, v1_state))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0 batch #: 1  of 300 sample #: 199 reconstruction error: 0.177314\n",
      "Epoch: 0 batch #: 2  of 300 sample #: 199 reconstruction error: 0.231510\n",
      "Epoch: 0 batch #: 3  of 300 sample #: 199 reconstruction error: 0.112321\n",
      "Epoch: 0 batch #: 4  of 300 sample #: 199 reconstruction error: 0.131889\n",
      "Epoch: 0 batch #: 5  of 300 sample #: 199 reconstruction error: 0.109967\n",
      "Epoch: 0 batch #: 6  of 300 sample #: 199 reconstruction error: 0.177350\n",
      "Epoch: 0 batch #: 7  of 300 sample #: 199 reconstruction error: 0.087693\n",
      "Epoch: 0 batch #: 8  of 300 sample #: 199 reconstruction error: 0.113766\n",
      "Epoch: 0 batch #: 9  of 300 sample #: 199 reconstruction error: 0.083736\n",
      "Epoch: 0 batch #: 10  of 300 sample #: 199 reconstruction error: 0.175486\n",
      "Epoch: 0 batch #: 11  of 300 sample #: 199 reconstruction error: 0.101725\n",
      "Epoch: 0 batch #: 12  of 300 sample #: 199 reconstruction error: 0.131739\n",
      "Epoch: 0 batch #: 13  of 300 sample #: 199 reconstruction error: 0.074012\n",
      "Epoch: 0 batch #: 14  of 300 sample #: 199 reconstruction error: 0.080929\n",
      "Epoch: 0 batch #: 15  of 300 sample #: 199 reconstruction error: 0.085945\n",
      "Epoch: 0 batch #: 16  of 300 sample #: 199 reconstruction error: 0.097949\n",
      "Epoch: 0 batch #: 17  of 300 sample #: 199 reconstruction error: 0.093595\n",
      "Epoch: 0 batch #: 18  of 300 sample #: 199 reconstruction error: 0.134595\n",
      "Epoch: 0 batch #: 19  of 300 sample #: 199 reconstruction error: 0.139515\n",
      "Epoch: 0 batch #: 20  of 300 sample #: 199 reconstruction error: 0.104037\n",
      "Epoch: 0 batch #: 21  of 300 sample #: 199 reconstruction error: 0.091814\n",
      "Epoch: 0 batch #: 22  of 300 sample #: 199 reconstruction error: 0.091194\n",
      "Epoch: 0 batch #: 23  of 300 sample #: 199 reconstruction error: 0.116202\n",
      "Epoch: 0 batch #: 24  of 300 sample #: 199 reconstruction error: 0.121738\n",
      "Epoch: 0 batch #: 25  of 300 sample #: 199 reconstruction error: 0.089726\n",
      "Epoch: 0 batch #: 26  of 300 sample #: 199 reconstruction error: 0.106730\n",
      "Epoch: 0 batch #: 27  of 300 sample #: 199 reconstruction error: 0.102391\n",
      "Epoch: 0 batch #: 28  of 300 sample #: 199 reconstruction error: 0.080999\n",
      "Epoch: 0 batch #: 29  of 300 sample #: 199 reconstruction error: 0.088311\n",
      "Epoch: 0 batch #: 30  of 300 sample #: 199 reconstruction error: 0.081650\n",
      "Epoch: 0 batch #: 31  of 300 sample #: 199 reconstruction error: 0.103248\n",
      "Epoch: 0 batch #: 32  of 300 sample #: 199 reconstruction error: 0.125031\n",
      "Epoch: 0 batch #: 33  of 300 sample #: 199 reconstruction error: 0.062674\n",
      "Epoch: 0 batch #: 34  of 300 sample #: 199 reconstruction error: 0.076446\n",
      "Epoch: 0 batch #: 35  of 300 sample #: 199 reconstruction error: 0.071864\n",
      "Epoch: 0 batch #: 36  of 300 sample #: 199 reconstruction error: 0.095838\n",
      "Epoch: 0 batch #: 37  of 300 sample #: 199 reconstruction error: 0.060332\n",
      "Epoch: 0 batch #: 38  of 300 sample #: 199 reconstruction error: 0.064931\n",
      "Epoch: 0 batch #: 39  of 300 sample #: 199 reconstruction error: 0.093057\n",
      "Epoch: 0 batch #: 40  of 300 sample #: 199 reconstruction error: 0.113585\n",
      "Epoch: 0 batch #: 41  of 300 sample #: 199 reconstruction error: 0.077743\n",
      "Epoch: 0 batch #: 42  of 300 sample #: 199 reconstruction error: 0.068441\n",
      "Epoch: 0 batch #: 43  of 300 sample #: 199 reconstruction error: 0.144776\n",
      "Epoch: 0 batch #: 44  of 300 sample #: 199 reconstruction error: 0.074091\n",
      "Epoch: 0 batch #: 45  of 300 sample #: 199 reconstruction error: 0.053589\n",
      "Epoch: 0 batch #: 46  of 300 sample #: 199 reconstruction error: 0.039699\n",
      "Epoch: 0 batch #: 47  of 300 sample #: 199 reconstruction error: 0.032322\n",
      "Epoch: 0 batch #: 48  of 300 sample #: 199 reconstruction error: 0.051351\n",
      "Epoch: 0 batch #: 49  of 300 sample #: 199 reconstruction error: 0.119241\n",
      "Epoch: 0 batch #: 50  of 300 sample #: 199 reconstruction error: 0.051208\n",
      "Epoch: 0 batch #: 51  of 300 sample #: 199 reconstruction error: 0.061710\n",
      "Epoch: 0 batch #: 52  of 300 sample #: 199 reconstruction error: 0.041043\n",
      "Epoch: 0 batch #: 53  of 300 sample #: 199 reconstruction error: 0.122544\n",
      "Epoch: 0 batch #: 54  of 300 sample #: 199 reconstruction error: 0.048903\n",
      "Epoch: 0 batch #: 55  of 300 sample #: 199 reconstruction error: 0.096929\n",
      "Epoch: 0 batch #: 56  of 300 sample #: 199 reconstruction error: 0.045506\n",
      "Epoch: 0 batch #: 57  of 300 sample #: 199 reconstruction error: 0.044373\n",
      "Epoch: 0 batch #: 58  of 300 sample #: 199 reconstruction error: 0.070165\n",
      "Epoch: 0 batch #: 59  of 300 sample #: 199 reconstruction error: 0.058947\n",
      "Epoch: 0 batch #: 60  of 300 sample #: 199 reconstruction error: 0.083875\n",
      "Epoch: 0 batch #: 61  of 300 sample #: 199 reconstruction error: 0.073959\n",
      "Epoch: 0 batch #: 62  of 300 sample #: 199 reconstruction error: 0.095379\n",
      "Epoch: 0 batch #: 63  of 300 sample #: 199 reconstruction error: 0.055466\n",
      "Epoch: 0 batch #: 64  of 300 sample #: 199 reconstruction error: 0.106266\n",
      "Epoch: 0 batch #: 65  of 300 sample #: 199 reconstruction error: 0.078972\n",
      "Epoch: 0 batch #: 66  of 300 sample #: 199 reconstruction error: 0.071285\n",
      "Epoch: 0 batch #: 67  of 300 sample #: 199 reconstruction error: 0.060334\n",
      "Epoch: 0 batch #: 68  of 300 sample #: 199 reconstruction error: 0.070841\n",
      "Epoch: 0 batch #: 69  of 300 sample #: 199 reconstruction error: 0.085108\n",
      "Epoch: 0 batch #: 70  of 300 sample #: 199 reconstruction error: 0.100075\n",
      "Epoch: 0 batch #: 71  of 300 sample #: 199 reconstruction error: 0.078991\n",
      "Epoch: 0 batch #: 72  of 300 sample #: 199 reconstruction error: 0.068338\n",
      "Epoch: 0 batch #: 73  of 300 sample #: 199 reconstruction error: 0.081281\n",
      "Epoch: 0 batch #: 74  of 300 sample #: 199 reconstruction error: 0.108585\n",
      "Epoch: 0 batch #: 75  of 300 sample #: 199 reconstruction error: 0.050744\n",
      "Epoch: 0 batch #: 76  of 300 sample #: 199 reconstruction error: 0.073799\n",
      "Epoch: 0 batch #: 77  of 300 sample #: 199 reconstruction error: 0.098373\n",
      "Epoch: 0 batch #: 78  of 300 sample #: 199 reconstruction error: 0.100443\n",
      "Epoch: 0 batch #: 79  of 300 sample #: 199 reconstruction error: 0.113749\n",
      "Epoch: 0 batch #: 80  of 300 sample #: 199 reconstruction error: 0.028650\n",
      "Epoch: 0 batch #: 81  of 300 sample #: 199 reconstruction error: 0.061914\n",
      "Epoch: 0 batch #: 82  of 300 sample #: 199 reconstruction error: 0.049574\n",
      "Epoch: 0 batch #: 83  of 300 sample #: 199 reconstruction error: 0.078116\n",
      "Epoch: 0 batch #: 84  of 300 sample #: 199 reconstruction error: 0.080619\n",
      "Epoch: 0 batch #: 85  of 300 sample #: 199 reconstruction error: 0.056883\n",
      "Epoch: 0 batch #: 86  of 300 sample #: 199 reconstruction error: 0.080752\n",
      "Epoch: 0 batch #: 87  of 300 sample #: 199 reconstruction error: 0.090086\n",
      "Epoch: 0 batch #: 88  of 300 sample #: 199 reconstruction error: 0.028702\n",
      "Epoch: 0 batch #: 89  of 300 sample #: 199 reconstruction error: 0.061324\n",
      "Epoch: 0 batch #: 90  of 300 sample #: 199 reconstruction error: 0.084093\n",
      "Epoch: 0 batch #: 91  of 300 sample #: 199 reconstruction error: 0.064057\n",
      "Epoch: 0 batch #: 92  of 300 sample #: 199 reconstruction error: 0.071146\n",
      "Epoch: 0 batch #: 93  of 300 sample #: 199 reconstruction error: 0.120374\n",
      "Epoch: 0 batch #: 94  of 300 sample #: 199 reconstruction error: 0.131782\n",
      "Epoch: 0 batch #: 95  of 300 sample #: 199 reconstruction error: 0.056317\n",
      "Epoch: 0 batch #: 96  of 300 sample #: 199 reconstruction error: 0.057001\n",
      "Epoch: 0 batch #: 97  of 300 sample #: 199 reconstruction error: 0.117685\n",
      "Epoch: 0 batch #: 98  of 300 sample #: 199 reconstruction error: 0.094651\n",
      "Epoch: 0 batch #: 99  of 300 sample #: 199 reconstruction error: 0.057956\n",
      "Epoch: 0 batch #: 100  of 300 sample #: 199 reconstruction error: 0.123954\n",
      "Epoch: 0 batch #: 101  of 300 sample #: 199 reconstruction error: 0.074317\n",
      "Epoch: 0 batch #: 102  of 300 sample #: 199 reconstruction error: 0.083677\n",
      "Epoch: 0 batch #: 103  of 300 sample #: 199 reconstruction error: 0.078705\n",
      "Epoch: 0 batch #: 104  of 300 sample #: 199 reconstruction error: 0.058442\n",
      "Epoch: 0 batch #: 105  of 300 sample #: 199 reconstruction error: 0.058985\n",
      "Epoch: 0 batch #: 106  of 300 sample #: 199 reconstruction error: 0.113435\n",
      "Epoch: 0 batch #: 107  of 300 sample #: 199 reconstruction error: 0.129980\n",
      "Epoch: 0 batch #: 108  of 300 sample #: 199 reconstruction error: 0.068111\n",
      "Epoch: 0 batch #: 109  of 300 sample #: 199 reconstruction error: 0.120424\n",
      "Epoch: 0 batch #: 110  of 300 sample #: 199 reconstruction error: 0.058686\n",
      "Epoch: 0 batch #: 111  of 300 sample #: 199 reconstruction error: 0.111196\n",
      "Epoch: 0 batch #: 112  of 300 sample #: 199 reconstruction error: 0.048419\n",
      "Epoch: 0 batch #: 113  of 300 sample #: 199 reconstruction error: 0.074259\n",
      "Epoch: 0 batch #: 114  of 300 sample #: 199 reconstruction error: 0.067641\n",
      "Epoch: 0 batch #: 115  of 300 sample #: 199 reconstruction error: 0.074131\n",
      "Epoch: 0 batch #: 116  of 300 sample #: 199 reconstruction error: 0.051736\n",
      "Epoch: 0 batch #: 117  of 300 sample #: 199 reconstruction error: 0.046774\n",
      "Epoch: 0 batch #: 118  of 300 sample #: 199 reconstruction error: 0.071285\n",
      "Epoch: 0 batch #: 119  of 300 sample #: 199 reconstruction error: 0.128470\n",
      "Epoch: 0 batch #: 120  of 300 sample #: 199 reconstruction error: 0.087446\n",
      "Epoch: 0 batch #: 121  of 300 sample #: 199 reconstruction error: 0.048710\n",
      "Epoch: 0 batch #: 122  of 300 sample #: 199 reconstruction error: 0.099707\n",
      "Epoch: 0 batch #: 123  of 300 sample #: 199 reconstruction error: 0.060872\n",
      "Epoch: 0 batch #: 124  of 300 sample #: 199 reconstruction error: 0.077511\n",
      "Epoch: 0 batch #: 125  of 300 sample #: 199 reconstruction error: 0.043986\n",
      "Epoch: 0 batch #: 126  of 300 sample #: 199 reconstruction error: 0.100183\n",
      "Epoch: 0 batch #: 127  of 300 sample #: 199 reconstruction error: 0.077205\n",
      "Epoch: 0 batch #: 128  of 300 sample #: 199 reconstruction error: 0.063185\n",
      "Epoch: 0 batch #: 129  of 300 sample #: 199 reconstruction error: 0.053036\n",
      "Epoch: 0 batch #: 130  of 300 sample #: 199 reconstruction error: 0.071597\n",
      "Epoch: 0 batch #: 131  of 300 sample #: 199 reconstruction error: 0.057895\n",
      "Epoch: 0 batch #: 132  of 300 sample #: 199 reconstruction error: 0.054774\n",
      "Epoch: 0 batch #: 133  of 300 sample #: 199 reconstruction error: 0.045311\n",
      "Epoch: 0 batch #: 134  of 300 sample #: 199 reconstruction error: 0.104135\n",
      "Epoch: 0 batch #: 135  of 300 sample #: 199 reconstruction error: 0.107854\n",
      "Epoch: 0 batch #: 136  of 300 sample #: 199 reconstruction error: 0.100599\n",
      "Epoch: 0 batch #: 137  of 300 sample #: 199 reconstruction error: 0.068751\n",
      "Epoch: 0 batch #: 138  of 300 sample #: 199 reconstruction error: 0.072977\n",
      "Epoch: 0 batch #: 139  of 300 sample #: 199 reconstruction error: 0.102232\n",
      "Epoch: 0 batch #: 140  of 300 sample #: 199 reconstruction error: 0.077086\n",
      "Epoch: 0 batch #: 141  of 300 sample #: 199 reconstruction error: 0.082890\n",
      "Epoch: 0 batch #: 142  of 300 sample #: 199 reconstruction error: 0.058350\n",
      "Epoch: 0 batch #: 143  of 300 sample #: 199 reconstruction error: 0.064174\n",
      "Epoch: 0 batch #: 144  of 300 sample #: 199 reconstruction error: 0.074281\n",
      "Epoch: 0 batch #: 145  of 300 sample #: 199 reconstruction error: 0.104256\n",
      "Epoch: 0 batch #: 146  of 300 sample #: 199 reconstruction error: 0.101839\n",
      "Epoch: 0 batch #: 147  of 300 sample #: 199 reconstruction error: 0.083155\n",
      "Epoch: 0 batch #: 148  of 300 sample #: 199 reconstruction error: 0.078741\n",
      "Epoch: 0 batch #: 149  of 300 sample #: 199 reconstruction error: 0.043782\n",
      "Epoch: 0 batch #: 150  of 300 sample #: 199 reconstruction error: 0.041809\n",
      "Epoch: 0 batch #: 151  of 300 sample #: 199 reconstruction error: 0.118756\n",
      "Epoch: 0 batch #: 152  of 300 sample #: 199 reconstruction error: 0.107914\n",
      "Epoch: 0 batch #: 153  of 300 sample #: 199 reconstruction error: 0.061613\n",
      "Epoch: 0 batch #: 154  of 300 sample #: 199 reconstruction error: 0.076973\n",
      "Epoch: 0 batch #: 155  of 300 sample #: 199 reconstruction error: 0.063578\n",
      "Epoch: 0 batch #: 156  of 300 sample #: 199 reconstruction error: 0.081654\n",
      "Epoch: 0 batch #: 157  of 300 sample #: 199 reconstruction error: 0.082500\n",
      "Epoch: 0 batch #: 158  of 300 sample #: 199 reconstruction error: 0.084200\n",
      "Epoch: 0 batch #: 159  of 300 sample #: 199 reconstruction error: 0.056445\n",
      "Epoch: 0 batch #: 160  of 300 sample #: 199 reconstruction error: 0.067952\n",
      "Epoch: 0 batch #: 161  of 300 sample #: 199 reconstruction error: 0.093489\n",
      "Epoch: 0 batch #: 162  of 300 sample #: 199 reconstruction error: 0.118305\n",
      "Epoch: 0 batch #: 163  of 300 sample #: 199 reconstruction error: 0.048640\n",
      "Epoch: 0 batch #: 164  of 300 sample #: 199 reconstruction error: 0.060288\n",
      "Epoch: 0 batch #: 165  of 300 sample #: 199 reconstruction error: 0.077118\n",
      "Epoch: 0 batch #: 166  of 300 sample #: 199 reconstruction error: 0.044564\n",
      "Epoch: 0 batch #: 167  of 300 sample #: 199 reconstruction error: 0.098689\n",
      "Epoch: 0 batch #: 168  of 300 sample #: 199 reconstruction error: 0.053538\n",
      "Epoch: 0 batch #: 169  of 300 sample #: 199 reconstruction error: 0.045474\n",
      "Epoch: 0 batch #: 170  of 300 sample #: 199 reconstruction error: 0.062200\n",
      "Epoch: 0 batch #: 171  of 300 sample #: 199 reconstruction error: 0.041465\n",
      "Epoch: 0 batch #: 172  of 300 sample #: 199 reconstruction error: 0.053608\n",
      "Epoch: 0 batch #: 173  of 300 sample #: 199 reconstruction error: 0.067777\n",
      "Epoch: 0 batch #: 174  of 300 sample #: 199 reconstruction error: 0.112980\n",
      "Epoch: 0 batch #: 175  of 300 sample #: 199 reconstruction error: 0.074799\n",
      "Epoch: 0 batch #: 176  of 300 sample #: 199 reconstruction error: 0.046221\n",
      "Epoch: 0 batch #: 177  of 300 sample #: 199 reconstruction error: 0.033336\n",
      "Epoch: 0 batch #: 178  of 300 sample #: 199 reconstruction error: 0.103379\n",
      "Epoch: 0 batch #: 179  of 300 sample #: 199 reconstruction error: 0.066866\n",
      "Epoch: 0 batch #: 180  of 300 sample #: 199 reconstruction error: 0.095571\n",
      "Epoch: 0 batch #: 181  of 300 sample #: 199 reconstruction error: 0.082042\n",
      "Epoch: 0 batch #: 182  of 300 sample #: 199 reconstruction error: 0.085580\n",
      "Epoch: 0 batch #: 183  of 300 sample #: 199 reconstruction error: 0.103877\n",
      "Epoch: 0 batch #: 184  of 300 sample #: 199 reconstruction error: 0.062923\n",
      "Epoch: 0 batch #: 185  of 300 sample #: 199 reconstruction error: 0.088384\n",
      "Epoch: 0 batch #: 186  of 300 sample #: 199 reconstruction error: 0.084221\n",
      "Epoch: 0 batch #: 187  of 300 sample #: 199 reconstruction error: 0.058584\n",
      "Epoch: 0 batch #: 188  of 300 sample #: 199 reconstruction error: 0.123320\n",
      "Epoch: 0 batch #: 189  of 300 sample #: 199 reconstruction error: 0.081963\n",
      "Epoch: 0 batch #: 190  of 300 sample #: 199 reconstruction error: 0.058391\n",
      "Epoch: 0 batch #: 191  of 300 sample #: 199 reconstruction error: 0.117468\n",
      "Epoch: 0 batch #: 192  of 300 sample #: 199 reconstruction error: 0.125951\n",
      "Epoch: 0 batch #: 193  of 300 sample #: 199 reconstruction error: 0.112122\n",
      "Epoch: 0 batch #: 194  of 300 sample #: 199 reconstruction error: 0.068890\n",
      "Epoch: 0 batch #: 195  of 300 sample #: 199 reconstruction error: 0.049270\n",
      "Epoch: 0 batch #: 196  of 300 sample #: 199 reconstruction error: 0.038385\n",
      "Epoch: 0 batch #: 197  of 300 sample #: 199 reconstruction error: 0.055280\n",
      "Epoch: 0 batch #: 198  of 300 sample #: 199 reconstruction error: 0.085527\n",
      "Epoch: 0 batch #: 199  of 300 sample #: 199 reconstruction error: 0.084256\n",
      "Epoch: 0 batch #: 200  of 300 sample #: 199 reconstruction error: 0.112240\n",
      "Epoch: 0 batch #: 201  of 300 sample #: 199 reconstruction error: 0.099811\n",
      "Epoch: 0 batch #: 202  of 300 sample #: 199 reconstruction error: 0.031745\n",
      "Epoch: 0 batch #: 203  of 300 sample #: 199 reconstruction error: 0.089001\n",
      "Epoch: 0 batch #: 204  of 300 sample #: 199 reconstruction error: 0.045287\n",
      "Epoch: 0 batch #: 205  of 300 sample #: 199 reconstruction error: 0.021085\n",
      "Epoch: 0 batch #: 206  of 300 sample #: 199 reconstruction error: 0.061266\n",
      "Epoch: 0 batch #: 207  of 300 sample #: 199 reconstruction error: 0.072119\n",
      "Epoch: 0 batch #: 208  of 300 sample #: 199 reconstruction error: 0.060007\n",
      "Epoch: 0 batch #: 209  of 300 sample #: 199 reconstruction error: 0.030261\n",
      "Epoch: 0 batch #: 210  of 300 sample #: 199 reconstruction error: 0.108951\n",
      "Epoch: 0 batch #: 211  of 300 sample #: 199 reconstruction error: 0.117259\n",
      "Epoch: 0 batch #: 212  of 300 sample #: 199 reconstruction error: 0.066920\n",
      "Epoch: 0 batch #: 213  of 300 sample #: 199 reconstruction error: 0.104439\n",
      "Epoch: 0 batch #: 214  of 300 sample #: 199 reconstruction error: 0.095216\n",
      "Epoch: 0 batch #: 215  of 300 sample #: 199 reconstruction error: 0.031748\n",
      "Epoch: 0 batch #: 216  of 300 sample #: 199 reconstruction error: 0.088442\n",
      "Epoch: 0 batch #: 217  of 300 sample #: 199 reconstruction error: 0.073018\n",
      "Epoch: 0 batch #: 218  of 300 sample #: 199 reconstruction error: 0.060348\n",
      "Epoch: 0 batch #: 219  of 300 sample #: 199 reconstruction error: 0.065654\n",
      "Epoch: 0 batch #: 220  of 300 sample #: 199 reconstruction error: 0.067269\n",
      "Epoch: 0 batch #: 221  of 300 sample #: 199 reconstruction error: 0.090299\n",
      "Epoch: 0 batch #: 222  of 300 sample #: 199 reconstruction error: 0.102278\n",
      "Epoch: 0 batch #: 223  of 300 sample #: 199 reconstruction error: 0.072993\n",
      "Epoch: 0 batch #: 224  of 300 sample #: 199 reconstruction error: 0.064196\n",
      "Epoch: 0 batch #: 225  of 300 sample #: 199 reconstruction error: 0.068627\n",
      "Epoch: 0 batch #: 226  of 300 sample #: 199 reconstruction error: 0.110055\n",
      "Epoch: 0 batch #: 227  of 300 sample #: 199 reconstruction error: 0.067022\n",
      "Epoch: 0 batch #: 228  of 300 sample #: 199 reconstruction error: 0.078375\n",
      "Epoch: 0 batch #: 229  of 300 sample #: 199 reconstruction error: 0.080492\n",
      "Epoch: 0 batch #: 230  of 300 sample #: 199 reconstruction error: 0.028016\n",
      "Epoch: 0 batch #: 231  of 300 sample #: 199 reconstruction error: 0.029626\n",
      "Epoch: 0 batch #: 232  of 300 sample #: 199 reconstruction error: 0.079153\n",
      "Epoch: 0 batch #: 233  of 300 sample #: 199 reconstruction error: 0.088687\n",
      "Epoch: 0 batch #: 234  of 300 sample #: 199 reconstruction error: 0.061536\n",
      "Epoch: 0 batch #: 235  of 300 sample #: 199 reconstruction error: 0.069775\n",
      "Epoch: 0 batch #: 236  of 300 sample #: 199 reconstruction error: 0.086046\n",
      "Epoch: 0 batch #: 237  of 300 sample #: 199 reconstruction error: 0.075653\n",
      "Epoch: 0 batch #: 238  of 300 sample #: 199 reconstruction error: 0.120498\n",
      "Epoch: 0 batch #: 239  of 300 sample #: 199 reconstruction error: 0.028300\n",
      "Epoch: 0 batch #: 240  of 300 sample #: 199 reconstruction error: 0.111587\n",
      "Epoch: 0 batch #: 241  of 300 sample #: 199 reconstruction error: 0.076198\n",
      "Epoch: 0 batch #: 242  of 300 sample #: 199 reconstruction error: 0.056960\n",
      "Epoch: 0 batch #: 243  of 300 sample #: 199 reconstruction error: 0.076723\n",
      "Epoch: 0 batch #: 244  of 300 sample #: 199 reconstruction error: 0.083579\n",
      "Epoch: 0 batch #: 245  of 300 sample #: 199 reconstruction error: 0.070141\n",
      "Epoch: 0 batch #: 246  of 300 sample #: 199 reconstruction error: 0.086069\n",
      "Epoch: 0 batch #: 247  of 300 sample #: 199 reconstruction error: 0.043119\n",
      "Epoch: 0 batch #: 248  of 300 sample #: 199 reconstruction error: 0.068667\n",
      "Epoch: 0 batch #: 249  of 300 sample #: 199 reconstruction error: 0.044040\n",
      "Epoch: 0 batch #: 250  of 300 sample #: 199 reconstruction error: 0.093749\n",
      "Epoch: 0 batch #: 251  of 300 sample #: 199 reconstruction error: 0.047651\n",
      "Epoch: 0 batch #: 252  of 300 sample #: 199 reconstruction error: 0.065470\n",
      "Epoch: 0 batch #: 253  of 300 sample #: 199 reconstruction error: 0.061928\n",
      "Epoch: 0 batch #: 254  of 300 sample #: 199 reconstruction error: 0.038046\n",
      "Epoch: 0 batch #: 255  of 300 sample #: 199 reconstruction error: 0.065337\n",
      "Epoch: 0 batch #: 256  of 300 sample #: 199 reconstruction error: 0.088324\n",
      "Epoch: 0 batch #: 257  of 300 sample #: 199 reconstruction error: 0.094725\n",
      "Epoch: 0 batch #: 258  of 300 sample #: 199 reconstruction error: 0.088699\n"
     ]
    }
   ],
   "source": [
    "#Parameters\n",
    "alpha = 0.01\n",
    "epochs = 1\n",
    "batchsize = 200\n",
    "weights = []\n",
    "errors = []\n",
    "batch_number = 0\n",
    "K = 1\n",
    "\n",
    "#creating datasets\n",
    "train_ds = \\\n",
    "    tf.data.Dataset.from_tensor_slices((trX, trY)).batch(batchsize)\n",
    "\n",
    "for epoch in range(epochs):\n",
    "    for batch_x, batch_y in train_ds:\n",
    "        batch_number += 1\n",
    "        for i_sample in range(batchsize):           \n",
    "            for k in range(K):\n",
    "                v0_state = batch_x[i_sample]\n",
    "                h0_state = hidden_layer(v0_state, W, hb)\n",
    "                v1_state = reconstructed_output(h0_state, W, vb)\n",
    "                h1_state = hidden_layer(v1_state, W, hb)\n",
    "\n",
    "                delta_W = tf.matmul(tf.transpose([v0_state]), h0_state) - tf.matmul(tf.transpose([v1_state]), h1_state)\n",
    "                W = W + alpha * delta_W\n",
    "\n",
    "                vb = vb + alpha * tf.reduce_mean(v0_state - v1_state, 0)\n",
    "                hb = hb + alpha * tf.reduce_mean(h0_state - h1_state, 0) \n",
    "\n",
    "                v0_state = v1_state\n",
    "\n",
    "            if i_sample == batchsize-1:\n",
    "                err = error(batch_x[i_sample], v1_state)\n",
    "                errors.append(err)\n",
    "                weights.append(W)\n",
    "                print ( 'Epoch: %d' % epoch, \n",
    "                       \"batch #: %i \" % batch_number, \"of %i\" % int(60e3/batchsize), \n",
    "                       \"sample #: %i\" % i_sample,\n",
    "                       'reconstruction error: %f' % err)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the errors at the end of each batch:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(errors)\n",
    "plt.xlabel(\"Batch Number\")\n",
    "plt.ylabel(\"Error\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is the final weight matrix $W$ after training?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(W.numpy()) # a weight matrix of shape (50,784)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ref5\"></a>\n",
    "<h3>Learned features</h3> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can take each hidden unit and visualize the connections between that hidden unit and each element in the input vector. In our case, we have 50 hidden units. Lets visualize those.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot the current weights:\n",
    "<b>tile_raster_images</b> helps in generating an easy to grasp image from a set of samples or weights. It transforms the <b>uw</b> (with one flattened image per row of size 784), into an array (of size $28\\times28$) in which images are reshaped and laid out like tiles on a floor.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tile_raster_images(X=W.numpy().T, img_shape=(28, 28), tile_shape=(5, 10), tile_spacing=(1, 1))\n",
    "import matplotlib.pyplot as plt\n",
    "from PIL import Image\n",
    "%matplotlib inline\n",
    "image = Image.fromarray(tile_raster_images(X=W.numpy().T, img_shape=(28, 28) ,tile_shape=(5, 10), tile_spacing=(1, 1)))\n",
    "### Plot image\n",
    "plt.rcParams['figure.figsize'] = (18.0, 18.0)\n",
    "imgplot = plt.imshow(image)\n",
    "imgplot.set_cmap('gray')  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each tile in the above visualization corresponds to a vector of connections between a hidden unit and visible layer's units. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at one of the learned weights corresponding to one of hidden units for example. In this particular square, the gray color represents weight = 0, and the whiter it is, the more positive the weights are (closer to 1). Conversely, the darker pixels are, the more negative the weights. The positive pixels will increase the probability of activation in hidden units (after multiplying by input/visible pixels), and negative pixels will decrease the probability of a unit hidden to be 1 (activated). So, why is this important?  So we can see that this specific square (hidden unit) can detect a feature (e.g. a \"/\" shape) and if it exists in the input.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from PIL import Image\n",
    "image = Image.fromarray(tile_raster_images(X =W.numpy().T[10:11], img_shape=(28, 28),tile_shape=(1, 1), tile_spacing=(1, 1)))\n",
    "### Plot image\n",
    "plt.rcParams['figure.figsize'] = (4.0, 4.0)\n",
    "imgplot = plt.imshow(image)\n",
    "imgplot.set_cmap('gray')  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the reconstruction of an image now. Imagine that we have a destructed image of figure 3. Lets see if our trained network can fix it:\n",
    "\n",
    "First we plot the image:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget -O destructed3.jpg  https://ibm.box.com/shared/static/vvm1b63uvuxq88vbw9znpwu5ol380mco.jpg\n",
    "img = Image.open('destructed3.jpg')\n",
    "img"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's pass this image through the neural net:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert the image to a 1d numpy array\n",
    "sample_case = np.array(img.convert('I').resize((28,28))).ravel().reshape((1, -1))/255.0\n",
    "\n",
    "sample_case = tf.cast(sample_case, dtype=tf.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feed the sample case into the network and reconstruct the output:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hh0_p = tf.nn.sigmoid(tf.matmul(sample_case, W) + hb)\n",
    "hh0_s = tf.round(hh0_p)\n",
    "\n",
    "print(\"Probability nodes in hidden layer:\" ,hh0_p)\n",
    "print(\"activated nodes in hidden layer:\" ,hh0_s)\n",
    "\n",
    "# reconstruct\n",
    "vv1_p = tf.nn.sigmoid(tf.matmul(hh0_s, tf.transpose(W)) + vb)\n",
    "\n",
    "print(vv1_p)\n",
    "#rec_prob = sess.run(vv1_p, feed_dict={ hh0_s: hh0_s_val, W: prv_w, vb: prv_vb})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we plot the reconstructed image:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "img = Image.fromarray(tile_raster_images(X=vv1_p.numpy(), img_shape=(28, 28),tile_shape=(1, 1), tile_spacing=(1, 1)))\n",
    "plt.rcParams['figure.figsize'] = (4.0, 4.0)\n",
    "imgplot = plt.imshow(img)\n",
    "imgplot.set_cmap('gray') "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "## Want to learn more?\n",
    "\n",
    "Also, you can use __Watson Studio__ to run these notebooks faster with bigger datasets.__Watson Studio__ is IBM’s leading cloud solution for data scientists, built by data scientists. With Jupyter notebooks, RStudio, Apache Spark and popular libraries pre-packaged in the cloud, __Watson Studio__ enables data scientists to collaborate on their projects without having to install anything. Join the fast-growing community of __Watson Studio__ users today with a free account at [Watson Studio](https://cocl.us/ML0120EN_DSX).This is the end of this lesson. Thank you for reading this notebook, and good luck on your studies.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Thanks for completing this lesson!\n",
    "\n",
    "Notebook created by: <a href = \"https://ca.linkedin.com/in/saeedaghabozorgi\">Saeed Aghabozorgi</a>\n",
    "\n",
    "Updated to TF 2.X by  <a href=\"https://ca.linkedin.com/in/nilmeier\"> Jerome Nilmeier</a><br />\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### References:\n",
    "https://en.wikipedia.org/wiki/Restricted_Boltzmann_machine  \n",
    "http://deeplearning.net/tutorial/rbm.html  \n",
    "http://www.cs.utoronto.ca/~hinton/absps/netflixICML.pdf<br>\n",
    "http://imonad.com/rbm/restricted-boltzmann-machine/  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "Copyright &copy; 2018 [Cognitive Class](https://cocl.us/DX0108EN_CC). This notebook and its source code are released under the terms of the [MIT License](https://bigdatauniversity.com/mit-license/).\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python",
   "language": "python",
   "name": "conda-env-python-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}