exercise_1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DATA20001 Deep Learning 2018 - Exercise 1\n",
    "\n",
    "**Due Sunday November 11, before 23:59**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 1.1. Simple MLP with hand-designed weights\n",
    "\n",
    "Hand-design a simple feedforward network (or MLP) that produces the desired $y$ given the inputs $x_1, x_2, x_3$ as given below:\n",
    "\n",
    "````\n",
    "x1 x2 x3 y\n",
    "0  0  0  0\n",
    "1  0  0  1\n",
    "0  1  0  1\n",
    "1  1  0  0\n",
    "0  0  1  1\n",
    "1  0  1  1\n",
    "0  1  1  1\n",
    "1  1  1  1\n",
    "````\n",
    "\n",
    "In this exercise we will not be training the MLP, but you will have to specify the weights yourself.  You can use any number of layers (although 2 should be enough) and any of the activation functions mentioned in the lectures.\n",
    "\n",
    "You should implement the neural network in Python code using numpy. You can perform each neural network layer as a matrix multiplication (you can use the `@` operator) followed by an activation function as shown in the lectures.\n",
    "\n",
    "Here are some definitions to get you started:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "# The inputs, each row is one input, i.e., x_1, x_2, and x_3\n",
    "X = np.array([[0, 0, 0],\n",
    "              [1, 0, 0],\n",
    "              [0, 1, 0],\n",
    "              [1, 1, 0],\n",
    "              [0, 0, 1],\n",
    "              [1, 0, 1],\n",
    "              [0, 1, 1],\n",
    "              [1, 1, 1]])\n",
    "    \n",
    "# The desired output as a column vector in 2-D array format (.T means transpose)\n",
    "y = np.array([[0,1,1,0,1,1,1,1]]).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Write your solution below as Python code, finally printing the end result. Please also add text fields (you can change a code field to text by selecting \"Markdown\" from the pull-down menu above), or at least some comments in the code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 1.2\n",
    "\n",
    "Suppose you are given some data pairs $(x,y)$, as shown in the picture.\n",
    "You guess that the data was generated by a cubic function:\n",
    "\n",
    "   $$y = g(x) = a * x**3 - b* x**2  - c * x + d$$\n",
    "\n",
    "(plus some random noise).  Use the mean squared error as the loss function,\n",
    "and use gradient descent to optimize the parameters $(a,b,c,d)$.\n",
    "\n",
    "What are your optimal parameters?\n",
    "What is the loss at its minimal value?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Data plot](images/exercise-1_2.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Data\n",
    "\n",
    "x = [ -3.00000000e+00,  -2.99000000e+00,  -2.98000000e+00,\n",
    "      -2.97000000e+00,  -2.96000000e+00,  -2.95000000e+00,\n",
    "      -2.94000000e+00,  -2.93000000e+00,  -2.92000000e+00,\n",
    "      -2.91000000e+00,  -2.90000000e+00,  -2.89000000e+00,\n",
    "      -2.88000000e+00,  -2.87000000e+00,  -2.86000000e+00,\n",
    "      -2.85000000e+00,  -2.84000000e+00,  -2.83000000e+00,\n",
    "      -2.82000000e+00,  -2.81000000e+00,  -2.80000000e+00,\n",
    "      -2.79000000e+00,  -2.78000000e+00,  -2.77000000e+00,\n",
    "      -2.76000000e+00,  -2.75000000e+00,  -2.74000000e+00,\n",
    "      -2.73000000e+00,  -2.72000000e+00,  -2.71000000e+00,\n",
    "      -2.70000000e+00,  -2.69000000e+00,  -2.68000000e+00,\n",
    "      -2.67000000e+00,  -2.66000000e+00,  -2.65000000e+00,\n",
    "      -2.64000000e+00,  -2.63000000e+00,  -2.62000000e+00,\n",
    "      -2.61000000e+00,  -2.60000000e+00,  -2.59000000e+00,\n",
    "      -2.58000000e+00,  -2.57000000e+00,  -2.56000000e+00,\n",
    "      -2.55000000e+00,  -2.54000000e+00,  -2.53000000e+00,\n",
    "      -2.52000000e+00,  -2.51000000e+00,  -2.50000000e+00,\n",
    "      -2.49000000e+00,  -2.48000000e+00,  -2.47000000e+00,\n",
    "      -2.46000000e+00,  -2.45000000e+00,  -2.44000000e+00,\n",
    "      -2.43000000e+00,  -2.42000000e+00,  -2.41000000e+00,\n",
    "      -2.40000000e+00,  -2.39000000e+00,  -2.38000000e+00,\n",
    "      -2.37000000e+00,  -2.36000000e+00,  -2.35000000e+00,\n",
    "      -2.34000000e+00,  -2.33000000e+00,  -2.32000000e+00,\n",
    "      -2.31000000e+00,  -2.30000000e+00,  -2.29000000e+00,\n",
    "      -2.28000000e+00,  -2.27000000e+00,  -2.26000000e+00,\n",
    "      -2.25000000e+00,  -2.24000000e+00,  -2.23000000e+00,\n",
    "      -2.22000000e+00,  -2.21000000e+00,  -2.20000000e+00,\n",
    "      -2.19000000e+00,  -2.18000000e+00,  -2.17000000e+00,\n",
    "      -2.16000000e+00,  -2.15000000e+00,  -2.14000000e+00,\n",
    "      -2.13000000e+00,  -2.12000000e+00,  -2.11000000e+00,\n",
    "      -2.10000000e+00,  -2.09000000e+00,  -2.08000000e+00,\n",
    "      -2.07000000e+00,  -2.06000000e+00,  -2.05000000e+00,\n",
    "      -2.04000000e+00,  -2.03000000e+00,  -2.02000000e+00,\n",
    "      -2.01000000e+00,  -2.00000000e+00,  -1.99000000e+00,\n",
    "      -1.98000000e+00,  -1.97000000e+00,  -1.96000000e+00,\n",
    "      -1.95000000e+00,  -1.94000000e+00,  -1.93000000e+00,\n",
    "      -1.92000000e+00,  -1.91000000e+00,  -1.90000000e+00,\n",
    "      -1.89000000e+00,  -1.88000000e+00,  -1.87000000e+00,\n",
    "      -1.86000000e+00,  -1.85000000e+00,  -1.84000000e+00,\n",
    "      -1.83000000e+00,  -1.82000000e+00,  -1.81000000e+00,\n",
    "      -1.80000000e+00,  -1.79000000e+00,  -1.78000000e+00,\n",
    "      -1.77000000e+00,  -1.76000000e+00,  -1.75000000e+00,\n",
    "      -1.74000000e+00,  -1.73000000e+00,  -1.72000000e+00,\n",
    "      -1.71000000e+00,  -1.70000000e+00,  -1.69000000e+00,\n",
    "      -1.68000000e+00,  -1.67000000e+00,  -1.66000000e+00,\n",
    "      -1.65000000e+00,  -1.64000000e+00,  -1.63000000e+00,\n",
    "      -1.62000000e+00,  -1.61000000e+00,  -1.60000000e+00,\n",
    "      -1.59000000e+00,  -1.58000000e+00,  -1.57000000e+00,\n",
    "      -1.56000000e+00,  -1.55000000e+00,  -1.54000000e+00,\n",
    "      -1.53000000e+00,  -1.52000000e+00,  -1.51000000e+00,\n",
    "      -1.50000000e+00,  -1.49000000e+00,  -1.48000000e+00,\n",
    "      -1.47000000e+00,  -1.46000000e+00,  -1.45000000e+00,\n",
    "      -1.44000000e+00,  -1.43000000e+00,  -1.42000000e+00,\n",
    "      -1.41000000e+00,  -1.40000000e+00,  -1.39000000e+00,\n",
    "      -1.38000000e+00,  -1.37000000e+00,  -1.36000000e+00,\n",
    "      -1.35000000e+00,  -1.34000000e+00,  -1.33000000e+00,\n",
    "      -1.32000000e+00,  -1.31000000e+00,  -1.30000000e+00,\n",
    "      -1.29000000e+00,  -1.28000000e+00,  -1.27000000e+00,\n",
    "      -1.26000000e+00,  -1.25000000e+00,  -1.24000000e+00,\n",
    "      -1.23000000e+00,  -1.22000000e+00,  -1.21000000e+00,\n",
    "      -1.20000000e+00,  -1.19000000e+00,  -1.18000000e+00,\n",
    "      -1.17000000e+00,  -1.16000000e+00,  -1.15000000e+00,\n",
    "      -1.14000000e+00,  -1.13000000e+00,  -1.12000000e+00,\n",
    "      -1.11000000e+00,  -1.10000000e+00,  -1.09000000e+00,\n",
    "      -1.08000000e+00,  -1.07000000e+00,  -1.06000000e+00,\n",
    "      -1.05000000e+00,  -1.04000000e+00,  -1.03000000e+00,\n",
    "      -1.02000000e+00,  -1.01000000e+00,  -1.00000000e+00,\n",
    "      -9.90000000e-01,  -9.80000000e-01,  -9.70000000e-01,\n",
    "      -9.60000000e-01,  -9.50000000e-01,  -9.40000000e-01,\n",
    "      -9.30000000e-01,  -9.20000000e-01,  -9.10000000e-01,\n",
    "      -9.00000000e-01,  -8.90000000e-01,  -8.80000000e-01,\n",
    "      -8.70000000e-01,  -8.60000000e-01,  -8.50000000e-01,\n",
    "      -8.40000000e-01,  -8.30000000e-01,  -8.20000000e-01,\n",
    "      -8.10000000e-01,  -8.00000000e-01,  -7.90000000e-01,\n",
    "      -7.80000000e-01,  -7.70000000e-01,  -7.60000000e-01,\n",
    "      -7.50000000e-01,  -7.40000000e-01,  -7.30000000e-01,\n",
    "      -7.20000000e-01,  -7.10000000e-01,  -7.00000000e-01,\n",
    "      -6.90000000e-01,  -6.80000000e-01,  -6.70000000e-01,\n",
    "      -6.60000000e-01,  -6.50000000e-01,  -6.40000000e-01,\n",
    "      -6.30000000e-01,  -6.20000000e-01,  -6.10000000e-01,\n",
    "      -6.00000000e-01,  -5.90000000e-01,  -5.80000000e-01,\n",
    "      -5.70000000e-01,  -5.60000000e-01,  -5.50000000e-01,\n",
    "      -5.40000000e-01,  -5.30000000e-01,  -5.20000000e-01,\n",
    "      -5.10000000e-01,  -5.00000000e-01,  -4.90000000e-01,\n",
    "      -4.80000000e-01,  -4.70000000e-01,  -4.60000000e-01,\n",
    "      -4.50000000e-01,  -4.40000000e-01,  -4.30000000e-01,\n",
    "      -4.20000000e-01,  -4.10000000e-01,  -4.00000000e-01,\n",
    "      -3.90000000e-01,  -3.80000000e-01,  -3.70000000e-01,\n",
    "      -3.60000000e-01,  -3.50000000e-01,  -3.40000000e-01,\n",
    "      -3.30000000e-01,  -3.20000000e-01,  -3.10000000e-01,\n",
    "      -3.00000000e-01,  -2.90000000e-01,  -2.80000000e-01,\n",
    "      -2.70000000e-01,  -2.60000000e-01,  -2.50000000e-01,\n",
    "      -2.40000000e-01,  -2.30000000e-01,  -2.20000000e-01,\n",
    "      -2.10000000e-01,  -2.00000000e-01,  -1.90000000e-01,\n",
    "      -1.80000000e-01,  -1.70000000e-01,  -1.60000000e-01,\n",
    "      -1.50000000e-01,  -1.40000000e-01,  -1.30000000e-01,\n",
    "      -1.20000000e-01,  -1.10000000e-01,  -1.00000000e-01,\n",
    "      -9.00000000e-02,  -8.00000000e-02,  -7.00000000e-02,\n",
    "      -6.00000000e-02,  -5.00000000e-02,  -4.00000000e-02,\n",
    "      -3.00000000e-02,  -2.00000000e-02,  -1.00000000e-02,\n",
    "      -6.39488462e-14,   1.00000000e-02,   2.00000000e-02,\n",
    "       3.00000000e-02,   4.00000000e-02,   5.00000000e-02,\n",
    "       6.00000000e-02,   7.00000000e-02,   8.00000000e-02,\n",
    "       9.00000000e-02,   1.00000000e-01,   1.10000000e-01,\n",
    "       1.20000000e-01,   1.30000000e-01,   1.40000000e-01,\n",
    "       1.50000000e-01,   1.60000000e-01,   1.70000000e-01,\n",
    "       1.80000000e-01,   1.90000000e-01,   2.00000000e-01,\n",
    "       2.10000000e-01,   2.20000000e-01,   2.30000000e-01,\n",
    "       2.40000000e-01,   2.50000000e-01,   2.60000000e-01,\n",
    "       2.70000000e-01,   2.80000000e-01,   2.90000000e-01,\n",
    "       3.00000000e-01,   3.10000000e-01,   3.20000000e-01,\n",
    "       3.30000000e-01,   3.40000000e-01,   3.50000000e-01,\n",
    "       3.60000000e-01,   3.70000000e-01,   3.80000000e-01,\n",
    "       3.90000000e-01,   4.00000000e-01,   4.10000000e-01,\n",
    "       4.20000000e-01,   4.30000000e-01,   4.40000000e-01,\n",
    "       4.50000000e-01,   4.60000000e-01,   4.70000000e-01,\n",
    "       4.80000000e-01,   4.90000000e-01,   5.00000000e-01,\n",
    "       5.10000000e-01,   5.20000000e-01,   5.30000000e-01,\n",
    "       5.40000000e-01,   5.50000000e-01,   5.60000000e-01,\n",
    "       5.70000000e-01,   5.80000000e-01,   5.90000000e-01,\n",
    "       6.00000000e-01,   6.10000000e-01,   6.20000000e-01,\n",
    "       6.30000000e-01,   6.40000000e-01,   6.50000000e-01,\n",
    "       6.60000000e-01,   6.70000000e-01,   6.80000000e-01,\n",
    "       6.90000000e-01,   7.00000000e-01,   7.10000000e-01,\n",
    "       7.20000000e-01,   7.30000000e-01,   7.40000000e-01,\n",
    "       7.50000000e-01,   7.60000000e-01,   7.70000000e-01,\n",
    "       7.80000000e-01,   7.90000000e-01,   8.00000000e-01,\n",
    "       8.10000000e-01,   8.20000000e-01,   8.30000000e-01,\n",
    "       8.40000000e-01,   8.50000000e-01,   8.60000000e-01,\n",
    "       8.70000000e-01,   8.80000000e-01,   8.90000000e-01,\n",
    "       9.00000000e-01,   9.10000000e-01,   9.20000000e-01,\n",
    "       9.30000000e-01,   9.40000000e-01,   9.50000000e-01,\n",
    "       9.60000000e-01,   9.70000000e-01,   9.80000000e-01,\n",
    "       9.90000000e-01,   1.00000000e+00,   1.01000000e+00,\n",
    "       1.02000000e+00,   1.03000000e+00,   1.04000000e+00,\n",
    "       1.05000000e+00,   1.06000000e+00,   1.07000000e+00,\n",
    "       1.08000000e+00,   1.09000000e+00,   1.10000000e+00,\n",
    "       1.11000000e+00,   1.12000000e+00,   1.13000000e+00,\n",
    "       1.14000000e+00,   1.15000000e+00,   1.16000000e+00,\n",
    "       1.17000000e+00,   1.18000000e+00,   1.19000000e+00,\n",
    "       1.20000000e+00,   1.21000000e+00,   1.22000000e+00,\n",
    "       1.23000000e+00,   1.24000000e+00,   1.25000000e+00,\n",
    "       1.26000000e+00,   1.27000000e+00,   1.28000000e+00,\n",
    "       1.29000000e+00,   1.30000000e+00,   1.31000000e+00,\n",
    "       1.32000000e+00,   1.33000000e+00,   1.34000000e+00,\n",
    "       1.35000000e+00,   1.36000000e+00,   1.37000000e+00,\n",
    "       1.38000000e+00,   1.39000000e+00,   1.40000000e+00,\n",
    "       1.41000000e+00,   1.42000000e+00,   1.43000000e+00,\n",
    "       1.44000000e+00,   1.45000000e+00,   1.46000000e+00,\n",
    "       1.47000000e+00,   1.48000000e+00,   1.49000000e+00,\n",
    "       1.50000000e+00,   1.51000000e+00,   1.52000000e+00,\n",
    "       1.53000000e+00,   1.54000000e+00,   1.55000000e+00,\n",
    "       1.56000000e+00,   1.57000000e+00,   1.58000000e+00,\n",
    "       1.59000000e+00,   1.60000000e+00,   1.61000000e+00,\n",
    "       1.62000000e+00,   1.63000000e+00,   1.64000000e+00,\n",
    "       1.65000000e+00,   1.66000000e+00,   1.67000000e+00,\n",
    "       1.68000000e+00,   1.69000000e+00,   1.70000000e+00,\n",
    "       1.71000000e+00,   1.72000000e+00,   1.73000000e+00,\n",
    "       1.74000000e+00,   1.75000000e+00,   1.76000000e+00,\n",
    "       1.77000000e+00,   1.78000000e+00,   1.79000000e+00,\n",
    "       1.80000000e+00,   1.81000000e+00,   1.82000000e+00,\n",
    "       1.83000000e+00,   1.84000000e+00,   1.85000000e+00,\n",
    "       1.86000000e+00,   1.87000000e+00,   1.88000000e+00,\n",
    "       1.89000000e+00,   1.90000000e+00,   1.91000000e+00,\n",
    "       1.92000000e+00,   1.93000000e+00,   1.94000000e+00,\n",
    "       1.95000000e+00,   1.96000000e+00,   1.97000000e+00,\n",
    "       1.98000000e+00,   1.99000000e+00,   2.00000000e+00,\n",
    "       2.01000000e+00,   2.02000000e+00,   2.03000000e+00,\n",
    "       2.04000000e+00,   2.05000000e+00,   2.06000000e+00,\n",
    "       2.07000000e+00,   2.08000000e+00,   2.09000000e+00,\n",
    "       2.10000000e+00,   2.11000000e+00,   2.12000000e+00,\n",
    "       2.13000000e+00,   2.14000000e+00,   2.15000000e+00,\n",
    "       2.16000000e+00,   2.17000000e+00,   2.18000000e+00,\n",
    "       2.19000000e+00,   2.20000000e+00,   2.21000000e+00,\n",
    "       2.22000000e+00,   2.23000000e+00,   2.24000000e+00,\n",
    "       2.25000000e+00,   2.26000000e+00,   2.27000000e+00,\n",
    "       2.28000000e+00,   2.29000000e+00,   2.30000000e+00,\n",
    "       2.31000000e+00,   2.32000000e+00,   2.33000000e+00,\n",
    "       2.34000000e+00,   2.35000000e+00,   2.36000000e+00,\n",
    "       2.37000000e+00,   2.38000000e+00,   2.39000000e+00,\n",
    "       2.40000000e+00,   2.41000000e+00,   2.42000000e+00,\n",
    "       2.43000000e+00,   2.44000000e+00,   2.45000000e+00,\n",
    "       2.46000000e+00,   2.47000000e+00,   2.48000000e+00,\n",
    "       2.49000000e+00,   2.50000000e+00,   2.51000000e+00,\n",
    "       2.52000000e+00,   2.53000000e+00,   2.54000000e+00,\n",
    "       2.55000000e+00,   2.56000000e+00,   2.57000000e+00,\n",
    "       2.58000000e+00,   2.59000000e+00,   2.60000000e+00,\n",
    "       2.61000000e+00,   2.62000000e+00,   2.63000000e+00,\n",
    "       2.64000000e+00,   2.65000000e+00,   2.66000000e+00,\n",
    "       2.67000000e+00,   2.68000000e+00,   2.69000000e+00,\n",
    "       2.70000000e+00,   2.71000000e+00,   2.72000000e+00,\n",
    "       2.73000000e+00,   2.74000000e+00,   2.75000000e+00,\n",
    "       2.76000000e+00,   2.77000000e+00,   2.78000000e+00,\n",
    "       2.79000000e+00,   2.80000000e+00,   2.81000000e+00,\n",
    "       2.82000000e+00,   2.83000000e+00,   2.84000000e+00,\n",
    "       2.85000000e+00,   2.86000000e+00,   2.87000000e+00,\n",
    "       2.88000000e+00,   2.89000000e+00,   2.90000000e+00,\n",
    "       2.91000000e+00,   2.92000000e+00,   2.93000000e+00,\n",
    "       2.94000000e+00,   2.95000000e+00,   2.96000000e+00,\n",
    "       2.97000000e+00,   2.98000000e+00,   2.99000000e+00]\n",
    "\n",
    "\n",
    "y = [-14.398283457389724, -15.235219477068169, -12.237534384422302, -12.588063675984927,\n",
    "-13.91934688119415, -13.349768954391656, -13.985789122001483, -11.738827178125542,\n",
    "-12.177225258090253, -11.845497832447453, -11.453845710873868, -12.758442790463523,\n",
    "-12.084911260512314, -9.2433084037824962, -12.175769361117002, -11.115928446393855,\n",
    "-12.002656815758435, -12.427384653325415, -10.545849556421615, -11.670092880943347,\n",
    "-9.7712605932460708, -8.220265771802989, -11.233187534360976, -11.498256503283221,\n",
    "-7.6402883730526643, -8.4549227614416704, -8.3453466192709627, -7.7879180827113919,\n",
    "-9.0002596862485653, -7.433618860005577, -6.7596208916379394, -6.5094620257447504,\n",
    "-7.6807464658898583, -6.8968337158223196, -6.7778428034609766, -6.7591327177512834,\n",
    "-5.6612645811682061, -5.1483894734231788, -6.6970946525514066, -5.942576928687636,\n",
    "-5.7144999881036638, -7.3110292177102147, -4.7863103913889198, -3.946846953716908,\n",
    "-5.5721788331925346, -4.7011390046766204, -3.6835426783737502, -5.0298853532378329,\n",
    "-2.8264195288483425, -4.2434799362406945, -4.1363005588874753, -4.729524642078486,\n",
    "-2.2203844992459185, -4.0461961916693543, -1.0383131202654718, -3.1122020034385787,\n",
    "-4.4858229842393325, -2.2783659762966431, -2.3887578048143863, -4.3980378954166364,\n",
    "-2.3746517682084001, -2.1726763686835131, -3.4678592224797624, -2.2059316475176609,\n",
    "-1.3467149438337294, -0.76580801058263537, 0.23373546202521145, -2.3888740040985001,\n",
    "-2.7044658871824874, 0.65815794646108228, -1.1657858238714109, -1.7015342909764504,\n",
    "-0.99495603919293019, -0.45420258791387297, 1.3961635653834958, 0.23795085370755531,\n",
    "-0.42816865757295319, 1.3415943025323143, -0.77823871495212638, 0.42836173086541074,\n",
    "0.50831567600375638, 1.7980013325723783, 0.24419892442210434, 0.89968632354471156,\n",
    "0.47692344226125094, 1.9461631027243185, 1.3699029376431919, 0.9332423534203399,\n",
    "2.3842216100036806, 2.3678813544511681, 1.7646986813788197, 1.4340377049048931,\n",
    "3.0528617561832769, 2.39227462001059, 2.348907336294888, 2.1317336613228788,\n",
    "2.5030180763932335, 1.6303590268789481, 3.7563943076005293, 3.124679751826362,\n",
    "3.356636899671726, 2.0292236414039406, 1.8661138414507148, 3.7363951707119725,\n",
    "3.3751522596060282, 3.1854576273122324, 2.1054234679591888, 2.7021110399199344,\n",
    "4.3627520812363736, 4.5383501629365774, 4.0012232791895768, 5.3629333511793096,\n",
    "4.8676433541851951, 3.4393242775397055, 3.4283619595069532, 5.2327031608089261,\n",
    "4.8364383564553162, 4.3069704609639592, 2.6767835091170431, 6.5760925171012099,\n",
    "3.4195039424140763, 4.8928521455019256, 5.9664257239381913, 5.7496033775823454,\n",
    "4.5573922551500843, 5.9171601200760735, 3.8147327064714611, 5.2716910574066027,\n",
    "5.8836632449048647, 6.3168433133782118, 5.5185711300753306, 3.8203441162754599,\n",
    "6.1852869643732191, 4.9304448248736401, 4.610837672563763, 5.1506548714533142,\n",
    "4.6721842722559419, 3.7549183064652776, 5.27081307294028, 4.6782078473347433,\n",
    "5.9490185437815306, 4.8888033781065223, 5.8883238059140162, 7.3069810063502674,\n",
    "5.7884380405838716, 7.114319374709682, 7.3865960651150404, 6.0569671786497787,\n",
    "8.1477023277032092, 5.9766752149628175, 6.7885558245451767, 6.8391743004482519,\n",
    "6.6977431016675943, 7.6402862610094253, 5.7957872012194587, 5.1171771644322037,\n",
    "6.389790740228265, 6.5706785219355277, 7.8182898203202145, 5.5756125055105716,\n",
    "6.376951756560084, 6.140041247212003, 6.6004733079068574, 7.2770255011658422,\n",
    "7.5704330802326458, 5.1006363516134208, 9.0866319704334799, 8.3936258554752623,\n",
    "8.0109117988581175, 5.8739526753259721, 7.025020328767261, 6.210412219350065,\n",
    "6.1400536799522039, 7.1351247026716438, 7.9279768063527376, 8.7886464053164843,\n",
    "6.2626137539551729, 7.3643141104275784, 7.2395049287791773, 6.6125223581970536,\n",
    "7.826381187266243, 7.9386074693981463, 5.6058017045993234, 7.7153055620525972,\n",
    "7.3151618330605954, 6.1940809685783309, 7.1067201314984754, 6.9088816846210319,\n",
    "6.6023174035237666, 7.4037119314694779, 5.0143225968989604, 6.8389183597763843,\n",
    "7.7724661778305419, 7.7885568846991582, 5.8418558739705295, 8.2563829950983045,\n",
    "6.1119858830950857, 7.1236975504361606, 7.1145620873004436, 8.6034636826373756,\n",
    "7.8440534122282974, 7.7637272210237569, 8.6583645704066221, 7.7503352932572218,\n",
    "7.0831398551933091, 8.0352010240910516, 7.4019133074720784, 8.4459046592848424,\n",
    "5.5519778457317068, 6.6148552795933737, 8.3168012437526322, 8.3475888331382482,\n",
    "7.4678695431127311, 7.6579079325710504, 6.5001116942702435, 8.1583348236123001,\n",
    "7.6249563154732547, 8.623190969041536, 7.5931762952528548, 8.3951911258097489,\n",
    "6.9537929790659616, 6.8665435677631645, 7.5679205429683503, 6.0919816997905034,\n",
    "9.156104634365084, 8.9910500466380601, 6.6363684098968889, 6.1532002518722333,\n",
    "6.6092069031073208, 6.7962113842658125, 5.8247869270365067, 7.0234061649915072,\n",
    "6.8245817466245136, 7.312041445241225, 8.1338130773460673, 6.7450748875686264,\n",
    "7.54714732645252, 7.2945347082648935, 5.9209060104783937, 7.8599541761161387,\n",
    "8.5198276520238352, 6.9606718543318467, 6.8127709483280929, 6.0592722091894844,\n",
    "7.7406630712986129, 5.6637747464057977, 6.8675559465976077, 6.6836363037336479,\n",
    "5.1210582875011221, 6.3238659359860803, 6.3177622466212942, 5.7411369826270828,\n",
    "6.5944401269040664, 5.6124350592005872, 6.7374622903420018, 5.9933229008348157,\n",
    "6.8595066928502009, 5.3829115482146248, 6.7519136231158763, 6.7327025138070073,\n",
    "6.2030833501665281, 5.0708083305100047, 6.024249796953022, 6.4367321474195052,\n",
    "7.7255760411915082, 6.2998145321591643, 6.1087807309332733, 6.6891072837068357,\n",
    "5.958712110623833, 7.5436702756008014, 5.5281269780231517, 5.6525778047538644,\n",
    "6.1217650432161363, 5.400194687242875, 6.0593243041128932, 4.7102012455880189,\n",
    "8.0693900957125244, 4.509906172902018, 5.6405460830273784, 4.3878127941796023,\n",
    "5.8458891816644334, 6.5465759828582435, 5.1586555896930948, 4.1411437958555242,\n",
    "3.8703880859456001, 4.5045728937994776, 3.9626853220361271, 4.7926549992771896,\n",
    "6.3178500618522166, 5.7890489785619721, 7.2132412120019183, 6.0978005941873361,\n",
    "6.7833469397023523, 5.4956813933419246, 4.2215292670050095, 6.1794633595262924,\n",
    "4.6952162852381543, 4.8896519252482822, 7.6813071163740769, 4.1268245277893749,\n",
    "7.19709757990457, 5.0251212923579658, 2.7647389007400398, 5.6162523674593672,\n",
    "5.8111799985144961, 3.7320702970253112, 8.1844889192163865, 6.5198427394791718,\n",
    "4.600203205729577, 5.5145322374601262, 4.2433709206689372, 4.0905690026353616,\n",
    "4.6326498814020667, 4.1334917342391959, 3.6388153722165346, 1.7962979141314288,\n",
    "4.5709925736324939, 3.7905591373721865, 1.8198732332285874, 4.2280593935694659,\n",
    "5.3800726542336736, 3.6359160113026112, 3.2065025789583093, 3.4552023740710993,\n",
    "5.1551168625842827, 4.7808584716881901, 4.2949046363099139, 3.8941390704753736,\n",
    "3.6669620278465049, 3.9730253074263149, 3.2083523677393191, 4.3676786176105393,\n",
    "3.3584680583022375, 4.9380552733349941, 1.6106963947537687, 4.5905137511837895,\n",
    "3.3758925014645675, 3.8868661103968374, 3.7334995801569644, 3.6869763280546448,\n",
    "5.5736798637775955, 2.0083607601902616, 3.3890020918966561, 5.0262111630923165,\n",
    "1.7872250902341138, 2.0322294012083142, 2.6469143799546013, 4.3827982506700458,\n",
    "2.4987894969616096, 2.9761100254992385, 2.6786653124976265, 2.7270544752054526,\n",
    "3.5541209171544064, 2.3731229418474142, 2.046837388497305, 1.2967031498899582,\n",
    "1.6773184572242958, 1.3427963965083503, 2.4332804779938471, 1.8620460310653626,\n",
    "2.7436930124763501, 4.2744577802775607, 1.5760228174091633, 3.0321160030109948,\n",
    "1.1116238470093771, 3.1086733467104879, 3.8352127473002788, 0.97142207505616951,\n",
    "2.5671040952887152, 2.4818851159351096, 0.38332579715675585, 2.2098526060047616,\n",
    "2.6383525492633151, 1.8903119154678034, 4.8507681501581121, 2.9001322981855244,\n",
    "2.3876735588123474, 2.1422647812147058, 1.2690485695536338, 0.84886253392496114,\n",
    "2.7975256572300653, 3.4917480961979552, 1.4848503912991979, 2.8009622434478261,\n",
    "2.0866471031379219, 3.5706020586532841, 1.4858078010875539, 1.2832442369112029,\n",
    "2.4892692176719526, 3.1345283141880147, 2.6311135665296645, 2.5558623494987573,\n",
    "2.3334803027936868, 2.805838351162266, -1.4133789068451597, 2.5948651612552132,\n",
    "1.5637238777850773, 2.1982870733899387, -0.32210682383320077, 1.3452611749466339,\n",
    "1.8634414504232923, 4.0333090585899569, 0.99824078356886736, 1.4299509808573687,\n",
    "2.9223730078503052, 1.7308649346356055, 0.32772314753286791, 1.3285404840759742,\n",
    "0.98577364891253327, 1.095520188705333, 2.2988175846090337, 1.3173503098007955,\n",
    "1.3544925678491879, 1.5639412192651103, 1.9632426138433754, 1.7883617378012455,\n",
    "0.95308762780400325, -0.60326409260389302, 0.336433999228689, 0.94369447930759942,\n",
    "0.014104413870604349, -0.29463245777542468, 2.1028371717375522, 1.2814296499892699,\n",
    "0.50667758940679808, 0.62306023751713147, 1.984136546337449, 2.9464120351017291,\n",
    "1.1946801940728118, 1.0395125316458136, 1.9999568541640529, 0.12842149835565198,\n",
    "0.77133994537631212, 1.5100928579862496, 0.63413472791095127, 1.8173221550320247,\n",
    "3.2741993127294853, 0.97667970113574643, 0.84368224083277288, 2.7276990101803351,\n",
    "-0.32120806238914956, 2.0989170154182695, 0.72142937114550287, 1.2263471487100803,\n",
    "-0.18356517248853166, 1.8713110008388325, 1.804811658703068, 0.96829168449998337,\n",
    "1.943980944874578, -1.7209136808409995, 3.1607005937973063, 0.73954607333527977,\n",
    "0.071281670533145158, -0.54978620355723851, -0.18651830141276027, 1.013858697178819,\n",
    "-0.11424752776217861, 1.1897488714876008, 1.91708693917365, 0.22953533341864296,\n",
    "1.9119626063042623, 2.1295333717157456, 2.9246496384293401, 2.4741035359616705,\n",
    "1.2526910705834837, 0.50205008334995971, 1.7783863779981415, 3.1371032779279275,\n",
    "1.3233100888129952, 0.53118869728110063, 1.6380054971371436, 1.7862547637534816,\n",
    "1.1832841992747867, 2.0333723737246014, 0.51873152197891437, 2.4669623065925648,\n",
    "1.2829624019395311, 1.7579582413727139, 2.0425088748361939, 1.1039937304466627,\n",
    "3.3485732126497405, 1.3589762534052645, 2.1447239717526161, 0.63722393290390955,\n",
    "1.9764771210457484, 1.8096892839631153, 2.2364123528049791, 1.5472651257422969,\n",
    "2.6376319033031477, 4.0606504753167023, 2.2246494111364523, 2.0681169642978072,\n",
    "1.8415586421434029, 1.6377403315262447, 3.268085910772375, 3.4300172733807051,\n",
    "3.1293825503919148, 3.1727218522579692, 3.3235280901995852, 4.7353070260629968,\n",
    "3.6721657542826356, 2.9622025299310213, 3.6046901942167664, 4.2855649480824471,\n",
    "4.4791805621992182, 3.7492703461407992, 3.7359416944399899, 3.2183643974774836,\n",
    "3.8610961208978987, 2.7673928644236874, 4.95594051920094, 2.3420853012007843,\n",
    "3.0622407639709794, 3.6889517487388357, 4.8561370970461395, 4.634589971089091,\n",
    "4.6518648015113744, 3.0787422491218219, 2.7848654802589192, 5.4055869250686168,\n",
    "2.9164279126408634, 4.6523830133025283, 4.9057271862734062, 4.8766713319018455,\n",
    "4.7835891910656585, 3.4245225800987051, 5.6972618371707169, 3.707815954275941,\n",
    "3.8413667029206318, 7.6951903563941979, 5.2180911560642205, 7.0284531944080602,\n",
    "5.4179281426162227, 5.4826301297725877, 7.8304515561567163, 5.0411630209363709,\n",
    "5.6971386597811273, 5.6304497813733256, 5.7521992508121098, 8.1459730019370689,\n",
    "5.621888964479977, 6.7967191249624994, 6.7341055092212683, 8.2072563753653576,\n",
    "7.1354953335890032, 7.6344303978151444, 8.4535975729144095, 9.0646350763332393,\n",
    "6.9612232249838195, 6.96482356859548, 7.9918542392039251, 8.4627730898931635,\n",
    "8.289786741310726, 10.637405810936741, 10.270005049826434, 8.0714760503431844,\n",
    "7.3902598568433442, 8.4227712410698494, 6.7537106707406496, 6.5314079582303677,\n",
    "9.0295094033250258, 8.7315661529747235, 9.9609213131099459, 6.5797406987036986,\n",
    "9.5843735284180571, 10.208628249675936, 7.4585072619076493, 8.9199385850499251,\n",
    "11.921919600104639, 10.357729526392594, 9.8172618694505278, 10.022731076097214,\n",
    "9.316397212266347, 10.56620896936596, 10.837171065604361, 11.075195358775174,\n",
    "12.697407291327917, 11.224639768836752, 10.654690890600639, 9.3520335261477001,\n",
    "11.308468217957545, 13.567949517097393, 13.679273456186404, 12.12579245215834,\n",
    "12.202912099386342, 11.676936801079698, 12.05250421564147, 11.216384456150669,\n",
    "13.833489892425087, 13.809565237698548, 12.419950109799089, 12.365750447335001,\n",
    "13.638589692982816, 14.711316606940278, 14.154676044856346, 16.413065677678446,\n",
    "14.435530834478991, 15.576028232468097, 16.525470706412023, 15.828468616174431] \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 1.3 \n",
    "\n",
    "In this exercise we will be implementing a simple feed-forward neural network with numpy in order to learn the XOR rule.\n",
    "\n",
    "Construct a neural network with 4 hidden and 1 output neurons, all using the sigmoid activation function. Implement training the network using gradient descent with backpropagation as described in the lectures and the course book. When training the network, use mean squared error as the cost function.\n",
    "\n",
    "Hint: You can check to make sure your network works correctly by picking weights that are known to work from previously, and running the network on the given X data to make sure it outputs something close to y before trying to implement the training.\n",
    "\n",
    "Hint: Initializing the weights randomly with 2*np.random.random(weight_mtx_shape) - 1 works well in this case.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "np.random.seed(42)\n",
    "\n",
    "# Learning rate\n",
    "eps = 0.5\n",
    "\n",
    "## XOR data, last column acts as bias\n",
    "X = np.array([[0,0,1],\n",
    "              [0,1,1],\n",
    "              [1,0,1],\n",
    "              [1,1,1]])\n",
    "\n",
    "y = np.array([[0,1,1,0]]).T\n",
    "\n",
    "def sigmoid(z):\n",
    "    # Implement the sigmoid function\n",
    "\n",
    "def cost(y, y_hat):\n",
    "    # Implement the cost function\n",
    "\n",
    "\n",
    "# Initialize the weights\n",
    "\n",
    "\n",
    "# Train the network   "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}