daal/decision-forest-example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Decision Forest Classification and Regression with Intel® Data Analytics Acceleration Library in Amazon SageMaker\n",
    "\n",
    "## Introduction\n",
    "\n",
    "Intel® Data Analytics Acceleration Library (Intel® DAAL) is the library of Intel® architecture optimized building blocks covering all stages of data analytics: data acquisition from a data source, preprocessing, transformation, data mining, modeling, validation, and decision making. One of its algorithms is Decision Forest.\n",
    "\n",
    "The library provides decision forest classification and regression algorithms based on an ensemble of tree-structured classifiers (decision trees) built using the general technique of bootstrap aggregation (bagging) and random choice of features. Decision tree is a binary tree graph. Its internal (split) nodes represent a decision function used to select the following (child) node at the prediction stage. Its leaf (terminal) nodes represent the corresponding response values, which are the result of the prediction from the tree.\n",
    "\n",
    "Intel® DAAL developer guide: https://software.intel.com/en-us/daal-programming-guide\n",
    "\n",
    "Intel® DAAL documentation for Decision Forest: https://software.intel.com/en-us/daal-programming-guide-decision-forest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decision Forest Usage with SageMaker Estimator\n",
    "Firstly, you need to import SageMaker package, get execution role and create session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "\n",
    "role = sagemaker.get_execution_role()\n",
    "sess = sagemaker.Session()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Secondly, you can specify parameters of Decision Forest.\n",
    "#### Hyperparameters\n",
    "<table style=\"border: 1px solid black;\">\n",
    "    <tr>\n",
    "        <td><strong>Parameter name</strong></td>\n",
    "        <td><strong>Type</strong></td>\n",
    "        <td><strong>Default value</strong></td>\n",
    "        <td><strong>Description</strong></td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>nClasses</td>\n",
    "        <td>int</td>\n",
    "        <td>2</td>\n",
    "        <td>Number of classes in data (only for classification)</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>fptype</td>\n",
    "        <td>str</td>\n",
    "        <td>\"double\"</td>\n",
    "        <td>The floating-point type that the algorithm uses for intermediate computations. Can be \"float\" or \"double\"</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>method</td>\n",
    "        <td>str</td>\n",
    "        <td>\"defaultDense\"</td>\n",
    "        <td>The only training method supported so far is the default dense method</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>nTrees</td>\n",
    "        <td>int</td>\n",
    "        <td>100</td>\n",
    "        <td>The number of trees in the forest</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>observationsPerTreeFraction</td>\n",
    "        <td>int</td>\n",
    "        <td>1</td>\n",
    "        <td>Fraction of the training set S used to form the bootstrap set for a single tree training, observationsPerTreeFraction in (0, 1]. The observations are sampled randomly with replacement</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>featuresPerNode</td>\n",
    "        <td>int</td>\n",
    "        <td>0</td>\n",
    "        <td>The number of features tried as possible splits per node. If the parameter is set to 0, the library uses the square root of the number of features for classification and (the number of features)/3 for regression</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>maxTreeDepth</td>\n",
    "        <td>int</td>\n",
    "        <td>0</td>\n",
    "        <td>Maximal tree depth. Default is 0 (unlimited).</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>minObservationsInLeafNode</td>\n",
    "        <td>int</td>\n",
    "        <td>1 for classification, 5 for regression</td>\n",
    "        <td>The number of neighbors</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>seed</td>\n",
    "        <td>int</td>\n",
    "        <td>777</td>\n",
    "        <td>The seed for random number generator, which is used to choose the bootstrap set, split features in every split node in a tree, and generate permutation required in computations of MDA variable importance</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>impurityThreshold</td>\n",
    "        <td>float</td>\n",
    "        <td>0</td>\n",
    "        <td>The threshold value used as stopping criteria: if the impurity value in the node is smaller than the threshold, the node is not split anymore</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>varImportance</td>\n",
    "        <td>str</td>\n",
    "        <td>\"None\"</td>\n",
    "        <td>The variable importance computation mode. Possible values:<br/>none – variable importance is not calculated<br/>MDI - Mean Decrease of Impurity, also known as the Gini importance or Mean Decrease Gini<br/>MDA_Raw - Mean Decrease of Accuracy (permutation importance)<br/>MDA_Scaled - the MDA_Raw value scaled by its standard deviation</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>resultsToCompute</td>\n",
    "        <td>str</td>\n",
    "        <td>\"None\"</td>\n",
    "        <td>Provide one of the following values to request a single characteristic or use bitwise OR to request a combination of the characteristics:<br/>computeOutOfBagError, computeOutOfBagErrorPerObservation</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>memorySavingMode</td>\n",
    "        <td>bool</td>\n",
    "        <td>False</td>\n",
    "        <td>If True, memory saving mode is enabled</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>bootstrap</td>\n",
    "        <td>bool</td>\n",
    "        <td>False for classification, True for regression</td>\n",
    "        <td>If True, bootstrap is enabled</td>\n",
    "    </tr>\n",
    "</table>\n",
    "\n",
    "Example of hyperparameters dictionary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "decision_forest_params = {\n",
    "    \"nClasses\": 5,\n",
    "    \"fptype\":\"double\",\n",
    "    \"method\":\"defaultDense\",\n",
    "    \"nTrees\":\"100\",\n",
    "    \"observationsPerTreeFraction\":\"1\",\n",
    "    \"featuresPerNode\":\"0\",\n",
    "    \"maxTreeDepth\":\"0\",\n",
    "    \"minObservationsInLeafNode\":\"1\",\n",
    "    \"seed\":\"777\",\n",
    "    \"impurityThreshold\":\"0\",\n",
    "    \"varImportance\":\"None\",\n",
    "    \"resultsToCompute\":\"None\",\n",
    "    \"memorySavingMode\":\"False\",\n",
    "    \"bootstrap\":\"False\"\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, you need to create SageMaker Estimator instance with following parameters:\n",
    "<table style=\"border: 1px solid black;\">\n",
    "    <tr>\n",
    "        <td><strong>Parameter name</strong></td>\n",
    "        <td><strong>Description</strong></td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>image_name</td>\n",
    "        <td>The container image to use for training</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>role</td>\n",
    "        <td>An AWS IAM role. The SageMaker training jobs and APIs that create SageMaker endpoints use this role to access training data and models</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>train_instance_count</td>\n",
    "        <td>Number of Amazon EC2 instances to use for training. Should be 1, because it is not distributed version of algorithm</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>train_instance_type</td>\n",
    "        <td>Type of EC2 instance to use for training. See available types on Amazon Marketplace page of algorithm</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>input_mode</td>\n",
    "        <td>The input mode that the algorithm supports. May be \"File\" or \"Pipe\"</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>output_path</td>\n",
    "        <td>S3 location for saving the trainig result (model artifacts and output files)</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>sagemaker_session</td>\n",
    "        <td>Session object which manages interactions with Amazon SageMaker APIs and any other AWS services needed</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>hyperparameters</td>\n",
    "        <td>Dictionary containing the hyperparameters to initialize this estimator with</td>\n",
    "    </tr>\n",
    "</table>\n",
    "Full SageMaker Estimator documentation: https://sagemaker.readthedocs.io/en/latest/estimators.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "daal_decision_forest_arn = \"<algorithm-arn>\" # you can find it on algorithm page in your subscriptions\n",
    "\n",
    "daal_decision_forest = sagemaker.algorithm.AlgorithmEstimator(\n",
    "    algorithm_arn=daal_decision_forest_arn,\n",
    "    role=role,\n",
    "    base_job_name=\"<base-job-name>\",\n",
    "    train_instance_count=1,\n",
    "    train_instance_type='ml.m4.xlarge',\n",
    "    input_mode=\"File\",\n",
    "    output_path=\"s3://<bucket-name>/<output-path>\",\n",
    "    sagemaker_session=sess,\n",
    "    hyperparameters=decision_forest_params\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training stage\n",
    "On training stage, Decision Forest algorithm consume input data from S3 location.\n",
    "This container supports only .csv (\"comma-separated values\") files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Creating training-job with name: testdaaldfcls2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2019-02-24 15:09:50 Starting - Starting the training job...\n",
      "2019-02-24 15:09:51 Starting - Launching requested ML instances......\n",
      "2019-02-24 15:11:18 Starting - Preparing the instances for training...\n",
      "2019-02-24 15:11:49 Downloading - Downloading input data...\n",
      "2019-02-24 15:11:56 Training - Downloading the training image.....\n",
      "\u001b[31m2019-02-24 15:12:57 INFO     Container setup completed, In Docker entrypoint - train... \u001b[0m\n",
      "\u001b[31m2019-02-24 15:12:57 INFO     Default Hyperparameters loaded: \u001b[0m\n",
      "\u001b[31m2019-02-24 15:12:57 INFO     {'nClasses': '2', 'task': 'none', 'fptype': 'double', 'method': 'defaultDense', 'nTrees': '100', 'observationsPerTreeFraction': '1', 'featuresPerNode': '0', 'maxTreeDepth': '0', 'minObservationsInLeafNode': '0', 'seed': '777', 'impurityThreshold': '0', 'varImportance': 'none', 'resultsToCompute': '', 'memorySavingMode': 'False', 'bootstrap': 'False'}\u001b[0m\n",
      "\u001b[31m2019-02-24 15:12:57 INFO     classification\u001b[0m\n",
      "\u001b[31m2019-02-24 15:12:57 INFO     Updated with user hyperparameters, Final Hyperparameters: \u001b[0m\n",
      "\u001b[31m2019-02-24 15:12:57 INFO     {'nClasses': '5', 'task': 'classification', 'fptype': 'double', 'method': 'defaultDense', 'nTrees': '100', 'observationsPerTreeFraction': '1.0', 'featuresPerNode': '0', 'maxTreeDepth': '0', 'minObservationsInLeafNode': '1', 'seed': '777', 'impurityThreshold': '0.0', 'varImportance': 'none', 'resultsToCompute': 'none', 'memorySavingMode': 'False', 'bootstrap': 'False'}\u001b[0m\n",
      "\u001b[31m2019-02-24 15:12:57 INFO     Reading training data... \u001b[0m\n",
      "\u001b[31m2019-02-24 15:12:57 INFO     Train data shape: (16000, 6)\u001b[0m\n",
      "\u001b[31m2019-02-24 15:12:57 INFO     Files loading time: 0.021045207977294922\u001b[0m\n",
      "\u001b[31m2019-02-24 15:12:57 INFO     Starting DAAL Decision Forest training...\u001b[0m\n",
      "\u001b[31m2019-02-24 15:12:58 INFO     Training time, sec: 0.35150837898254395\u001b[0m\n",
      "\u001b[31m2019-02-24 15:12:58 INFO     Saving model results...\u001b[0m\n",
      "\u001b[31m2019-02-24 15:12:58 INFO     To get outOfBagErrorPerObservation use the method 'outOfBagErrorPerObservation' on the training model object\u001b[0m\n",
      "\u001b[31m2019-02-24 15:12:58 INFO     Parameters saved at /opt/ml/model/parameters.json\u001b[0m\n",
      "\n",
      "2019-02-24 15:13:08 Uploading - Uploading generated training model\n",
      "2019-02-24 15:13:08 Completed - Training job completed\n",
      "Billable seconds: 79\n"
     ]
    }
   ],
   "source": [
    "daal_decision_forest.fit({\"training\": \"s3://<bucket-name>/<training-data-path>\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Real-time prediction\n",
    "Firstly, you need to deploy SageMaker endpoint that consumes data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Creating model package with name: daal-df-cls-2019-02-24-15-13-43-176\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".........."
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Creating model with name: daal-df-cls-2019-02-24-15-13-43-176-2019-02-24-15-14-28-639\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Creating endpoint with name testdaaldfcls2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------------------------!"
     ]
    }
   ],
   "source": [
    "predictor = daal_decision_forest.deploy(1, \"ml.m4.xlarge\", serializer=sagemaker.predictor.csv_serializer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Secondly, you should pass data as numpy array to endpoint and get predictions.\n",
    "\n",
    "In this example we are passing random data, but you can use any numpy 2D array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4\n",
      "2\n",
      "2\n",
      "1\n",
      "3\n",
      "2\n",
      "3\n",
      "3\n",
      "3\n",
      "3\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "predict_data = np.random.random(size=(10,10))\n",
    "print(predictor.predict(predict_data).decode(\"utf-8\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Don't forget to delete endpoint if you don't need it anymore."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Deleting endpoint with name: testdaaldfcls2\n"
     ]
    }
   ],
   "source": [
    "sess.delete_endpoint(predictor.endpoint)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Batch transform job\n",
    "If you don't need real-time prediction, you can use transform job. It uses saved model, compute predictions one time and saves it in specified or auto-generated output path.\n",
    "\n",
    "More about transform jobs: https://docs.aws.amazon.com/sagemaker/latest/dg/how-it-works-batch.html\n",
    "\n",
    "Transformer API: https://sagemaker.readthedocs.io/en/latest/transformer.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Creating model package with name: daal-df-cls-2019-02-24-15-22-38-863\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".........."
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Creating model with name: daal-df-cls-2019-02-24-15-22-38-863-2019-02-24-15-23-24-300\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Creating transform job with name: testdfclst1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "........................................!\n",
      "s3://sagemaker-us-east-2-123123123123/testdfclst1\n"
     ]
    }
   ],
   "source": [
    "transformer = daal_decision_forest.transformer(1, 'ml.m4.xlarge')\n",
    "transformer.transform(\"s3://<bucket-name>/<prediction-data-path>\", content_type='text/csv', job_name=\"<job-name>\")\n",
    "transformer.wait()\n",
    "print(transformer.output_path)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}