UseCases/TitanicSurvival_HyperParameter/Predict_TitanicSurvival_Hyperparameter_Tuning_Python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<header>\n",
    "   <p  style='font-size:36px;font-family:Arial; color:#F0F0F0; background-color: #00233c; padding-left: 20pt; padding-top: 20pt;padding-bottom: 10pt; padding-right: 20pt;'>\n",
    "       Predict Survival on the Titanic Disaster using HyperParameter Tuning\n",
    "  <br>\n",
    "       <img id=\"teradata-logo\" src=\"https://storage.googleapis.com/clearscape_analytics_demo_data/DEMO_Logo/teradata.svg\" alt=\"Teradata\" style=\"width: 125px; height: auto; margin-top: 20pt;\">\n",
    "    </p>\n",
    "</header>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>Introduction:</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The sinking of the RMS Titanic is one of the most infamous shipwrecks in history. On April 15, 1912, during her maiden voyage, the Titanic sank after colliding with an iceberg, killing 1502 out of 2224 passengers and crew. This sensational tragedy shocked the international community and led to better safety regulations for ships.</p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "One of the reasons that the shipwreck led to such loss of life was that there were not enough lifeboats for the passengers and crew. Although there was some element of luck involved in surviving the sinking, some groups of people were more likely to survive than others, such as women, children, and the upper-class.</p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "Researchers are still drawn to the Titanic disaster, even though it happened just over a century ago, as they try to figure out how some people survived while others couldn't. Fortunately, Teradata Vantage and ClearScape Analytics provide the ideal platform to create these predictions. ClearScape Analytics combines these analytic disciplines into a single, massively scalable platform which enables unique business outcomes and more accurate analytic and predictive models. With Vantage’s advanced in-database analytics, time series functions, and AI/ML capabilities, researchers can increase their confidence in these predictions. </p>\n",
    "\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Business Value</b></p>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>Improve speed, performance, and time-to value by minimizing data movement by fully integrating data for faster results and trusted outcomes</li>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>Easily deploys new, more accurate models to production</li>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>Deploy preferred AI/ML tools and models directly to the VantageCloud platform</li>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>Identify patterns and reasons leading to survival of passengers.</li>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>Advanced research and development stemming from the results of the data and models produced.</li></p>\n",
    "\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Why Teradata? </b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Traditional ML and AI development and deployment pipelines require users to manually combine various tools and techniques across the lifecycle.  This leads to lengthy, fragile, manual, error-prone processes that are, in many cases, impossible to migrate out of the lab and into production in order to realize business value. ClearScape Analytics helps to solve this “development to deployment gap” by providing highly scalable, performant, and easy-to-use analytic capabilities that address all aspects of the development lifecycle.  The same tools and techniques that data scientists use in development can be seamlessly deployed into production using the same code, platform, and operational pipeline.</p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>A critical strategy for Vantage and ClearScape Analytics is to embrace the value and innovation in the open-source and partner ML and AI community. This provides enterprises with the most scalable option for deploying custom machine learning pipelines. Users can leverage the innovation and familiarity of a broad range of tools and techniques, with the ability to prepare and score new data in near-real-time and at any scale; allowing the products of machine learning to become pervasive across all applications, reporting tools, and consumers in an organization. </p>\n",
    "\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "The goal of this demo is to create a predictive algorithm that can identify whether or not Titanic passengers survived the ship's sinking with the use of titanic passenger data. Here we are analyzing data for 891 passengers.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>1. Connect to Vantage.</b></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>In the section, we import the required libraries and set environment variables and environment paths (if required).</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import required packages.\n",
    "import random\n",
    "from getpass import getpass\n",
    "from teradataml import *\n",
    "from teradataml.hyperparameter_tuner import *\n",
    "from matplotlib import pyplot as plt\n",
    "display.max_rows = 5\n",
    "warnings.filterwarnings('ignore')\n",
    "warnings.simplefilter(action='ignore', category=DeprecationWarning)\n",
    "warnings.simplefilter(action='ignore', category=RuntimeWarning)\n",
    "warnings.simplefilter(action='ignore', category=FutureWarning)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>You will be prompted to provide the password. Enter your password, press the Enter key, then use down arrow to go to next cell. Begin running steps with Shift + Enter keys.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run -i ../startup.ipynb\n",
    "eng = create_context(host = 'host.docker.internal', username='demo_user', password = password)\n",
    "print(eng)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "execute_sql('''SET query_band='DEMO=Predict_TitanicSurvival_Hyperparameter_Tuning_Python.ipynb;' UPDATE FOR SESSION; ''')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>2. Getting Data for This Demo</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>We have provided data for this demo on cloud storage.  You have the option of either running the demo using foreign tables to access the data without using any storage on your environment or downloading the data to local storage which may yield somewhat faster execution, but there could be considerations of available storage.  There are two statements in the following cell, and one is commented out.  You may switch which mode you choose by changing the comment string.</p>   \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %run -i ../run_procedure.py \"call get_data('DEMO_TitanicSurvival_cloud');\"\n",
    " # Takes about 30 seconds\n",
    "%run -i ../run_procedure.py \"call get_data('DEMO_TitanicSurvival_local');\"\n",
    " # Takes about 50 seconds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "<p style = 'font-size:16px;font-family:Arial'>Next is an optional step – if you want to see status of databases/tables created and space used.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run -i ../run_procedure.py \"call space_report();\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>3. Analyze the raw data set</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Create a DataFrame to get the data from the table created.  We'll create a virtual dataframe to keep the date in Vantage and not copy it down to the client.</p>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "titanic = DataFrame(in_schema(\"DEMO_TitanicSurvival\", \"Passenger_Data\"))\n",
    "titanic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Check the shape of the dataset.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Shape of the dataframe.\n",
    "titanic.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The dataset contains 891 rows and 12 columns.</p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Check the number of nulls in each of the columns.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Info about dataframe and null values.\n",
    "titanic.info(null_counts=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>We observe that there are no NULLS in any of the columns.</p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Describe is used to find the statistics of the numeric columns in the dataframe.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generates statistics for numeric columns in titanic data. \n",
    "titanic.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Check the number of passengers who survived and the number of passengers who did not survive.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Count of survived passengers.\n",
    "survived_count = titanic[titanic.survived == 1]\n",
    "survived_count.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>As per the count 342 passengers survived.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Count of lost passengers.\n",
    "non_survived_count = titanic[titanic.survived == 0]\n",
    "non_survived_count.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>As per the count 549 passengers did not survive.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>4. Data Preparation</b></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Column selection.</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>We drop the columns which are not needed for further analysis. </p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Dropping unwanted columns.\n",
    "titanic = titanic.drop([\"passengername\", \"ticket\", \"cabin\"], axis=1)\n",
    "titanic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:1px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>4.1 Ordinal Encoding.</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>OrdinalEncodingFit function identifies distinct categorical values from an input table or a user-defined list and returns the distinct categorical values along with the ordinal value for each category.</p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>OrdinalEncodingFit is useful in use cases where categorical data needs to be converted into numerical data for analysis or machine learning algorithms. For example, in a dataset with categorical variables such as \"color\" or \"size\", TD_OrdinalEncodingFit can assign numerical values to each category, making it possible to perform mathematical operations on the data. This helps in tasks such as clustering, classification, and regression analysis.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform OrdinalEncoding for 'sex' column.\n",
    "ordinal_obj = OrdinalEncodingFit(target_column=['sex', 'embarked'],\n",
    "                                 data=titanic)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The OrdinalEncodingTransform function maps the categorical value to a specified ordinal value using the OrdinalEncodingFit output.\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "OrdinalEncodingTransform follows this process:- </p>\n",
    "    <li style = 'font-size:16px;font-family:Arial;color:#00233C'>Select the table and columns to be encoded by the OrdinalEncodingFit function.</li>\n",
    "    <li style = 'font-size:16px;font-family:Arial;color:#00233C'>Use OrdinalEncodingTransform to map each category value to a specified ordinal value, using OrdinalEncodingTransform output.</li>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Transforming the encoded data.\n",
    "df = ordinal_obj.transform(data=titanic,\n",
    "                           accumulate=['passenger', 'survived', 'pclass', 'age', 'sibsp', 'parch', 'fare']).result\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The OrdinalEncodingTransform function maps the categorical value to a specified ordinal value using the OrdinalEncodingFit output.\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "As we can observe the categorical values for the column 'sex' are converted into numeric values. female is replaced by 0 and male is replaced by 1. </p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:1px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>4.2 Train-Test split.</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The TrainTestSplit() function simulates how a model would perform on new data. The function divides the dataset into train and test subsets to evaluate machine learning algorithms and validate processes. The first subset is used to train the model. The second subset is used to make predictions and compare the predictions to actual values.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sample 5% of data for model validation.\n",
    "df_sample = df.sample(frac=[0.95, 0.05], randomize=True)\n",
    "df_sample"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Train dataset is created using sampleid = 1.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create train dataset from sample 1 by filtering on \"sampleid\" and drop \"sampleid\" column as it is required for training model.\n",
    "data_train = df_sample[df_sample.sampleid == \"1\"].drop(\"sampleid\", axis = 1)\n",
    "data_train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Test dataset is created using sampleid = 2.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create validation dataset from sample 2 by filtering on \"sampleid\" and drop \"sampleid\" column as it is required for validating model.\n",
    "data_val = df_sample[df_sample.sampleid == \"2\"].drop(\"sampleid\", axis = 1)\n",
    "data_val"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>5. Hyper-Parametrization of SimpleImpute.</b>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>GridSearch is an exhaustive search algorithm that covers all possible parameter values to identify optimal hyperparameters. It works for teradataml analytic functions from Analytics Database, BYOM, VAL, and UAF features.</p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "teradataml GridSearch allows you to perform hyperparameter tuning for all model trainer and non-model trainer functions.</p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "When used for model trainer functions:\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>Based on evaluation metrics, search determines best model.</li>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>All methods and properties can be used.</li></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>When used for non-model trainer functions:\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>You can choose the best output as you see fit to use this.</li>\n",
    "    <li style = 'font-size:16px;font-family:Arial;color:#00233C'>Only fit method is supported.</li></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>teradataml GridSearch also allows you to use input data as the hyperparameter. This option can be suitable when the you want to identify the best models for a set of input data. When you pass set of data as hyperparameter for model trainer function, the search determines the best data along with the best model based on the evaluation metrics.</p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>GridSearch offers hyper-parameterization for Non-Model Trainer functions. \"age\" and \"embarked\" columns contains 'NaN' values. Hence, Impute 'NaN' value with special metrics, for example, mean, mode or median. And use those imputed data to build the best model.</p>\n",
    "\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Define Hyperparameters for SimpleImputeFit </b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>GridSearch perform imputation on \"data_train\" for specified combination of parameters and returns imputed data.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "si_params = {\"data\":data_train,\n",
    "            \"stats_columns\":[\"age\", \"embarked\"],\n",
    "            \"stats\":(\"median\", \"mean\", \"mode\")}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Perform GridSearch on SimpleImputeFit function.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "si_gs_obj = GridSearch(func=SimpleImputeFit, params=si_params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The fit() method is used to run the teradataml analytic function for all sets of hyperparameters. Sets of hyperparameters chosen for execution from the parameter grid is populated based on search algorithm.</p>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>In model trainer function, the best parameters are selected based on training results.</li>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>In non- model trainer function, first execution parameter set is selected as the best parameters.</li>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Start the imputation task.\n",
    "si_gs_obj.fit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Imputation task metadata shows three variants of imputation results.\n",
    "si_gs_obj.models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>As seen above the models property returns the generated models metadata.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "models = si_gs_obj.models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Using each of the models the tranform function is applied on the data .</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Perform SimpleImpute transform and structure the data in dictionary format with labels.\n",
    "imputed_data = dict((model, si_gs_obj.get_model(model).transform(data = df,\n",
    "                    accumulate=['passenger', 'survived', 'pclass', 'age', 'sibsp', 'parch', 'fare']).result) \\\n",
    "                    for model in models[\"MODEL_ID\"])\n",
    "imputed_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# SimpleImpute performed on validation data.\n",
    "si_obj_val = SimpleImputeFit(data=data_val, stats_columns=[\"age\", \"embarked\"], stats=\"mean\")\n",
    "val_df = si_obj_val.transform(data=data_val).result\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>6. Hyperparameter-Tuning to create optimal predictive model.</b></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>RandomSearch algorithm performs random sampling on hyperparameter space to identify optimal hyperparameters. It works for teradataml analytic functions from Analytics Database, BYOM, VAL, and UAF features.>/p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>teradataml RandomSearch allows user to perform hyperparameter tuning for all model trainer and non-model trainer functions.</p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>When used for model trainer functions:\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>Based on evaluation metrics, search determines best model.</li>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>All methods and properties can be used.</li>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>When used for non-model trainer functions:\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>You can choose the best output as you see fit to use this.</li>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>Only fit method is supported.</li>\n",
    "    </p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Define XGBoost hyperparameter space with 4000 parameter combinations for XGBoost model. Any combination specified with in hyperparameter space is used for hyperparameter tuning task.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "XGB_params = {\"input_columns\":['pclass', 'age', 'sibsp', 'parch', 'fare', 'sex', 'embarked'],\n",
    "              \"response_column\" : 'survived',\n",
    "              \"max_depth\":tuple(random.randrange(3, 50) for i in range(10)),\n",
    "              \"lambda1\" : tuple(round(random.uniform(0.001, 1.0), 3) for i in range(10)),\n",
    "              \"model_type\" : \"classification\",\n",
    "              \"num_boosted_trees\": 50,\n",
    "              \"shrinkage_factor\":tuple(round(random.uniform(0.001, 1.0), 3) for i in range(10)),\n",
    "              \"iter_num\":( 35,40,45,50)}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Define Evaluation parameters which is used for model evaluation.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_params = {\"id_column\": \"passenger\",\n",
    "               \"model_type\": \"classification\",\n",
    "               \"accumulate\": \"survived\",\n",
    "               \"object_order_column\": ['task_index', 'tree_num', 'iter', 'class_num', 'tree_order']}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Initialize the RandomSearch for XGBoost model. In addition, Though hyperparameter space contains 4000 parameters based on \"n_iter\" value hyperparameter combinations are selected randomly. selected set of hyperparameters are used for model optimization.</p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Note: Chosen hyperparameter combinations are used on hyper-parameterized data for model optimization.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rs_obj = RandomSearch(func=XGBoost, params=XGB_params, n_iter=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The fit() method is used to run the teradataml analytic function for all sets of hyperparameters. Sets of hyperparameters chosen for execution from the parameter grid is populated based on search algorithm.</p>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>In model trainer function, the best parameters are selected based on training results.</li>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>In non- model trainer function, first execution parameter set is selected as the best parameters.</li></p>\n",
    "<p style = 'font-size:14px;font-family:Arial;color:#00233C'><b><i>**Note: Since this step searches for all model variations it might take time, around 3-4 minutes</i></b></p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Start the RandomSearch optimization.\n",
    "rs_obj.fit(data=imputed_data,\n",
    "           verbose=1, frac=0.85,run_parallel=False,\n",
    "           **eval_params\n",
    "            )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Trained model metadata explains 4 models build for each hyper-parameterized data. Hence, Total of 12 models generated in RandomSearch optimization.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rs_obj.models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>There are various properties of Randomsearch that can be used to analyze the models created by the fit() method. We will be using some of these to get the details for the models created </p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Check <b>model_stats:</b> The model_stats property returns a pandas DataFrame representing the model statistics of the model with best score.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# RandomSearch model stats for XGBoost.\n",
    "rs_obj.model_stats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Check <b>best_model_id:</b> The best_model_id property returns a string representing the model id of the model with the best score.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Best identified XGBoost model id.\n",
    "rs_obj.best_model_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Check <b>best_data_id:</b> The best_data_id property returns a string representing the \"data_id\" of a sampled data used for training the best model.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Best identified data id.\n",
    "rs_obj.best_data_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Check <b>best_score_:</b> The best_score_ property returns a string representing the best score of the model out of all generated models.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Best identified model score.\n",
    "rs_obj.best_score_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Check <b>best_params_:</b> The best_params_ property returns a dictionary of the the parameters used for the model with best score.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Best identified model hyperparameters.\n",
    "rs_obj.best_params_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>7. Perform validation on the best model.</b>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The evaluate() method is used for evaluation using trained models from Analytics Database, VAL, and UAF features. Evaluation are done using the default trained model.</p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Validating the best model.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rs_obj.evaluate(newdata=val_df,\n",
    "                **eval_params)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>8. Perform classification using best model.</b></p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The evaluate() method is used for evaluation using trained models from Analytics Database, VAL, and UAF features. Evaluation are done using the default trained model.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict passenger survival using the best model.\n",
    "result = rs_obj.predict(newdata=val_df,\n",
    "                        **eval_params)\n",
    "df_pred=result.result\n",
    "df_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>9. Evaluate the model</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The ClassificationEvaluator() function evaluates and emits various metrics of classification model based on its predictions on the data. Apart from accuracy, the secondary output data returns micro, macro, and weighted-averaged metrics of precision, recall, and F1-score values.\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>The function works for multi-class scenarios as well. In any case, the primary output data contains class-level metrics, whereas the secondary output data contains metrics that are applicable across classes.</li>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>The function works only when columns specified in 'observation_column' and 'prediction_column' has same teradata types.</li></p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_pred_1=df_pred\n",
    "df_pred_1 = df_pred_1.assign(Prediction = df_pred_1.Prediction.cast(type_ = VARCHAR(2)))\n",
    "df_pred_1 = df_pred_1.assign(survived = df_pred_1.survived.cast(type_ = VARCHAR(2)))\n",
    "df_pred_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ClassificationEvaluator_obj = ClassificationEvaluator(\n",
    "                                                        data = df_pred_1,\n",
    "                                                        observation_column = 'survived',\n",
    "                                                        prediction_column = 'Prediction',\n",
    "                                                        labels = ['0','1'])\n",
    "df_result=ClassificationEvaluator_obj.result\n",
    "df_result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Show AUC-ROC Curve</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The <a href = 'https://docs.teradata.com/search/all?query=TD_ROC&content-lang=en-US'>ROC</a> curve shows the performance of a binary classification model as its discrimination threshold varies. For a range of thresholds, the curve plots the true positive rate against false-positive rate.</p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>This function accepts a set of prediction-actual pairs as input and calculates the following values for a range of discrimination thresholds.</p>\n",
    "    <ul style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "        <li>True-positive rate (TPR)</li>\n",
    "        <li>False-positive rate (FPR)</li>\n",
    "        <li>The area under the ROC curve (AUC)</li>\n",
    "        <li>Gini coefficient</li>\n",
    "        <li>Other details are mentioned in the documentation</li>\n",
    "    </ul>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import metrics\n",
    "df_cm=df_pred.to_pandas()\n",
    "fpr, tpr, thresholds = metrics.roc_curve(df_cm['survived'], df_cm['Prediction'])\n",
    "auc = metrics.auc(fpr, tpr)\n",
    "print(\"AUC=\"+str(auc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Plot the predictions.</b></p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(fpr,tpr,label=\"AUC=\"+str(auc))\n",
    "plt.legend(loc=4)\n",
    "plt.xlabel(\"FPR\")\n",
    "plt.ylabel(\"TPR\")\n",
    "plt.title(\"AUC-ROC Curve\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Show Confusion Matrix</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The Confusion Matrix shows the actual and the Predicted values. Based on model the matrix shows the predicted and actual value comparison for people who survived and those who did not survive the titanic disaster.</p>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "from sklearn.metrics import confusion_matrix, roc_curve, roc_auc_score, recall_score, ConfusionMatrixDisplay\n",
    "# Compute confusion matrix\n",
    "cm = confusion_matrix(df_cm['survived'], df_cm['Prediction'])\n",
    "\n",
    "# Plot confusion matrix\n",
    "disp = ConfusionMatrixDisplay(confusion_matrix = cm, display_labels = ['Survived', 'Did not Survive'])\n",
    "fig, ax = plt.subplots(figsize = (8, 8))\n",
    "disp.plot(ax = ax, cmap = 'Blues', colorbar = True)\n",
    "\n",
    "# Add labels and annotations\n",
    "plt.title('Confusion Matrix')\n",
    "plt.xlabel('Predicted Label')\n",
    "plt.ylabel('True Label')\n",
    "plt.xticks(ticks = [0, 1], labels = ['Survived', 'Did not Survive'])\n",
    "plt.yticks(ticks = [0, 1], labels = ['Survived', 'Did not Survive'])\n",
    "\n",
    "# Add text to the plot to show the actual values of the confusion matrix\n",
    "for i in range(cm.shape[0]):\n",
    "    for j in range(cm.shape[1]):\n",
    "        plt.text(j, i, f'{cm[i, j]}', ha = 'center', va = 'center', color = 'white' if cm[i, j] > cm.max()/1.4 else 'black')\n",
    "\n",
    "# Remove grid lines\n",
    "ax.grid(False)\n",
    "\n",
    "# Show the plot\n",
    "plt.show()\n",
    "\n",
    "print(f'''\n",
    "This means that out of all the actual survived cases ({cm[0][0] + cm[0][1]}),\n",
    "{round(cm[0][0]/(cm[0][0] + cm[0][1])*100, 2)}% were correctly classified as survived, while\n",
    "{round(cm[0][1]/(cm[0][0] + cm[0][1])*100, 2)}% were incorrectly classified as survived.\n",
    "Similarly, out of all the actual death cases ({cm[1][0] + cm[1][1]}),\n",
    "{round(cm[1][1]/(cm[1][0] + cm[1][1])*100, 2)}% were correctly classified as did not survive, while\n",
    "{round(cm[1][0]/(cm[1][0] + cm[1][1])*100, 2)}% were incorrectly classified as did not survive.\n",
    "''')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Conclusion</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Thus using the Hyperparameter tuning we select the best data preparation model and training model for the required data and try to predict the correct value of target variables for the data. Vantage's easy-to-use analytic and AI/ML capabilities help researchers and datascientist use the best model and provide more accurate predictions. </p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>10. Cleanup</b></p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Databases and Tables</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The following code will clean up tables and databases created above.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run -i ../run_procedure.py \"call remove_data('DEMO_TitanicSurvival');\" \n",
    "#Takes 40 seconds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "remove_context()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<b style = 'font-size:20px;font-family:Arial;color:#00233C'>Required Materials</b>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Let’s look at the elements we have available for reference for this notebook:</p>\n",
    "\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Dataset</b></p>\n",
    "\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'>Passenger Data </p>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>Survived: Survival\t0 = No, 1 = Yes</li>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>Passenger: Unique ID of each passenger (integer) \n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>Pclass: Ticket class\t(1 = 1st, 2 = 2nd, 3 = 3rd) </li>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>Sex: Sex\t('male' 'female')</li>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>Age: Age in years\t</li>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>SibSp: Number of siblings / spouses aboard the Titanic\t</li>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>Parch: Number of parents / children aboard the Titanic\t</li>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>Ticket: Ticket number\t</li>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>Fare: Passenger fare\t</li>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>Cabin: Cabin number\t</li>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'>Embarked: Port of Embarkation\t(C = Cherbourg, Q = Queenstown, S = Southampton)</li>\n",
    "<p></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Filters:</b></p>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'><b>Industry:</b> Travel and Transportation</li>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'><b>Functionality:</b> Hyperparameter Tuning</li>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'><b>Use Case:</b> Titanic Survival Prediction</li>\n",
    "</p>\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Related Resources:</b></p>\n",
    "<li style = 'font-size:16px;font-family:Arial;color:#00233C'><a href = 'https://www.teradata.com/blogs/nps-is-a-metric-not-the-goal'>In the fight to improve customer experience, NPS is a metric, not the goal</a></li>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<footer style=\"padding-bottom:35px; background:#f9f9f9; border-bottom:3px solid #00233C\">\n",
    "    <div style=\"float:left;margin-top:14px\">ClearScape Analytics™</div>\n",
    "    <div style=\"float:right;\">\n",
    "        <div style=\"float:left; margin-top:14px\">\n",
    "            Copyright © Teradata Corporation - 2023,2024. All Rights Reserved\n",
    "        </div>\n",
    "    </div>\n",
    "</footer>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}