ModelOps/07_ModelOps_GIT_PIMA_Python_Indb_GLM.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d0e00e6e",
   "metadata": {},
   "source": [
    "<header>\n",
    "   <p  style='font-size:36px;font-family:Arial; color:#F0F0F0; background-color: #00233c; padding-left: 20pt; padding-top: 20pt;padding-bottom: 10pt; padding-right: 20pt;'>\n",
    "       ModelOps demo - Python In-database Scaling and GLM using Git\n",
    "  <br>\n",
    "       <img id=\"teradata-logo\" src=\"https://storage.googleapis.com/clearscape_analytics_demo_data/DEMO_Logo/teradata.svg\" alt=\"Teradata\" style=\"width: 125px; height: auto; margin-top: 20pt;\">\n",
    "    </p>\n",
    "</header>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8caf2ead-72c0-4a2c-bbe1-f71fd8a54e7d",
   "metadata": {
    "tags": []
   },
   "source": [
    "![image](images/git_meth.png) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a96f35c",
   "metadata": {},
   "source": [
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>Introduction</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "This Notebook will show you how to work with ClearScape Analytics in-database functions with ModelOps. With in-database analytics you can solve your scalable challenges by using Vantage to train and score your models. Whether you have a big volume of data or you want to avoid the data movement implementation to train models outside Vantage, you can use ModelOps to manage your Catalog of Models from multiple platforms including in-database algorithms.<br>To know more about in-database algorithms review teradata official documentation.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c0bb3df",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Steps in this Notebook</b></p>\n",
    "<ol style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "<li>Configure the Environment </li>\n",
    " <li>Connect to Vantage</li>\n",
    " <li>Define Training function</li>\n",
    " <li>Define Evaluate function</li>\n",
    " <li>Define Scoring function</li>\n",
    " <li>Define Model Metadata</li>\n",
    " <li>Commit and Push to Git to let ModelOps manage</li>\n",
    " <li>ModelOps full lifecycle till deployment</li>\n",
    " <li>ModelOps Monitoring</li>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f9eaa38",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial;color:#00233C'>1. Configure the Environment</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Here, we import the required libraries, set environment variables and environment paths (if required).</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f7606f8",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>1.1 Libraries installation</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'><b>A restart of the Kernel is needed to confirm changes</b>. We use -q parameter for a non-verbose log of the installation command, you may remove this parameter if you want to know all the steps of the pip installation.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "471a044b",
   "metadata": {},
   "outputs": [],
   "source": [
    "#%pip install -q teradataml==17.20.0.6 teradatamodelops==7.0.3 matplotlib==3.8.2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a210b4c9-7d3d-4daa-9ada-28f2591351ae",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'><b>Hint:</b><i>The easy way to restart the kernel to bring the above installed software into memory is to type zero zero (<b> 0 0 </b>). </i></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "600ef23e",
   "metadata": {},
   "source": [
    "<hr style=\"height:1px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>1.2 Libraries import</b></p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b744bc29",
   "metadata": {},
   "outputs": [],
   "source": [
    "from teradataml import *\n",
    "import os\n",
    "import getpass\n",
    "import logging\n",
    "import sys\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59bf237b",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial;color:#00233C'>2. Connect to Vantage</b></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bcc6d73b-bb60-4124-ad09-33ec9eaaa203",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>You will be prompted to provide the password. Enter your password, press the Enter key, then use down arrow to go to next cell. Begin running steps with Shift + Enter keys.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "80378f0d-a464-40bf-9180-03da16c0e96c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%run -i ../UseCases/startup.ipynb\n",
    "eng = create_context(host = 'host.docker.internal', username='demo_user', password = password)\n",
    "print(eng)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c81eaf79-ccd8-421d-9a75-046607a69a0c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%capture\n",
    "execute_sql('''SET query_band='DEMO=07_ModelOps_GIT_PIMA_Python_Indb_GLM.ipynb;' UPDATE FOR SESSION; ''')\n",
    "\n",
    "# configure byom/val installation\n",
    "configure.val_install_location = \"VAL\"\n",
    "configure.byom_install_location = \"MLDB\"\n",
    "\n",
    "# set the path to the local project repository for this model demo\n",
    "model_local_path = '~/modelops-demo-models/model_definitions/pima_indb_glm'\n",
    "res = os.system(f'mkdir -p {model_local_path}/model_modules')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0dd2607-a300-477a-a1bd-f3f59ba5c12a",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Getting Data for This Demo</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>We have provided data for this demo on cloud storage. You can either run the demo using foreign tables to access the data without any storage on your environment or download the data to local storage, which may yield faster execution. Still, there could be considerations of available storage. Two statements are in the following cell, and one is commented out. You may switch which mode you choose by changing the comment string.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d33878bf-fd84-4bdb-80c0-6ce93d2b7145",
   "metadata": {},
   "outputs": [],
   "source": [
    "#%run -i ../UseCases/run_procedure.py \"call get_data('DEMO_ModelOps_cloud');\"        # Takes 10 seconds\n",
    "%run -i ../UseCases/run_procedure.py \"call get_data('DEMO_ModelOps_local');\"        # Takes 30 seconds"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3276b9aa-0bb5-461f-b4e1-e6ef4afd3e97",
   "metadata": {},
   "source": [
    "<hr style=\"height:1px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Creating predictions and model table</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>We will create a predictions table where we get our model predictions and the model table where we will upload the model created.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "39a5acfa-9dbd-4e7b-bc5c-1a9f115ca14f",
   "metadata": {},
   "outputs": [],
   "source": [
    "#ddl for Aoa_Byom_Models \n",
    "query = '''CREATE SET TABLE Aoa_Byom_Models \n",
    "     (\n",
    "      model_version VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,\n",
    "      model_id VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,\n",
    "      model_type VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,\n",
    "      project_id VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,\n",
    "      deployed_at TIMESTAMP(6) DEFAULT CURRENT_TIMESTAMP(6),\n",
    "      model BLOB(2097088000))\n",
    "UNIQUE PRIMARY INDEX ( model_version );\n",
    "'''\n",
    "try:\n",
    "    execute_sql(query)\n",
    "except:\n",
    "    db_drop_table('Aoa_Byom_Models')\n",
    "    execute_sql(query) \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0223483a-5f1b-430d-87f3-db0a8ab2466f",
   "metadata": {},
   "outputs": [],
   "source": [
    "#ddl for Pima_Patient_Predictions\n",
    "query = '''CREATE MULTISET TABLE Pima_Patient_Predictions \n",
    "     (\n",
    "      job_id VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,\n",
    "      PatientId BIGINT,\n",
    "      HasDiabetes BIGINT,\n",
    "      json_report CLOB(1048544000) CHARACTER SET LATIN)\n",
    "PRIMARY INDEX ( job_id );;\n",
    "'''\n",
    "try:\n",
    "    execute_sql(query)\n",
    "except:\n",
    "    db_drop_table('Pima_Patient_Predictions')\n",
    "    execute_sql(query) \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad889bda-ec31-4967-990f-9837e50aef45",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Next is an optional step – if you want to see the status of databases/tables created and space used.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b43f2753-26a0-4b4f-8080-fc2221603e9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "%run -i ../UseCases/run_procedure.py \"call space_report();\"        # Takes 10 seconds"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fec57466",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial;color:#00233C'>3. Define Training Function</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The training function takes the following shape </p>\n",
    "\n",
    "```python\n",
    "def train(context: ModelContext, **kwargs):\n",
    "    aoa_create_context()\n",
    "    \n",
    "    # your training code using teradataml indDB function\n",
    "    model = <InDB Function>(...)\n",
    "    \n",
    "    # save your model\n",
    "    model.result.to_sql(f\"model_${context.model_version}\", if_exists=\"replace\")  \n",
    "    \n",
    "    record_training_stats(...)\n",
    "```\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>You can execute this from the CLI or directly within the notebook as shown.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "51bb236e",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile $model_local_path/model_modules/training.py\n",
    "from teradataml import (\n",
    "    DataFrame,\n",
    "    GLM,\n",
    "    ScaleFit,\n",
    "    ScaleTransform\n",
    ")\n",
    "from aoa import (\n",
    "    record_training_stats,\n",
    "    save_plot,\n",
    "    aoa_create_context,\n",
    "    ModelContext\n",
    ")\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "def plot_feature_importance(fi, img_filename):\n",
    "    import pandas as pd\n",
    "    import matplotlib.pyplot as plt\n",
    "    feat_importances = pd.Series(fi)\n",
    "    feat_importances.nlargest(10).plot(kind='barh').set_title('Feature Importance')\n",
    "    fig = plt.gcf()\n",
    "    fig.savefig(img_filename, dpi=500)\n",
    "    plt.clf()\n",
    "\n",
    "\n",
    "def train(context: ModelContext, **kwargs):\n",
    "    aoa_create_context()\n",
    "\n",
    "    feature_names = context.dataset_info.feature_names\n",
    "    target_name = context.dataset_info.target_names[0]\n",
    "    entity_key = context.dataset_info.entity_key\n",
    "\n",
    "    # read training dataset from Teradata and convert to pandas\n",
    "    train_df = DataFrame.from_query(context.dataset_info.sql)\n",
    "\n",
    "    print (\"Scaling using InDB Functions...\")\n",
    "    \n",
    "    scaler = ScaleFit(\n",
    "        data=train_df,\n",
    "        target_columns = feature_names,\n",
    "        scale_method = context.hyperparams[\"scale_method\"],\n",
    "        miss_value = context.hyperparams[\"miss_value\"],\n",
    "        global_scale = context.hyperparams[\"global_scale\"].lower() in ['true', '1'],\n",
    "        multiplier = context.hyperparams[\"multiplier\"],\n",
    "        intercept = context.hyperparams[\"intercept\"]\n",
    "    )\n",
    "\n",
    "    scaled_train = ScaleTransform(\n",
    "        data=train_df,\n",
    "        object=scaler.output,\n",
    "        accumulate = [target_name,entity_key]\n",
    "    )\n",
    "    \n",
    "    scaler.output.to_sql(f\"scaler_${context.model_version}\", if_exists=\"replace\")\n",
    "    print(\"Saved scaler\")\n",
    "    \n",
    "    print(\"Starting training...\")\n",
    "\n",
    "    model = GLM(\n",
    "        input_columns = feature_names,\n",
    "        response_column = target_name,\n",
    "        data = scaled_train.result,\n",
    "        family = context.hyperparams[\"family\"],\n",
    "        learning_rate = context.hyperparams[\"learning_rate\"],\n",
    "        momentum = context.hyperparams[\"momentum\"],\n",
    "        initial_eta = context.hyperparams[\"initial_eta\"],\n",
    "        local_sgd_iterations = context.hyperparams[\"local_sgd_iterations\"],\n",
    "        iter_max = context.hyperparams[\"iter_max\"],\n",
    "        batch_size = context.hyperparams[\"batch_size\"],\n",
    "        iter_num_no_change = context.hyperparams[\"iter_num_no_change\"]\n",
    "    )\n",
    "    \n",
    "    model.result.to_sql(f\"model_${context.model_version}\", if_exists=\"replace\")    \n",
    "    print(\"Saved trained model\")\n",
    "\n",
    "    # Calculate feature importance and generate plot\n",
    "    model_pdf = model.result.to_pandas()[['predictor','estimate']]\n",
    "    predictor_dict = {}\n",
    "    \n",
    "    for index, row in model_pdf.iterrows():\n",
    "        if row['predictor'] in feature_names:\n",
    "            value = row['estimate']\n",
    "            predictor_dict[row['predictor']] = value\n",
    "    \n",
    "    feature_importance = dict(sorted(predictor_dict.items(), key=lambda x: x[1], reverse=True))\n",
    "    keys, values = zip(*feature_importance.items())\n",
    "    norm_values = (values-np.min(values))/(np.max(values)-np.min(values))\n",
    "    feature_importance = {keys[i]: float(norm_values[i]*1000) for i in range(len(keys))}\n",
    "    plot_feature_importance(feature_importance, f\"{context.artifact_output_path}/feature_importance\")\n",
    "\n",
    "    record_training_stats(\n",
    "        train_df,\n",
    "        features=feature_names,\n",
    "        targets=[target_name],\n",
    "        categorical=[target_name],\n",
    "        feature_importance=feature_importance,\n",
    "        context=context\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b1483fa",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the ModelContext to test with. The ModelContext is created and managed automatically by ModelOps \n",
    "# when it executes your code via CLI / UI. However, for testing in the notebook, you can define as follows\n",
    "\n",
    "# define the training dataset \n",
    "sql = \"\"\"\n",
    "SELECT \n",
    "    F.*, D.hasdiabetes\n",
    "FROM Demo_ModelOps.PIMA_PATIENT_FEATURES F \n",
    "JOIN Demo_ModelOps.PIMA_PATIENT_DIAGNOSES D\n",
    "ON F.patientid = D.patientid\n",
    "    WHERE D.patientid MOD 5 <> 0\n",
    "\"\"\"\n",
    "\n",
    "feature_metadata =  {\n",
    "    \"database\": \"demo_modelops\",\n",
    "    \"table\": \"aoa_statistics_metadata\"\n",
    "}\n",
    "\n",
    "hyperparams = {\n",
    "    \"scale_method\":\"STD\",\n",
    "    \"miss_value\":\"KEEP\",\n",
    "    \"global_scale\":\"False\",\n",
    "    \"multiplier\":\"1\",\n",
    "    \"intercept\":\"0\",\n",
    "    \"family\": \"BINOMIAL\", \n",
    "    \"learning_rate\": \"OPTIMAL\",\n",
    "    \"momentum\": 0.80,\n",
    "    \"initial_eta\": 0.05,\n",
    "    \"local_sgd_iterations\": 10,\n",
    "    \"iter_max\": 100,\n",
    "    \"batch_size\": 50,\n",
    "    \"iter_num_no_change\": 5\n",
    "}\n",
    "\n",
    "entity_key = \"PatientId\"\n",
    "target_names = [\"HasDiabetes\"]\n",
    "feature_names = [\"NumTimesPrg\", \"PlGlcConc\", \"BloodP\", \"SkinThick\", \"TwoHourSerIns\", \"BMI\", \"DiPedFunc\", \"Age\"]\n",
    "\n",
    "from aoa import ModelContext, DatasetInfo\n",
    "\n",
    "dataset_info = DatasetInfo(\n",
    "    sql=sql,\n",
    "    entity_key=entity_key,\n",
    "    feature_names=feature_names,\n",
    "    target_names=target_names,\n",
    "    feature_metadata=feature_metadata\n",
    ")\n",
    "\n",
    "ctx = ModelContext(\n",
    "    hyperparams=hyperparams,\n",
    "    dataset_info=dataset_info,\n",
    "    artifact_output_path=\"artifacts/\",\n",
    "    model_version=\"InDB_v1\",\n",
    "    model_table=\"model_InDB_v1\"\n",
    ")\n",
    "\n",
    "sys.path.append(os.path.expanduser(f\"{model_local_path}/model_modules\"))\n",
    "import training\n",
    "training.train(context=ctx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0f5f3e9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check the generated files\n",
    "!ls -lh artifacts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "334f052a",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial;color:#00233C'>4. Define Evaluation Function</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The evaluation function takes the following shape</p>\n",
    "\n",
    "```python\n",
    "def evaluate(context: ModelContext, **kwargs):\n",
    "    aoa_create_context()\n",
    "\n",
    "    # read your model from Vantage\n",
    "    model = DataFrame(f\"model_${context.model_version}\")\n",
    "    \n",
    "    # your evaluation logic\n",
    "    \n",
    "    record_evaluation_stats(...)\n",
    "```\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>You can execute this from the CLI or directly within the notebook as shown.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a9e017d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile $model_local_path/model_modules/evaluation.py\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from teradataml import (\n",
    "    copy_to_sql,\n",
    "    DataFrame,\n",
    "    TDGLMPredict,\n",
    "    ScaleTransform,\n",
    "    ClassificationEvaluator,\n",
    "    ConvertTo,\n",
    "    ROC\n",
    ")\n",
    "from aoa import (\n",
    "    record_evaluation_stats,\n",
    "    save_plot,\n",
    "    aoa_create_context,\n",
    "    ModelContext\n",
    ")\n",
    "import json\n",
    "import os\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "def plot_feature_importance(fi, img_filename):\n",
    "    import pandas as pd\n",
    "    import matplotlib.pyplot as plt\n",
    "    feat_importances = pd.Series(fi)\n",
    "    feat_importances.nlargest(10).plot(kind='barh').set_title('Feature Importance')\n",
    "    fig = plt.gcf()\n",
    "    fig.savefig(img_filename, dpi=500)\n",
    "    plt.clf()\n",
    "    \n",
    "    \n",
    "def plot_confusion_matrix(cf, img_filename):\n",
    "    import matplotlib.pyplot as plt\n",
    "    fig, ax = plt.subplots(figsize=(7.5, 7.5))\n",
    "    ax.matshow(cf, cmap=plt.cm.Blues, alpha=0.3)\n",
    "    for i in range(cf.shape[0]):\n",
    "        for j in range(cf.shape[1]):\n",
    "            ax.text(x=j, y=i,s=cf[i, j], va='center', ha='center', size='xx-large')\n",
    "    ax.set_xlabel('Predicted labels');\n",
    "    ax.set_ylabel('True labels'); \n",
    "    ax.set_title('Confusion Matrix');\n",
    "    fig = plt.gcf()\n",
    "    fig.savefig(img_filename, dpi=500)\n",
    "    plt.clf()\n",
    "\n",
    "    \n",
    "def plot_roc_curve(roc_out, img_filename):\n",
    "    import matplotlib.pyplot as plt\n",
    "    auc = roc_out.result.to_pandas().reset_index()['AUC'][0]\n",
    "    roc_results = roc_out.output_data.to_pandas()\n",
    "    plt.plot(roc_results['fpr'], roc_results['tpr'], color='darkorange', lw=2, label='ROC curve (AUC = %0.2f)' % 0.27)\n",
    "    plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--')\n",
    "    plt.xlim([0.0, 1.0])\n",
    "    plt.ylim([0.0, 1.05])\n",
    "    plt.xlabel('False Positive Rate')\n",
    "    plt.ylabel('True Positive Rate')\n",
    "    plt.title('Receiver Operating Characteristic (ROC) Curve')\n",
    "    plt.legend(loc=\"lower right\")\n",
    "    fig = plt.gcf()\n",
    "    fig.savefig(img_filename, dpi=500)\n",
    "    plt.clf()\n",
    "\n",
    "    \n",
    "def evaluate(context: ModelContext, **kwargs):\n",
    "\n",
    "    aoa_create_context()\n",
    "\n",
    "    model = DataFrame(f\"model_${context.model_version}\")\n",
    "\n",
    "    feature_names = context.dataset_info.feature_names\n",
    "    target_name = context.dataset_info.target_names[0]\n",
    "    entity_key = context.dataset_info.entity_key\n",
    "\n",
    "    test_df = DataFrame.from_query(context.dataset_info.sql)\n",
    "\n",
    "    # Scaling the test set\n",
    "    print (\"Loading scaler...\")\n",
    "    scaler = DataFrame(f\"scaler_${context.model_version}\")\n",
    "\n",
    "    scaled_test = ScaleTransform(\n",
    "        data=test_df,\n",
    "        object=scaler,\n",
    "        accumulate = [target_name,entity_key]\n",
    "    )\n",
    "    \n",
    "    print(\"Scoring\")\n",
    "    predictions = TDGLMPredict(\n",
    "        object=model,\n",
    "        newdata=scaled_test.result,\n",
    "        accumulate=target_name,\n",
    "        id_column=entity_key,\n",
    "        output_prob=True,\n",
    "        output_responses=['0','1']\n",
    "    )\n",
    "\n",
    "    predicted_data = ConvertTo(\n",
    "        data = predictions.result,\n",
    "        target_columns = [target_name,'prediction'],\n",
    "        target_datatype = [\"INTEGER\"]\n",
    "    )\n",
    "\n",
    "    ClassificationEvaluator_obj = ClassificationEvaluator(\n",
    "        data=predicted_data.result,\n",
    "        observation_column=target_name,\n",
    "        prediction_column='prediction',\n",
    "        num_labels=2\n",
    "    )\n",
    "\n",
    "    metrics_pd = ClassificationEvaluator_obj.output_data.to_pandas()\n",
    "\n",
    "    evaluation = {\n",
    "        'Accuracy': '{:.2f}'.format(metrics_pd.MetricValue[0]),\n",
    "        'Micro-Precision': '{:.2f}'.format(metrics_pd.MetricValue[1]),\n",
    "        'Micro-Recall': '{:.2f}'.format(metrics_pd.MetricValue[2]),\n",
    "        'Micro-F1': '{:.2f}'.format(metrics_pd.MetricValue[3]),\n",
    "        'Macro-Precision': '{:.2f}'.format(metrics_pd.MetricValue[4]),\n",
    "        'Macro-Recall': '{:.2f}'.format(metrics_pd.MetricValue[5]),\n",
    "        'Macro-F1': '{:.2f}'.format(metrics_pd.MetricValue[6]),\n",
    "        'Weighted-Precision': '{:.2f}'.format(metrics_pd.MetricValue[7]),\n",
    "        'Weighted-Recall': '{:.2f}'.format(metrics_pd.MetricValue[8]),\n",
    "        'Weighted-F1': '{:.2f}'.format(metrics_pd.MetricValue[9]),\n",
    "    }\n",
    "\n",
    "    with open(f\"{context.artifact_output_path}/metrics.json\", \"w+\") as f:\n",
    "        json.dump(evaluation, f)\n",
    "        \n",
    "    cm = confusion_matrix(predicted_data.result.to_pandas()['HasDiabetes'], predicted_data.result.to_pandas()['prediction'])\n",
    "    plot_confusion_matrix(cm, f\"{context.artifact_output_path}/confusion_matrix\")\n",
    "\n",
    "    roc_out = ROC(\n",
    "        data=predictions.result,\n",
    "        probability_column='prob_1',\n",
    "        observation_column=target_name,\n",
    "        positive_class='1',\n",
    "        num_thresholds=1000\n",
    "    )\n",
    "    plot_roc_curve(roc_out, f\"{context.artifact_output_path}/roc_curve\")\n",
    "\n",
    "    # Calculate feature importance and generate plot\n",
    "    model_pdf = model.to_pandas()[['predictor','estimate']]\n",
    "    predictor_dict = {}\n",
    "    \n",
    "    for index, row in model_pdf.iterrows():\n",
    "        if row['predictor'] in feature_names:\n",
    "            value = row['estimate']\n",
    "            predictor_dict[row['predictor']] = value\n",
    "    \n",
    "    feature_importance = dict(sorted(predictor_dict.items(), key=lambda x: x[1], reverse=True))\n",
    "    keys, values = zip(*feature_importance.items())\n",
    "    norm_values = (values-np.min(values))/(np.max(values)-np.min(values))\n",
    "    feature_importance = {keys[i]: float(norm_values[i]*1000) for i in range(len(keys))}\n",
    "    plot_feature_importance(feature_importance, f\"{context.artifact_output_path}/feature_importance\")\n",
    "\n",
    "    predictions_table = \"predictions_tmp\"\n",
    "    copy_to_sql(df=predicted_data.result, table_name=predictions_table, index=False, if_exists=\"replace\", temporary=True)\n",
    "\n",
    "    # calculate stats if training stats exist\n",
    "    if os.path.exists(f\"{context.artifact_input_path}/data_stats.json\"):\n",
    "        record_evaluation_stats(\n",
    "            features_df=test_df,\n",
    "            predicted_df=DataFrame.from_query(f\"SELECT * FROM {predictions_table}\"),\n",
    "            feature_importance=feature_importance,\n",
    "            context=context\n",
    "        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b9e3a1c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the ModelContext to test with. The ModelContext is created and managed automatically by ModelOps \n",
    "# when it executes your code via CLI / UI. However, for testing in the notebook, you can define as follows\n",
    "\n",
    "# define the evaluation dataset \n",
    "sql = \"\"\"\n",
    "SELECT \n",
    "    F.*, D.hasdiabetes \n",
    "FROM Demo_ModelOps.PIMA_PATIENT_FEATURES F \n",
    "JOIN Demo_ModelOps.PIMA_PATIENT_DIAGNOSES D\n",
    "ON F.patientid = D.patientid\n",
    "    WHERE D.patientid MOD 5 = 0\n",
    "\"\"\"\n",
    "\n",
    "entity_key = \"PatientId\"\n",
    "target_names = [\"HasDiabetes\"]\n",
    "feature_names = [\"NumTimesPrg\", \"PlGlcConc\", \"BloodP\", \"SkinThick\", \"TwoHourSerIns\", \"BMI\", \"DiPedFunc\", \"Age\"]\n",
    " \n",
    "dataset_info = DatasetInfo(\n",
    "    sql=sql,\n",
    "    entity_key=entity_key,\n",
    "    feature_names=feature_names,\n",
    "    target_names=target_names,\n",
    "    feature_metadata=feature_metadata\n",
    ")\n",
    "\n",
    "ctx = ModelContext(\n",
    "    hyperparams=hyperparams,\n",
    "    dataset_info=dataset_info,\n",
    "    artifact_output_path=\"artifacts/\",\n",
    "    artifact_input_path=\"artifacts/\",\n",
    "    model_version=\"InDB_v1\",\n",
    "    model_table=\"model_InDB_v1\"\n",
    ")\n",
    "\n",
    "import evaluation\n",
    "evaluation.evaluate(context=ctx)\n",
    "\n",
    "# view evaluation results\n",
    "import json\n",
    "with open(f\"{ctx.artifact_output_path}/metrics.json\") as f:\n",
    "    print(json.load(f))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7dd20501",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check the generated files\n",
    "!ls -lh artifacts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47dcdfc8",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial;color:#00233C'>5. Define Scoring Function</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The scoring function takes the following shape</p>\n",
    "\n",
    "```python\n",
    "def score(context: ModelContext, **kwargs):\n",
    "    aoa_create_context()\n",
    "\n",
    "    # read your model\n",
    "    model = DataFrame(f\"model_${context.model_version}\")\n",
    "    \n",
    "    # your evaluation logic\n",
    "    \n",
    "    record_scoring_stats(...)\n",
    "```\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>You can execute this from the CLI or directly within the notebook as shown.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f77825ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile $model_local_path/model_modules/scoring.py\n",
    "from teradataml import (\n",
    "    copy_to_sql,\n",
    "    DataFrame,\n",
    "    TDGLMPredict,\n",
    "    ScaleTransform\n",
    ")\n",
    "from aoa import (\n",
    "    record_scoring_stats,\n",
    "    aoa_create_context,\n",
    "    ModelContext\n",
    ")\n",
    "import pandas as pd\n",
    "\n",
    "\n",
    "def score(context: ModelContext, **kwargs):\n",
    "\n",
    "    aoa_create_context()\n",
    "\n",
    "    model = DataFrame(f\"model_${context.model_version}\")\n",
    "\n",
    "    feature_names = context.dataset_info.feature_names\n",
    "    target_name = context.dataset_info.target_names[0]\n",
    "    entity_key = context.dataset_info.entity_key\n",
    "\n",
    "    features_tdf = DataFrame.from_query(context.dataset_info.sql)\n",
    "    features_pdf = features_tdf.to_pandas(all_rows=True)\n",
    "\n",
    "    # Scaling the scoring set\n",
    "    print (\"Loading scaler...\")\n",
    "    scaler = DataFrame(f\"scaler_${context.model_version}\")\n",
    "\n",
    "    scaled_features = ScaleTransform(\n",
    "        data=features_tdf,\n",
    "        object=scaler,\n",
    "        accumulate = entity_key\n",
    "    )\n",
    "    \n",
    "    print(\"Scoring\")\n",
    "    predictions = TDGLMPredict(\n",
    "        object=model,\n",
    "        newdata=scaled_features.result,\n",
    "        id_column=entity_key\n",
    "    )\n",
    "\n",
    "    predictions_pdf = predictions.result.to_pandas(all_rows=True).rename(columns={\"prediction\": target_name}).astype(int)\n",
    "\n",
    "    print(\"Finished Scoring\")\n",
    "\n",
    "    # store the predictions\n",
    "    predictions_pdf = pd.DataFrame(predictions_pdf, columns=[target_name])\n",
    "    predictions_pdf[entity_key] = features_pdf.index.values\n",
    "    # add job_id column so we know which execution this is from if appended to predictions table\n",
    "    predictions_pdf[\"job_id\"] = context.job_id\n",
    "\n",
    "    # teradataml doesn't match column names on append.. and so to match / use same table schema as for byom predict\n",
    "    # example (see README.md), we must add empty json_report column and change column order manually (v17.0.0.4)\n",
    "    # CREATE MULTISET TABLE pima_patient_predictions\n",
    "    # (\n",
    "    #     job_id VARCHAR(255), -- comes from airflow on job execution\n",
    "    #     PatientId BIGINT,    -- entity key as it is in the source data\n",
    "    #     HasDiabetes BIGINT,   -- if model automatically extracts target\n",
    "    #     json_report CLOB(1048544000) CHARACTER SET UNICODE  -- output of\n",
    "    # )\n",
    "    # PRIMARY INDEX ( job_id );\n",
    "    predictions_pdf[\"json_report\"] = \"\"\n",
    "    predictions_pdf = predictions_pdf[[\"job_id\", entity_key, target_name, \"json_report\"]]\n",
    "\n",
    "    copy_to_sql(\n",
    "        df=predictions_pdf,\n",
    "        schema_name=context.dataset_info.predictions_database,\n",
    "        table_name=context.dataset_info.predictions_table,\n",
    "        index=False,\n",
    "        if_exists=\"append\"\n",
    "    )\n",
    "    \n",
    "    print(\"Saved predictions in Teradata\")\n",
    "\n",
    "    # calculate stats\n",
    "    predictions_df = DataFrame.from_query(f\"\"\"\n",
    "        SELECT \n",
    "            * \n",
    "        FROM {context.dataset_info.get_predictions_metadata_fqtn()} \n",
    "            WHERE job_id = '{context.job_id}'\n",
    "    \"\"\")\n",
    "\n",
    "    record_scoring_stats(features_df=features_tdf, predicted_df=predictions_df, context=context)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d04af97b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the ModelContext to test with. The ModelContext is created and managed automatically by ModelOps \n",
    "# when it executes your code via CLI / UI. However, for testing in the notebook, you can define as follows\n",
    "\n",
    "# define the scoring dataset \n",
    "\n",
    "sql = \"\"\"\n",
    "SELECT \n",
    "    F.*\n",
    "FROM Demo_ModelOps.PIMA_PATIENT_FEATURES F \n",
    "    WHERE F.patientid MOD 5 = 0\n",
    "\"\"\"\n",
    "\n",
    "# where to store predictions\n",
    "predictions = {\n",
    "    \"database\": \"demo_user\",\n",
    "    \"table\": \"pima_patient_predictions_tmp\"\n",
    "}\n",
    "\n",
    "import uuid\n",
    "job_id=str(uuid.uuid4())\n",
    "\n",
    "dataset_info = DatasetInfo(sql=sql,\n",
    "                           entity_key=entity_key,\n",
    "                           feature_names=feature_names,\n",
    "                           target_names=target_names,\n",
    "                           feature_metadata=feature_metadata,\n",
    "                           predictions=predictions)\n",
    "\n",
    "ctx = ModelContext(hyperparams=hyperparams,\n",
    "                   dataset_info=dataset_info,\n",
    "                   artifact_output_path=\"artifacts/\",\n",
    "                   artifact_input_path=\"artifacts/\",\n",
    "                   model_version=\"InDB_v1\",\n",
    "                   model_table=\"model_InDB_v1\",\n",
    "                   job_id=job_id)\n",
    "\n",
    "import scoring\n",
    "scoring.score(context=ctx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d57f577",
   "metadata": {},
   "outputs": [],
   "source": [
    "DataFrame.from_query(f\"SELECT * FROM Demo_User.pima_patient_predictions_tmp WHERE job_id='{job_id}'\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "582e8c51",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Clean up\n",
    "\n",
    "os.system('rm -f artifacts/*')\n",
    "\n",
    "try:\n",
    "    db_drop_table('model_InDB_v1')\n",
    "except: \n",
    "    pass\n",
    "\n",
    "try:\n",
    "    db_drop_table('scaler_InDB_v1')\n",
    "except: \n",
    "    pass\n",
    "\n",
    "try:\n",
    "    db_drop_table('pima_patient_predictions_tmp')\n",
    "except: \n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b5d9a73",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial;color:#00233C'>6. Define Model Metadata</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Now let's create the configuration files.<br>Requirements file with the dependencies and versions:</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f946e187",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile $model_local_path/model_modules/requirements.txt\n",
    "teradataml==17.20.0.6\n",
    "teradatamodelops==7.0.3\n",
    "pandas==2.1.4\n",
    "scikit-learn==1.3.2\n",
    "matplotlib==3.8.2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9af179c7",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The hyper parameter configuration (default values):</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37165d41",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile $model_local_path/config.json\n",
    "{\n",
    "   \"hyperParameters\": {\n",
    "        \"scale_method\":\"STD\",\n",
    "        \"miss_value\":\"KEEP\",\n",
    "        \"global_scale\": \"False\",\n",
    "        \"multiplier\":\"1\",\n",
    "        \"intercept\":\"0\",\n",
    "        \"family\": \"BINOMIAL\", \n",
    "        \"learning_rate\": \"OPTIMAL\",\n",
    "        \"momentum\": 0.80,\n",
    "        \"initial_eta\": 0.05,\n",
    "        \"local_sgd_iterations\": 10,\n",
    "        \"iter_max\": 100,\n",
    "        \"batch_size\": 50,\n",
    "        \"iter_num_no_change\": 5\n",
    "   }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbf4467f",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The model configuration:</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "236decde",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile $model_local_path/model.json\n",
    "{\n",
    "    \"id\": \"f8df0bec-12d1-4d2d-920f-4448503df82d\",\n",
    "    \"name\": \"Python PIMA InDB GLM\",\n",
    "    \"description\": \"Python PIMA InDB GLM for Diabetes Prediction\",\n",
    "    \"language\": \"python\"\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "395b17a7",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial;color:#00233C'>7. Commit and Push to Git to let ModelOps manage</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Run the command below to commit and push changes to our forked repository, so ModelOps can fetch the changes to the model.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a45c4ca7",
   "metadata": {},
   "outputs": [],
   "source": [
    "!cd $model_local_path/../.. && git add . && git commit -m \"Added Python PIMA InDB GLM demo model 🚀\" && git push"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2e7cf2a",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Now that changes are pushed, you can make the lifecycle inside <b>ModelOps User Interface</b>, plan for new trainings, evaluations and scorings. Compare models and operationalize into Production with automated Monitoring and alerting capabilities.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f5a160f",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial;color:#00233C'>8. ModelOps full lifecycle till deployment</b></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c8fbc02",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Use or Create a Project with the git code repository with the model code, then you should see the model in the catalog already created</p>\n",
    "\n",
    "<img src=\"images/08_01.png\" alt=\"Model Catalog with inDB\"/>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Select the Model and then click Train a new Model. Use default hyper-parameters. This will launch the training job with the training script we generated and pushed to Git.</p>\n",
    "\n",
    "<img src=\"images/08_02.png\" alt=\"Train\"/>\n",
    "\n",
    "<img src=\"images/08_03.png\" alt=\"Train job\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<img src=\"images/08_04.png\" alt=\"Train finished\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>When Model is trained a new Model Id is created and you can get inside the Model Lifecycle screen to review artifacts and other details</p>\n",
    "\n",
    "<img src=\"images/08_06.png\" alt=\"Model lifecycle\"/>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Now, let's evaluate the Model, click the button and select the evaluation dataset. This will launch the evaluation job with the training script we generated and pushed to Git.</p>\n",
    "\n",
    "<img src=\"images/08_07.png\" alt=\"Evaluation\" width=\"500\" height=\"500\"/> <img src=\"images/08_08.png\" alt=\"Evaluation job\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>When evaluation job is finished a Model evaluation Report is generated with the metrics and charts that evaluation script generates</p>\n",
    "\n",
    "<img src=\"images/08_26.png\" alt=\"Model Report\" />\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Now, let's approve the model and provide an approval description</p>\n",
    "\n",
    "<img src=\"images/08_09.png\" alt=\"Approval\" />\n",
    "\n",
    "<img src=\"images/08_10.png\" alt=\"Approval description\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The model is ready to be deployed. Let's deploy using a Batch scheduling option - Run it manual</p>\n",
    "\n",
    "<img src=\"images/08_11.png\" alt=\"Deployment Engine\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<img src=\"images/08_12.png\" alt=\"Deployment Publish\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<img src=\"images/08_13.png\" alt=\"Deployment Schedule\" width=\"500\" height=\"500\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88476c06-359f-41ea-b65e-67327de32ed7",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Go and try this Step by yourself. Launch ModelOps from this button below:</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3116c610-7119-41f6-973f-349e2081bdda",
   "metadata": {},
   "source": [
    "[![image](images/launchModelOps.png)](/modelops)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b73db1c",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial;color:#00233C'>9. ModelOps Monitoring</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Now the model is deployed and a new Deployment appears in the deployment screen</p>\n",
    "\n",
    "\n",
    "<img src=\"images/08_15.png\" alt=\"Deploymet\" />\n",
    "\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>You can run jobs manually from here, review history of executions and view the predictions for a specific job</p>\n",
    "\n",
    "<img src=\"images/08_16.png\" alt=\"Deployment Run\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<img src=\"images/08_17.png\" alt=\"Deployment Jobs\" />\n",
    "\n",
    "<img src=\"images/08_18.png\" alt=\"Deployment view\" width=\"500\" height=\"500\" />\n",
    "\n",
    "<img src=\"images/08_19.png\" alt=\"Deployment predictions\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<img src=\"images/08_20.png\" alt=\"Deployment\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>From the Feature Drift and Prediction Drift tabs you can check on the monitoring of the data drift</p>\n",
    "\n",
    "<img src=\"images/08_22.png\" alt=\"Feature Drift\" />\n",
    "\n",
    "<img src=\"images/08_21.png\" alt=\"Prediction Drift\" />\n",
    "\n",
    "<img src=\"images/08_23.png\" alt=\"Performance Monitoring\" />\n",
    "\n",
    "\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>From the Performance Drift, you can review multiple evaluations, let's evaluate the model with a new dataset. We create a new evaluation dataset with this query:</p>\n",
    "    \n",
    "```sql\n",
    "SELECT * FROM pima_patient_diagnoses F WHERE F.patientid MOD 8 <> 0  \n",
    "```\n",
    "\n",
    "<img src=\"images/08_24.png\" alt=\"Evaluate\" width=\"500\" height=\"500\" />\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>and now see the evolution of the metrics</p>\n",
    "\n",
    "<img src=\"images/08_25.png\" alt=\"Metrics monitoring\" />\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "With ModelOps you can close the cycle and review make decisions when you need to replace yor model in production, For example, You could get alerting from Data Drift of Performance Drift and you can create multiple versions and compare them, select a champion and deploy new versions that replace existing in Production.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2c1554f-4c37-4a3a-a333-a96a505dafdf",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Go and try this Step by yourself. Launch ModelOps from this button below:</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7bbfe9b7-ab64-43b7-97e8-8c8457269820",
   "metadata": {},
   "source": [
    "[![image](images/launchModelOps.png)](/modelops)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3046954d-65b1-4939-9ff9-2e12411f3e7e",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial;color:#00233C'>10. Cleanup</b></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b210b57-bbc3-485b-a6cc-fbd01f06253f",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>If you are done with ModelOps usecase, please uncomment and run the below cleanup section.</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5bcd8c6-7d73-46e3-8a22-f81c30119256",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Work Tables</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Cleanup work tables to prevent errors next time.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "745e49a6-3cd3-49cd-9cfa-c0342c9d7fa0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# db_drop_table(table_name = 'aoa_byom_models', schema_name = 'demo_user')\n",
    "# db_drop_table(table_name = 'pima_patient_predictions', schema_name = 'demo_user')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9afd8630-f040-4583-af23-d0a30f8901ed",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'> <b>Databases and Tables </b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The following code will clean up tables and databases created above.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "22fbc041-b2d5-4926-bae2-0b1cee3d3b46",
   "metadata": {},
   "outputs": [],
   "source": [
    "# %run -i ../UseCases/run_procedure.py \"call remove_data('DEMO_ModelOps');\"        # Takes 10 seconds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ae5f8688-e4e8-46b8-9bba-2c88d9eefe59",
   "metadata": {},
   "outputs": [],
   "source": [
    "remove_context()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d459036-095d-4dc9-a082-1575ada88fa9",
   "metadata": {},
   "source": [
    "[<< Back to GIT Project Setup](./06_ModelOps_GIT_Project_Setup.ipynb) | [Continue to GIT PIMA H2OAutoML >>](./08_ModelOps_GIT_PIMA_Python_H2OAutoML.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c67c179b",
   "metadata": {},
   "source": [
    "<footer style=\"padding-bottom:35px; background:#f9f9f9; border-bottom:3px solid #00233C\">\n",
    "    <div style=\"float:left;margin-top:14px\">ClearScape Analytics™</div>\n",
    "    <div style=\"float:right;\">\n",
    "        <div style=\"float:left; margin-top:14px\">\n",
    "            Copyright © Teradata Corporation - 2023. All Rights Reserved\n",
    "        </div>\n",
    "    </div>\n",
    "</footer>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}