ModelOps/05_ModelOps_BYOM_PIMA_H2O.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<header>\n",
    "   <p  style='font-size:36px;font-family:Arial; color:#F0F0F0; background-color: #00233c; padding-left: 20pt; padding-top: 20pt;padding-bottom: 10pt; padding-right: 20pt;'>\n",
    "       ModelOps demo: H2O with ModelOps BYOM (no code)\n",
    "  <br>\n",
    "       <img id=\"teradata-logo\" src=\"https://storage.googleapis.com/clearscape_analytics_demo_data/DEMO_Logo/teradata.svg\" alt=\"Teradata\" style=\"width: 125px; height: auto; margin-top: 20pt;\">\n",
    "    </p>\n",
    "</header>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "![image](images/byom_meth.png) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>Introduction</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "This Notebook, we will show you how to work with the Bring Your Own Model (BYOM) pattern and BYOM In-Vantage Scoring. This deployment pattern allows you to use whatever data science platform you want to perform model training and import back to Vantage to do a direct deployment, scoring. For understanding more about BYOM you can review official user documentation.<br><br>\n",
    "When you use BYOM mechanisms you can deploy models directly from IDE environment, or make a proper operationalization through ModelOps. With ModelOps, you get <strong>full governance</strong> around your models deployment and you can <strong>leverage ModelOps automations</strong> for Validation, Scoring and Monitoring with an intuitive user interface that let you audit at any time all the information from your models and provide dashboards to monitor and review alerts from your deployments.<br><br>\n",
    "This notebook will cover the Operationalization of the PIMA diabetes use case with H2O model format. <strong>H2O</strong> is an open source, distributed in-memory machine learning library with linear scalability. H2O supports the most widely used statistical & machine learning algorithms including gradient boosted machines, generalized linear models, deep learning and more. The most used capabilities in H2O are the <strong>AutoML</strong> capabilities that provides. In this example we generate a H2O xgboost and operationalize it through ModelOps in the same Model Catalog than other trained models based on other libraries.\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Steps in this Notebook</b></p>\n",
    "\n",
    "<ol style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "    <li>Configure the Environment </li>\n",
    "    <li>Connect to Vantage</li>\n",
    "    <li>Train a model and export to H2O native format</li>\n",
    "    <li>Import the H2O into ModelOps</li>\n",
    "    <li>Go through Lifecycle - Evaluation (Automated Model Report)</li>\n",
    "    <li>Go through Lifecycle - Approve </li>\n",
    "    <li>Go through Lifecycle - Deploy (Publish and Schedule)</li>\n",
    "    <li>Go through Lifecycle - Monitor (Data Drift and Performance)</li>\n",
    "    <li>Configure Monitoring alert threshold (Optional) </li>\n",
    "    <li>On demand Scoring from SQL (Optional)</li>\n",
    "</ol>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<b style = 'font-size:20px;font-family:Arial;color:#00233C'>1. Configure the Environment</b>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Here, we import the required libraries, set environment variables and environment paths (if required).</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>1.1 Libraries installation</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'><b>A restart of the Kernel is needed to confirm changes</b>. We use -q parameter for a non-verbose log of the installation command, you may remove this parameter if you want to know all the steps of the pip installation.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#%pip install -q teradataml==17.20.0.6 teradatamodelops==7.0.3 h2o==3.44.0.3 install-jdk==1.0.4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'><b>Hint:</b><i>The easy way to restart the kernel to bring the above installed software into memory is to type zero zero (<b> 0 0 </b>). </i></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:1px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>1.2 Libraries import</b></p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from teradataml import *\n",
    "from teradatasqlalchemy.types import *\n",
    "import os\n",
    "import pandas as pd\n",
    "import getpass\n",
    "import logging\n",
    "import sys\n",
    "from xgboost import XGBClassifier\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "import h2o\n",
    "import os\n",
    "import jdk\n",
    "from h2o.estimators import H2OXGBoostEstimator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>2. Connect to Vantage and configure VAL-BYOM</b></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>You will be prompted to provide the password. Enter your password, press the Enter key, then use down arrow to go to next cell. Begin running steps with Shift + Enter keys.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%run -i ../UseCases/startup.ipynb\n",
    "eng = create_context(host = 'host.docker.internal', username = 'demo_user', password = password)\n",
    "print(eng)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%capture\n",
    "execute_sql('''SET query_band='DEMO=05_ModelOps_BYOM_PIMA_H2O;' UPDATE FOR SESSION;''')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# configure byom/val installation\n",
    "configure.val_install_location = \"VAL\"\n",
    "configure.byom_install_location = \"MLDB\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Getting Data for This Demo</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>We have provided data for this demo on cloud storage. You can either run the demo using foreign tables to access the data without any storage on your environment or download the data to local storage, which may yield faster execution. Still, there could be considerations of available storage. Two statements are in the following cell, and one is commented out. You may switch which mode you choose by changing the comment string.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#%run -i ../UseCases/run_procedure.py \"call get_data('DEMO_ModelOps_cloud');\"        # Takes 10 seconds\n",
    "%run -i ../UseCases/run_procedure.py \"call get_data('DEMO_ModelOps_local');\"        # Takes 30 seconds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:1px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Creating predictions and model table</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>We will create a predictions table where we get our model predictions and the model table where we will upload the model created.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#ddl for Aoa_Byom_Models \n",
    "query = '''CREATE SET TABLE Aoa_Byom_Models \n",
    "     (\n",
    "      model_version VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,\n",
    "      model_id VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,\n",
    "      model_type VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,\n",
    "      project_id VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,\n",
    "      deployed_at TIMESTAMP(6) DEFAULT CURRENT_TIMESTAMP(6),\n",
    "      model BLOB(2097088000))\n",
    "UNIQUE PRIMARY INDEX ( model_version );\n",
    "'''\n",
    " \n",
    "try:\n",
    "    execute_sql(query)\n",
    "except:\n",
    "    db_drop_table('Aoa_Byom_Models')\n",
    "    execute_sql(query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#ddl for Pima_Patient_Predictions\n",
    "query = '''CREATE MULTISET TABLE Pima_Patient_Predictions \n",
    "     (\n",
    "      job_id VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,\n",
    "      PatientId BIGINT,\n",
    "      HasDiabetes BIGINT,\n",
    "      json_report CLOB(1048544000) CHARACTER SET LATIN)\n",
    "PRIMARY INDEX ( job_id );;\n",
    "'''\n",
    " \n",
    "try:\n",
    "    execute_sql(query)\n",
    "except:\n",
    "    db_drop_table('Pima_Patient_Predictions')\n",
    "    execute_sql(query)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Next is an optional step – if you want to see the status of databases/tables created and space used.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run -i ../UseCases/run_procedure.py \"call space_report();\"        # Takes 10 seconds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>3. Train a model using H2O libraries</b></p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import h2o\n",
    "import os\n",
    "import jdk\n",
    "from h2o.estimators import H2OXGBoostEstimator\n",
    "\n",
    "# Check Java installation, installs if required\n",
    "try:\n",
    "    print(f\"Java is installed at {os.environ['JAVA_HOME']}\")\n",
    "except:\n",
    "    os.environ['JAVA_HOME'] = '/home/jovyan/.jdk/jdk-17.0.9+9'\n",
    "    if os.path.isdir(os.environ['JAVA_HOME']) is False:\n",
    "        print ('Installing Java...')\n",
    "        import jdk\n",
    "        jdk.install('17', path='/home/jovyan/.jdk')\n",
    "\n",
    "train_pdf = DataFrame.from_query(\"\"\"\n",
    "SELECT \n",
    "    F.*, D.hasdiabetes \n",
    "FROM Demo_Modelops.pima_patient_features F\n",
    "JOIN Demo_Modelops.pima_patient_diagnoses D\n",
    "    ON F.patientid = D.patientid \n",
    "    WHERE F.patientid MOD 5 <> 0\n",
    "\"\"\").to_pandas(all_rows=True)\n",
    "\n",
    "features = [\"NumTimesPrg\", \"Age\", \"PlGlcConc\", \"BloodP\", \"SkinThick\", \"TwoHourSerIns\", \"BMI\", \"DiPedFunc\"]\n",
    "target = \"HasDiabetes\"\n",
    "\n",
    "# Initialize H2O engine\n",
    "h2o.init()\n",
    "\n",
    "# Convert dataset to H2O\n",
    "train_hf=h2o.H2OFrame(train_pdf)\n",
    "\n",
    "# convert target column to categorical\n",
    "train_hf[target] = train_hf[target].asfactor()\n",
    "\n",
    "model = H2OXGBoostEstimator(eta=0.2, max_depth=6)\n",
    "model.train(x = features, y = target, training_frame = train_hf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:1px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Convert the model to H2O</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>You can save the model as h2o format.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "current_path = os.path.abspath(os.getcwd())\n",
    "artifacts_path = \"artifacts\"\n",
    "mojo = model.download_mojo(path=artifacts_path, get_genmodel_jar=True)\n",
    "new_mojo = os.path.join(current_path, artifacts_path, \"model.h2o\")\n",
    "if os.path.isfile(new_mojo):\n",
    "    print(\"The file already exists\")\n",
    "else:\n",
    "    # Rename the file\n",
    "    os.rename(mojo, new_mojo)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>4. Import into ModelOps to Operationalize</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "Note some of the images may contain PMML because the operationalization steps are the same. Go to the ModelOps UI and import this as a new model inside the <strong>ModelOps Getting Started</strong> project.\n",
    "</p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "Navigate to the projects screen and select the <strong>ModelOps Getting Started</strong> project. Click on it to navigate to the models screen of the project.\n",
    "</p>\n",
    "\n",
    "<img src=\"images/04_01.png\" alt=\"Projects screen\" />\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "If you haven't added a default connection, you will be prompted to add one. Check how to <a href=\"link here\">add a default connection</a>.\n",
    "</p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "Once there, click on the <strong>DEFINE BYOM MODEL</strong> button on the top right of the screen.\n",
    "</p>\n",
    "\n",
    "<img src=\"images/04_02.png\" alt=\"Models screen\" />\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "Ensure that <strong>Enable Model Monitoring</strong> is checked and classification as model type is selected. Then set the table the one you have been using (probably your username) and use as Prediction Expression:\n",
    "</p>\n",
    "\n",
    "```sql\n",
    "CAST(prediction AS INT)\n",
    "```\n",
    "\n",
    "<img src=\"images/04_03.png\" alt=\"BYOM model import sidesheet\" />\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "Then click on <strong>SAVE</strong>, and the import job will start.\n",
    "</p>\n",
    "\n",
    "<img src=\"images/04_04.png\" alt=\"BYOM model import job log\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Go and try this Step by yourself. Launch ModelOps from this button below:</p>\n",
    "<a href=\"/modelops\"><img src=\"images/launchModelOps.png\" alt=\"Launch ModelOps\" /></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>5. Go through Lifecycle - Evaluation (Automated Model Report)</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "Open the imported ModelID Lifecycle Screen. Review imported artifacts and evaluate the model with default logic (No code needed)\n",
    "</p>\n",
    "\n",
    "<img src=\"images/04_05.png\" alt=\"BYOM model lifecycle\"/>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "Select Evaluation dataset and run the evaluation process\n",
    "</p>\n",
    "\n",
    "<img src=\"images/04_06.png\" alt=\"Evaluation\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "Now you can review the Model evaluation report generated with default metrics and confusion matrix plot - this can be customized with a custom evaluation logic (later in this notebook)\n",
    "</p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "Click on View Report from ModelID Lifecycle screen\n",
    "</p>\n",
    "\n",
    "<img src=\"images/04_07.png\" alt=\"Model Evaluation Report\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Go and try this Step by yourself. Launch ModelOps from this button below:</p>\n",
    "<a href=\"/modelops\"><img src=\"images/launchModelOps.png\" alt=\"Launch ModelOps\" /></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>6. Go through Lifecycle - Approve</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "Open the imported ModelID Lifecycle Screen. Click on Approve button to move forward on the next stage.\n",
    "</p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "Include description of approval and accept.\n",
    "</p>\n",
    "\n",
    "<img src=\"images/04_08.png\" alt=\"Approval\" width=\"500\" height=\"500\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>7. Go through Lifecycle - Deploy (Publish and Schedule)</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "Open the imported ModelID Lifecycle Screen. Click on Deploy button to publish Model in Vantage.\n",
    "</p>\n",
    "\n",
    "<img src=\"images/04_09.png\" alt=\"Model Lifecycle screen\"/>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "Now select In-Vantage engine and click next.\n",
    "</p>\n",
    "\n",
    "<img src=\"images/04_10.png\" alt=\"Deployment Engine\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "Now select which Connection (here you could use a Service Connection to run in Production optionally), Database and table you want to publish your BYOM model. Use your user and the table \"aoa_byom_models\".\n",
    "</p>\n",
    "\n",
    "<img src=\"images/04_11.png\" alt=\"Deployment Publish\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "Now select if you want to schedule your model scoring, dataset template to gather where you want to store your predictions, and run the deploy process.\n",
    "</p>\n",
    "\n",
    "<img src=\"images/04_12.png\" alt=\"Deployment Schedule\" width=\"500\" height=\"500\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Go and try this Step by yourself. Launch ModelOps from this button below:</p>\n",
    "<a href=\"/modelops\"><img src=\"images/launchModelOps.png\" alt=\"Launch ModelOps\" /></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>8. Go through Lifecycle - Monitor (Data Drift and Performance)</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "Open the Deployment and review details of the deployed model.\n",
    "</p>\n",
    "\n",
    "<img src=\"images/04_14.png\" alt=\"Deployment details\"/>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "You can run the Prediction job from here. Click the button to run the prediction job.\n",
    "</p>\n",
    "\n",
    "<img src=\"images/04_15.png\" alt=\"Deployment Scoring Job\" />\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "You can also review the predictions. You can take this query and run it in your SQL IDE or in a Notebook later.\n",
    "</p>\n",
    "\n",
    "<img src=\"images/04_16.png\" alt=\"Deployment Predictions\" width=\"500\" height=\"500\"/>\n",
    "<img src=\"images/04_17.png\" alt=\"Deployment Predictions query\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "Review your Feature Drift and Prediction Drift. Here, ModelOps shows the comparison of the distribution of data between training and evaluation/scoring. Every scoring, this gets updated and different KPIs are calculated. The one we used by default for monitoring is the Population Stability Index (PSI).\n",
    "</p>\n",
    "\n",
    "<img src=\"images/04_18.png\" alt=\"Feature Drift\" />\n",
    "<img src=\"images/04_19.png\" alt=\"Prediction Drift\" />\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "For Performance Monitoring, we track over time the metrics of the model. To generate new metrics, we need to create a new evaluation dataset and run an evaluation job.\n",
    "</p>\n",
    "\n",
    "<img src=\"images/04_20.png\" alt=\"Performance Drift\" />\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "This has been generated using this query for the evaluation dataset target:\n",
    "</p>\n",
    "\n",
    "```sql\n",
    "SELECT * FROM pima_patient_diagnoses F WHERE F.patientid MOD 8 <> 0\n",
    "```\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "Now the performance has changed:\n",
    "</p>\n",
    "\n",
    "<img src=\"images/04_21.png\" alt=\"Performance Drift change\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Go and try this Step by yourself. Launch ModelOps from this button below:</p>\n",
    "<a href=\"/modelops\"><img src=\"images/launchModelOps.png\" alt=\"Launch ModelOps\" /></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>9. Configure Monitoring alert threshold (Optional)</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "We can update the Alerts configuration for the models.\n",
    "</p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "First, enable Alert from Model catalog.\n",
    "</p>\n",
    "\n",
    "<img src=\"images/04_23.png\" alt=\"Alert Configuration\" />\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "Now, you can change your default alerting mechanism.\n",
    "</p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "Go to your Model and find the Tab \"Alert Configuration\".\n",
    "</p>\n",
    "\n",
    "<img src=\"images/04_22.png\" alt=\"Alert Configuration\" />\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "Let's change the PSI value to 0.01 instead of 0.2. Use the Edit option:\n",
    "</p>\n",
    "\n",
    "<img src=\"images/04_24.png\" alt=\"Alert Configuration PSI\" width=\"500\" height=\"500\"/>\n",
    "<img src=\"images/04_25.png\" alt=\"Alert Configuration PSI\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "After a few minutes, this has generated new alerts on the deployed model.\n",
    "</p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "Go to Menu -> Alert and review the alerts generated for your model.\n",
    "</p>\n",
    "\n",
    "<img src=\"images/04_26.png\" alt=\"Alerts\" />\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "Now, go to ModelID and click on View Model Drift.\n",
    "</p>\n",
    "\n",
    "<img src=\"images/04_27.png\" alt=\"Model Lifecycle View Model Drift\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "Now, review the Feature drift. You can check this screen from your deployments as well.\n",
    "</p>\n",
    "\n",
    "<img src=\"images/04_28.png\" alt=\"Feature Drift\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Go and try this Step by yourself. Launch ModelOps from this button below:</p>\n",
    "<a href=\"/modelops\"><img src=\"images/launchModelOps.png\" alt=\"Launch ModelOps\" /></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>10. On demand Scoring from SQL (Optional)</b></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>10.1 View Published Models</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Once deployed via ModelOps, we can view the models published to vantage by querying the table they are published to. Note this information is available via the AOA APIs also.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "DataFrame.from_query(\"SELECT TOP 10 * FROM aoa_byom_models WHERE model_type='H2O'\").head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:1px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>10.2 On-Demand Scoring</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Replace model_version with the one we have operationalized:</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#teradataml version\n",
    "model_version=\"c14a3544-1cb3-4f96-aeec-9a6cfd67daad\"\n",
    "\n",
    "model = DataFrame.from_query(f\"\"\"\n",
    "SELECT * FROM aoa_byom_models \n",
    "    WHERE model_version='{model_version}'\n",
    "\"\"\")\n",
    "\n",
    "\n",
    "preds = H2OPredict(\n",
    "        modeldata=model,\n",
    "        newdata=DataFrame.from_query(\"SELECT * FROM Demo_ModelOps.pima_patient_features WHERE patientid MOD 5 = 0\"),\n",
    "        accumulate=['PatientId'])\n",
    "\n",
    "preds.result.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#SQL version\n",
    "query = f\"\"\"\n",
    "SELECT * FROM MLDB.H2OPredict (\n",
    "    ON (SELECT * FROM Demo_ModelOps.pima_patient_features WHERE patientid MOD 5 = 0) AS InputTable\n",
    "    ON (SELECT * FROM aoa_byom_models \n",
    "            WHERE model_version='{model_version}') AS ModelTable DIMENSION\n",
    "    USING\n",
    "      Accumulate ('patientid')\n",
    ") AS td;\n",
    "\"\"\"\n",
    "\n",
    "DataFrame.from_query(query).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>11. Cleanup</b></p>\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "    <p style = 'font-size:16px;font-family:Arial;color:#00233C'>If you are done with ModelOps usecase, please uncomment and run the below cleanup section.</p>\n",
    "</div>\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Work Tables</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Cleanup work tables to prevent errors next time.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# db_drop_table(table_name = 'aoa_byom_models', schema_name = 'demo_user')\n",
    "# db_drop_table(table_name = 'pima_patient_predictions', schema_name = 'demo_user')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'> <b>Databases and Tables </b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The following code will clean up tables and databases created above.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %run -i ../UseCases/run_procedure.py \"call remove_data('DEMO_ModelOps');\"        # Takes 10 seconds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "remove_context()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[<< Back to PIMA ONNX BYOM](./04_ModelOps_BYOM_PIMA_ONNX.ipynb) | [Continue to Git Project Setup >>](./06_ModelOps_GIT_Project_Setup.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<footer style=\"padding-bottom:35px; background:#f9f9f9; border-bottom:3px solid #00233C\">\n",
    "    <div style=\"float:left;margin-top:14px\">ClearScape Analytics™</div>\n",
    "    <div style=\"float:right;\">\n",
    "        <div style=\"float:left; margin-top:14px\">\n",
    "            Copyright © Teradata Corporation - 2023. All Rights Reserved\n",
    "        </div>\n",
    "    </div>\n",
    "</footer>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}