ModelOps/09_ModelOps_GIT_PIMA_Python_XGboost.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0ed076ee",
   "metadata": {},
   "source": [
    "<header>\n",
    "   <p  style='font-size:36px;font-family:Arial; color:#F0F0F0; background-color: #00233c; padding-left: 20pt; padding-top: 20pt;padding-bottom: 10pt; padding-right: 20pt;'>\n",
    "       ModelOps demo: Python XGBoost using Git\n",
    "  <br>\n",
    "       <img id=\"teradata-logo\" src=\"https://storage.googleapis.com/clearscape_analytics_demo_data/DEMO_Logo/teradata.svg\" alt=\"Teradata\" style=\"width: 125px; height: auto; margin-top: 20pt;\">\n",
    "    </p>\n",
    "</header>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "260bf2ce-6b27-468d-9200-1e928d8709c2",
   "metadata": {
    "tags": []
   },
   "source": [
    "![image](images/git_meth.png) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e628a88",
   "metadata": {},
   "source": [
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>Introduction</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>This notebook will cover the Operationalization of the PIMA diabetes use case with Python XGBoost model format. <strong>XGBoost</strong> is an optimized distributed gradient boosting library designed to be highly efficient, flexible and portable. It is one of the most used libraries by the community that solve many data science problems in a fast and accurate way.</p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>In this example, we will use the XGBoost algorithm to generate both <strong>Python Joblib and PMML</strong> model formats and operationalize them through ModelOps in the same Model Catalog as other trained models based on other libraries.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e89a61f-22f2-4ea0-8189-aee3e232a487",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Steps in this Notebook</b></p>\n",
    "\n",
    "<ol style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "    <li>Configure the Environment </li>\n",
    "    <li>Connect to Vantage</li>\n",
    "    <li>Define Training function </li>\n",
    "    <li>Define Evaluate function </li>\n",
    "    <li>Define Scoring function</li>\n",
    "    <li>Define Model Metadata</li>\n",
    "    <li>Commit and Push to Git to let ModelOps manage</li>\n",
    "    <li>ModelOps full lifecycle till deployment</li>\n",
    "    <li>ModelOps Monitoring</li>\n",
    "</ol>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e20e6a1-d566-46a7-a7ca-c01470689998",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>1. Configure the Environment</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Here, we import the required libraries, set environment variables and environment paths (if required).</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5311af5-5087-44e6-a538-8e3d59dec9e0",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>1.1 Libraries installation</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'><b>A restart of the Kernel is needed to confirm changes</b>. We use -q parameter for a non-verbose log of the installation command, you may remove this parameter if you want to know all the steps of the pip installation.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31866de8",
   "metadata": {},
   "outputs": [],
   "source": [
    "#%pip install -q teradatamodelops==7.0.3 nyoka==4.3.0 matplotlib==3.8.2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "753dcf0f-7daf-4e80-beae-7cf340d6259a",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'><b>Hint:</b><i>The easy way to restart the kernel to bring the above installed software into memory is to type zero zero (<b> 0 0 </b>). </i></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe998588",
   "metadata": {},
   "source": [
    "<hr style=\"height:1px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>1.2 Libraries import</b></p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c6fbde24",
   "metadata": {},
   "outputs": [],
   "source": [
    "from teradataml import *\n",
    "import os\n",
    "import getpass\n",
    "import logging\n",
    "import sys"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95564a11-c549-43b8-9904-f5a06241cd67",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>2. Connect to Vantage</b></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4a1f46c-7798-4fc9-a710-750c56e8a299",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>You will be prompted to provide the password. Enter your password, press the Enter key, then use down arrow to go to next cell. Begin running steps with Shift + Enter keys.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f50e37d9-3c0b-470f-b50b-378c24b7eda4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%run -i ../UseCases/startup.ipynb\n",
    "eng = create_context(host = 'host.docker.internal', username='demo_user', password = password)\n",
    "print(eng)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "65f6d9c5-471c-4d9e-9863-f20daa6a8d95",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%capture\n",
    "execute_sql('''SET query_band='DEMO=09_ModelOps_GIT_PIMA_Python_XGboost.ipynb;' UPDATE FOR SESSION; ''')\n",
    "\n",
    "# configure byom/val installation\n",
    "configure.val_install_location = \"VAL\"\n",
    "configure.byom_install_location = \"MLDB\"\n",
    "\n",
    "# set the path to the local project repository for this model demo\n",
    "model_local_path = '~/modelops-demo-models/model_definitions/pima_python_xgboost'\n",
    "res = os.system(f'mkdir -p {model_local_path}/model_modules')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13a38043-9878-424d-b459-6b196735e5ca",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Getting Data for This Demo</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>We have provided data for this demo on cloud storage. You can either run the demo using foreign tables to access the data without any storage on your environment or download the data to local storage, which may yield faster execution. Still, there could be considerations of available storage. Two statements are in the following cell, and one is commented out. You may switch which mode you choose by changing the comment string.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "23f3b523-2df3-4f13-afbc-1f7a88d60221",
   "metadata": {},
   "outputs": [],
   "source": [
    "#%run -i ../UseCases/run_procedure.py \"call get_data('DEMO_ModelOps_cloud');\"        # Takes 10 seconds\n",
    "%run -i ../UseCases/run_procedure.py \"call get_data('DEMO_ModelOps_local');\"        # Takes 30 seconds"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "645f20f0-efbc-42fb-abfc-514fc24bb5f4",
   "metadata": {},
   "source": [
    "<hr style=\"height:1px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Creating predictions and model table</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>We will create a predictions table where we get our model predictions and the model table where we will upload the model created.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "07a3e8f5-2ee8-4ace-83b7-e56c083effab",
   "metadata": {},
   "outputs": [],
   "source": [
    "#ddl for Aoa_Byom_Models \n",
    "query = '''CREATE SET TABLE Aoa_Byom_Models \n",
    "     (\n",
    "      model_version VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,\n",
    "      model_id VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,\n",
    "      model_type VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,\n",
    "      project_id VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,\n",
    "      deployed_at TIMESTAMP(6) DEFAULT CURRENT_TIMESTAMP(6),\n",
    "      model BLOB(2097088000))\n",
    "UNIQUE PRIMARY INDEX ( model_version );\n",
    "'''\n",
    " \n",
    "try:\n",
    "    execute_sql(query)\n",
    "except:\n",
    "    db_drop_table('Aoa_Byom_Models')\n",
    "    execute_sql(query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b3d6c0b9-f6b9-416c-b788-11cd8a6a9723",
   "metadata": {},
   "outputs": [],
   "source": [
    "#ddl for Pima_Patient_Predictions\n",
    "query = '''CREATE MULTISET TABLE Pima_Patient_Predictions \n",
    "     (\n",
    "      job_id VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,\n",
    "      PatientId BIGINT,\n",
    "      HasDiabetes BIGINT,\n",
    "      json_report CLOB(1048544000) CHARACTER SET LATIN)\n",
    "PRIMARY INDEX ( job_id );;\n",
    "'''\n",
    " \n",
    "try:\n",
    "    execute_sql(query)\n",
    "except:\n",
    "    db_drop_table('Pima_Patient_Predictions')\n",
    "    execute_sql(query)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4c1c700-ba57-4c31-9fd2-3434023e6640",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Next is an optional step – if you want to see the status of databases/tables created and space used.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be4b98a7-ed3a-4873-89b9-538bfbdc51a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "%run -i ../UseCases/run_procedure.py \"call space_report();\"        # Takes 10 seconds"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "614f11fd",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>3. Define Training Function</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The training function takes the following shape</p>\n",
    "\n",
    "```python\n",
    "def train(context: ModelContext, **kwargs):\n",
    "    aoa_create_context()\n",
    "    \n",
    "    # your training code\n",
    "    \n",
    "    # save your model\n",
    "    joblib.dump(model, f\"{context.artifact_output_path}/model.joblib\")\n",
    "    \n",
    "    record_training_stats(...)\n",
    "```\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>You can execute this from the CLI or directly within the notebook as shown.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a7fb724",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile $model_local_path/model_modules/training.py\n",
    "from xgboost import XGBClassifier\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "from sklearn.pipeline import Pipeline\n",
    "from nyoka import xgboost_to_pmml\n",
    "from teradataml import DataFrame\n",
    "from aoa import (\n",
    "    record_training_stats,\n",
    "    save_plot,\n",
    "    aoa_create_context,\n",
    "    ModelContext\n",
    ")\n",
    "\n",
    "import joblib\n",
    "\n",
    "\n",
    "def train(context: ModelContext, **kwargs):\n",
    "    aoa_create_context()\n",
    "\n",
    "    feature_names = context.dataset_info.feature_names\n",
    "    target_name = context.dataset_info.target_names[0]\n",
    "\n",
    "    # read training dataset from Teradata and convert to pandas\n",
    "    train_df = DataFrame.from_query(context.dataset_info.sql)\n",
    "    train_pdf = train_df.to_pandas(all_rows=True)\n",
    "\n",
    "    # split data into X and y\n",
    "    X_train = train_pdf[feature_names]\n",
    "    y_train = train_pdf[target_name]\n",
    "\n",
    "    print(\"Starting training...\")\n",
    "\n",
    "    # fit model to training data\n",
    "    model = Pipeline([('scaler', MinMaxScaler()),\n",
    "                      ('xgb', XGBClassifier(eta=context.hyperparams[\"eta\"],\n",
    "                                            max_depth=context.hyperparams[\"max_depth\"]))])\n",
    "\n",
    "    model.fit(X_train, y_train)\n",
    "\n",
    "    print(\"Finished training\")\n",
    "\n",
    "    # export model artefacts\n",
    "    joblib.dump(model, f\"{context.artifact_output_path}/model.joblib\")\n",
    "\n",
    "    # we can also save as pmml so it can be used for In-Vantage scoring etc.\n",
    "    xgboost_to_pmml(pipeline=model, col_names=feature_names, target_name=target_name,\n",
    "                    pmml_f_name=f\"{context.artifact_output_path}/model.pmml\")\n",
    "\n",
    "    print(\"Saved trained model\")\n",
    "\n",
    "    from xgboost import plot_importance\n",
    "    model[\"xgb\"].get_booster().feature_names = feature_names\n",
    "    plot_importance(model[\"xgb\"].get_booster(), max_num_features=10)\n",
    "    save_plot(\"feature_importance.png\", context=context)\n",
    "\n",
    "    feature_importance = model[\"xgb\"].get_booster().get_score(importance_type=\"weight\")\n",
    "\n",
    "    print(\"Recording training stats\")\n",
    "\n",
    "    record_training_stats(train_df,\n",
    "                          features=feature_names,\n",
    "                          targets=[target_name],\n",
    "                          categorical=[target_name],\n",
    "                          feature_importance=feature_importance,\n",
    "                          context=context)\n",
    "    \n",
    "    print(\"All done!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c437521",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the ModelContext to test with. The ModelContext is created and managed automatically by ModelOps \n",
    "# when it executes your code via CLI / UI. However, for testing in the notebook, you can define as follows\n",
    "\n",
    "# define the training dataset \n",
    "sql = \"\"\"\n",
    "SELECT \n",
    "    F.*, D.hasdiabetes\n",
    "FROM DEMO_ModelOps.PIMA_PATIENT_FEATURES F \n",
    "JOIN DEMO_ModelOps.PIMA_PATIENT_DIAGNOSES D\n",
    "ON F.patientid = D.patientid\n",
    "    WHERE D.patientid MOD 5 <> 0\n",
    "\"\"\"\n",
    "\n",
    "feature_metadata =  {\n",
    "    \"database\": \"DEMO_ModelOps\",\n",
    "    \"table\": \"aoa_statistics_metadata\"\n",
    "}\n",
    "hyperparams = {\"max_depth\": 5, \"eta\": 0.2}\n",
    "\n",
    "entity_key = \"PatientId\"\n",
    "target_names = [\"HasDiabetes\"]\n",
    "feature_names = [\"NumTimesPrg\", \"PlGlcConc\", \"BloodP\", \"SkinThick\", \"TwoHourSerIns\", \"BMI\", \"DiPedFunc\", \"Age\"]\n",
    " \n",
    "from aoa import ModelContext, DatasetInfo\n",
    "\n",
    "dataset_info = DatasetInfo(sql=sql,\n",
    "                           entity_key=entity_key,\n",
    "                           feature_names=feature_names,\n",
    "                           target_names=target_names,\n",
    "                           feature_metadata=feature_metadata)\n",
    "\n",
    "ctx = ModelContext(hyperparams=hyperparams,\n",
    "                   dataset_info=dataset_info,\n",
    "                   artifact_output_path=\"./artifacts\",\n",
    "                   model_version=\"python_xgboost_v1\",\n",
    "                   model_table=\"model_python_xgboost_v1\")\n",
    "\n",
    "sys.path.append(os.path.expanduser(f\"{model_local_path}/model_modules\"))\n",
    "import training\n",
    "training.train(context=ctx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15cf5f8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check the generated files\n",
    "!ls -lh artifacts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f35e4a9",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>4. Define Evaluation Function</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The evaluation function takes the following shape</p>\n",
    "\n",
    "```python\n",
    "def evaluate(context: ModelContext, **kwargs):\n",
    "    aoa_create_context()\n",
    "\n",
    "    # read your model\n",
    "    model = joblib.load(f\"{context.artifact_input_path}/model.joblib\")\n",
    "    \n",
    "    # your evaluation logic\n",
    "    \n",
    "    record_evaluation_stats(...)\n",
    "```\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>You can execute this from the CLI or directly within the notebook as shown.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d751b6b",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile $model_local_path/model_modules/evaluation.py\n",
    "from sklearn import metrics\n",
    "from teradataml import DataFrame, copy_to_sql\n",
    "from aoa import (\n",
    "    record_evaluation_stats,\n",
    "    save_plot,\n",
    "    aoa_create_context,\n",
    "    ModelContext\n",
    ")\n",
    "\n",
    "import joblib\n",
    "import json\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "\n",
    "def evaluate(context: ModelContext, **kwargs):\n",
    "\n",
    "    aoa_create_context()\n",
    "\n",
    "    model = joblib.load(f\"{context.artifact_input_path}/model.joblib\")\n",
    "\n",
    "    feature_names = context.dataset_info.feature_names\n",
    "    target_name = context.dataset_info.target_names[0]\n",
    "\n",
    "    test_df = DataFrame.from_query(context.dataset_info.sql)\n",
    "    test_pdf = test_df.to_pandas(all_rows=True)\n",
    "\n",
    "    X_test = test_pdf[feature_names]\n",
    "    y_test = test_pdf[target_name]\n",
    "\n",
    "    print(\"Scoring\")\n",
    "    y_pred = model.predict(X_test)\n",
    "\n",
    "    y_pred_tdf = pd.DataFrame(y_pred, columns=[target_name])\n",
    "    y_pred_tdf[\"PatientId\"] = test_pdf[\"PatientId\"].values\n",
    "\n",
    "    evaluation = {\n",
    "        'Accuracy': '{:.2f}'.format(metrics.accuracy_score(y_test, y_pred)),\n",
    "        'Recall': '{:.2f}'.format(metrics.recall_score(y_test, y_pred)),\n",
    "        'Precision': '{:.2f}'.format(metrics.precision_score(y_test, y_pred)),\n",
    "        'f1-score': '{:.2f}'.format(metrics.f1_score(y_test, y_pred))\n",
    "    }\n",
    "\n",
    "    with open(f\"{context.artifact_output_path}/metrics.json\", \"w+\") as f:\n",
    "        json.dump(evaluation, f)\n",
    "\n",
    "    metrics.plot_confusion_matrix(model, X_test, y_test)\n",
    "    save_plot('Confusion Matrix', context=context)\n",
    "\n",
    "    metrics.plot_roc_curve(model, X_test, y_test)\n",
    "    save_plot('ROC Curve', context=context)\n",
    "\n",
    "    from xgboost import plot_importance\n",
    "    model[\"xgb\"].get_booster().feature_names = feature_names\n",
    "    plot_importance(model[\"xgb\"].get_booster(), max_num_features=10)\n",
    "    save_plot(\"feature_importance.png\", context=context)\n",
    "\n",
    "    feature_importance = model[\"xgb\"].get_booster().get_score(importance_type=\"weight\")\n",
    "\n",
    "    predictions_table = \"predictions_tmp\"\n",
    "    copy_to_sql(df=y_pred_tdf, table_name=predictions_table, index=False, if_exists=\"replace\", temporary=True)\n",
    "\n",
    "    record_evaluation_stats(features_df=test_df,\n",
    "                            predicted_df=DataFrame.from_query(f\"SELECT * FROM {predictions_table}\"),\n",
    "                            feature_importance=feature_importance,\n",
    "                            context=context)\n",
    "\n",
    "    print(\"All done!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "12ba7a9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the ModelContext to test with. The ModelContext is created and managed automatically by ModelOps \n",
    "# when it executes your code via CLI / UI. However, for testing in the notebook, you can define as follows\n",
    "\n",
    "# define the evaluation dataset \n",
    "sql = \"\"\"\n",
    "SELECT \n",
    "    F.*, D.hasdiabetes \n",
    "FROM DEMO_ModelOps.PIMA_PATIENT_FEATURES F \n",
    "JOIN DEMO_ModelOps.PIMA_PATIENT_DIAGNOSES D\n",
    "ON F.patientid = D.patientid\n",
    "    WHERE D.patientid MOD 5 = 0\n",
    "\"\"\"\n",
    "\n",
    "dataset_info = DatasetInfo(sql=sql,\n",
    "                           entity_key=entity_key,\n",
    "                           feature_names=feature_names,\n",
    "                           target_names=target_names,\n",
    "                           feature_metadata=feature_metadata)\n",
    "\n",
    "ctx = ModelContext(hyperparams=hyperparams,\n",
    "                   dataset_info=dataset_info,\n",
    "                   artifact_output_path=\"./artifacts\",\n",
    "                   artifact_input_path=\"./artifacts\",\n",
    "                   model_version=\"python_xgboost_v1\",\n",
    "                   model_table=\"model_python_xgboost_v1\")\n",
    "\n",
    "import evaluation\n",
    "evaluation.evaluate(context=ctx)\n",
    "\n",
    "# view evaluation results\n",
    "import json\n",
    "with open(f\"{ctx.artifact_output_path}/metrics.json\") as f:\n",
    "    print(json.load(f))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ce4ba2d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check the generated files\n",
    "!ls -lh artifacts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6beeb59",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>5. Define Scoring Function</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The scoring function takes the following shape</p>\n",
    "\n",
    "```python\n",
    "def score(context: ModelContext, **kwargs):\n",
    "    aoa_create_context()\n",
    "\n",
    "    # read your model\n",
    "    model = joblib.load(f\"{context.artifact_input_path}/model.joblib\")\n",
    "    \n",
    "    # your evaluation logic\n",
    "    \n",
    "    record_scoring_stats(...)\n",
    "```\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>You can execute this from the CLI or directly within the notebook as shown.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cb4d4c1c",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile $model_local_path/model_modules/scoring.py\n",
    "from teradataml import copy_to_sql, DataFrame\n",
    "from aoa import (\n",
    "    record_scoring_stats,\n",
    "    aoa_create_context,\n",
    "    ModelContext\n",
    ")\n",
    "\n",
    "import joblib\n",
    "import pandas as pd\n",
    "\n",
    "\n",
    "def score(context: ModelContext, **kwargs):\n",
    "\n",
    "    aoa_create_context()\n",
    "\n",
    "    model = joblib.load(f\"{context.artifact_input_path}/model.joblib\")\n",
    "\n",
    "    feature_names = context.dataset_info.feature_names\n",
    "    target_name = context.dataset_info.target_names[0]\n",
    "    entity_key = context.dataset_info.entity_key\n",
    "\n",
    "    features_tdf = DataFrame.from_query(context.dataset_info.sql)\n",
    "    features_pdf = features_tdf.to_pandas(all_rows=True)\n",
    "\n",
    "    print(\"Scoring\")\n",
    "    predictions_pdf = model.predict(features_pdf[feature_names])\n",
    "\n",
    "    print(\"Finished Scoring\")\n",
    "\n",
    "    # store the predictions\n",
    "    predictions_pdf = pd.DataFrame(predictions_pdf, columns=[target_name])\n",
    "    predictions_pdf[entity_key] = features_pdf.index.values\n",
    "    # add job_id column so we know which execution this is from if appended to predictions table\n",
    "    predictions_pdf[\"job_id\"] = context.job_id\n",
    "\n",
    "    # teradataml doesn't match column names on append.. and so to match / use same table schema as for byom predict\n",
    "    # example (see README.md), we must add empty json_report column and change column order manually (v17.0.0.4)\n",
    "    # CREATE MULTISET TABLE pima_patient_predictions\n",
    "    # (\n",
    "    #     job_id VARCHAR(255), -- comes from airflow on job execution\n",
    "    #     PatientId BIGINT,    -- entity key as it is in the source data\n",
    "    #     HasDiabetes BIGINT,   -- if model automatically extracts target\n",
    "    #     json_report CLOB(1048544000) CHARACTER SET UNICODE  -- output of\n",
    "    # )\n",
    "    # PRIMARY INDEX ( job_id );\n",
    "    predictions_pdf[\"json_report\"] = \"\"\n",
    "    predictions_pdf = predictions_pdf[[\"job_id\", entity_key, target_name, \"json_report\"]]\n",
    "\n",
    "    copy_to_sql(df=predictions_pdf,\n",
    "                schema_name=context.dataset_info.predictions_database,\n",
    "                table_name=context.dataset_info.predictions_table,\n",
    "                index=False,\n",
    "                if_exists=\"append\")\n",
    "    \n",
    "    print(\"Saved predictions in Teradata\")\n",
    "\n",
    "    # calculate stats\n",
    "    predictions_df = DataFrame.from_query(f\"\"\"\n",
    "        SELECT \n",
    "            * \n",
    "        FROM {context.dataset_info.get_predictions_metadata_fqtn()} \n",
    "            WHERE job_id = '{context.job_id}'\n",
    "    \"\"\")\n",
    "\n",
    "    record_scoring_stats(features_df=features_tdf,\n",
    "                         predicted_df=predictions_df,\n",
    "                         context=context)\n",
    "\n",
    "    print(\"All done!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0edf735d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the ModelContext to test with. The ModelContext is created and managed automatically by ModelOps \n",
    "# when it executes your code via CLI / UI. However, for testing in the notebook, you can define as follows\n",
    "\n",
    "# define the scoring dataset \n",
    "\n",
    "sql = \"\"\"\n",
    "SELECT \n",
    "    F.*\n",
    "FROM DEMO_ModelOps.PIMA_PATIENT_FEATURES F \n",
    "    WHERE F.patientid MOD 5 = 0\n",
    "\"\"\"\n",
    "\n",
    "# where to store predictions\n",
    "predictions = {\n",
    "    \"database\": \"demo_user\",\n",
    "    \"table\": \"pima_patient_predictions_tmp\"\n",
    "}\n",
    "\n",
    "import uuid\n",
    "job_id=str(uuid.uuid4())\n",
    "\n",
    "dataset_info = DatasetInfo(sql=sql,\n",
    "                           entity_key=entity_key,\n",
    "                           feature_names=feature_names,\n",
    "                           target_names=target_names,\n",
    "                           feature_metadata=feature_metadata,\n",
    "                           predictions=predictions)\n",
    "\n",
    "ctx = ModelContext(hyperparams=hyperparams,\n",
    "                   dataset_info=dataset_info,\n",
    "                   artifact_output_path=\"./artifacts\",\n",
    "                   artifact_input_path=\"./artifacts\",\n",
    "                   model_version=\"python_xgboost_v1\",\n",
    "                   model_table=\"model_python_xgboost_v1\",\n",
    "                   job_id=job_id)\n",
    "\n",
    "import scoring\n",
    "scoring.score(context=ctx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e27b2370",
   "metadata": {},
   "outputs": [],
   "source": [
    "DataFrame.from_query(f\"SELECT * FROM pima_patient_predictions_tmp WHERE job_id='{job_id}'\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "06db76e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Clean up\n",
    "\n",
    "os.system('rm -f artifacts/*')\n",
    "\n",
    "try:\n",
    "    db_drop_table('model_python_xgboost_v1')\n",
    "except: \n",
    "    pass\n",
    "\n",
    "try:\n",
    "    db_drop_table('pima_patient_predictions_tmp')\n",
    "except: \n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "818ec1cf",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>6. Define Model Metadata</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Now let's create the configuration files.</p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Requirements file with the dependencies and versions:</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11ee3f52",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile $model_local_path/model_modules/requirements.txt\n",
    "xgboost==0.90\n",
    "scikit-learn==1.3.2\n",
    "teradataml==17.20.0.6\n",
    "nyoka==4.3.0\n",
    "teradatamodelops==7.0.3\n",
    "matplotlib==3.8.2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55f9f7d1",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The hyper parameter configuration (default values):</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8711bb8e",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile $model_local_path/config.json\n",
    "{\n",
    "   \"hyperParameters\": {\n",
    "      \"eta\": 0.2,\n",
    "      \"max_depth\": 6\n",
    "   }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00ed3d03",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The model configuration:</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "224d5f9b",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile $model_local_path/model.json\n",
    "{\n",
    "    \"id\": \"5b41d4d4-7236-54ab-846a-01c3151e1fd9\",\n",
    "    \"name\": \"Python PIMA Quickstart\",\n",
    "    \"description\": \"Python PIMA Quickstart for Diabetes Prediction\",\n",
    "    \"language\": \"python\"\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8cffb5da",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>7.Commit and push changes into GIT to manage from ModelOps</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Run the command below to commit and push changes to our forked repository, so ModelOps can fetch the changes to the model.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "36decf79",
   "metadata": {},
   "outputs": [],
   "source": [
    "!cd $model_local_path/../.. && git add . && git commit -m \"Added PIMA Quickstart demo model ⚡\" && git push"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6c7d023",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Now that changes are pushed, you can make the lifecycle inside <strong>ModelOps User Interface</strong>, plan for new trainings, evaluations, scorings. Compare models and operationalize into Production with automated Monitoring and alerting capabilities.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ea5da5e",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>8. ModelOps full lifecycle till deployment</b></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4a34b63",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Use or Create a Project with the git code repository with the model code, then you should see the model in the catalog already created</p>\n",
    "\n",
    "<img src=\"images/08_01.png\" alt=\"Model Catalog with inDB\"/>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Select the Model and then click Train a new Model. Use default hyper-parameters. This will launch the training job with the training script we generated and pushed to Git.</p>\n",
    "\n",
    "<img src=\"images/08_02.png\" alt=\"Train\"/>\n",
    "\n",
    "<img src=\"images/08_03.png\" alt=\"Train job\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<img src=\"images/08_04.png\" alt=\"Train finished\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>When Model is trained a new Model Id is created and you can get inside the Model Lifecycle screen to review artifacts and other details</p>\n",
    "\n",
    "<img src=\"images/08_06.png\" alt=\"Model lifecycle\"/>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Now, let's evaluate the Model, click the button and select the evaluation dataset. This will launch the evaluation job with the training script we generated and pushed to Git.</p>\n",
    "\n",
    "<img src=\"images/08_07.png\" alt=\"Evaluation\" width=\"500\" height=\"500\"/> <img src=\"images/08_08.png\" alt=\"Evaluation job\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>When evaluation job is finished a Model evaluation Report is generated with the metrics and charts that evaluation script generates</p>\n",
    "\n",
    "<img src=\"images/08_26.png\" alt=\"Model Report\" />\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Now, let's approve the model and provide an approval description</p>\n",
    "\n",
    "<img src=\"images/08_09.png\" alt=\"Approval\" />\n",
    "\n",
    "<img src=\"images/08_10.png\" alt=\"Approval description\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The model is ready to be deployed. Let's deploy using a Batch scheduling option - Run it manual</p>\n",
    "\n",
    "<img src=\"images/08_11.png\" alt=\"Deployment Engine\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<img src=\"images/08_12.png\" alt=\"Deployment Publish\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<img src=\"images/08_13.png\" alt=\"Deployment Schedule\" width=\"500\" height=\"500\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fe60441-f62e-46c1-8baa-ff3cb0f1b809",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Go and try this Step by yourself. Launch ModelOps from this button below:</p>\n",
    "<a href=\"/modelops\"><img src=\"images/launchModelOps.png\" alt=\"Launch ModelOps\" /></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "429cab80",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>9. ModelOps Monitoring</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Now the model is deployed and a new Deployment appears in the deployment screen</p>\n",
    "\n",
    "<img src=\"images/08_15.png\" alt=\"Deploymet\" />\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>You can run jobs manually from here, review history of executions and view the predictions for a specific job</p>\n",
    "\n",
    "<img src=\"images/08_16.png\" alt=\"Deployment Run\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<img src=\"images/08_17.png\" alt=\"Deployment Jobs\" />\n",
    "\n",
    "<img src=\"images/08_18.png\" alt=\"Deployment view\" width=\"500\" height=\"500\" />\n",
    "\n",
    "<img src=\"images/08_19.png\" alt=\"Deployment predictions\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<img src=\"images/08_20.png\" alt=\"Deployment\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>From the Feature Drift and Prediction Drift tabs you can check on the monitoring of the data drift</p>\n",
    "\n",
    "<img src=\"images/08_22.png\" alt=\"Feature Drift\" />\n",
    "\n",
    "<img src=\"images/08_21.png\" alt=\"Prediction Drift\" />\n",
    "\n",
    "<img src=\"images/08_23.png\" alt=\"Performance Monitoring\" />\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>From the Performance Drift, you can review multiple evaluations, let's evaluate the model with a new dataset. We create a new evaluation dataset with this query:</p>\n",
    "    \n",
    "```sql\n",
    "SELECT * FROM pima_patient_diagnoses F WHERE F.patientid MOD 8 <> 0\n",
    "```\n",
    "\n",
    "<img src=\"images/08_24.png\" alt=\"Evaluate\" width=\"500\" height=\"500\" />\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>and now see the evolution of the metrics</p>\n",
    "\n",
    "<img src=\"images/08_25.png\" alt=\"Metrics monitoring\" />\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>With ModelOps you can close the cycle and review make decisions when you need to replace yor model in production, For example, You could get alerting from Data Drift of Performance Drift and you can create multiple versions and compare them, select a champion and deploy new versions that replace existing in Production.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97beb6b4-ce0a-4787-bb8f-1653b9794dcd",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Go and try this Step by yourself. Launch ModelOps from this button below:</p>\n",
    "<a href=\"/modelops\"><img src=\"images/launchModelOps.png\" alt=\"Launch ModelOps\" /></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a8cd489-e87f-4f69-af6d-2cc81dcb79e8",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<b style = 'font-size:20px;font-family:Arial;color:#00233C'>8. Cleanup</b>\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "    <p style = 'font-size:16px;font-family:Arial;color:#00233C'>If you are done with ModelOps usecase, please uncomment and run the below cleanup section.</p>\n",
    "</div>\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Work Tables</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Cleanup work tables to prevent errors next time.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf95b26c-02bc-495c-8759-ba41e25b5b92",
   "metadata": {},
   "outputs": [],
   "source": [
    "# db_drop_table(table_name = 'aoa_byom_models', schema_name = 'demo_user')\n",
    "# db_drop_table(table_name = 'pima_patient_predictions', schema_name = 'demo_user')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d58f754e-6fca-46ee-b187-11df9d0658c6",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'> <b>Databases and Tables </b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The following code will clean up tables and databases created above.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "62fc78de-172c-4426-81ac-011326623902",
   "metadata": {},
   "outputs": [],
   "source": [
    "# %run -i ../UseCases/run_procedure.py \"call remove_data('DEMO_ModelOps');\"        # Takes 10 seconds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b526c448-1ad5-4044-9273-96335bf020bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "remove_context()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65b0a274-1b4c-4f9a-91a5-7c0550e674d5",
   "metadata": {},
   "source": [
    "[<< Back to Git PIMA H2OAutoML](./08_ModelOps_GIT_PIMA_Python_H2OAutoML.ipynb) | [Continue to Git PIMA R GBM >>](./10_ModelOps_GIT_PIMA_R_GBM.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2b46c6d-eab2-4fa4-9e40-2d9064305c46",
   "metadata": {},
   "source": [
    "<footer style=\"padding-bottom:35px; background:#f9f9f9; border-bottom:3px solid #00233C\">\n",
    "    <div style=\"float:left;margin-top:14px\">ClearScape Analytics™</div>\n",
    "    <div style=\"float:right;\">\n",
    "        <div style=\"float:left; margin-top:14px\">\n",
    "            Copyright © Teradata Corporation - 2023. All Rights Reserved\n",
    "        </div>\n",
    "    </div>\n",
    "</footer>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}