docs/get-started/xgboost-examples/csp/databricks/init-notebook-for-rapids-spark-xgboost-on-databricks-gpu-7.0-ml.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download latest Jars"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "dbutils.fs.mkdirs(\"dbfs:/FileStore/jars/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "%sh\n",
    "cd ../../dbfs/FileStore/jars/\n",
    "wget -O cudf-21.08.2.jar https://search.maven.org/remotecontent?filepath=ai/rapids/cudf/21.08.2/cudf-21.08.2.jar\n",
    "wget -O rapids-4-spark_2.12-21.08.0.jar https://search.maven.org/remotecontent?filepath=com/nvidia/rapids-4-spark_2.12/21.08.0/rapids-4-spark_2.12-21.08.0.jar\n",
    "wget -O xgboost4j_3.0-1.4.2-0.1.0.jar https://search.maven.org/remotecontent?filepath=com/nvidia/xgboost4j_3.0/1.4.2-0.1.0/xgboost4j_3.0-1.4.2-0.1.0.jar\n",
    "wget -O xgboost4j-spark_3.0-1.4.2-0.1.0.jar https://search.maven.org/remotecontent?filepath=com/nvidia/xgboost4j-spark_3.0/1.4.2-0.1.0/xgboost4j-spark_3.0-1.4.2-0.1.0.jar\n",
    "ls -ltr\n",
    "\n",
    "# Your Jars are downloaded in dbfs:/FileStore/jars directory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a Directory for your init script"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "dbutils.fs.mkdirs(\"dbfs:/databricks/init_scripts/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "dbutils.fs.put(\"/databricks/init_scripts/init.sh\",\"\"\"\n",
    "#!/bin/bash\n",
    "sudo cp /dbfs/FileStore/jars/xgboost4j_3.0-1.4.2-0.1.0.jar /databricks/jars/spark--maven-trees--ml--7.x--xgboost--ml.dmlc--xgboost4j_2.12--ml.dmlc__xgboost4j_2.12__1.0.0.jar\n",
    "sudo cp /dbfs/FileStore/jars/cudf-21.08.2.jar /databricks/jars/\n",
    "sudo cp /dbfs/FileStore/jars/rapids-4-spark_2.12-21.08.0.jar /databricks/jars/\n",
    "sudo cp /dbfs/FileStore/jars/xgboost4j-spark_3.0-1.4.2-0.1.0.jar /databricks/jars/spark--maven-trees--ml--7.x--xgboost--ml.dmlc--xgboost4j-spark_2.12--ml.dmlc__xgboost4j-spark_2.12__1.0.0.jar\"\"\", True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Confirm your init script is in the new directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "%sh\n",
    "cd ../../dbfs/databricks/init_scripts\n",
    "pwd\n",
    "ls -ltr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download the Mortgage Dataset into your local machine and upload Data using import Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "dbutils.fs.mkdirs(\"dbfs:/FileStore/tables/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "%sh\n",
    "cd /dbfs/FileStore/tables/\n",
    "wget -O mortgage.zip https://rapidsai-data.s3.us-east-2.amazonaws.com/spark/mortgage.zip\n",
    "ls\n",
    "unzip mortgage.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "%sh\n",
    "pwd\n",
    "cd ../../dbfs/FileStore/tables\n",
    "ls -ltr mortgage/csv/*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Next steps\n",
    "\n",
    "1. Edit your cluster, adding an initialization script from `dbfs:/databricks/init_scripts/init.sh` in the \"Advanced Options\" under \"Init Scripts\" tab\n",
    "2. Reboot the cluster\n",
    "3. Go to \"Libraries\" tab under your cluster and install `dbfs:/FileStore/jars/xgboost4j-spark_3.0-1.4.2-0.1.0.jar` in your cluster by selecting the \"DBFS\" option for installing jars\n",
    "4. Import the mortgage example notebook from `https://github.com/NVIDIA/spark-rapids-examples/blob/branch-21.08/examples/mortgage/notebooks/python/mortgage-gpu.ipynb`\n",
    "5. Inside the mortgage example notebook, update the data paths\n",
    "  `train_data = reader.schema(schema).option('header', True).csv('/data/mortgage/csv/small-train.csv')`\n",
    "  `trans_data = reader.schema(schema).option('header', True).csv('/data/mortgage/csv/small-trans.csv')`"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "name": "Init Scripts_demo",
  "notebookId": 2585487876834616
 },
 "nbformat": 4,
 "nbformat_minor": 1
}