house-prices-advanced-regression-techniques/house-prices-advanced-regression-techniques.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "square-charger",
   "metadata": {
    "id": "square-charger"
   },
   "source": [
    "# House Prices - Advanced Regression Techniques\n",
    "Predict sales prices and practice feature engineering, RFs, and gradient boosting\n",
    "\n",
    "### Start here if...\n",
    "\n",
    "You have some experience with R or Python and machine learning basics. This is a perfect competition for data science students who have completed an online course in machine learning and are looking to expand their skill set before trying a featured competition. \n",
    "\n",
    "### Competition Description\n",
    "\n",
    "![](https://storage.googleapis.com/kaggle-competitions/kaggle/5407/media/housesbanner.png)\n",
    "\n",
    "Ask a home buyer to describe their dream house, and they probably won't begin with the height of the basement ceiling or the proximity to an east-west railroad. But this playground competition's dataset proves that much more influences price negotiations than the number of bedrooms or a white-picket fence.\n",
    "\n",
    "With 79 explanatory variables describing (almost) every aspect of residential homes in Ames, Iowa, this competition challenges you to predict the final price of each home.\n",
    "\n",
    "### Practice Skills\n",
    "\n",
    "*   Creative feature engineering \n",
    "*   Advanced regression techniques like random forest and gradient boosting\n",
    "\n",
    "Acknowledgments\n",
    "---------------\n",
    "\n",
    "The [Ames Housing dataset](http://www.amstat.org/publications/jse/v19n3/decock.pdf) was compiled by Dean De Cock for use in data science education. It's an incredible alternative for data scientists looking for a modernized and expanded version of the often cited Boston Housing dataset. \n",
    "\n",
    "Photo by [Tom Thain](https://unsplash.com/@tthfilms) on Unsplash.\n",
    "\n",
    "Dataset Description\n",
    "-------------------\n",
    "\n",
    "File descriptions\n",
    "-----------------\n",
    "\n",
    "*   **train.csv** - the training set\n",
    "*   **test.csv** - the test set\n",
    "*   **data\\_description.txt** - full description of each column, originally prepared by Dean De Cock but lightly edited to match the column names used here\n",
    "*   **sample\\_submission.csv** - a benchmark submission from a linear regression on year and month of sale, lot square footage, and number of bedrooms\n",
    "\n",
    "Data fields\n",
    "-----------\n",
    "\n",
    "Here's a brief version of what you'll find in the data description file.\n",
    "\n",
    "*   **SalePrice** - the property's sale price in dollars. This is the target variable that you're trying to predict.\n",
    "*   **MSSubClass**: The building class\n",
    "*   **MSZoning**: The general zoning classification\n",
    "*   **LotFrontage**: Linear feet of street connected to property\n",
    "*   **LotArea**: Lot size in square feet\n",
    "*   **Street**: Type of road access\n",
    "*   **Alley**: Type of alley access\n",
    "*   **LotShape**: General shape of property\n",
    "*   **LandContour**: Flatness of the property\n",
    "*   **Utilities**: Type of utilities available\n",
    "*   **LotConfig**: Lot configuration\n",
    "*   **LandSlope**: Slope of property\n",
    "*   **Neighborhood**: Physical locations within Ames city limits\n",
    "*   **Condition1**: Proximity to main road or railroad\n",
    "*   **Condition2**: Proximity to main road or railroad (if a second is present)\n",
    "*   **BldgType**: Type of dwelling\n",
    "*   **HouseStyle**: Style of dwelling\n",
    "*   **OverallQual**: Overall material and finish quality\n",
    "*   **OverallCond**: Overall condition rating\n",
    "*   **YearBuilt**: Original construction date\n",
    "*   **YearRemodAdd**: Remodel date\n",
    "*   **RoofStyle**: Type of roof\n",
    "*   **RoofMatl**: Roof material\n",
    "*   **Exterior1st**: Exterior covering on house\n",
    "*   **Exterior2nd**: Exterior covering on house (if more than one material)\n",
    "*   **MasVnrType**: Masonry veneer type\n",
    "*   **MasVnrArea**: Masonry veneer area in square feet\n",
    "*   **ExterQual**: Exterior material quality\n",
    "*   **ExterCond**: Present condition of the material on the exterior\n",
    "*   **Foundation**: Type of foundation\n",
    "*   **BsmtQual**: Height of the basement\n",
    "*   **BsmtCond**: General condition of the basement\n",
    "*   **BsmtExposure**: Walkout or garden level basement walls\n",
    "*   **BsmtFinType1**: Quality of basement finished area\n",
    "*   **BsmtFinSF1**: Type 1 finished square feet\n",
    "*   **BsmtFinType2**: Quality of second finished area (if present)\n",
    "*   **BsmtFinSF2**: Type 2 finished square feet\n",
    "*   **BsmtUnfSF**: Unfinished square feet of basement area\n",
    "*   **TotalBsmtSF**: Total square feet of basement area\n",
    "*   **Heating**: Type of heating\n",
    "*   **HeatingQC**: Heating quality and condition\n",
    "*   **CentralAir**: Central air conditioning\n",
    "*   **Electrical**: Electrical system\n",
    "*   **1stFlrSF**: First Floor square feet\n",
    "*   **2ndFlrSF**: Second floor square feet\n",
    "*   **LowQualFinSF**: Low quality finished square feet (all floors)\n",
    "*   **GrLivArea**: Above grade (ground) living area square feet\n",
    "*   **BsmtFullBath**: Basement full bathrooms\n",
    "*   **BsmtHalfBath**: Basement half bathrooms\n",
    "*   **FullBath**: Full bathrooms above grade\n",
    "*   **HalfBath**: Half baths above grade\n",
    "*   **Bedroom**: Number of bedrooms above basement level\n",
    "*   **Kitchen**: Number of kitchens\n",
    "*   **KitchenQual**: Kitchen quality\n",
    "*   **TotRmsAbvGrd**: Total rooms above grade (does not include bathrooms)\n",
    "*   **Functional**: Home functionality rating\n",
    "*   **Fireplaces**: Number of fireplaces\n",
    "*   **FireplaceQu**: Fireplace quality\n",
    "*   **GarageType**: Garage location\n",
    "*   **GarageYrBlt**: Year garage was built\n",
    "*   **GarageFinish**: Interior finish of the garage\n",
    "*   **GarageCars**: Size of garage in car capacity\n",
    "*   **GarageArea**: Size of garage in square feet\n",
    "*   **GarageQual**: Garage quality\n",
    "*   **GarageCond**: Garage condition\n",
    "*   **PavedDrive**: Paved driveway\n",
    "*   **WoodDeckSF**: Wood deck area in square feet\n",
    "*   **OpenPorchSF**: Open porch area in square feet\n",
    "*   **EnclosedPorch**: Enclosed porch area in square feet\n",
    "*   **3SsnPorch**: Three season porch area in square feet\n",
    "*   **ScreenPorch**: Screen porch area in square feet\n",
    "*   **PoolArea**: Pool area in square feet\n",
    "*   **PoolQC**: Pool quality\n",
    "*   **Fence**: Fence quality\n",
    "*   **MiscFeature**: Miscellaneous feature not covered in other categories\n",
    "*   **MiscVal**: $Value of miscellaneous feature\n",
    "*   **MoSold**: Month Sold\n",
    "*   **YrSold**: Year Sold\n",
    "*   **SaleType**: Type of sale\n",
    "*   **SaleCondition**: Condition of sale\n",
    "\n",
    "Link: https://www.kaggle.com/c/house-prices-advanced-regression-techniques\n",
    "\n",
    "Help: https://www.kaggle.com/serigne/stacked-regressions-top-4-on-leaderboard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "limited-mongolia",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-22T09:32:41.109728Z",
     "start_time": "2022-12-22T09:32:40.455549Z"
    },
    "id": "limited-mongolia"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "from catboost import (\n",
    "    CatBoostRegressor,\n",
    "    EFeaturesSelectionAlgorithm,\n",
    "    EShapCalcType,\n",
    "    Pool,\n",
    "    sum_models,\n",
    "    to_regressor,\n",
    ")\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.metrics import mean_squared_error\n",
    "from sklearn.model_selection import KFold, train_test_split\n",
    "from tqdm.notebook import tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e779f8f9",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-22T09:32:41.157832Z",
     "start_time": "2022-12-22T09:32:41.110989Z"
    }
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 2;\n",
       "                var nbb_unformatted_code = \"%load_ext nb_black\";\n",
       "                var nbb_formatted_code = \"%load_ext nb_black\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%load_ext nb_black"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "secondary-hypothesis",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-22T09:32:41.164497Z",
     "start_time": "2022-12-22T09:32:41.161369Z"
    },
    "id": "secondary-hypothesis"
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 3;\n",
       "                var nbb_unformatted_code = \"scaler = StandardScaler()\";\n",
       "                var nbb_formatted_code = \"scaler = StandardScaler()\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "scaler = StandardScaler()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "above-rebecca",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-22T09:32:41.178535Z",
     "start_time": "2022-12-22T09:32:41.165487Z"
    },
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 455
    },
    "id": "above-rebecca",
    "outputId": "658ef4cd-edda-4212-e9ca-5dd84c1e226a"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SalePrice</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Id</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1461</th>\n",
       "      <td>169277.052498</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1462</th>\n",
       "      <td>187758.393989</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1463</th>\n",
       "      <td>183583.683570</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1464</th>\n",
       "      <td>179317.477511</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1465</th>\n",
       "      <td>150730.079977</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2915</th>\n",
       "      <td>167081.220949</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2916</th>\n",
       "      <td>164788.778231</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2917</th>\n",
       "      <td>219222.423400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2918</th>\n",
       "      <td>184924.279659</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2919</th>\n",
       "      <td>187741.866657</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1459 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          SalePrice\n",
       "Id                 \n",
       "1461  169277.052498\n",
       "1462  187758.393989\n",
       "1463  183583.683570\n",
       "1464  179317.477511\n",
       "1465  150730.079977\n",
       "...             ...\n",
       "2915  167081.220949\n",
       "2916  164788.778231\n",
       "2917  219222.423400\n",
       "2918  184924.279659\n",
       "2919  187741.866657\n",
       "\n",
       "[1459 rows x 1 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 4;\n",
       "                var nbb_unformatted_code = \"sample_submission_df = pd.read_csv(\\n    \\\"../../data/house-prices-advanced-regression-techniques/sample_submission.csv\\\"\\n).set_index(\\\"Id\\\")\\nsample_submission_df\";\n",
       "                var nbb_formatted_code = \"sample_submission_df = pd.read_csv(\\n    \\\"../../data/house-prices-advanced-regression-techniques/sample_submission.csv\\\"\\n).set_index(\\\"Id\\\")\\nsample_submission_df\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sample_submission_df = pd.read_csv(\n",
    "    \"../../data/house-prices-advanced-regression-techniques/sample_submission.csv\"\n",
    ").set_index(\"Id\")\n",
    "sample_submission_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "amended-virus",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-22T09:32:41.212806Z",
     "start_time": "2022-12-22T09:32:41.179691Z"
    },
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 475
    },
    "id": "amended-virus",
    "outputId": "c8b80c0f-a6a3-4851-c26e-a791723dab3f"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>MSZoning</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>Street</th>\n",
       "      <th>Alley</th>\n",
       "      <th>LotShape</th>\n",
       "      <th>LandContour</th>\n",
       "      <th>Utilities</th>\n",
       "      <th>LotConfig</th>\n",
       "      <th>...</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>YrSold</th>\n",
       "      <th>SaleType</th>\n",
       "      <th>SaleCondition</th>\n",
       "      <th>SalePrice</th>\n",
       "      <th>isTest</th>\n",
       "      <th>BsmtSFExt</th>\n",
       "      <th>2ndFlrExt</th>\n",
       "      <th>PoolExt</th>\n",
       "      <th>TotalLivBsmtSF</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>65.0</td>\n",
       "      <td>8450</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>...</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>4</td>\n",
       "      <td>208500.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2566.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>80.0</td>\n",
       "      <td>9600</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "      <td>4</td>\n",
       "      <td>181500.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2524.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>68.0</td>\n",
       "      <td>11250</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>...</td>\n",
       "      <td>9</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>4</td>\n",
       "      <td>223500.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2706.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>60.0</td>\n",
       "      <td>9550</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>140000.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2473.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>84.0</td>\n",
       "      <td>14260</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td>12</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>4</td>\n",
       "      <td>250000.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3343.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2915</th>\n",
       "      <td>13</td>\n",
       "      <td>4</td>\n",
       "      <td>21.0</td>\n",
       "      <td>1936</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>...</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1638.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2916</th>\n",
       "      <td>13</td>\n",
       "      <td>4</td>\n",
       "      <td>21.0</td>\n",
       "      <td>1894</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>...</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1638.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2917</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>160.0</td>\n",
       "      <td>20000</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>...</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2448.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2918</th>\n",
       "      <td>9</td>\n",
       "      <td>3</td>\n",
       "      <td>62.0</td>\n",
       "      <td>10441</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>...</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1882.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2919</th>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>74.0</td>\n",
       "      <td>9627</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>...</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2996.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2919 rows × 85 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      MSSubClass  MSZoning  LotFrontage  LotArea  Street  Alley  LotShape  \\\n",
       "Id                                                                          \n",
       "1              5         3         65.0     8450       1      2         3   \n",
       "2              0         3         80.0     9600       1      2         3   \n",
       "3              5         3         68.0    11250       1      2         0   \n",
       "4              6         3         60.0     9550       1      2         0   \n",
       "5              5         3         84.0    14260       1      2         0   \n",
       "...          ...       ...          ...      ...     ...    ...       ...   \n",
       "2915          13         4         21.0     1936       1      2         3   \n",
       "2916          13         4         21.0     1894       1      2         3   \n",
       "2917           0         3        160.0    20000       1      2         3   \n",
       "2918           9         3         62.0    10441       1      2         3   \n",
       "2919           5         3         74.0     9627       1      2         3   \n",
       "\n",
       "      LandContour  Utilities  LotConfig  ...  MoSold  YrSold  SaleType  \\\n",
       "Id                                       ...                             \n",
       "1               3          0          4  ...       2       2         8   \n",
       "2               3          0          2  ...       5       1         8   \n",
       "3               3          0          4  ...       9       2         8   \n",
       "4               3          0          0  ...       2       0         8   \n",
       "5               3          0          2  ...      12       2         8   \n",
       "...           ...        ...        ...  ...     ...     ...       ...   \n",
       "2915            3          0          4  ...       6       0         8   \n",
       "2916            3          0          4  ...       4       0         8   \n",
       "2917            3          0          4  ...       9       0         8   \n",
       "2918            3          0          4  ...       7       0         8   \n",
       "2919            3          0          4  ...      11       0         8   \n",
       "\n",
       "      SaleCondition  SalePrice  isTest  BsmtSFExt  2ndFlrExt  PoolExt  \\\n",
       "Id                                                                      \n",
       "1                 4   208500.0       0          1          1        0   \n",
       "2                 4   181500.0       0          1          0        0   \n",
       "3                 4   223500.0       0          1          1        0   \n",
       "4                 0   140000.0       0          1          1        0   \n",
       "5                 4   250000.0       0          1          1        0   \n",
       "...             ...        ...     ...        ...        ...      ...   \n",
       "2915              4        NaN       1          1          1        0   \n",
       "2916              0        NaN       1          1          1        0   \n",
       "2917              0        NaN       1          1          0        0   \n",
       "2918              4        NaN       1          1          0        0   \n",
       "2919              4        NaN       1          1          1        0   \n",
       "\n",
       "      TotalLivBsmtSF  \n",
       "Id                    \n",
       "1             2566.0  \n",
       "2             2524.0  \n",
       "3             2706.0  \n",
       "4             2473.0  \n",
       "5             3343.0  \n",
       "...              ...  \n",
       "2915          1638.0  \n",
       "2916          1638.0  \n",
       "2917          2448.0  \n",
       "2918          1882.0  \n",
       "2919          2996.0  \n",
       "\n",
       "[2919 rows x 85 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 5;\n",
       "                var nbb_unformatted_code = \"df = pd.read_csv(\\n    \\\"../../data/house-prices-advanced-regression-techniques/prepared.csv\\\"\\n).set_index(\\\"Id\\\")\\ndf\";\n",
       "                var nbb_formatted_code = \"df = pd.read_csv(\\n    \\\"../../data/house-prices-advanced-regression-techniques/prepared.csv\\\"\\n).set_index(\\\"Id\\\")\\ndf\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df = pd.read_csv(\n",
    "    \"../../data/house-prices-advanced-regression-techniques/prepared.csv\"\n",
    ").set_index(\"Id\")\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "elegant-committee",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-22T09:32:41.227118Z",
     "start_time": "2022-12-22T09:32:41.214177Z"
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "elegant-committee",
    "outputId": "66c3afe9-d0a6-4f0c-8578-34d118679dd2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 2919 entries, 1 to 2919\n",
      "Data columns (total 85 columns):\n",
      " #   Column          Non-Null Count  Dtype  \n",
      "---  ------          --------------  -----  \n",
      " 0   MSSubClass      2919 non-null   int64  \n",
      " 1   MSZoning        2919 non-null   int64  \n",
      " 2   LotFrontage     2919 non-null   float64\n",
      " 3   LotArea         2919 non-null   int64  \n",
      " 4   Street          2919 non-null   int64  \n",
      " 5   Alley           2919 non-null   int64  \n",
      " 6   LotShape        2919 non-null   int64  \n",
      " 7   LandContour     2919 non-null   int64  \n",
      " 8   Utilities       2919 non-null   int64  \n",
      " 9   LotConfig       2919 non-null   int64  \n",
      " 10  LandSlope       2919 non-null   int64  \n",
      " 11  Neighborhood    2919 non-null   int64  \n",
      " 12  Condition1      2919 non-null   int64  \n",
      " 13  Condition2      2919 non-null   int64  \n",
      " 14  BldgType        2919 non-null   int64  \n",
      " 15  HouseStyle      2919 non-null   int64  \n",
      " 16  OverallQual     2919 non-null   int64  \n",
      " 17  OverallCond     2919 non-null   int64  \n",
      " 18  YearBuilt       2919 non-null   int64  \n",
      " 19  YearRemodAdd    2919 non-null   int64  \n",
      " 20  RoofStyle       2919 non-null   int64  \n",
      " 21  RoofMatl        2919 non-null   int64  \n",
      " 22  Exterior1st     2919 non-null   int64  \n",
      " 23  Exterior2nd     2919 non-null   int64  \n",
      " 24  MasVnrType      2919 non-null   int64  \n",
      " 25  MasVnrArea      2919 non-null   float64\n",
      " 26  ExterQual       2919 non-null   int64  \n",
      " 27  ExterCond       2919 non-null   int64  \n",
      " 28  Foundation      2919 non-null   int64  \n",
      " 29  BsmtQual        2919 non-null   int64  \n",
      " 30  BsmtCond        2919 non-null   int64  \n",
      " 31  BsmtExposure    2919 non-null   int64  \n",
      " 32  BsmtFinType1    2919 non-null   int64  \n",
      " 33  BsmtFinSF1      2919 non-null   float64\n",
      " 34  BsmtFinType2    2919 non-null   int64  \n",
      " 35  BsmtFinSF2      2919 non-null   float64\n",
      " 36  BsmtUnfSF       2919 non-null   float64\n",
      " 37  TotalBsmtSF     2919 non-null   float64\n",
      " 38  Heating         2919 non-null   int64  \n",
      " 39  HeatingQC       2919 non-null   int64  \n",
      " 40  CentralAir      2919 non-null   int64  \n",
      " 41  Electrical      2919 non-null   int64  \n",
      " 42  1stFlrSF        2919 non-null   int64  \n",
      " 43  2ndFlrSF        2919 non-null   int64  \n",
      " 44  LowQualFinSF    2919 non-null   int64  \n",
      " 45  GrLivArea       2919 non-null   int64  \n",
      " 46  BsmtFullBath    2919 non-null   int64  \n",
      " 47  BsmtHalfBath    2919 non-null   int64  \n",
      " 48  FullBath        2919 non-null   int64  \n",
      " 49  HalfBath        2919 non-null   int64  \n",
      " 50  BedroomAbvGr    2919 non-null   int64  \n",
      " 51  KitchenAbvGr    2919 non-null   int64  \n",
      " 52  KitchenQual     2919 non-null   int64  \n",
      " 53  TotRmsAbvGrd    2919 non-null   int64  \n",
      " 54  Functional      2919 non-null   int64  \n",
      " 55  Fireplaces      2919 non-null   int64  \n",
      " 56  FireplaceQu     2919 non-null   int64  \n",
      " 57  GarageType      2919 non-null   int64  \n",
      " 58  GarageYrBlt     2919 non-null   int64  \n",
      " 59  GarageFinish    2919 non-null   int64  \n",
      " 60  GarageCars      2919 non-null   int64  \n",
      " 61  GarageArea      2919 non-null   float64\n",
      " 62  GarageQual      2919 non-null   int64  \n",
      " 63  GarageCond      2919 non-null   int64  \n",
      " 64  PavedDrive      2919 non-null   int64  \n",
      " 65  WoodDeckSF      2919 non-null   int64  \n",
      " 66  OpenPorchSF     2919 non-null   int64  \n",
      " 67  EnclosedPorch   2919 non-null   int64  \n",
      " 68  3SsnPorch       2919 non-null   int64  \n",
      " 69  ScreenPorch     2919 non-null   int64  \n",
      " 70  PoolArea        2919 non-null   int64  \n",
      " 71  PoolQC          2919 non-null   int64  \n",
      " 72  Fence           2919 non-null   int64  \n",
      " 73  MiscFeature     2919 non-null   int64  \n",
      " 74  MiscVal         2919 non-null   int64  \n",
      " 75  MoSold          2919 non-null   int64  \n",
      " 76  YrSold          2919 non-null   int64  \n",
      " 77  SaleType        2919 non-null   int64  \n",
      " 78  SaleCondition   2919 non-null   int64  \n",
      " 79  SalePrice       1460 non-null   float64\n",
      " 80  isTest          2919 non-null   int64  \n",
      " 81  BsmtSFExt       2919 non-null   int64  \n",
      " 82  2ndFlrExt       2919 non-null   int64  \n",
      " 83  PoolExt         2919 non-null   int64  \n",
      " 84  TotalLivBsmtSF  2919 non-null   float64\n",
      "dtypes: float64(9), int64(76)\n",
      "memory usage: 1.9 MB\n"
     ]
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 6;\n",
       "                var nbb_unformatted_code = \"df.info()\";\n",
       "                var nbb_formatted_code = \"df.info()\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9aaef0cb",
   "metadata": {},
   "source": [
    "# Prepare"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "f0cdade2",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-22T09:32:41.235992Z",
     "start_time": "2022-12-22T09:32:41.228584Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['MSSubClass', 'MSZoning', 'LotFrontage', 'LotArea', 'Street', 'Alley',\n",
       "       'LotShape', 'LandContour', 'Utilities', 'LotConfig', 'LandSlope',\n",
       "       'Neighborhood', 'Condition1', 'Condition2', 'BldgType', 'HouseStyle',\n",
       "       'OverallQual', 'OverallCond', 'YearBuilt', 'YearRemodAdd', 'RoofStyle',\n",
       "       'RoofMatl', 'Exterior1st', 'Exterior2nd', 'MasVnrType', 'MasVnrArea',\n",
       "       'ExterQual', 'ExterCond', 'Foundation', 'BsmtQual', 'BsmtCond',\n",
       "       'BsmtExposure', 'BsmtFinType1', 'BsmtFinSF1', 'BsmtFinType2',\n",
       "       'BsmtFinSF2', 'BsmtUnfSF', 'TotalBsmtSF', 'Heating', 'HeatingQC',\n",
       "       'CentralAir', 'Electrical', '1stFlrSF', '2ndFlrSF', 'LowQualFinSF',\n",
       "       'GrLivArea', 'BsmtFullBath', 'BsmtHalfBath', 'FullBath', 'HalfBath',\n",
       "       'BedroomAbvGr', 'KitchenAbvGr', 'KitchenQual', 'TotRmsAbvGrd',\n",
       "       'Functional', 'Fireplaces', 'FireplaceQu', 'GarageType', 'GarageYrBlt',\n",
       "       'GarageFinish', 'GarageCars', 'GarageArea', 'GarageQual', 'GarageCond',\n",
       "       'PavedDrive', 'WoodDeckSF', 'OpenPorchSF', 'EnclosedPorch', '3SsnPorch',\n",
       "       'ScreenPorch', 'PoolArea', 'PoolQC', 'Fence', 'MiscFeature', 'MiscVal',\n",
       "       'MoSold', 'YrSold', 'SaleType', 'SaleCondition', 'BsmtSFExt',\n",
       "       '2ndFlrExt', 'PoolExt', 'TotalLivBsmtSF'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 7;\n",
       "                var nbb_unformatted_code = \"num_columns = df.drop([\\\"isTest\\\", \\\"SalePrice\\\"], axis=1).columns\\nnum_columns\";\n",
       "                var nbb_formatted_code = \"num_columns = df.drop([\\\"isTest\\\", \\\"SalePrice\\\"], axis=1).columns\\nnum_columns\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "num_columns = df.drop([\"isTest\", \"SalePrice\"], axis=1).columns\n",
    "num_columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "f2d978cc",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-22T09:32:41.278859Z",
     "start_time": "2022-12-22T09:32:41.237463Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>MSZoning</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>Street</th>\n",
       "      <th>Alley</th>\n",
       "      <th>LotShape</th>\n",
       "      <th>LandContour</th>\n",
       "      <th>Utilities</th>\n",
       "      <th>LotConfig</th>\n",
       "      <th>...</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>YrSold</th>\n",
       "      <th>SaleType</th>\n",
       "      <th>SaleCondition</th>\n",
       "      <th>SalePrice</th>\n",
       "      <th>isTest</th>\n",
       "      <th>BsmtSFExt</th>\n",
       "      <th>2ndFlrExt</th>\n",
       "      <th>PoolExt</th>\n",
       "      <th>TotalLivBsmtSF</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.168840</td>\n",
       "      <td>-0.042157</td>\n",
       "      <td>-0.202068</td>\n",
       "      <td>-0.217879</td>\n",
       "      <td>0.064249</td>\n",
       "      <td>0.257283</td>\n",
       "      <td>0.746669</td>\n",
       "      <td>0.316671</td>\n",
       "      <td>-0.030863</td>\n",
       "      <td>0.588555</td>\n",
       "      <td>...</td>\n",
       "      <td>-1.552184</td>\n",
       "      <td>0.157646</td>\n",
       "      <td>0.319527</td>\n",
       "      <td>0.204967</td>\n",
       "      <td>208500.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.166784</td>\n",
       "      <td>1.154701</td>\n",
       "      <td>-0.066884</td>\n",
       "      <td>0.023003</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.982129</td>\n",
       "      <td>-0.042157</td>\n",
       "      <td>0.501870</td>\n",
       "      <td>-0.072044</td>\n",
       "      <td>0.064249</td>\n",
       "      <td>0.257283</td>\n",
       "      <td>0.746669</td>\n",
       "      <td>0.316671</td>\n",
       "      <td>-0.030863</td>\n",
       "      <td>-0.658174</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.446925</td>\n",
       "      <td>-0.602962</td>\n",
       "      <td>0.319527</td>\n",
       "      <td>0.204967</td>\n",
       "      <td>181500.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.166784</td>\n",
       "      <td>-0.866025</td>\n",
       "      <td>-0.066884</td>\n",
       "      <td>-0.029172</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.168840</td>\n",
       "      <td>-0.042157</td>\n",
       "      <td>-0.061280</td>\n",
       "      <td>0.137197</td>\n",
       "      <td>0.064249</td>\n",
       "      <td>0.257283</td>\n",
       "      <td>-1.381776</td>\n",
       "      <td>0.316671</td>\n",
       "      <td>-0.030863</td>\n",
       "      <td>0.588555</td>\n",
       "      <td>...</td>\n",
       "      <td>1.026753</td>\n",
       "      <td>0.157646</td>\n",
       "      <td>0.319527</td>\n",
       "      <td>0.204967</td>\n",
       "      <td>223500.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.166784</td>\n",
       "      <td>1.154701</td>\n",
       "      <td>-0.066884</td>\n",
       "      <td>0.196920</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.399034</td>\n",
       "      <td>-0.042157</td>\n",
       "      <td>-0.436714</td>\n",
       "      <td>-0.078385</td>\n",
       "      <td>0.064249</td>\n",
       "      <td>0.257283</td>\n",
       "      <td>-1.381776</td>\n",
       "      <td>0.316671</td>\n",
       "      <td>-0.030863</td>\n",
       "      <td>-1.904903</td>\n",
       "      <td>...</td>\n",
       "      <td>-1.552184</td>\n",
       "      <td>-1.363569</td>\n",
       "      <td>0.319527</td>\n",
       "      <td>-3.505415</td>\n",
       "      <td>140000.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.166784</td>\n",
       "      <td>1.154701</td>\n",
       "      <td>-0.066884</td>\n",
       "      <td>-0.092527</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.168840</td>\n",
       "      <td>-0.042157</td>\n",
       "      <td>0.689587</td>\n",
       "      <td>0.518903</td>\n",
       "      <td>0.064249</td>\n",
       "      <td>0.257283</td>\n",
       "      <td>-1.381776</td>\n",
       "      <td>0.316671</td>\n",
       "      <td>-0.030863</td>\n",
       "      <td>-0.658174</td>\n",
       "      <td>...</td>\n",
       "      <td>2.132012</td>\n",
       "      <td>0.157646</td>\n",
       "      <td>0.319527</td>\n",
       "      <td>0.204967</td>\n",
       "      <td>250000.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.166784</td>\n",
       "      <td>1.154701</td>\n",
       "      <td>-0.066884</td>\n",
       "      <td>0.988241</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2915</th>\n",
       "      <td>2.010392</td>\n",
       "      <td>1.477041</td>\n",
       "      <td>-2.266952</td>\n",
       "      <td>-1.043937</td>\n",
       "      <td>0.064249</td>\n",
       "      <td>0.257283</td>\n",
       "      <td>0.746669</td>\n",
       "      <td>0.316671</td>\n",
       "      <td>-0.030863</td>\n",
       "      <td>0.588555</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.078505</td>\n",
       "      <td>-1.363569</td>\n",
       "      <td>0.319527</td>\n",
       "      <td>0.204967</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>0.166784</td>\n",
       "      <td>1.154701</td>\n",
       "      <td>-0.066884</td>\n",
       "      <td>-1.129816</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2916</th>\n",
       "      <td>2.010392</td>\n",
       "      <td>1.477041</td>\n",
       "      <td>-2.266952</td>\n",
       "      <td>-1.049263</td>\n",
       "      <td>0.064249</td>\n",
       "      <td>0.257283</td>\n",
       "      <td>0.746669</td>\n",
       "      <td>0.316671</td>\n",
       "      <td>-0.030863</td>\n",
       "      <td>0.588555</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.815344</td>\n",
       "      <td>-1.363569</td>\n",
       "      <td>0.319527</td>\n",
       "      <td>-3.505415</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>0.166784</td>\n",
       "      <td>1.154701</td>\n",
       "      <td>-0.066884</td>\n",
       "      <td>-1.129816</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2917</th>\n",
       "      <td>-0.982129</td>\n",
       "      <td>-0.042157</td>\n",
       "      <td>4.256207</td>\n",
       "      <td>1.246808</td>\n",
       "      <td>0.064249</td>\n",
       "      <td>0.257283</td>\n",
       "      <td>0.746669</td>\n",
       "      <td>0.316671</td>\n",
       "      <td>-0.030863</td>\n",
       "      <td>0.588555</td>\n",
       "      <td>...</td>\n",
       "      <td>1.026753</td>\n",
       "      <td>-1.363569</td>\n",
       "      <td>0.319527</td>\n",
       "      <td>-3.505415</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>0.166784</td>\n",
       "      <td>-0.866025</td>\n",
       "      <td>-0.066884</td>\n",
       "      <td>-0.123584</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2918</th>\n",
       "      <td>1.089616</td>\n",
       "      <td>-0.042157</td>\n",
       "      <td>-0.342855</td>\n",
       "      <td>0.034605</td>\n",
       "      <td>0.064249</td>\n",
       "      <td>0.257283</td>\n",
       "      <td>0.746669</td>\n",
       "      <td>0.316671</td>\n",
       "      <td>-0.030863</td>\n",
       "      <td>0.588555</td>\n",
       "      <td>...</td>\n",
       "      <td>0.289914</td>\n",
       "      <td>-1.363569</td>\n",
       "      <td>0.319527</td>\n",
       "      <td>0.204967</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>0.166784</td>\n",
       "      <td>-0.866025</td>\n",
       "      <td>-0.066884</td>\n",
       "      <td>-0.826704</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2919</th>\n",
       "      <td>0.168840</td>\n",
       "      <td>-0.042157</td>\n",
       "      <td>0.220295</td>\n",
       "      <td>-0.068620</td>\n",
       "      <td>0.064249</td>\n",
       "      <td>0.257283</td>\n",
       "      <td>0.746669</td>\n",
       "      <td>0.316671</td>\n",
       "      <td>-0.030863</td>\n",
       "      <td>0.588555</td>\n",
       "      <td>...</td>\n",
       "      <td>1.763592</td>\n",
       "      <td>-1.363569</td>\n",
       "      <td>0.319527</td>\n",
       "      <td>0.204967</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>0.166784</td>\n",
       "      <td>1.154701</td>\n",
       "      <td>-0.066884</td>\n",
       "      <td>0.557176</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2919 rows × 85 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      MSSubClass  MSZoning  LotFrontage   LotArea    Street     Alley  \\\n",
       "Id                                                                      \n",
       "1       0.168840 -0.042157    -0.202068 -0.217879  0.064249  0.257283   \n",
       "2      -0.982129 -0.042157     0.501870 -0.072044  0.064249  0.257283   \n",
       "3       0.168840 -0.042157    -0.061280  0.137197  0.064249  0.257283   \n",
       "4       0.399034 -0.042157    -0.436714 -0.078385  0.064249  0.257283   \n",
       "5       0.168840 -0.042157     0.689587  0.518903  0.064249  0.257283   \n",
       "...          ...       ...          ...       ...       ...       ...   \n",
       "2915    2.010392  1.477041    -2.266952 -1.043937  0.064249  0.257283   \n",
       "2916    2.010392  1.477041    -2.266952 -1.049263  0.064249  0.257283   \n",
       "2917   -0.982129 -0.042157     4.256207  1.246808  0.064249  0.257283   \n",
       "2918    1.089616 -0.042157    -0.342855  0.034605  0.064249  0.257283   \n",
       "2919    0.168840 -0.042157     0.220295 -0.068620  0.064249  0.257283   \n",
       "\n",
       "      LotShape  LandContour  Utilities  LotConfig  ...    MoSold    YrSold  \\\n",
       "Id                                                 ...                       \n",
       "1     0.746669     0.316671  -0.030863   0.588555  ... -1.552184  0.157646   \n",
       "2     0.746669     0.316671  -0.030863  -0.658174  ... -0.446925 -0.602962   \n",
       "3    -1.381776     0.316671  -0.030863   0.588555  ...  1.026753  0.157646   \n",
       "4    -1.381776     0.316671  -0.030863  -1.904903  ... -1.552184 -1.363569   \n",
       "5    -1.381776     0.316671  -0.030863  -0.658174  ...  2.132012  0.157646   \n",
       "...        ...          ...        ...        ...  ...       ...       ...   \n",
       "2915  0.746669     0.316671  -0.030863   0.588555  ... -0.078505 -1.363569   \n",
       "2916  0.746669     0.316671  -0.030863   0.588555  ... -0.815344 -1.363569   \n",
       "2917  0.746669     0.316671  -0.030863   0.588555  ...  1.026753 -1.363569   \n",
       "2918  0.746669     0.316671  -0.030863   0.588555  ...  0.289914 -1.363569   \n",
       "2919  0.746669     0.316671  -0.030863   0.588555  ...  1.763592 -1.363569   \n",
       "\n",
       "      SaleType  SaleCondition  SalePrice  isTest  BsmtSFExt  2ndFlrExt  \\\n",
       "Id                                                                       \n",
       "1     0.319527       0.204967   208500.0       0   0.166784   1.154701   \n",
       "2     0.319527       0.204967   181500.0       0   0.166784  -0.866025   \n",
       "3     0.319527       0.204967   223500.0       0   0.166784   1.154701   \n",
       "4     0.319527      -3.505415   140000.0       0   0.166784   1.154701   \n",
       "5     0.319527       0.204967   250000.0       0   0.166784   1.154701   \n",
       "...        ...            ...        ...     ...        ...        ...   \n",
       "2915  0.319527       0.204967        NaN       1   0.166784   1.154701   \n",
       "2916  0.319527      -3.505415        NaN       1   0.166784   1.154701   \n",
       "2917  0.319527      -3.505415        NaN       1   0.166784  -0.866025   \n",
       "2918  0.319527       0.204967        NaN       1   0.166784  -0.866025   \n",
       "2919  0.319527       0.204967        NaN       1   0.166784   1.154701   \n",
       "\n",
       "       PoolExt  TotalLivBsmtSF  \n",
       "Id                              \n",
       "1    -0.066884        0.023003  \n",
       "2    -0.066884       -0.029172  \n",
       "3    -0.066884        0.196920  \n",
       "4    -0.066884       -0.092527  \n",
       "5    -0.066884        0.988241  \n",
       "...        ...             ...  \n",
       "2915 -0.066884       -1.129816  \n",
       "2916 -0.066884       -1.129816  \n",
       "2917 -0.066884       -0.123584  \n",
       "2918 -0.066884       -0.826704  \n",
       "2919 -0.066884        0.557176  \n",
       "\n",
       "[2919 rows x 85 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 8;\n",
       "                var nbb_unformatted_code = \"df[num_columns] = scaler.fit_transform(df[num_columns])\\ndf\";\n",
       "                var nbb_formatted_code = \"df[num_columns] = scaler.fit_transform(df[num_columns])\\ndf\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df[num_columns] = scaler.fit_transform(df[num_columns])\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "758d673b",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-22T09:32:41.296528Z",
     "start_time": "2022-12-22T09:32:41.281854Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((1460, 83), (1460,), (1459, 83))"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 9;\n",
       "                var nbb_unformatted_code = \"# RMSLE - https://www.kaggle.com/code/carlolepelaars/understanding-the-metric-rmsle/notebook\\ny_train = np.log1p(df[df[\\\"isTest\\\"] == 0][\\\"SalePrice\\\"])\\n\\nX_train = df[df[\\\"isTest\\\"] == 0][num_columns]\\nX_test = df[df[\\\"isTest\\\"] == 1][num_columns]\\n\\nX_train.shape, y_train.shape, X_test.shape\";\n",
       "                var nbb_formatted_code = \"# RMSLE - https://www.kaggle.com/code/carlolepelaars/understanding-the-metric-rmsle/notebook\\ny_train = np.log1p(df[df[\\\"isTest\\\"] == 0][\\\"SalePrice\\\"])\\n\\nX_train = df[df[\\\"isTest\\\"] == 0][num_columns]\\nX_test = df[df[\\\"isTest\\\"] == 1][num_columns]\\n\\nX_train.shape, y_train.shape, X_test.shape\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# RMSLE - https://www.kaggle.com/code/carlolepelaars/understanding-the-metric-rmsle/notebook\n",
    "y_train = np.log1p(df[df[\"isTest\"] == 0][\"SalePrice\"])\n",
    "\n",
    "X_train = df[df[\"isTest\"] == 0][num_columns]\n",
    "X_test = df[df[\"isTest\"] == 1][num_columns]\n",
    "\n",
    "X_train.shape, y_train.shape, X_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "58d22dbb",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-22T09:32:41.304218Z",
     "start_time": "2022-12-22T09:32:41.297518Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((1314, 83), (146, 83), (1314,), (146,))"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 10;\n",
       "                var nbb_unformatted_code = \"X_train, X_true, y_train, y_true = train_test_split(\\n    X_train, y_train, test_size=0.1, random_state=42\\n)\\nX_train.shape, X_true.shape, y_train.shape, y_true.shape\";\n",
       "                var nbb_formatted_code = \"X_train, X_true, y_train, y_true = train_test_split(\\n    X_train, y_train, test_size=0.1, random_state=42\\n)\\nX_train.shape, X_true.shape, y_train.shape, y_true.shape\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "X_train, X_true, y_train, y_true = train_test_split(\n",
    "    X_train, y_train, test_size=0.1, random_state=42\n",
    ")\n",
    "X_train.shape, X_true.shape, y_train.shape, y_true.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "d8be0fb7",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-22T09:32:41.320125Z",
     "start_time": "2022-12-22T09:32:41.305299Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>MSZoning</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>Street</th>\n",
       "      <th>Alley</th>\n",
       "      <th>LotShape</th>\n",
       "      <th>LandContour</th>\n",
       "      <th>Utilities</th>\n",
       "      <th>LotConfig</th>\n",
       "      <th>...</th>\n",
       "      <th>MiscFeature</th>\n",
       "      <th>MiscVal</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>YrSold</th>\n",
       "      <th>SaleType</th>\n",
       "      <th>SaleCondition</th>\n",
       "      <th>BsmtSFExt</th>\n",
       "      <th>2ndFlrExt</th>\n",
       "      <th>PoolExt</th>\n",
       "      <th>TotalLivBsmtSF</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>908</th>\n",
       "      <td>-0.061354</td>\n",
       "      <td>-0.042157</td>\n",
       "      <td>0.783446</td>\n",
       "      <td>0.168900</td>\n",
       "      <td>0.064249</td>\n",
       "      <td>0.257283</td>\n",
       "      <td>-1.381776</td>\n",
       "      <td>0.316671</td>\n",
       "      <td>-0.030863</td>\n",
       "      <td>0.588555</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.181028</td>\n",
       "      <td>-0.089592</td>\n",
       "      <td>-0.078505</td>\n",
       "      <td>-1.363569</td>\n",
       "      <td>0.319527</td>\n",
       "      <td>0.204967</td>\n",
       "      <td>0.166784</td>\n",
       "      <td>1.154701</td>\n",
       "      <td>-0.066884</td>\n",
       "      <td>0.654073</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>783</th>\n",
       "      <td>-0.982129</td>\n",
       "      <td>-0.042157</td>\n",
       "      <td>-0.108209</td>\n",
       "      <td>0.775699</td>\n",
       "      <td>0.064249</td>\n",
       "      <td>0.257283</td>\n",
       "      <td>-0.672294</td>\n",
       "      <td>0.316671</td>\n",
       "      <td>-0.030863</td>\n",
       "      <td>0.588555</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.181028</td>\n",
       "      <td>-0.089592</td>\n",
       "      <td>-0.078505</td>\n",
       "      <td>0.918253</td>\n",
       "      <td>0.319527</td>\n",
       "      <td>0.204967</td>\n",
       "      <td>0.166784</td>\n",
       "      <td>-0.866025</td>\n",
       "      <td>-0.066884</td>\n",
       "      <td>0.367110</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>953</th>\n",
       "      <td>1.089616</td>\n",
       "      <td>-0.042157</td>\n",
       "      <td>-0.436714</td>\n",
       "      <td>-0.376395</td>\n",
       "      <td>0.064249</td>\n",
       "      <td>0.257283</td>\n",
       "      <td>0.746669</td>\n",
       "      <td>0.316671</td>\n",
       "      <td>-0.030863</td>\n",
       "      <td>0.588555</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.181028</td>\n",
       "      <td>-0.089592</td>\n",
       "      <td>-0.815344</td>\n",
       "      <td>0.918253</td>\n",
       "      <td>0.319527</td>\n",
       "      <td>0.204967</td>\n",
       "      <td>0.166784</td>\n",
       "      <td>-0.866025</td>\n",
       "      <td>-0.066884</td>\n",
       "      <td>-1.256527</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>621</th>\n",
       "      <td>-0.751935</td>\n",
       "      <td>-0.042157</td>\n",
       "      <td>-1.140652</td>\n",
       "      <td>-0.243495</td>\n",
       "      <td>0.064249</td>\n",
       "      <td>-4.466046</td>\n",
       "      <td>0.746669</td>\n",
       "      <td>0.316671</td>\n",
       "      <td>-0.030863</td>\n",
       "      <td>0.588555</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.181028</td>\n",
       "      <td>-0.089592</td>\n",
       "      <td>1.026753</td>\n",
       "      <td>0.157646</td>\n",
       "      <td>0.319527</td>\n",
       "      <td>0.204967</td>\n",
       "      <td>0.166784</td>\n",
       "      <td>-0.866025</td>\n",
       "      <td>-0.066884</td>\n",
       "      <td>-1.018013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>670</th>\n",
       "      <td>-0.751935</td>\n",
       "      <td>-0.042157</td>\n",
       "      <td>0.501870</td>\n",
       "      <td>0.181581</td>\n",
       "      <td>0.064249</td>\n",
       "      <td>0.257283</td>\n",
       "      <td>0.746669</td>\n",
       "      <td>0.316671</td>\n",
       "      <td>-0.030863</td>\n",
       "      <td>0.588555</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.181028</td>\n",
       "      <td>-0.089592</td>\n",
       "      <td>0.289914</td>\n",
       "      <td>-1.363569</td>\n",
       "      <td>0.319527</td>\n",
       "      <td>0.204967</td>\n",
       "      <td>0.166784</td>\n",
       "      <td>-0.866025</td>\n",
       "      <td>-0.066884</td>\n",
       "      <td>-0.829189</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1096</th>\n",
       "      <td>-0.982129</td>\n",
       "      <td>-0.042157</td>\n",
       "      <td>0.408012</td>\n",
       "      <td>-0.107932</td>\n",
       "      <td>0.064249</td>\n",
       "      <td>0.257283</td>\n",
       "      <td>-1.381776</td>\n",
       "      <td>0.316671</td>\n",
       "      <td>-0.030863</td>\n",
       "      <td>0.588555</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.181028</td>\n",
       "      <td>-0.089592</td>\n",
       "      <td>-1.183764</td>\n",
       "      <td>-0.602962</td>\n",
       "      <td>0.319527</td>\n",
       "      <td>0.204967</td>\n",
       "      <td>0.166784</td>\n",
       "      <td>-0.866025</td>\n",
       "      <td>-0.066884</td>\n",
       "      <td>0.100024</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1131</th>\n",
       "      <td>-0.061354</td>\n",
       "      <td>-0.042157</td>\n",
       "      <td>-0.202068</td>\n",
       "      <td>-0.299800</td>\n",
       "      <td>0.064249</td>\n",
       "      <td>0.257283</td>\n",
       "      <td>0.746669</td>\n",
       "      <td>0.316671</td>\n",
       "      <td>-0.030863</td>\n",
       "      <td>0.588555</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.181028</td>\n",
       "      <td>-0.089592</td>\n",
       "      <td>2.132012</td>\n",
       "      <td>0.918253</td>\n",
       "      <td>0.319527</td>\n",
       "      <td>0.204967</td>\n",
       "      <td>0.166784</td>\n",
       "      <td>1.154701</td>\n",
       "      <td>-0.066884</td>\n",
       "      <td>0.690098</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1295</th>\n",
       "      <td>-0.982129</td>\n",
       "      <td>-0.042157</td>\n",
       "      <td>-0.436714</td>\n",
       "      <td>-0.253133</td>\n",
       "      <td>0.064249</td>\n",
       "      <td>0.257283</td>\n",
       "      <td>0.746669</td>\n",
       "      <td>0.316671</td>\n",
       "      <td>-0.030863</td>\n",
       "      <td>0.588555</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.181028</td>\n",
       "      <td>-0.089592</td>\n",
       "      <td>-0.815344</td>\n",
       "      <td>-1.363569</td>\n",
       "      <td>0.319527</td>\n",
       "      <td>0.204967</td>\n",
       "      <td>0.166784</td>\n",
       "      <td>-0.866025</td>\n",
       "      <td>-0.066884</td>\n",
       "      <td>-1.018013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>861</th>\n",
       "      <td>-0.061354</td>\n",
       "      <td>-0.042157</td>\n",
       "      <td>-0.671360</td>\n",
       "      <td>-0.320343</td>\n",
       "      <td>0.064249</td>\n",
       "      <td>0.257283</td>\n",
       "      <td>0.746669</td>\n",
       "      <td>0.316671</td>\n",
       "      <td>-0.030863</td>\n",
       "      <td>-1.904903</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.181028</td>\n",
       "      <td>-0.089592</td>\n",
       "      <td>-0.078505</td>\n",
       "      <td>-0.602962</td>\n",
       "      <td>0.319527</td>\n",
       "      <td>0.204967</td>\n",
       "      <td>0.166784</td>\n",
       "      <td>1.154701</td>\n",
       "      <td>-0.066884</td>\n",
       "      <td>-0.260232</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1127</th>\n",
       "      <td>1.550004</td>\n",
       "      <td>-0.042157</td>\n",
       "      <td>-0.765218</td>\n",
       "      <td>-0.822268</td>\n",
       "      <td>0.064249</td>\n",
       "      <td>0.257283</td>\n",
       "      <td>0.746669</td>\n",
       "      <td>0.316671</td>\n",
       "      <td>-0.030863</td>\n",
       "      <td>0.588555</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.181028</td>\n",
       "      <td>-0.089592</td>\n",
       "      <td>-0.078505</td>\n",
       "      <td>0.918253</td>\n",
       "      <td>0.319527</td>\n",
       "      <td>0.204967</td>\n",
       "      <td>0.166784</td>\n",
       "      <td>-0.866025</td>\n",
       "      <td>-0.066884</td>\n",
       "      <td>0.472702</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1314 rows × 83 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      MSSubClass  MSZoning  LotFrontage   LotArea    Street     Alley  \\\n",
       "Id                                                                      \n",
       "908    -0.061354 -0.042157     0.783446  0.168900  0.064249  0.257283   \n",
       "783    -0.982129 -0.042157    -0.108209  0.775699  0.064249  0.257283   \n",
       "953     1.089616 -0.042157    -0.436714 -0.376395  0.064249  0.257283   \n",
       "621    -0.751935 -0.042157    -1.140652 -0.243495  0.064249 -4.466046   \n",
       "670    -0.751935 -0.042157     0.501870  0.181581  0.064249  0.257283   \n",
       "...          ...       ...          ...       ...       ...       ...   \n",
       "1096   -0.982129 -0.042157     0.408012 -0.107932  0.064249  0.257283   \n",
       "1131   -0.061354 -0.042157    -0.202068 -0.299800  0.064249  0.257283   \n",
       "1295   -0.982129 -0.042157    -0.436714 -0.253133  0.064249  0.257283   \n",
       "861    -0.061354 -0.042157    -0.671360 -0.320343  0.064249  0.257283   \n",
       "1127    1.550004 -0.042157    -0.765218 -0.822268  0.064249  0.257283   \n",
       "\n",
       "      LotShape  LandContour  Utilities  LotConfig  ...  MiscFeature   MiscVal  \\\n",
       "Id                                                 ...                          \n",
       "908  -1.381776     0.316671  -0.030863   0.588555  ...    -0.181028 -0.089592   \n",
       "783  -0.672294     0.316671  -0.030863   0.588555  ...    -0.181028 -0.089592   \n",
       "953   0.746669     0.316671  -0.030863   0.588555  ...    -0.181028 -0.089592   \n",
       "621   0.746669     0.316671  -0.030863   0.588555  ...    -0.181028 -0.089592   \n",
       "670   0.746669     0.316671  -0.030863   0.588555  ...    -0.181028 -0.089592   \n",
       "...        ...          ...        ...        ...  ...          ...       ...   \n",
       "1096 -1.381776     0.316671  -0.030863   0.588555  ...    -0.181028 -0.089592   \n",
       "1131  0.746669     0.316671  -0.030863   0.588555  ...    -0.181028 -0.089592   \n",
       "1295  0.746669     0.316671  -0.030863   0.588555  ...    -0.181028 -0.089592   \n",
       "861   0.746669     0.316671  -0.030863  -1.904903  ...    -0.181028 -0.089592   \n",
       "1127  0.746669     0.316671  -0.030863   0.588555  ...    -0.181028 -0.089592   \n",
       "\n",
       "        MoSold    YrSold  SaleType  SaleCondition  BsmtSFExt  2ndFlrExt  \\\n",
       "Id                                                                        \n",
       "908  -0.078505 -1.363569  0.319527       0.204967   0.166784   1.154701   \n",
       "783  -0.078505  0.918253  0.319527       0.204967   0.166784  -0.866025   \n",
       "953  -0.815344  0.918253  0.319527       0.204967   0.166784  -0.866025   \n",
       "621   1.026753  0.157646  0.319527       0.204967   0.166784  -0.866025   \n",
       "670   0.289914 -1.363569  0.319527       0.204967   0.166784  -0.866025   \n",
       "...        ...       ...       ...            ...        ...        ...   \n",
       "1096 -1.183764 -0.602962  0.319527       0.204967   0.166784  -0.866025   \n",
       "1131  2.132012  0.918253  0.319527       0.204967   0.166784   1.154701   \n",
       "1295 -0.815344 -1.363569  0.319527       0.204967   0.166784  -0.866025   \n",
       "861  -0.078505 -0.602962  0.319527       0.204967   0.166784   1.154701   \n",
       "1127 -0.078505  0.918253  0.319527       0.204967   0.166784  -0.866025   \n",
       "\n",
       "       PoolExt  TotalLivBsmtSF  \n",
       "Id                              \n",
       "908  -0.066884        0.654073  \n",
       "783  -0.066884        0.367110  \n",
       "953  -0.066884       -1.256527  \n",
       "621  -0.066884       -1.018013  \n",
       "670  -0.066884       -0.829189  \n",
       "...        ...             ...  \n",
       "1096 -0.066884        0.100024  \n",
       "1131 -0.066884        0.690098  \n",
       "1295 -0.066884       -1.018013  \n",
       "861  -0.066884       -0.260232  \n",
       "1127 -0.066884        0.472702  \n",
       "\n",
       "[1314 rows x 83 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 11;\n",
       "                var nbb_unformatted_code = \"X_train\";\n",
       "                var nbb_formatted_code = \"X_train\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "X_train"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "stretch-trauma",
   "metadata": {
    "id": "stretch-trauma"
   },
   "source": [
    "# Train"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63e44bcb",
   "metadata": {},
   "source": [
    "## Hyperparameter tuning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "85d270c6",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-22T09:32:50.179977Z",
     "start_time": "2022-12-22T09:32:41.321706Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fff343703b0f444d9fb890b306b1a135",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "MetricVisualizer(layout=Layout(align_self='stretch', height='500px'))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 12;\n",
       "                var nbb_unformatted_code = \"model = CatBoostRegressor(logging_level=\\\"Silent\\\")\\n\\n# https://catboost.ai/en/docs/concepts/parameter-tuning\\n# https://docs.aws.amazon.com/sagemaker/latest/dg/catboost-tuning.html\\ntuned_params = {\\n    \\\"learning_rate\\\": np.concatenate(\\n        (\\n            np.arange(1, 10, 1) / 10,\\n            np.arange(1, 10, 1) / 100,\\n            np.arange(1, 10, 1) / 1000,\\n        )\\n    ),\\n    \\\"depth\\\": np.arange(1, 11),\\n    \\\"l2_leaf_reg\\\": np.concatenate((np.arange(1, 10), np.arange(10, 110, 10))),\\n    \\\"random_strength\\\": np.arange(1, 11),\\n    \\\"iterations\\\": np.arange(100, 1100, 100),\\n}\\n\\ngrid_search_result = model.randomized_search(\\n    tuned_params, Pool(X_train, y_train), cv=3, verbose=False, plot=True\\n)\";\n",
       "                var nbb_formatted_code = \"model = CatBoostRegressor(logging_level=\\\"Silent\\\")\\n\\n# https://catboost.ai/en/docs/concepts/parameter-tuning\\n# https://docs.aws.amazon.com/sagemaker/latest/dg/catboost-tuning.html\\ntuned_params = {\\n    \\\"learning_rate\\\": np.concatenate(\\n        (\\n            np.arange(1, 10, 1) / 10,\\n            np.arange(1, 10, 1) / 100,\\n            np.arange(1, 10, 1) / 1000,\\n        )\\n    ),\\n    \\\"depth\\\": np.arange(1, 11),\\n    \\\"l2_leaf_reg\\\": np.concatenate((np.arange(1, 10), np.arange(10, 110, 10))),\\n    \\\"random_strength\\\": np.arange(1, 11),\\n    \\\"iterations\\\": np.arange(100, 1100, 100),\\n}\\n\\ngrid_search_result = model.randomized_search(\\n    tuned_params, Pool(X_train, y_train), cv=3, verbose=False, plot=True\\n)\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model = CatBoostRegressor(logging_level=\"Silent\")\n",
    "\n",
    "# https://catboost.ai/en/docs/concepts/parameter-tuning\n",
    "# https://docs.aws.amazon.com/sagemaker/latest/dg/catboost-tuning.html\n",
    "tuned_params = {\n",
    "    \"learning_rate\": np.concatenate(\n",
    "        (\n",
    "            np.arange(1, 10, 1) / 10,\n",
    "            np.arange(1, 10, 1) / 100,\n",
    "            np.arange(1, 10, 1) / 1000,\n",
    "        )\n",
    "    ),\n",
    "    \"depth\": np.arange(1, 11),\n",
    "    \"l2_leaf_reg\": np.concatenate((np.arange(1, 10), np.arange(10, 110, 10))),\n",
    "    \"random_strength\": np.arange(1, 11),\n",
    "    \"iterations\": np.arange(100, 1100, 100),\n",
    "}\n",
    "\n",
    "grid_search_result = model.randomized_search(\n",
    "    tuned_params, Pool(X_train, y_train), cv=3, verbose=False, plot=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "19abbc07",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-22T09:32:50.184941Z",
     "start_time": "2022-12-22T09:32:50.181079Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'random_strength': 1,\n",
       " 'depth': 3,\n",
       " 'l2_leaf_reg': 6,\n",
       " 'iterations': 800,\n",
       " 'learning_rate': 0.09}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 13;\n",
       "                var nbb_unformatted_code = \"best_model_params = grid_search_result[\\\"params\\\"]\\nbest_model_params\";\n",
       "                var nbb_formatted_code = \"best_model_params = grid_search_result[\\\"params\\\"]\\nbest_model_params\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "best_model_params = grid_search_result[\"params\"]\n",
    "best_model_params"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e35ce62",
   "metadata": {},
   "source": [
    "## Feature selections"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "aef38fe0",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-22T09:32:50.192956Z",
     "start_time": "2022-12-22T09:32:50.185872Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((1182, 83), (132, 83), (1182,), (132,))"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 14;\n",
       "                var nbb_unformatted_code = \"X_sf_train, X_sf_val, y_sf_train, y_sf_val = train_test_split(\\n    X_train, y_train, test_size=0.1, random_state=42\\n)\\nX_sf_train.shape, X_sf_val.shape, y_sf_train.shape, y_sf_val.shape\";\n",
       "                var nbb_formatted_code = \"X_sf_train, X_sf_val, y_sf_train, y_sf_val = train_test_split(\\n    X_train, y_train, test_size=0.1, random_state=42\\n)\\nX_sf_train.shape, X_sf_val.shape, y_sf_train.shape, y_sf_val.shape\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "X_sf_train, X_sf_val, y_sf_train, y_sf_val = train_test_split(\n",
    "    X_train, y_train, test_size=0.1, random_state=42\n",
    ")\n",
    "X_sf_train.shape, X_sf_val.shape, y_sf_train.shape, y_sf_val.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "4ebe4da2",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-22T09:32:50.592566Z",
     "start_time": "2022-12-22T09:32:50.193867Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<catboost.core.CatBoostRegressor at 0x7fafb438caf0>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 15;\n",
       "                var nbb_unformatted_code = \"sf_model = CatBoostRegressor(**best_model_params, logging_level=\\\"Silent\\\")\\nsf_model.fit(\\n    Pool(X_sf_train, y_sf_train),\\n    eval_set=Pool(X_sf_val, y_sf_val),\\n    verbose=False,\\n)\";\n",
       "                var nbb_formatted_code = \"sf_model = CatBoostRegressor(**best_model_params, logging_level=\\\"Silent\\\")\\nsf_model.fit(\\n    Pool(X_sf_train, y_sf_train),\\n    eval_set=Pool(X_sf_val, y_sf_val),\\n    verbose=False,\\n)\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sf_model = CatBoostRegressor(**best_model_params, logging_level=\"Silent\")\n",
    "sf_model.fit(\n",
    "    Pool(X_sf_train, y_sf_train),\n",
    "    eval_set=Pool(X_sf_val, y_sf_val),\n",
    "    verbose=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "28f11efc",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-22T09:32:50.602727Z",
     "start_time": "2022-12-22T09:32:50.593535Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Score</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Column</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>TotalLivBsmtSF</th>\n",
       "      <td>26.527573</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>OverallQual</th>\n",
       "      <td>21.739055</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GrLivArea</th>\n",
       "      <td>5.129957</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GarageCars</th>\n",
       "      <td>3.791661</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Fireplaces</th>\n",
       "      <td>3.753480</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ExterQual</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ExterCond</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BsmtFinType2</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BsmtHalfBath</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Electrical</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>83 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                    Score\n",
       "Column                   \n",
       "TotalLivBsmtSF  26.527573\n",
       "OverallQual     21.739055\n",
       "GrLivArea        5.129957\n",
       "GarageCars       3.791661\n",
       "Fireplaces       3.753480\n",
       "...                   ...\n",
       "ExterQual        0.000000\n",
       "ExterCond        0.000000\n",
       "BsmtFinType2     0.000000\n",
       "BsmtHalfBath     0.000000\n",
       "Electrical       0.000000\n",
       "\n",
       "[83 rows x 1 columns]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 16;\n",
       "                var nbb_unformatted_code = \"importance_df = (\\n    pd.DataFrame(\\n        {\\n            \\\"Column\\\": X_train.columns,\\n            \\\"Score\\\": sf_model.get_feature_importance(),\\n        }\\n    )\\n    .sort_values(by=\\\"Score\\\", ascending=False)\\n    .set_index(\\\"Column\\\")\\n)\\nimportance_df\";\n",
       "                var nbb_formatted_code = \"importance_df = (\\n    pd.DataFrame(\\n        {\\n            \\\"Column\\\": X_train.columns,\\n            \\\"Score\\\": sf_model.get_feature_importance(),\\n        }\\n    )\\n    .sort_values(by=\\\"Score\\\", ascending=False)\\n    .set_index(\\\"Column\\\")\\n)\\nimportance_df\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "importance_df = (\n",
    "    pd.DataFrame(\n",
    "        {\n",
    "            \"Column\": X_train.columns,\n",
    "            \"Score\": sf_model.get_feature_importance(),\n",
    "        }\n",
    "    )\n",
    "    .sort_values(by=\"Score\", ascending=False)\n",
    "    .set_index(\"Column\")\n",
    ")\n",
    "importance_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "9127a0cf",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-22T09:32:50.608371Z",
     "start_time": "2022-12-22T09:32:50.603894Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((1314, 83), (1314,))"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 17;\n",
       "                var nbb_unformatted_code = \"X_sf = X_train[importance_df.index]\\ny_sf = y_train\\n\\nX_sf.shape, y_sf.shape\";\n",
       "                var nbb_formatted_code = \"X_sf = X_train[importance_df.index]\\ny_sf = y_train\\n\\nX_sf.shape, y_sf.shape\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "X_sf = X_train[importance_df.index]\n",
    "y_sf = y_train\n",
    "\n",
    "X_sf.shape, y_sf.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "cc370b7b",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-22T09:32:50.614011Z",
     "start_time": "2022-12-22T09:32:50.609424Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([10, 13, 16, 19, 22, 25, 28, 31, 34, 37, 40, 43, 46, 49, 52, 55, 58,\n",
       "       61, 64, 67, 70, 73, 76, 79, 82])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 18;\n",
       "                var nbb_unformatted_code = \"num_list = np.arange(10, X_sf.shape[1], 3)\\nnum_list\";\n",
       "                var nbb_formatted_code = \"num_list = np.arange(10, X_sf.shape[1], 3)\\nnum_list\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "num_list = np.arange(10, X_sf.shape[1], 3)\n",
    "num_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "032504a8",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-22T09:32:50.620204Z",
     "start_time": "2022-12-22T09:32:50.615032Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'learn': {'RMSE': 0.06328158506600852}}"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 19;\n",
       "                var nbb_unformatted_code = \"model.best_score_\";\n",
       "                var nbb_formatted_code = \"model.best_score_\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "79c1a4b6",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-22T09:32:58.801302Z",
     "start_time": "2022-12-22T09:32:50.621193Z"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "db7c658283e94c0a825fc96f0f69975a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>num_features</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>0.103163</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>0.104580</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>0.104714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>0.106384</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>70</th>\n",
       "      <td>0.108365</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>0.109520</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>0.109703</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52</th>\n",
       "      <td>0.109904</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>0.110051</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>0.110483</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 score\n",
       "num_features          \n",
       "31            0.103163\n",
       "28            0.104580\n",
       "37            0.104714\n",
       "25            0.106384\n",
       "70            0.108365\n",
       "22            0.109520\n",
       "49            0.109703\n",
       "52            0.109904\n",
       "64            0.110051\n",
       "34            0.110483"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 20;\n",
       "                var nbb_unformatted_code = \"def select_features_loop(X, y, num_features=10):\\n    X = X.iloc[:, :num_features]\\n\\n    X_sub_train, X_sub_val, y_sub_train, y_sub_val = train_test_split(\\n        X, y, test_size=0.1, shuffle=False, random_state=42\\n    )\\n\\n    model = CatBoostRegressor(**best_model_params, logging_level=\\\"Silent\\\")\\n    model.fit(\\n        Pool(X_sub_train, y_sub_train),\\n        eval_set=Pool(X_sub_val, y_sub_val),\\n        verbose=False,\\n    )\\n\\n    score = mean_squared_error(y_true, model.predict(X_true), squared=False)\\n    # score = model.best_score_[\\\"validation\\\"][\\\"RMSE\\\"]\\n\\n    return [num_features, score]\\n\\n\\nloss_list = []\\nfor num_features in tqdm(num_list):\\n    loss_values = select_features_loop(X_sf, y_sf, num_features)\\n    loss_list.append(loss_values)\\n\\nnum_features_df = (\\n    pd.DataFrame(loss_list, columns=[\\\"num_features\\\", \\\"score\\\"])\\n    .set_index(\\\"num_features\\\")\\n    .sort_values(by=\\\"score\\\")\\n)\\nnum_features_df.head(10)\";\n",
       "                var nbb_formatted_code = \"def select_features_loop(X, y, num_features=10):\\n    X = X.iloc[:, :num_features]\\n\\n    X_sub_train, X_sub_val, y_sub_train, y_sub_val = train_test_split(\\n        X, y, test_size=0.1, shuffle=False, random_state=42\\n    )\\n\\n    model = CatBoostRegressor(**best_model_params, logging_level=\\\"Silent\\\")\\n    model.fit(\\n        Pool(X_sub_train, y_sub_train),\\n        eval_set=Pool(X_sub_val, y_sub_val),\\n        verbose=False,\\n    )\\n\\n    score = mean_squared_error(y_true, model.predict(X_true), squared=False)\\n    # score = model.best_score_[\\\"validation\\\"][\\\"RMSE\\\"]\\n\\n    return [num_features, score]\\n\\n\\nloss_list = []\\nfor num_features in tqdm(num_list):\\n    loss_values = select_features_loop(X_sf, y_sf, num_features)\\n    loss_list.append(loss_values)\\n\\nnum_features_df = (\\n    pd.DataFrame(loss_list, columns=[\\\"num_features\\\", \\\"score\\\"])\\n    .set_index(\\\"num_features\\\")\\n    .sort_values(by=\\\"score\\\")\\n)\\nnum_features_df.head(10)\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def select_features_loop(X, y, num_features=10):\n",
    "    X = X.iloc[:, :num_features]\n",
    "\n",
    "    X_sub_train, X_sub_val, y_sub_train, y_sub_val = train_test_split(\n",
    "        X, y, test_size=0.1, shuffle=False, random_state=42\n",
    "    )\n",
    "\n",
    "    model = CatBoostRegressor(**best_model_params, logging_level=\"Silent\")\n",
    "    model.fit(\n",
    "        Pool(X_sub_train, y_sub_train),\n",
    "        eval_set=Pool(X_sub_val, y_sub_val),\n",
    "        verbose=False,\n",
    "    )\n",
    "\n",
    "    score = mean_squared_error(y_true, model.predict(X_true), squared=False)\n",
    "    # score = model.best_score_[\"validation\"][\"RMSE\"]\n",
    "\n",
    "    return [num_features, score]\n",
    "\n",
    "\n",
    "loss_list = []\n",
    "for num_features in tqdm(num_list):\n",
    "    loss_values = select_features_loop(X_sf, y_sf, num_features)\n",
    "    loss_list.append(loss_values)\n",
    "\n",
    "num_features_df = (\n",
    "    pd.DataFrame(loss_list, columns=[\"num_features\", \"score\"])\n",
    "    .set_index(\"num_features\")\n",
    "    .sort_values(by=\"score\")\n",
    ")\n",
    "num_features_df.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "2ca358a5",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-22T09:32:58.804744Z",
     "start_time": "2022-12-22T09:32:58.802302Z"
    }
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 21;\n",
       "                var nbb_unformatted_code = \"#X_train = X_train[X_train.columns[: num_features_df.index[0]]]\\n#X_train.shape\";\n",
       "                var nbb_formatted_code = \"# X_train = X_train[X_train.columns[: num_features_df.index[0]]]\\n# X_train.shape\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# X_train = X_train[X_train.columns[: num_features_df.index[0]]]\n",
    "# X_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ef647d1",
   "metadata": {},
   "source": [
    "## KFold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "BMilHM9JzzcG",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-22T09:32:58.811044Z",
     "start_time": "2022-12-22T09:32:58.805680Z"
    },
    "id": "BMilHM9JzzcG"
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 22;\n",
       "                var nbb_unformatted_code = \"kf = KFold(n_splits=5)\";\n",
       "                var nbb_formatted_code = \"kf = KFold(n_splits=5)\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "kf = KFold(n_splits=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "Y6MwSzWpK_FR",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-22T09:33:00.764120Z",
     "start_time": "2022-12-22T09:32:58.812039Z"
    },
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Y6MwSzWpK_FR",
    "outputId": "190af34d-cf23-4c59-8d34-2d5e65df4368"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'learn': {'RMSE': 0.05748547476006802}, 'validation': {'RMSE': 0.11718502167285287}}\n",
      "{'learn': {'RMSE': 0.057552164786365774}, 'validation': {'RMSE': 0.1456234144557095}}\n",
      "{'learn': {'RMSE': 0.05624911044848918}, 'validation': {'RMSE': 0.1443633666154224}}\n",
      "{'learn': {'RMSE': 0.05785965091942618}, 'validation': {'RMSE': 0.10734540528524772}}\n",
      "{'learn': {'RMSE': 0.057446421395864936}, 'validation': {'RMSE': 0.1109771435398126}}\n"
     ]
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 23;\n",
       "                var nbb_unformatted_code = \"ensemble = []\\n\\nfor train_index, val_index in kf.split(X_train):\\n    X_sub_train, X_sub_valid = X_train.iloc[train_index], X_train.iloc[val_index]\\n    y_sub_train, y_sub_valid = y_train.iloc[train_index], y_train.iloc[val_index]\\n\\n    train_pool = Pool(X_sub_train, y_sub_train)\\n    valid_pool = Pool(X_sub_valid, y_sub_valid)\\n\\n    model = CatBoostRegressor(**best_model_params, logging_level=\\\"Silent\\\")\\n    model.fit(train_pool, eval_set=valid_pool, verbose=False)\\n\\n    ensemble.append(model)\\n    print(model.best_score_)\";\n",
       "                var nbb_formatted_code = \"ensemble = []\\n\\nfor train_index, val_index in kf.split(X_train):\\n    X_sub_train, X_sub_valid = X_train.iloc[train_index], X_train.iloc[val_index]\\n    y_sub_train, y_sub_valid = y_train.iloc[train_index], y_train.iloc[val_index]\\n\\n    train_pool = Pool(X_sub_train, y_sub_train)\\n    valid_pool = Pool(X_sub_valid, y_sub_valid)\\n\\n    model = CatBoostRegressor(**best_model_params, logging_level=\\\"Silent\\\")\\n    model.fit(train_pool, eval_set=valid_pool, verbose=False)\\n\\n    ensemble.append(model)\\n    print(model.best_score_)\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ensemble = []\n",
    "\n",
    "for train_index, val_index in kf.split(X_train):\n",
    "    X_sub_train, X_sub_valid = X_train.iloc[train_index], X_train.iloc[val_index]\n",
    "    y_sub_train, y_sub_valid = y_train.iloc[train_index], y_train.iloc[val_index]\n",
    "\n",
    "    train_pool = Pool(X_sub_train, y_sub_train)\n",
    "    valid_pool = Pool(X_sub_valid, y_sub_valid)\n",
    "\n",
    "    model = CatBoostRegressor(**best_model_params, logging_level=\"Silent\")\n",
    "    model.fit(train_pool, eval_set=valid_pool, verbose=False)\n",
    "\n",
    "    ensemble.append(model)\n",
    "    print(model.best_score_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "40e2e9d1",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-22T09:33:00.774364Z",
     "start_time": "2022-12-22T09:33:00.765206Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<catboost.core.CatBoostRegressor at 0x7fafb42673d0>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 24;\n",
       "                var nbb_unformatted_code = \"models_avrg = to_regressor(\\n    sum_models(ensemble, weights=[1.0 / len(ensemble)] * len(ensemble))\\n)\\nmodels_avrg\";\n",
       "                var nbb_formatted_code = \"models_avrg = to_regressor(\\n    sum_models(ensemble, weights=[1.0 / len(ensemble)] * len(ensemble))\\n)\\nmodels_avrg\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "models_avrg = to_regressor(\n",
    "    sum_models(ensemble, weights=[1.0 / len(ensemble)] * len(ensemble))\n",
    ")\n",
    "models_avrg"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3750e0b9",
   "metadata": {},
   "source": [
    "# Validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "bca6bb6c",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-22T09:33:00.780592Z",
     "start_time": "2022-12-22T09:33:00.775348Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Id\n",
       "908     12.429220\n",
       "783     12.139404\n",
       "953     11.804856\n",
       "621     11.112463\n",
       "670     11.831386\n",
       "          ...    \n",
       "1096    12.080696\n",
       "1131    11.813037\n",
       "1295    11.652696\n",
       "861     12.154521\n",
       "1127    12.066816\n",
       "Name: SalePrice, Length: 1314, dtype: float64"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 25;\n",
       "                var nbb_unformatted_code = \"y_train\";\n",
       "                var nbb_formatted_code = \"y_train\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "y_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "25af2c88",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-22T09:33:00.788058Z",
     "start_time": "2022-12-22T09:33:00.781944Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.10890748208456814"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 26;\n",
       "                var nbb_unformatted_code = \"y_pred = models_avrg.predict(X_true)\\nmean_squared_error(y_pred, y_true, squared=False)\";\n",
       "                var nbb_formatted_code = \"y_pred = models_avrg.predict(X_true)\\nmean_squared_error(y_pred, y_true, squared=False)\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "y_pred = models_avrg.predict(X_true)\n",
    "mean_squared_error(y_pred, y_true, squared=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "YAefLlet94Be",
   "metadata": {
    "id": "YAefLlet94Be"
   },
   "source": [
    "# Submission"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "lEqikTpl93kX",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-22T09:33:00.804366Z",
     "start_time": "2022-12-22T09:33:00.790359Z"
    },
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 455
    },
    "id": "lEqikTpl93kX",
    "outputId": "dc792b09-d89d-48d9-ba4b-d8538060fc93"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SalePrice</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Id</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1461</th>\n",
       "      <td>123216.619126</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1462</th>\n",
       "      <td>160641.511760</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1463</th>\n",
       "      <td>188682.702150</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1464</th>\n",
       "      <td>196853.726502</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1465</th>\n",
       "      <td>190235.392462</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2915</th>\n",
       "      <td>78912.438462</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2916</th>\n",
       "      <td>82007.397294</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2917</th>\n",
       "      <td>161159.003586</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2918</th>\n",
       "      <td>116505.470645</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2919</th>\n",
       "      <td>229343.382610</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1459 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          SalePrice\n",
       "Id                 \n",
       "1461  123216.619126\n",
       "1462  160641.511760\n",
       "1463  188682.702150\n",
       "1464  196853.726502\n",
       "1465  190235.392462\n",
       "...             ...\n",
       "2915   78912.438462\n",
       "2916   82007.397294\n",
       "2917  161159.003586\n",
       "2918  116505.470645\n",
       "2919  229343.382610\n",
       "\n",
       "[1459 rows x 1 columns]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 27;\n",
       "                var nbb_unformatted_code = \"submission = pd.DataFrame(\\n    {\\\"Id\\\": X_test.index, \\\"SalePrice\\\": np.exp(models_avrg.predict(X_test))}\\n).set_index(\\\"Id\\\")\\nsubmission\";\n",
       "                var nbb_formatted_code = \"submission = pd.DataFrame(\\n    {\\\"Id\\\": X_test.index, \\\"SalePrice\\\": np.exp(models_avrg.predict(X_test))}\\n).set_index(\\\"Id\\\")\\nsubmission\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "submission = pd.DataFrame(\n",
    "    {\"Id\": X_test.index, \"SalePrice\": np.exp(models_avrg.predict(X_test))}\n",
    ").set_index(\"Id\")\n",
    "submission"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "sSqEovlD93nV",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-12-22T09:33:00.811346Z",
     "start_time": "2022-12-22T09:33:00.805485Z"
    },
    "id": "sSqEovlD93nV"
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "\n",
       "            setTimeout(function() {\n",
       "                var nbb_cell_id = 28;\n",
       "                var nbb_unformatted_code = \"submission.to_csv(\\n    \\\"../../data/house-prices-advanced-regression-techniques/submission.csv\\\"\\n)\";\n",
       "                var nbb_formatted_code = \"submission.to_csv(\\n    \\\"../../data/house-prices-advanced-regression-techniques/submission.csv\\\"\\n)\";\n",
       "                var nbb_cells = Jupyter.notebook.get_cells();\n",
       "                for (var i = 0; i < nbb_cells.length; ++i) {\n",
       "                    if (nbb_cells[i].input_prompt_number == nbb_cell_id) {\n",
       "                        if (nbb_cells[i].get_text() == nbb_unformatted_code) {\n",
       "                             nbb_cells[i].set_text(nbb_formatted_code);\n",
       "                        }\n",
       "                        break;\n",
       "                    }\n",
       "                }\n",
       "            }, 500);\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "submission.to_csv(\n",
    "    \"../../data/house-prices-advanced-regression-techniques/submission.csv\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7ea6a46",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "44f9e8fa",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3060db7d",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "include_colab_link": true,
   "name": "house-prices-advanced-regression-techniques.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}