tutorials/notebooks/21_stac_extensions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# STAC generation with extensions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the [previous notebook](20_stac.ipynb) we generated STAC metadata from Sentinel-2 imagery. Despithe this, the resulting metadata is kind of incomplete, as it lacks a powerful STAC feature: [extensions](https://stac-extensions.github.io/).\n",
    "\n",
    "Uncomment the following line to install eotdl if needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install eotdl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get a `STACDataFrame`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from eotdl.curation.stac.stac import STACGenerator\n",
    "from eotdl.curation.stac.assets import STACAssetGenerator\n",
    "from eotdl.curation.stac.parsers import UnestructuredParser\n",
    "from eotdl.curation.stac.dataframe_labeling import LabeledStrategy\n",
    "\n",
    "stac_generator = STACGenerator(item_parser=UnestructuredParser, \n",
    "                               assets_generator=STACAssetGenerator, \n",
    "                               labeling_strategy=LabeledStrategy,\n",
    "                               image_format='tif'\n",
    "                               )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>image</th>\n",
       "      <th>label</th>\n",
       "      <th>ix</th>\n",
       "      <th>collection</th>\n",
       "      <th>extensions</th>\n",
       "      <th>bands</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>example_data/jaca_dataset/Jaca_1.tif</td>\n",
       "      <td>Jaca</td>\n",
       "      <td>0</td>\n",
       "      <td>example_data/jaca_dataset/source</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>example_data/jaca_dataset/Jaca_2.tif</td>\n",
       "      <td>Jaca</td>\n",
       "      <td>0</td>\n",
       "      <td>example_data/jaca_dataset/source</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>example_data/jaca_dataset/Jaca_3.tif</td>\n",
       "      <td>Jaca</td>\n",
       "      <td>0</td>\n",
       "      <td>example_data/jaca_dataset/source</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>example_data/jaca_dataset/Jaca_4.tif</td>\n",
       "      <td>Jaca</td>\n",
       "      <td>0</td>\n",
       "      <td>example_data/jaca_dataset/source</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                  image label  ix  \\\n",
       "0  example_data/jaca_dataset/Jaca_1.tif  Jaca   0   \n",
       "1  example_data/jaca_dataset/Jaca_2.tif  Jaca   0   \n",
       "2  example_data/jaca_dataset/Jaca_3.tif  Jaca   0   \n",
       "3  example_data/jaca_dataset/Jaca_4.tif  Jaca   0   \n",
       "\n",
       "                         collection extensions bands  \n",
       "0  example_data/jaca_dataset/source       None  None  \n",
       "1  example_data/jaca_dataset/source       None  None  \n",
       "2  example_data/jaca_dataset/source       None  None  \n",
       "3  example_data/jaca_dataset/source       None  None  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = stac_generator.get_stac_dataframe('example_data/jaca_dataset/')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A key feature is the `label` column. Using the label of every image we are going to assign parameters like the STAC extensions that this image's item is going to have, or the bands we want to extract using the `BandsAssetGenerator`. We can obtain the existing labels in the STACDataFrame before adding new information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Jaca']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labels = df.label.unique().tolist()\n",
    "labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Starting from the found label we are going to define the STAC extensions. As STAC extensions we are going to implement the [proj](https://github.com/stac-extensions/projection), [raster](https://github.com/stac-extensions/raster) and [eo](https://github.com/stac-extensions/eo) STAC extensions. \n",
    "\n",
    "> Note: the supported extensiones are `('eo', 'sar', 'proj', 'raster')`.\n",
    "\n",
    "On the other hand, although we don't want to extract the image bands, we can define them to see their metadata using the `eo` STAC extension. To simplify, let's only define the bands `B04`, `B03` and `B02`, which are the RGB bands.\n",
    "\n",
    "To define these parameters for each label, we simply have to declare a dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "extensions = {'Jaca': ('proj', 'raster', 'eo')}\n",
    "bands = {'Jaca': ('B02', 'B03', 'B04')}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are ready to generate a `STACDataFrame` with relevant information. Some extra parameters to take into account:\n",
    "- `path`: is the root path where the images are located at. In our case is `data/sentinel_2`.\n",
    "- `collections`: we can use this parameter to define the STAC collection to which we want each item with a specific label to go. There are several options:\n",
    "    - The default option puts all the STAC items in a single collection called `source`.\n",
    "    - The `*` option will consider folders located directly under the root folder as collections, so it will create a collection for each of them.\n",
    "\n",
    "    <p align=\"center\">\n",
    "        <img src=\"assets/collection.png\" alt=\"* collection\" style=\"height:170px; width:200px;\"/>\n",
    "    </p>\n",
    "\n",
    "    - You can decide the collection you want an image to go to through its label, as we have seen in the case of extensions and bands. To give an example, we are going to define it like this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "collection = {'Jaca': 'sentinel-2-l2a'}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>image</th>\n",
       "      <th>label</th>\n",
       "      <th>ix</th>\n",
       "      <th>collection</th>\n",
       "      <th>extensions</th>\n",
       "      <th>bands</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>example_data/jaca_dataset/Jaca_1.tif</td>\n",
       "      <td>Jaca</td>\n",
       "      <td>0</td>\n",
       "      <td>example_data/jaca_dataset/sentinel-2-l2a</td>\n",
       "      <td>(proj, raster, eo)</td>\n",
       "      <td>(B02, B03, B04)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>example_data/jaca_dataset/Jaca_2.tif</td>\n",
       "      <td>Jaca</td>\n",
       "      <td>0</td>\n",
       "      <td>example_data/jaca_dataset/sentinel-2-l2a</td>\n",
       "      <td>(proj, raster, eo)</td>\n",
       "      <td>(B02, B03, B04)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>example_data/jaca_dataset/Jaca_3.tif</td>\n",
       "      <td>Jaca</td>\n",
       "      <td>0</td>\n",
       "      <td>example_data/jaca_dataset/sentinel-2-l2a</td>\n",
       "      <td>(proj, raster, eo)</td>\n",
       "      <td>(B02, B03, B04)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>example_data/jaca_dataset/Jaca_4.tif</td>\n",
       "      <td>Jaca</td>\n",
       "      <td>0</td>\n",
       "      <td>example_data/jaca_dataset/sentinel-2-l2a</td>\n",
       "      <td>(proj, raster, eo)</td>\n",
       "      <td>(B02, B03, B04)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                  image label  ix  \\\n",
       "0  example_data/jaca_dataset/Jaca_1.tif  Jaca   0   \n",
       "1  example_data/jaca_dataset/Jaca_2.tif  Jaca   0   \n",
       "2  example_data/jaca_dataset/Jaca_3.tif  Jaca   0   \n",
       "3  example_data/jaca_dataset/Jaca_4.tif  Jaca   0   \n",
       "\n",
       "                                 collection          extensions  \\\n",
       "0  example_data/jaca_dataset/sentinel-2-l2a  (proj, raster, eo)   \n",
       "1  example_data/jaca_dataset/sentinel-2-l2a  (proj, raster, eo)   \n",
       "2  example_data/jaca_dataset/sentinel-2-l2a  (proj, raster, eo)   \n",
       "3  example_data/jaca_dataset/sentinel-2-l2a  (proj, raster, eo)   \n",
       "\n",
       "             bands  \n",
       "0  (B02, B03, B04)  \n",
       "1  (B02, B03, B04)  \n",
       "2  (B02, B03, B04)  \n",
       "3  (B02, B03, B04)  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = stac_generator.get_stac_dataframe('example_data/jaca_dataset/', collections=collection, extensions=extensions, bands=bands)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating sentinel-2-l2a collection...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:00<00:00, 243.99it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validating and saving catalog...\n",
      "Success!\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "stac_generator.generate_stac_metadata(stac_id='jaca-dataset-extensions',\n",
    "                                      description='Jaca dataset with STAC extensions',\n",
    "                                      output_folder='data/jaca_dataset_stac_extensions')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check our new STAC catalog!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "\n",
       "<style>\n",
       ".pystac-summary {\n",
       "    cursor: pointer;\n",
       "    display:list-item;\n",
       "}\n",
       ".pystac-key {\n",
       "    color: rgb(0, 128, 0);\n",
       "    font-weight: 700;\n",
       "}\n",
       ".pystac-key-value {\n",
       "    display: inline-block;\n",
       "    margin: 0px 0.5em 0px 0px;\n",
       "}\n",
       "</style>\n",
       "<div class=\"jp-RenderedJSON jp-mod-trusted jp-OutputArea-output\">\n",
       "    <div class=\"container\" style=\"line-height: normal;\">\n",
       "        <ul style=\"padding: 0px; margin: 0px; list-style: none; display: block;\">\n",
       "            \n",
       "                \n",
       "                    \n",
       "        <li style=\"overflow-wrap: break-word; padding-left: 2.125em; text-indent: -0.5em;\">\n",
       "            <span class=\"pystac-key pystac-key-value\">type</span>\n",
       "            <span style=\"color: rgb(186, 33, 33);\">\"Catalog\"</span>\n",
       "        </li>\n",
       "    \n",
       "                \n",
       "            \n",
       "                \n",
       "                    \n",
       "        <li style=\"overflow-wrap: break-word; padding-left: 2.125em; text-indent: -0.5em;\">\n",
       "            <span class=\"pystac-key pystac-key-value\">id</span>\n",
       "            <span style=\"color: rgb(186, 33, 33);\">\"jaca-dataset-extensions\"</span>\n",
       "        </li>\n",
       "    \n",
       "                \n",
       "            \n",
       "                \n",
       "                    \n",
       "        <li style=\"overflow-wrap: break-word; padding-left: 2.125em; text-indent: -0.5em;\">\n",
       "            <span class=\"pystac-key pystac-key-value\">stac_version</span>\n",
       "            <span style=\"color: rgb(186, 33, 33);\">\"1.0.0\"</span>\n",
       "        </li>\n",
       "    \n",
       "                \n",
       "            \n",
       "                \n",
       "                    \n",
       "        <li style=\"overflow-wrap: break-word; padding-left: 2.125em; text-indent: -0.5em;\">\n",
       "            <span class=\"pystac-key pystac-key-value\">description</span>\n",
       "            <span style=\"color: rgb(186, 33, 33);\">\"Jaca dataset with STAC extensions\"</span>\n",
       "        </li>\n",
       "    \n",
       "                \n",
       "            \n",
       "                \n",
       "                    <li><details>\n",
       "        <summary class=\"pystac-summary\">\n",
       "            <span class=\"pystac-key\">links</span>\n",
       "            <span style=\"padding-left: 0.5em; color: rgb(64, 128, 128); font-style: italic;\">[] 3 items</span>\n",
       "        </summary>\n",
       "        \n",
       "            <ul style=\"margin: 0px; padding: 0px 0px 0px 1.75em; list-style: none; display: block;\">\n",
       "        \n",
       "            \n",
       "                \n",
       "        <li><details>\n",
       "            <summary class=\"pystac-summary\"><span class=\"pystac-key\">0</span></summary>\n",
       "            <ul style=\"margin: 0px; padding: 0px 0px 0px 1.75em; list-style: none; display: block;\">\n",
       "        \n",
       "            \n",
       "                \n",
       "        <li style=\"overflow-wrap: break-word; padding-left: 2.125em; text-indent: -0.5em;\">\n",
       "            <span class=\"pystac-key pystac-key-value\">rel</span>\n",
       "            <span style=\"color: rgb(186, 33, 33);\">\"self\"</span>\n",
       "        </li>\n",
       "    \n",
       "            \n",
       "        \n",
       "            \n",
       "                \n",
       "        <li style=\"overflow-wrap: break-word; padding-left: 2.125em; text-indent: -0.5em;\">\n",
       "            <span class=\"pystac-key pystac-key-value\">href</span>\n",
       "            <span style=\"color: rgb(186, 33, 33);\">\"/Users/fran/Documents/Projects/eotdl/tutorials/notebooks/data/jaca_dataset_stac_extensions/catalog.json\"</span>\n",
       "        </li>\n",
       "    \n",
       "            \n",
       "        \n",
       "            \n",
       "                \n",
       "        <li style=\"overflow-wrap: break-word; padding-left: 2.125em; text-indent: -0.5em;\">\n",
       "            <span class=\"pystac-key pystac-key-value\">type</span>\n",
       "            <span style=\"color: rgb(186, 33, 33);\">\"application/json\"</span>\n",
       "        </li>\n",
       "    \n",
       "            \n",
       "        \n",
       "    </ul>\n",
       "        </details></li>\n",
       "    \n",
       "            \n",
       "        \n",
       "    </ul>\n",
       "        \n",
       "            <ul style=\"margin: 0px; padding: 0px 0px 0px 1.75em; list-style: none; display: block;\">\n",
       "        \n",
       "            \n",
       "                \n",
       "        <li><details>\n",
       "            <summary class=\"pystac-summary\"><span class=\"pystac-key\">1</span></summary>\n",
       "            <ul style=\"margin: 0px; padding: 0px 0px 0px 1.75em; list-style: none; display: block;\">\n",
       "        \n",
       "            \n",
       "                \n",
       "        <li style=\"overflow-wrap: break-word; padding-left: 2.125em; text-indent: -0.5em;\">\n",
       "            <span class=\"pystac-key pystac-key-value\">rel</span>\n",
       "            <span style=\"color: rgb(186, 33, 33);\">\"root\"</span>\n",
       "        </li>\n",
       "    \n",
       "            \n",
       "        \n",
       "            \n",
       "                \n",
       "        <li style=\"overflow-wrap: break-word; padding-left: 2.125em; text-indent: -0.5em;\">\n",
       "            <span class=\"pystac-key pystac-key-value\">href</span>\n",
       "            <span style=\"color: rgb(186, 33, 33);\">\"/Users/fran/Documents/Projects/eotdl/tutorials/notebooks/data/jaca_dataset_stac_extensions/catalog.json\"</span>\n",
       "        </li>\n",
       "    \n",
       "            \n",
       "        \n",
       "            \n",
       "                \n",
       "        <li style=\"overflow-wrap: break-word; padding-left: 2.125em; text-indent: -0.5em;\">\n",
       "            <span class=\"pystac-key pystac-key-value\">type</span>\n",
       "            <span style=\"color: rgb(186, 33, 33);\">\"application/json\"</span>\n",
       "        </li>\n",
       "    \n",
       "            \n",
       "        \n",
       "    </ul>\n",
       "        </details></li>\n",
       "    \n",
       "            \n",
       "        \n",
       "    </ul>\n",
       "        \n",
       "            <ul style=\"margin: 0px; padding: 0px 0px 0px 1.75em; list-style: none; display: block;\">\n",
       "        \n",
       "            \n",
       "                \n",
       "        <li><details>\n",
       "            <summary class=\"pystac-summary\"><span class=\"pystac-key\">2</span></summary>\n",
       "            <ul style=\"margin: 0px; padding: 0px 0px 0px 1.75em; list-style: none; display: block;\">\n",
       "        \n",
       "            \n",
       "                \n",
       "        <li style=\"overflow-wrap: break-word; padding-left: 2.125em; text-indent: -0.5em;\">\n",
       "            <span class=\"pystac-key pystac-key-value\">rel</span>\n",
       "            <span style=\"color: rgb(186, 33, 33);\">\"child\"</span>\n",
       "        </li>\n",
       "    \n",
       "            \n",
       "        \n",
       "            \n",
       "                \n",
       "        <li style=\"overflow-wrap: break-word; padding-left: 2.125em; text-indent: -0.5em;\">\n",
       "            <span class=\"pystac-key pystac-key-value\">href</span>\n",
       "            <span style=\"color: rgb(186, 33, 33);\">\"./sentinel-2-l2a/collection.json\"</span>\n",
       "        </li>\n",
       "    \n",
       "            \n",
       "        \n",
       "            \n",
       "                \n",
       "        <li style=\"overflow-wrap: break-word; padding-left: 2.125em; text-indent: -0.5em;\">\n",
       "            <span class=\"pystac-key pystac-key-value\">type</span>\n",
       "            <span style=\"color: rgb(186, 33, 33);\">\"application/json\"</span>\n",
       "        </li>\n",
       "    \n",
       "            \n",
       "        \n",
       "    </ul>\n",
       "        </details></li>\n",
       "    \n",
       "            \n",
       "        \n",
       "    </ul>\n",
       "        \n",
       "    </details></li>\n",
       "                \n",
       "            \n",
       "        </ul>\n",
       "    </div>\n",
       "</div>"
      ],
      "text/plain": [
       "<Catalog id=jaca-dataset-extensions>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from pystac import Catalog\n",
    "\n",
    "Catalog.from_file('data/jaca_dataset_stac_extensions/catalog.json')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "sen12floods",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}