tutorials/notebooks/02_ingesting.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ingest an existing Dataset or Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook we are going to showcase how to ingest an existing dataset or model into EOTDL.\n",
    "\n",
    "Once it is ingested, you can use it in the same way as any other dataset or model in EOTDL (exploring, downloading, etc.)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ingesting through the CLI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The recommended way to ingest a dataset is using the CLI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m                                                                                \u001b[0m\n",
      "\u001b[1m \u001b[0m\u001b[1;33mUsage: \u001b[0m\u001b[1meotdl datasets ingest [OPTIONS]\u001b[0m\u001b[1m                                        \u001b[0m\u001b[1m \u001b[0m\n",
      "\u001b[1m                                                                                \u001b[0m\n",
      " Ingest a dataset to the EOTDL.                                                 \n",
      " \u001b[2mThis command ingests the dataset to the EOTDL. The dataset must be a folder \u001b[0m   \n",
      " \u001b[2mwith the dataset files, and at least a metadata.yml file or a catalog.json \u001b[0m    \n",
      " \u001b[2mfile. If there are not these files, the ingestion will not work. All the files\u001b[0m \n",
      " \u001b[2min the folder will be uploaded to the EOTDL.\u001b[0m                                   \n",
      "                                                                                \n",
      " \u001b[2mThe following constraints apply to the dataset name:\u001b[0m                           \n",
      " \u001b[2m- It must be unique\u001b[0m                                                            \n",
      " \u001b[2m- It must be between 3 and 45 characters long\u001b[0m                                  \n",
      " \u001b[2m- It can only contain alphanumeric characters and dashes.\u001b[0m                      \n",
      "                                                                                \n",
      " \u001b[2mThe metadata.yml file must contain the following fields:\u001b[0m                       \n",
      " \u001b[2m- name: the name of the dataset\u001b[0m                                                \n",
      " \u001b[2m- authors: the author or authors of the dataset\u001b[0m                                \n",
      " \u001b[2m- license: the license of the dataset\u001b[0m                                          \n",
      " \u001b[2m- source: the source of the dataset\u001b[0m                                            \n",
      "                                                                                \n",
      " \u001b[2mIf using \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-verbose\u001b[0m\u001b[2m, it will print the progress of the ingestion.\u001b[0m               \n",
      "                                                                                \n",
      " \u001b[2mExamples\u001b[0m                                                                       \n",
      " \u001b[1;2;36m--------\u001b[0m                                                                       \n",
      " \u001b[2m$ eotdl dataset ingest \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-path\u001b[0m\u001b[2m /path/to/folder-with-dataset \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-verbose\u001b[0m\u001b[2m True\u001b[0m      \n",
      "                                                                                \n",
      "\u001b[2m╭─\u001b[0m\u001b[2m Options \u001b[0m\u001b[2m───────────────────────────────────────────────────────────────────\u001b[0m\u001b[2m─╮\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[31m*\u001b[0m  \u001b[1;36m-\u001b[0m\u001b[1;36m-path\u001b[0m     \u001b[1;32m-p\u001b[0m      \u001b[1;33mPATH\u001b[0m  Path to the dataset to ingest \u001b[2m[default: None]\u001b[0m    \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m                             \u001b[2;31m[required]                   \u001b[0m                    \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m    \u001b[1;36m-\u001b[0m\u001b[1;36m-verbose\u001b[0m          \u001b[1;33m    \u001b[0m  Verbose output. This will print the progress of  \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m                             the ingestion                                    \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m    \u001b[1;36m-\u001b[0m\u001b[1;36m-help\u001b[0m             \u001b[1;33m    \u001b[0m  Show this message and exit.                      \u001b[2m│\u001b[0m\n",
      "\u001b[2m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!eotdl datasets ingest --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to ingest a dataset you will need a folder in your system with the data you want to upload."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "eurosat_rgb_dataset\t jaca_dataset\t    jaca_dataset_stac_labels\n",
      "eurosat_rgb_stac\t jaca_dataset_q2    labels_scaneo\n",
      "eurosat_rgb_stac_labels  jaca_dataset_stac  sample_stacdataframe.csv\n"
     ]
    }
   ],
   "source": [
    "!ls example_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this tutorial we are going to work with a subsample of the [EuroSAT](https://www.eotdl.com/datasets/EuroSAT-RGB) dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['example_data/EuroSAT-small/metadata.yml',\n",
       " 'example_data/EuroSAT-small/Forest/Forest_3.tif',\n",
       " 'example_data/EuroSAT-small/Forest/Forest_1.tif',\n",
       " 'example_data/EuroSAT-small/Forest/Forest_2.tif',\n",
       " 'example_data/EuroSAT-small/AnnualCrop/AnnualCrop_3.tif',\n",
       " 'example_data/EuroSAT-small/AnnualCrop/AnnualCrop_1.tif',\n",
       " 'example_data/EuroSAT-small/AnnualCrop/AnnualCrop_2.tif']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from glob import glob \n",
    "\n",
    "files = glob('example_data/EuroSAT-small/**/*.*', recursive=True)\n",
    "files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A `metadata.yml` file is required for Q0 datasets and models, containing some basic required information (dataset authors, licens, link to source and dataset name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "authors:\n",
      "- Patrick Helber\n",
      "license: open\n",
      "source: http://madm.dfki.de/downloads\n",
      "name: EuroSAT-small\n"
     ]
    }
   ],
   "source": [
    "!cat example_data/EuroSAT-small/metadata.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The chosen name is the one that will appear in the repository, hence it must be unique, between 3 and 45 characters long and can only contain alphanumeric characters and dashes (learn more at [https://www.eotdl.com/docs/datasets/ingest](https://www.eotdl.com/docs/datasets/ingest)).\n",
    "\n",
    "Trying to ingest a dataset without a `metadata.yml` file will fail.\n",
    "\n",
    "If everything is correct, the ingestion process should suceed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Uploading directory example_data/EuroSAT-small...\n",
      "generating list of files to upload...\n",
      "100%|███████████████████████████████████████████| 7/7 [00:00<00:00, 4106.31it/s]\n",
      "No new files to upload\n"
     ]
    }
   ],
   "source": [
    "!eotdl datasets ingest -p example_data/EuroSAT-small/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now your dataset is avilable at EOTDL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['EuroSAT-small']\n"
     ]
    }
   ],
   "source": [
    "!eotdl datasets list -n eurosat-small"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Since the `EuroSAT-small` name is already taken, this process should fail for you. To solve it, just upload the dataset with a different name. However, this will polute the EOTDL with test datasets so we encourage you to try the ingestion process with a real dataset that you want to ingest (or overwrite your test dataset in the future with useful data)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to ingest Q1+ datasets, a valid STAC catalog is required instead of the `metadata.yml` file. We will explore this in the [data curation](tutorials/workshops/bids23/05_STAC_metadata.ipynb) notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can ingest a model exactly in the same way"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m                                                                                \u001b[0m\n",
      "\u001b[1m \u001b[0m\u001b[1;33mUsage: \u001b[0m\u001b[1meotdl models ingest [OPTIONS]\u001b[0m\u001b[1m                                          \u001b[0m\u001b[1m \u001b[0m\n",
      "\u001b[1m                                                                                \u001b[0m\n",
      " Ingest a model to the EOTDL.                                                   \n",
      " \u001b[2mThis command ingests the model to the EOTDL. The model must be a folder with \u001b[0m  \n",
      " \u001b[2mthe model files, and at least a metadata.yml file or a catalog.json file. If \u001b[0m  \n",
      " \u001b[2mthere are not these files, the ingestion will not work. All the files in the \u001b[0m  \n",
      " \u001b[2mfolder will be uploaded to the EOTDL.\u001b[0m                                          \n",
      "                                                                                \n",
      " \u001b[2mThe following constraints apply to the model name:\u001b[0m                             \n",
      " \u001b[2m- It must be unique\u001b[0m                                                            \n",
      " \u001b[2m- It must be between 3 and 45 characters long\u001b[0m                                  \n",
      " \u001b[2m- It can only contain alphanumeric characters and dashes.\u001b[0m                      \n",
      "                                                                                \n",
      " \u001b[2mThe metadata.yml file must contain the following fields:\u001b[0m                       \n",
      " \u001b[2m- name: the name of the model\u001b[0m                                                  \n",
      " \u001b[2m- authors: the author or authors of the model\u001b[0m                                  \n",
      " \u001b[2m- license: the license of the model\u001b[0m                                            \n",
      " \u001b[2m- source: the source of the model\u001b[0m                                              \n",
      "                                                                                \n",
      " \u001b[2mIf using \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-verbose\u001b[0m\u001b[2m, it will print the progress of the ingestion.\u001b[0m               \n",
      "                                                                                \n",
      " \u001b[2mExamples\u001b[0m                                                                       \n",
      " \u001b[1;2;36m--------\u001b[0m                                                                       \n",
      " \u001b[2m$ eotdl models ingest \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-path\u001b[0m\u001b[2m /path/to/folder-with-model \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-verbose\u001b[0m\u001b[2m True\u001b[0m         \n",
      "                                                                                \n",
      "\u001b[2m╭─\u001b[0m\u001b[2m Options \u001b[0m\u001b[2m───────────────────────────────────────────────────────────────────\u001b[0m\u001b[2m─╮\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[31m*\u001b[0m  \u001b[1;36m-\u001b[0m\u001b[1;36m-path\u001b[0m     \u001b[1;32m-p\u001b[0m      \u001b[1;33mPATH\u001b[0m  Path to the model to ingest \u001b[2m[default: None]\u001b[0m      \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m                             \u001b[2;31m[required]                 \u001b[0m                      \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m    \u001b[1;36m-\u001b[0m\u001b[1;36m-verbose\u001b[0m          \u001b[1;33m    \u001b[0m  Verbose output. This will print the progress of  \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m                             the ingestion                                    \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m    \u001b[1;36m-\u001b[0m\u001b[1;36m-help\u001b[0m             \u001b[1;33m    \u001b[0m  Show this message and exit.                      \u001b[2m│\u001b[0m\n",
      "\u001b[2m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!eotdl models ingest --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Versioning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, every time you re-upload a dataset or model a new version is created.\n",
    "\n",
    "When you download a dataset, the latest version is used by default."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset `EuroSAT-small v10` already exists at /home/juan/.cache/eotdl/datasets/EuroSAT-small/v10. To force download, use force=True or -f in the CLI.\n"
     ]
    }
   ],
   "source": [
    "!eotdl datasets get EuroSAT-small"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, you can specify the version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset `EuroSAT-small v1` already exists at /home/juan/.cache/eotdl/datasets/EuroSAT-small/v1. To force download, use force=True or -f in the CLI.\n"
     ]
    }
   ],
   "source": [
    "!eotdl datasets get EuroSAT-small -v 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "v1  v10  v2  v3  v9\n"
     ]
    }
   ],
   "source": [
    "!ls $HOME/.cache/eotdl/datasets/EuroSAT-small"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We apply versioning at dataset/model and file level, meaning only new or modified files will be uploaded in future re-uploads, downloading the appropriate files for each version.\n",
    "\n",
    "You can explore the different versions in the user interface."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ingesting through the Library"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can ingest datasets and models using the library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Uploading directory example_data/EuroSAT-small...\n",
      "generating list of files to upload...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 7/7 [00:00<00:00, 6440.04it/s]\n"
     ]
    },
    {
     "ename": "Exception",
     "evalue": "No new files to upload",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mException\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m/home/juan/Desktop/eotdl/tutorials/notebooks/02_ingesting.ipynb Cell 29\u001b[0m line \u001b[0;36m3\n\u001b[1;32m      <a href='vscode-notebook-cell://ssh-remote%2Bharley/home/juan/Desktop/eotdl/tutorials/notebooks/02_ingesting.ipynb#X45sdnNjb2RlLXJlbW90ZQ%3D%3D?line=0'>1</a>\u001b[0m \u001b[39mfrom\u001b[39;00m \u001b[39meotdl\u001b[39;00m\u001b[39m.\u001b[39;00m\u001b[39mdatasets\u001b[39;00m \u001b[39mimport\u001b[39;00m ingest_dataset\n\u001b[0;32m----> <a href='vscode-notebook-cell://ssh-remote%2Bharley/home/juan/Desktop/eotdl/tutorials/notebooks/02_ingesting.ipynb#X45sdnNjb2RlLXJlbW90ZQ%3D%3D?line=2'>3</a>\u001b[0m ingest_dataset(\u001b[39m\"\u001b[39;49m\u001b[39mexample_data/EuroSAT-small\u001b[39;49m\u001b[39m\"\u001b[39;49m)\n",
      "File \u001b[0;32m~/miniconda3/envs/eotdl/lib/python3.8/site-packages/eotdl/datasets/ingest.py:16\u001b[0m, in \u001b[0;36mingest_dataset\u001b[0;34m(path, verbose, logger)\u001b[0m\n\u001b[1;32m     13\u001b[0m     \u001b[39mraise\u001b[39;00m \u001b[39mException\u001b[39;00m(\u001b[39m\"\u001b[39m\u001b[39mPath must be a folder\u001b[39m\u001b[39m\"\u001b[39m)\n\u001b[1;32m     14\u001b[0m \u001b[39m# if \"catalog.json\" in [f.name for f in path.iterdir()]:\u001b[39;00m\n\u001b[1;32m     15\u001b[0m \u001b[39m#     return ingest_stac(path / \"catalog.json\", logger)\u001b[39;00m\n\u001b[0;32m---> 16\u001b[0m \u001b[39mreturn\u001b[39;00m ingest_folder(path, verbose, logger)\n",
      "File \u001b[0;32m~/miniconda3/envs/eotdl/lib/python3.8/site-packages/eotdl/auth/auth.py:47\u001b[0m, in \u001b[0;36mwith_auth.<locals>.wrapper\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m     45\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39mwrapper\u001b[39m(\u001b[39m*\u001b[39margs, \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mkwargs):\n\u001b[1;32m     46\u001b[0m     user \u001b[39m=\u001b[39m auth()\n\u001b[0;32m---> 47\u001b[0m     \u001b[39mreturn\u001b[39;00m func(\u001b[39m*\u001b[39;49margs, \u001b[39m*\u001b[39;49m\u001b[39m*\u001b[39;49mkwargs, user\u001b[39m=\u001b[39;49muser)\n",
      "File \u001b[0;32m~/miniconda3/envs/eotdl/lib/python3.8/site-packages/eotdl/datasets/ingest.py:44\u001b[0m, in \u001b[0;36mingest_folder\u001b[0;34m(folder, verbose, logger, user)\u001b[0m\n\u001b[1;32m     42\u001b[0m dataset_id \u001b[39m=\u001b[39m retrieve_dataset(metadata, user)\n\u001b[1;32m     43\u001b[0m \u001b[39m# ingest files\u001b[39;00m\n\u001b[0;32m---> 44\u001b[0m \u001b[39mreturn\u001b[39;00m ingest_files(\n\u001b[1;32m     45\u001b[0m     repo, dataset_id, folder, verbose, logger, user, endpoint\u001b[39m=\u001b[39;49m\u001b[39m\"\u001b[39;49m\u001b[39mdatasets\u001b[39;49m\u001b[39m\"\u001b[39;49m\n\u001b[1;32m     46\u001b[0m )\n",
      "File \u001b[0;32m~/miniconda3/envs/eotdl/lib/python3.8/site-packages/eotdl/files/ingest.py:102\u001b[0m, in \u001b[0;36mingest_files\u001b[0;34m(repo, dataset_or_model_id, folder, verbose, logger, user, endpoint)\u001b[0m\n\u001b[1;32m    100\u001b[0m items \u001b[39m=\u001b[39m retrieve_files(folder)\n\u001b[1;32m    101\u001b[0m \u001b[39m# retrieve files\u001b[39;00m\n\u001b[0;32m--> 102\u001b[0m upload_files, existing_files, large_files \u001b[39m=\u001b[39m generate_files_lists(\n\u001b[1;32m    103\u001b[0m     items, folder, dataset_or_model_id, endpoint, logger\n\u001b[1;32m    104\u001b[0m )\n\u001b[1;32m    105\u001b[0m logger(\u001b[39mf\u001b[39m\u001b[39m\"\u001b[39m\u001b[39m{\u001b[39;00m\u001b[39mlen\u001b[39m(upload_files)\u001b[39m \u001b[39m\u001b[39m+\u001b[39m\u001b[39m \u001b[39m\u001b[39mlen\u001b[39m(large_files)\u001b[39m}\u001b[39;00m\u001b[39m new files will be ingested\u001b[39m\u001b[39m\"\u001b[39m)\n\u001b[1;32m    106\u001b[0m logger(\u001b[39mf\u001b[39m\u001b[39m\"\u001b[39m\u001b[39m{\u001b[39;00m\u001b[39mlen\u001b[39m(existing_files)\u001b[39m}\u001b[39;00m\u001b[39m files already exist in dataset\u001b[39m\u001b[39m\"\u001b[39m)\n",
      "File \u001b[0;32m~/miniconda3/envs/eotdl/lib/python3.8/site-packages/eotdl/files/ingest.py:86\u001b[0m, in \u001b[0;36mgenerate_files_lists\u001b[0;34m(items, folder, dataset_or_model_id, endpoint, logger, max_size)\u001b[0m\n\u001b[1;32m     84\u001b[0m             upload_files\u001b[39m.\u001b[39mappend(data)\n\u001b[1;32m     85\u001b[0m \u001b[39mif\u001b[39;00m \u001b[39mlen\u001b[39m(upload_files) \u001b[39m==\u001b[39m \u001b[39m0\u001b[39m \u001b[39mand\u001b[39;00m \u001b[39mlen\u001b[39m(large_files) \u001b[39m==\u001b[39m \u001b[39m0\u001b[39m:\n\u001b[0;32m---> 86\u001b[0m     \u001b[39mraise\u001b[39;00m \u001b[39mException\u001b[39;00m(\u001b[39m\"\u001b[39m\u001b[39mNo new files to upload\u001b[39m\u001b[39m\"\u001b[39m)\n\u001b[1;32m     87\u001b[0m \u001b[39mreturn\u001b[39;00m upload_files, existing_files, large_files\n",
      "\u001b[0;31mException\u001b[0m: No new files to upload"
     ]
    }
   ],
   "source": [
    "from eotdl.datasets import ingest_dataset\n",
    "\n",
    "ingest_dataset(\"example_data/EuroSAT-small\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ingesting through the API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ingesting a dataset or model through the API is a multi step (and error prone) process:\n",
    "\n",
    "1. Create/Retrieve a dataset\n",
    "2. Create a version\n",
    "3. Ingest files to version\n",
    "\t1. Ingest small files in batches\n",
    "\t2. Ingest large files in chunks as multipart upload\n",
    "\t\t1. Create multipart upload\n",
    "\t\t2. Ingest chunks\n",
    "\t\t3. Complete multipart upload\n",
    "\t3. Ingest existing files in batches to new version\n",
    "\n",
    "The library/CLI will take care of these steps, so it is the recommended way to ingest a dataset. \n",
    "\n",
    "However, if you still want to ingest datasets with the API, we recommend following the previous steps using the API [documentation](https://api.eotdl.com/docs) or reading the implementation of the ingestion functions in the library. If you need further help, reach out to us at the Discord server.\n",
    "\n",
    "This is a process we would like to simplify in the future."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "eotdl",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}