Intro_to_Argovis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0b065678",
   "metadata": {},
   "source": [
    "# Introduction to Argovis's API"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1769e82",
   "metadata": {},
   "source": [
    "Argovis provides an API that indexes and distributes numerous oceanographic datasets with detailed query parameters, enabling you to search and download only and exactly data of interest. In this notebook, we'll tour some of the standard usage patterns enabled by Argovis."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a7ba8dc",
   "metadata": {},
   "source": [
    "## Note on how to navigate this and other notebooks in the repository"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "effaed84",
   "metadata": {},
   "source": [
    "We suggest to:\n",
    "1. Start from this notebook to have an overview of the standard usage patterns enabled by Argovis.\n",
    "2. Try some tasks that Argovis supports via the Argovis API, running e.g. **Argovis_explore_ocean_vertical_structure** (and in general current and upcoming notebooks with file name starting in Argovis_explore_*): if the task of interest for you is included in these notebooks, we reccomend starting your code from the one used in these notebooks.\n",
    "3. Visit the folder **dataset_specific_notebooks** for notebooks with examples that are specific to individual Argovis datasets (some of the code here will be modified and/or moved to the parent folder to allow for usage with multiple datasets).\n",
    "\n",
    "Notebooks in the folder work in progress are under development and not guaranteed to work as is."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c62091b",
   "metadata": {},
   "source": [
    "## Setup: Register an API key"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97168963",
   "metadata": {},
   "source": [
    "In order to allocate Argovis's limited computing resources fairly, users are encouraged to register and request a free API key. This works like a password that identifies your requests to Argovis. To do so:\n",
    "\n",
    " - Visit [https://argovis-keygen.colorado.edu/](https://argovis-keygen.colorado.edu/)\n",
    " - Fill out the form under _New Account Registration_\n",
    " - An API key will be emailed to you shortly.\n",
    " \n",
    "Treat this API key like a password - don't share it or leave it anywhere public. If you ever forget it or accidentally reveal it to a third party, see the same website above to change or deactivate your token.\n",
    "\n",
    "Put your API key in the quotes in the variable below before moving on:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "2f0cc646",
   "metadata": {},
   "outputs": [],
   "source": [
    "API_ROOT='https://argovis-api.colorado.edu/'\n",
    "API_KEY=''"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a51b0c32",
   "metadata": {},
   "source": [
    "# Argovis data structures"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c677f7d",
   "metadata": {},
   "source": [
    "Argovis standard data structures divide measurements into _data_ and _metadata_ documents. Typically, a data document corresponds to measurements or gridded data associated with a discreet temporospatial column - a time, latitude and longitude. A single such document may contain measurements at multiple depths or altitudes, provided they share the same latitude, longitude, and time.\n",
    "\n",
    "Each of these data documents will refer to at least one corresponding metadata document that captures additional information about the measurement. Argovis divides information between data and metadata documents in order to minimize redundancy in the data you download: many data documents will point to the same metadata document, allowing you to only download that metadata once. Typically, these metadata groupings will refer to some meaningful characteristic of the data; Argo metadata documents correspond to physical floats, while CCHDO metadata documents correspond to cruises, for example.\n",
    "\n",
    "For more detail and specifications on the data and metadata documents for each collection, see [https://argovis.colorado.edu/docs/documentation/_build/html/database/schema.html](https://argovis.colorado.edu/docs/documentation/_build/html/database/schema.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c70c044",
   "metadata": {},
   "source": [
    "# The standard data routes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d329222",
   "metadata": {},
   "source": [
    "## What datasets does Argovis index?\n",
    "\n",
    "Argovis supports several different data sets with the API and data structures described here. They and their corresponding routes are:\n",
    "\n",
    " - Argo profiling float data, `/argo`\n",
    " - CCHDO ship-based profile data, `/cchdo`\n",
    " - tropical cyclone data from HURDAT and JTWC, `/tc`\n",
    " - Global Drifter Program data, `/drifters`\n",
    " - Easy Ocean, `/easyocean`\n",
    " - several gridded products:\n",
    "   - Roemmich-Gilson total temperature and salinity, `/grids/rg09`\n",
    "   - ocean heat content, `/grids/kg21`\n",
    "   - GLODAP, `/grids/glodap`\n",
    " - Argone Argo float position forecast model data, `/argone`\n",
    " - Argo trajectory data, `/argotrajectories`\n",
    " - several satellite-based timeseries:\n",
    "   - NOAA sea surface temperature, `/timeseries/noaasst`\n",
    "   - Copernicus sea surface height, `/timeseries/copernicussla`\n",
    "   - CCMP wind vector product, `/timeseries/ccmpwind`\n",
    "   \n",
    "The examples that follow apply equally to all these routes; they all support similar query options and follow similar behavior patterns."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c898a1f",
   "metadata": {},
   "source": [
    "## Using Swagger and the `argovisHelpers` package to download data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b175d99f",
   "metadata": {},
   "source": [
    "In order to successfully explore Argovis data, there are two important tools to introduce in this section: Swagger, our API documentation engine, and `argovisHelpers`, our Python package of fuctions to help you access and interpret Argovis data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ae17340",
   "metadata": {},
   "source": [
    "### Using Swagger docs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e65db44",
   "metadata": {},
   "source": [
    "Argovis' API documentation is found at [https://argovis-api.colorado.edu/docs/](https://argovis-api.colorado.edu/docs/). These docs are split into several categories; what follows applies to all categories _not_ marked experimental; the experimental categories are under development and may change or be removed at any time.\n",
    "\n",
    "Categories have three typical routes:\n",
    " - The main _data route_, like `/argo`, or `/cchdo`. These routes provide the data documents for the dataset named in the route.\n",
    " - The _metadata route_, like `/argo/meta`. These routes provide the metadata documents referred to by data documents.\n",
    " - The _vocabulary route_, like `/argo/vocabulary`. These routes provide lists of possible options for search parameters used in the corresponding data and metadata routes.\n",
    " \n",
    "Click on any of the routes, like `/argo` - a list of possible query string parameters are presented, with a short explanation of what they mean.\n",
    "\n",
    "If you're familiar with REST APIs, this is enough information for you to construct a query string and issue a request in any programming environment that can facilitate an HTTP GET request. If you're working in Python, we provide a helper library, `argovisHelpers`, to manage these requests for you. Let's try it out by making our first request for Argo data, for profiles found within 100 km of a point in the South Atlantic in May 2011 (users of Python's `requests` module will notice a familiar pattern, providing the query string parameters listed in the Swagger docs and associated values as a dictionary):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2e2414e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from argovisHelpers import helpers as avh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "8a522910",
   "metadata": {},
   "outputs": [],
   "source": [
    "argoSearch = {\n",
    "    'startDate': '2011-05-01T00:00:00Z',\n",
    "    'endDate': '2011-06-01T00:00:00Z',\n",
    "    'center': '-22.5,0',\n",
    "    'radius': 100\n",
    "}\n",
    "\n",
    "argoProfiles = avh.query('argo', options=argoSearch, apikey=API_KEY, apiroot=API_ROOT)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85127028",
   "metadata": {},
   "source": [
    "Let's have a look at what we get from the first profile returned:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "8fad8197",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'_id': '1901094_109',\n",
       " 'geolocation': {'type': 'Point', 'coordinates': [-21.851, 0.602]},\n",
       " 'basin': 1,\n",
       " 'timestamp': '2011-05-31T04:48:00.000Z',\n",
       " 'date_updated_argovis': '2023-01-25T06:43:16.110Z',\n",
       " 'source': [{'source': ['argo_core'],\n",
       "   'url': 'ftp://ftp.ifremer.fr/ifremer/argo/dac/coriolis/1901094/profiles/D1901094_109.nc',\n",
       "   'date_updated': '2015-06-10T11:16:52.000Z'}],\n",
       " 'cycle_number': 109,\n",
       " 'geolocation_argoqc': 1,\n",
       " 'profile_direction': 'A',\n",
       " 'timestamp_argoqc': 1,\n",
       " 'vertical_sampling_scheme': 'Primary sampling: averaged [10 sec sampling, 25 dbar average from 2000 dbar to 200 dbar; 10 sec sampling, 10 dbar average from 200 dbar to 10.0 dbar]',\n",
       " 'data_info': [['pressure',\n",
       "   'pressure_argoqc',\n",
       "   'salinity',\n",
       "   'salinity_argoqc',\n",
       "   'temperature',\n",
       "   'temperature_argoqc'],\n",
       "  ['units', 'data_keys_mode'],\n",
       "  [['decibar', 'D'],\n",
       "   [None, None],\n",
       "   ['psu', 'D'],\n",
       "   [None, None],\n",
       "   ['degree_Celsius', 'D'],\n",
       "   [None, None]]],\n",
       " 'metadata': ['1901094_m0']}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "argoProfiles[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b151408",
   "metadata": {},
   "source": [
    "This is a data document for Argo, matching the specification at [https://argovis.colorado.edu/docs/documentation/_build/html/database/schema.html](https://argovis.colorado.edu/docs/documentation/_build/html/database/schema.html). It contains the `timestamp` and `geolocation` properties that place this profile geospatially, and other parameters that typically change from point to point.\n",
    "\n",
    "All data documents bear a `metadata` key, which is a pointer to the appropriate metadata document to find out more about this measurement. Let's fetch that document for this first profile by querying the `argo/meta` route for a doument with an `id` that matches this `metadata` pointer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "2b667cd5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'_id': '1901094_m0',\n",
       "  'data_type': 'oceanicProfile',\n",
       "  'data_center': 'IF',\n",
       "  'instrument': 'profiling_float',\n",
       "  'pi_name': ['Christine Coatanoan'],\n",
       "  'platform': '1901094',\n",
       "  'platform_type': 'PROVOR',\n",
       "  'fleetmonitoring': 'https://fleetmonitoring.euro-argo.eu/float/1901094',\n",
       "  'oceanops': 'https://www.ocean-ops.org/board/wa/Platform?ref=1901094',\n",
       "  'positioning_system': 'ARGOS',\n",
       "  'wmo_inst_type': '841'}]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metaOptions = {\n",
    "    'id': argoProfiles[0]['metadata'][0]\n",
    "}\n",
    "\n",
    "argoMeta = avh.query('argo/meta', options=metaOptions, apikey=API_KEY, apiroot=API_ROOT)\n",
    "argoMeta"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0d0af84",
   "metadata": {},
   "source": [
    "In addition to temporospatial searches, data and metadata routes typically support _category searches_, which are searches for documents that belong to certain categories. Which categories are available to search by changes logically from dataset to dataset; Argo floats can be searched by platform number, for example, while tropical cyclones can be searched by storm name. See the swagger docs for the full set of possibilities for each category; let's now use argo's platform category search to get all profiles collected by the same platform as the first profile above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "c059c9d8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "176\n"
     ]
    }
   ],
   "source": [
    "platformSearch = {\n",
    "    'platform': argoMeta[0]['platform']\n",
    "}\n",
    "\n",
    "platformProfiles = avh.query('argo', options=platformSearch, apikey=API_KEY, apiroot=API_ROOT)\n",
    "print(len(platformProfiles))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "854aed0f",
   "metadata": {},
   "source": [
    "At the time of writing, 125 profiles are found for this platform in this way.\n",
    "\n",
    "For all category searches, we may wish to know the full list of all possible values a category can take on; for this, there are the _vocabulary_ routes. All vocabulary routes support a parameter `enum`, to list what other categorical parameters are available to filter this dataset by:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "6b1957d9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['platform', 'source', 'data', 'metadata', 'platform_type', 'position_qc']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab_enum = {\n",
    "    'parameter': 'enum'\n",
    "}\n",
    "\n",
    "avh.query('argo/vocabulary', options=vocab_enum, apikey=API_KEY, apiroot=API_ROOT)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a881b5e5",
   "metadata": {},
   "source": [
    "Evidently we can filter Argo data by platform, for example. Let's see what platforms are available:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "f8a7f8cf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://argovis-api.colorado.edu/argo/vocabulary?parameter=platform\n",
      "['13857', '13858', '13859', '15819', '15820', '15851', '15852', '15853', '15854', '15855']\n"
     ]
    }
   ],
   "source": [
    "platformVocabSearch = {\n",
    "    'parameter': 'platform'\n",
    "}\n",
    "\n",
    "platforms = avh.query('argo/vocabulary', options=platformVocabSearch, apikey=API_KEY, apiroot=API_ROOT, verbose=True)\n",
    "print(platforms[0:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d16a6969",
   "metadata": {},
   "source": [
    "Here we just print out the first 10 platform IDs found, but all 17 thousand or so are present."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92315691",
   "metadata": {},
   "source": [
    "## Using the `data` query option"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c0c1840",
   "metadata": {},
   "source": [
    "The astute reader may have noticed something about the data document shown above: there's no actual measurements included in it! By default, only the non-measurement data is returned, in order to minimize bandwidth consumed; in order to get back actual measurements and their QC flags, we must query and filter including the `data` parameter, the behavior of which we'll see in this section.\n",
    "\n",
    "### Basic data request\n",
    "\n",
    "Let's start by asking for one particular profile by ID, and ask for some temperature data to go with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "cf3bbfb8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[28.669001, 28.667999, 28.722, 28.816, 28.823, 28.826, 28.830999, 28.783001, 28.775999, 28.740999, 28.694, 28.551001, 28.497, 28.489, 28.414, 28.191999, 28.087999, 28.044001, 27.836, 27.715, 27.655001, 27.41, 27.125999, 26.805, 26.500999, 26.311001, 26.075001, 25.448, 25.120001, 24.632999, 23.709, 22.400999, 21.893, 21.523001, 21.122, 20.861, 20.657, 20.104, 19.707001, 19.681, 19.573999, 19.396999, 19.181, 18.747, 18.438999, 18.240999, 18.045, 17.882, 17.783001, 17.664, 17.563, 17.474001, 17.370001, 17.250999, 17.006001, 16.861, 16.638, 16.465, 16.337999, 16.142, 15.902, 15.721, 15.585, 15.415, 15.28, 15.214, 15.158, 15.084, 15.039, 14.989, 14.91, 14.792, 14.726, 14.698, 14.681, 14.654, 14.618, 14.589, 14.546, 14.459, 14.389, 14.329, 14.219, 14.137, 14.074, 14.039, 14, 13.966, 13.928, 13.851, 13.83, 13.826, 13.823, 13.819, 13.812, 13.808, 13.809, 13.81, 13.808, 13.769, 13.753, 13.736, 13.721, 13.694, 13.657, 13.625, 13.608, 13.584, 13.547, 13.524, 13.508, 13.485, 13.457, 13.418, 13.364, 13.291, 13.23, 13.148, 12.999, 12.903, 12.843, 12.819, 12.807, 12.767, 12.714, 12.692, 12.691, 12.689, 12.688, 12.682, 12.668, 12.639, 12.573, 12.476, 12.336, 12.064, 11.854, 11.74, 11.69, 11.638, 11.583, 11.478, 11.419, 11.361, 11.282, 11.241, 11.178, 11.107, 11.059, 11.055, 11.045, 11.034, 11.015, 11, 10.985, 10.975, 10.956, 10.936, 10.864, 10.482, 10.09, 9.975, 9.848, 9.83, 9.829, 9.807, 9.784, 9.768, 9.749, 9.725, 9.672, 9.634, 9.629, 9.617, 9.587, 9.535, 9.465, 9.442, 9.411, 9.385, 9.36, 9.33, 9.278, 9.21, 9.138, 9.091, 9.051, 9, 8.964, 8.918, 8.88, 8.82, 8.732, 8.658, 8.607, 8.576, 8.558, 8.554, 8.549, 8.541, 8.535, 8.527, 8.514, 8.49, 8.476, 8.462, 8.432, 8.372, 8.297, 8.198, 8.156, 8.111, 8.024, 7.953, 7.884, 7.775, 7.755, 7.746, 7.734, 7.692, 7.665, 7.635, 7.606, 7.602, 7.598, 7.593, 7.586, 7.581, 7.577, 7.572, 7.563, 7.551, 7.539, 7.521, 7.516, 7.507, 7.495, 7.479, 7.433, 7.396, 7.368, 7.344, 7.299, 7.26, 7.227, 7.227, 7.227, 7.195, 7.138, 7.109, 7.097, 7.088, 7.072, 7.05, 7.032, 7.012, 6.994, 6.984, 6.982, 6.981, 6.979, 6.978, 6.975, 6.971, 6.955, 6.896, 6.884, 6.878, 6.853, 6.833, 6.809, 6.794, 6.78, 6.772, 6.768, 6.765, 6.763, 6.758, 6.756, 6.75, 6.739, 6.713, 6.685, 6.62, 6.577, 6.555, 6.545, 6.532, 6.518, 6.506, 6.491, 6.473, 6.459, 6.449, 6.442, 6.429, 6.401, 6.37, 6.351, 6.344, 6.34, 6.338, 6.335, 6.331, 6.319, 6.314, 6.311, 6.305, 6.291, 6.281, 6.274, 6.266, 6.257, 6.25, 6.247, 6.244, 6.241, 6.236, 6.22, 6.204, 6.187, 6.181, 6.176, 6.174, 6.17, 6.167, 6.165, 6.163, 6.147, 6.13, 6.127, 6.126, 6.123, 6.116, 6.11, 6.094, 6.085, 6.052, 6.024, 5.986, 5.968, 5.958, 5.939, 5.923, 5.912, 5.888, 5.858, 5.834, 5.822, 5.802, 5.788, 5.767, 5.619, 5.357, 5.272, 5.173, 5.101, 5.089, 5.084, 5.064, 5.013, 4.994, 4.986, 4.983, 4.978, 4.962, 4.972, 4.98, 4.954, 4.934, 4.889, 4.853, 4.817, 4.767, 4.73, 4.72, 4.72, 4.719, 4.713, 4.706, 4.702, 4.699, 4.679, 4.653, 4.647, 4.639, 4.63, 4.626, 4.619, 4.616, 4.61, 4.601, 4.598, 4.592, 4.582, 4.574, 4.548, 4.517, 4.492, 4.471, 4.452, 4.444, 4.489, 4.51, 4.531, 4.509, 4.483, 4.488, 4.498, 4.51, 4.522, 4.534, 4.53, 4.522, 4.513, 4.51, 4.514, 4.516, 4.517, 4.516, 4.514, 4.511, 4.51], [2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46, 48, 50, 52, 54, 56, 58, 60, 62, 64, 66, 68, 70, 72, 74, 76, 78, 80, 82, 84, 86, 88, 90, 92, 94, 96, 98, 100, 102, 104, 106, 108, 110, 112, 114, 116, 118, 120, 122, 124, 126, 128, 130, 132, 134, 136, 138, 140, 142, 144, 146, 148, 150, 152, 154, 156, 158, 160, 162, 164, 166, 168, 170, 172, 174, 176, 178, 180, 182, 184, 186, 188, 190, 192, 194, 196, 198, 200, 202, 204, 206, 208, 210, 212, 214, 216, 218, 220, 222, 224, 226, 228, 230, 232, 234, 236, 238, 240, 242, 244, 246, 248, 250, 252, 254, 256, 258, 260, 262, 264, 266, 268, 270, 272, 274, 276, 278, 280, 282, 284, 286, 288, 290, 292, 294, 296, 298, 300, 302, 304, 306, 308, 310, 312, 314, 316, 318, 320, 322, 324, 326, 328, 330, 332, 334, 336, 338, 340, 342, 344, 346, 348, 350, 352, 354, 356, 358, 360, 362, 364, 366, 368, 370, 372, 374, 376, 378, 380, 382, 384, 386, 388, 390, 392, 394, 396, 398, 400, 402, 404, 406, 408, 410, 412, 414, 416, 418, 420, 422, 424, 426, 428, 430, 432, 434, 436, 438, 440, 442, 444, 446, 448, 450, 452, 454, 456, 458, 460, 462, 464, 466, 468, 470, 472, 474, 476, 478, 480, 482, 484, 486, 488, 490, 492, 494, 496, 498, 500, 502, 504, 506, 508, 510, 512, 514, 516, 518, 520, 522, 524, 526, 528, 530, 532, 534, 536, 538, 540, 542, 544, 546, 548, 550, 552, 554, 556, 558, 560, 562, 564, 566, 568, 570, 572, 574, 576, 578, 580, 582, 584, 586, 588, 590, 592, 594, 596, 598, 600, 602, 604, 606, 608, 610, 612, 614, 616, 618, 620, 622, 624, 626, 628, 630, 632, 634, 636, 638, 640, 642, 644, 646, 648, 650, 652, 654, 656, 658, 660, 662, 664, 666, 668, 670, 672, 674, 676, 678, 680, 682, 684, 686, 688, 690, 692, 694, 696, 698, 700, 705, 710, 715, 720, 725, 730, 735, 740, 745, 750, 755, 760, 765, 770, 775, 780, 785, 790, 795, 800, 805, 810, 815, 820, 825, 830, 835, 840, 845, 850, 855, 860, 865, 870, 875, 880, 885, 890, 895, 900, 905, 910, 915, 920, 925, 930, 935, 940, 945, 950, 955, 960, 965, 970, 975, 980, 985, 990, 995, 1000, 1005, 1010, 1015, 1020, 1025, 1030, 1035, 1040, 1045, 1050, 1055, 1060, 1065]]\n"
     ]
    }
   ],
   "source": [
    "dataQuery = {\n",
    "    'id': '4901283_003',\n",
    "    'data': 'temperature'\n",
    "}\n",
    "\n",
    "profile = avh.query('argo', options=dataQuery, apikey=API_KEY, apiroot=API_ROOT)\n",
    "print(profile[0]['data'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa107616",
   "metadata": {},
   "source": [
    "The `data` key contains a list of lists of measurements. To interpret them, we use the `data_info` key:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "9b80b011",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['temperature', 'pressure'], ['units', 'data_keys_mode'], [['degree_Celsius', 'D'], ['decibar', 'D']]]\n"
     ]
    }
   ],
   "source": [
    "print(profile[0]['data_info'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a6aaa6c",
   "metadata": {},
   "source": [
    "`data_info` is always a list that contains exactly three items:\n",
    "\n",
    " - `data_info[0]` is the list of measurements returned in our `data` object, in the same order as `data`. So in the example above, `data[0]` are pressure measurements, while `data[1]` are temperature measurements. Note we got back pressures even though we only asked for temperatures; pressures are always provided where available as they are needed to meaningfully interpret all other data variables.\n",
    " - `data_info[1]` is a list of per-measurement variables. In the example above, pressure and temperature both have a `units` and a `data_keys_mode` associated with them.\n",
    " - `data_info[2]` is a rank 2 matrix with rows labeled by `data_info[0]` and columns by `data_info[1]`. So for the example above, this matrix indicates pressure has units 'decibar', and temperature has `data_keys_mode` 'D'.\n",
    " \n",
    "With this information, we now understand how to interpret the `data` key above: the first list is a list of pressures measured in decibar, and the second list are corresponding temperature measurements measured in degrees C. Note that the ith elements in the data lists all correspond to the same level - in other words, `data[0][i],  data[1][i], data[2][1], ....` are all measurements corresponding to the ith level of this object.\n",
    " \n",
    "> **Data and metadata precedence**: sometimes, you might see a given key on *both* a data document and its corresponding metadata document; when this happens, the value on the data document always takes precedence. `data_info` is a common example of this, which we'll see again below."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49c7f219",
   "metadata": {},
   "source": [
    "### Data inflation\n",
    "\n",
    "If you find this format difficult to consume, another option is to use the `data_inflate` function from the argovis helpers package. This function will turn your data array into a list of dictionaries, one dictionary per level, with keys corresponding to the data values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "ae814c00",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'temperature': 28.669001, 'pressure': 2},\n",
       " {'temperature': 28.667999, 'pressure': 4},\n",
       " {'temperature': 28.722, 'pressure': 6},\n",
       " {'temperature': 28.816, 'pressure': 8},\n",
       " {'temperature': 28.823, 'pressure': 10},\n",
       " {'temperature': 28.826, 'pressure': 12},\n",
       " {'temperature': 28.830999, 'pressure': 14},\n",
       " {'temperature': 28.783001, 'pressure': 16},\n",
       " {'temperature': 28.775999, 'pressure': 18},\n",
       " {'temperature': 28.740999, 'pressure': 20}]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inflated_data = avh.data_inflate(profile[0])\n",
    "inflated_data[0:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e85c27cc",
   "metadata": {},
   "source": [
    "This format is inefficient to download, but easy to read and work with. Long-time users of previous versions of Argovis may recognize this as similar to the legacy format of some of our data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d90836f9",
   "metadata": {},
   "source": [
    "### Getting absolutely everything\n",
    "\n",
    "What we've seen above allows us to be very targeted in the data we download; rather than being forced to spend time and bandwidth downloading data we aren't interested in, we can focus on just what we need. On the other hand, somtimes we really do want everything, and for that there's `data=all`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "f8b1d181",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'doxy': None,\n",
       "  'doxy_argoqc': 4,\n",
       "  'pressure': 2,\n",
       "  'pressure_argoqc': 1,\n",
       "  'salinity': 35.574966,\n",
       "  'salinity_argoqc': 1,\n",
       "  'salinity_sfile': 35.574966,\n",
       "  'salinity_sfile_argoqc': 1,\n",
       "  'temperature': 28.669001,\n",
       "  'temperature_argoqc': 1,\n",
       "  'temperature_sfile': 28.669001,\n",
       "  'temperature_sfile_argoqc': 1},\n",
       " {'doxy': None,\n",
       "  'doxy_argoqc': 4,\n",
       "  'pressure': 4,\n",
       "  'pressure_argoqc': 1,\n",
       "  'salinity': 35.573761,\n",
       "  'salinity_argoqc': 1,\n",
       "  'salinity_sfile': 35.573761,\n",
       "  'salinity_sfile_argoqc': 1,\n",
       "  'temperature': 28.667999,\n",
       "  'temperature_argoqc': 1,\n",
       "  'temperature_sfile': 28.667999,\n",
       "  'temperature_sfile_argoqc': 1},\n",
       " {'doxy': None,\n",
       "  'doxy_argoqc': 4,\n",
       "  'pressure': 6,\n",
       "  'pressure_argoqc': 1,\n",
       "  'salinity': 35.626602,\n",
       "  'salinity_argoqc': 1,\n",
       "  'salinity_sfile': 35.626602,\n",
       "  'salinity_sfile_argoqc': 1,\n",
       "  'temperature': 28.722,\n",
       "  'temperature_argoqc': 1,\n",
       "  'temperature_sfile': 28.722,\n",
       "  'temperature_sfile_argoqc': 1},\n",
       " {'doxy': None,\n",
       "  'doxy_argoqc': 4,\n",
       "  'pressure': 8,\n",
       "  'pressure_argoqc': 1,\n",
       "  'salinity': 35.752064,\n",
       "  'salinity_argoqc': 1,\n",
       "  'salinity_sfile': 35.752064,\n",
       "  'salinity_sfile_argoqc': 1,\n",
       "  'temperature': 28.816,\n",
       "  'temperature_argoqc': 1,\n",
       "  'temperature_sfile': 28.816,\n",
       "  'temperature_sfile_argoqc': 1},\n",
       " {'doxy': None,\n",
       "  'doxy_argoqc': 4,\n",
       "  'pressure': 10,\n",
       "  'pressure_argoqc': 1,\n",
       "  'salinity': 35.763969,\n",
       "  'salinity_argoqc': 1,\n",
       "  'salinity_sfile': 35.763969,\n",
       "  'salinity_sfile_argoqc': 1,\n",
       "  'temperature': 28.823,\n",
       "  'temperature_argoqc': 1,\n",
       "  'temperature_sfile': 28.823,\n",
       "  'temperature_sfile_argoqc': 1},\n",
       " {'doxy': None,\n",
       "  'doxy_argoqc': 4,\n",
       "  'pressure': 12,\n",
       "  'pressure_argoqc': 1,\n",
       "  'salinity': 35.769985,\n",
       "  'salinity_argoqc': 1,\n",
       "  'salinity_sfile': 35.769985,\n",
       "  'salinity_sfile_argoqc': 1,\n",
       "  'temperature': 28.826,\n",
       "  'temperature_argoqc': 1,\n",
       "  'temperature_sfile': 28.826,\n",
       "  'temperature_sfile_argoqc': 1},\n",
       " {'doxy': None,\n",
       "  'doxy_argoqc': 4,\n",
       "  'pressure': 14,\n",
       "  'pressure_argoqc': 1,\n",
       "  'salinity': 35.777859,\n",
       "  'salinity_argoqc': 1,\n",
       "  'salinity_sfile': 35.777859,\n",
       "  'salinity_sfile_argoqc': 1,\n",
       "  'temperature': 28.830999,\n",
       "  'temperature_argoqc': 1,\n",
       "  'temperature_sfile': 28.830999,\n",
       "  'temperature_sfile_argoqc': 1},\n",
       " {'doxy': None,\n",
       "  'doxy_argoqc': 4,\n",
       "  'pressure': 16,\n",
       "  'pressure_argoqc': 1,\n",
       "  'salinity': 35.805309,\n",
       "  'salinity_argoqc': 1,\n",
       "  'salinity_sfile': 35.805309,\n",
       "  'salinity_sfile_argoqc': 1,\n",
       "  'temperature': 28.783001,\n",
       "  'temperature_argoqc': 1,\n",
       "  'temperature_sfile': 28.783001,\n",
       "  'temperature_sfile_argoqc': 1},\n",
       " {'doxy': None,\n",
       "  'doxy_argoqc': 4,\n",
       "  'pressure': 18,\n",
       "  'pressure_argoqc': 1,\n",
       "  'salinity': 35.822716,\n",
       "  'salinity_argoqc': 1,\n",
       "  'salinity_sfile': 35.822716,\n",
       "  'salinity_sfile_argoqc': 1,\n",
       "  'temperature': 28.775999,\n",
       "  'temperature_argoqc': 1,\n",
       "  'temperature_sfile': 28.775999,\n",
       "  'temperature_sfile_argoqc': 1},\n",
       " {'doxy': None,\n",
       "  'doxy_argoqc': 4,\n",
       "  'pressure': 20,\n",
       "  'pressure_argoqc': 1,\n",
       "  'salinity': 35.869289,\n",
       "  'salinity_argoqc': 1,\n",
       "  'salinity_sfile': 35.869289,\n",
       "  'salinity_sfile_argoqc': 1,\n",
       "  'temperature': 28.740999,\n",
       "  'temperature_argoqc': 1,\n",
       "  'temperature_sfile': 28.740999,\n",
       "  'temperature_sfile_argoqc': 1}]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataQuery = {\n",
    "    'id': '4901283_003',\n",
    "    'data': 'all'\n",
    "}\n",
    "\n",
    "profile = avh.query('argo', options=dataQuery, apikey=API_KEY, apiroot=API_ROOT)\n",
    "avh.data_inflate(profile[0])[0:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42d88ec5",
   "metadata": {},
   "source": [
    "> **Downloading only what you need:**\n",
    "Some objects, like Argo BGC probes, measure many values. Your downoads will often be dramatically faster if you specify your variables of interest, rather than using `data=all` unnecessarily. Recall that the `data` parameter can also accept a comma-separated list of variable names, if there are a few that you'd like."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fed0e8b",
   "metadata": {},
   "source": [
    "### Filtering behavior of data requests\n",
    "\n",
    "Note that adding a specific data filter is a _firm requirement_ that all returned profiles have some meaningful data for _all_ variables listed. Try demanding chlorophyl-a in addition to temperature for our current profile of interest:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "44a02570",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[]\n"
     ]
    }
   ],
   "source": [
    "dataQuery = {\n",
    "    'id': '4901283_003',\n",
    "    'data': 'temperature,chla'\n",
    "}\n",
    "\n",
    "profile = avh.query('argo', options=dataQuery, apikey=API_KEY, apiroot=API_ROOT)\n",
    "print(profile)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89f38a06",
   "metadata": {},
   "source": [
    "We get nothing in our array of profiles; even though we asked for profile id '4901283_003' and we know it exists, `data=temperature,chla` filters our query down to _only_ profiles that have both temperature and chla reported; since the profile requested doesn't have any chla measurements, it is dropped from the returns in this case. This is useful if you only want to download profiles that definitely have data of interest; for example, try the same thing on our regional search from above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "8c07acbe",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n"
     ]
    }
   ],
   "source": [
    "argoSearch = {\n",
    "    'startDate': '2011-05-01T00:00:00Z',\n",
    "    'endDate': '2011-06-01T00:00:00Z',\n",
    "    'center': '-22.5,0',\n",
    "    'radius': 100,\n",
    "    'data': 'temperature,chla'\n",
    "}\n",
    "\n",
    "argoProfiles = avh.query('argo', options=argoSearch, apikey=API_KEY, apiroot=API_ROOT)\n",
    "print(len(argoProfiles))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "743680c7",
   "metadata": {},
   "source": [
    "Evidently Argo made no chlorophyl-a measurements in May 2011 within 100 km of our point of interest - a fact which we found using the data api without having to download or reduce any data at all. One final point on data filtering in this manner: it's not enough for a profile to nominally have a variable defined for it; it must have at least one non-null value reported for that variable somewhere in the search results. For example, when we did `data=all` for our profile of interest above, we saw dissolved oxygen, `doxy`, was defined for it. But:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "59082d13",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[]\n"
     ]
    }
   ],
   "source": [
    "dataQuery = {\n",
    "    'id': '4901283_003',\n",
    "    'data': 'doxy'\n",
    "}\n",
    "\n",
    "profile = avh.query('argo', options=dataQuery, apikey=API_KEY, apiroot=API_ROOT)\n",
    "print(profile)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "582ad322",
   "metadata": {},
   "source": [
    "Again our search is filtered down to nothing, since every level in that profile reported `None` for `doxy`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "316186cf",
   "metadata": {},
   "source": [
    "### Search negation\n",
    "\n",
    "Let's find some profiles that do actually have dissolved oxygen in them, this time with a slightly different geography search: let's look for everything in August 2017 within a polygon region, defined as a list of `[longitude, latitude]` points: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "ebe30ddf",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataQuery = {\n",
    "    'startDate': '2017-08-01T00:00:00Z',\n",
    "    'endDate': '2017-09-01T00:00:00Z',\n",
    "    'polygon': [[-150,-30],[-155,-30],[-155,-35],[-150,-35],[-150,-30]],\n",
    "    'data': 'doxy'\n",
    "}\n",
    "\n",
    "profiles = avh.query('argo', options=dataQuery, apikey=API_KEY, apiroot=API_ROOT)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d60f52d5",
   "metadata": {},
   "source": [
    "We find one profile with meaningful dissolved oxygen data in the region of interest.\n",
    "\n",
    "The `data` key also accepts _tilde negation_, meaning 'filter for profiles that _don't_ contain this data', for example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "4eedb30e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "19\n"
     ]
    }
   ],
   "source": [
    "dataQuery = {\n",
    "    'startDate': '2017-08-01T00:00:00Z',\n",
    "    'endDate': '2017-09-01T00:00:00Z',\n",
    "    'polygon': [[-150,-30],[-155,-30],[-155,-35],[-150,-35],[-150,-30]],\n",
    "    'data': 'temperature,~doxy'\n",
    "}\n",
    "\n",
    "profiles = avh.query('argo', options=dataQuery, apikey=API_KEY, apiroot=API_ROOT)\n",
    "print(len(profiles))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa08c136",
   "metadata": {},
   "source": [
    "We get a collection of profiles that appear in the region of interest, and have temperature but _not_ dissolved oxygen. In this way, we can split up our downloads into groups of related and interesting profiles without re-downloading the same profiles over and over."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "282f1b5a",
   "metadata": {},
   "source": [
    "### QC filtering\n",
    "\n",
    "In addition to querying and filtering by what data is available, we can also make demands on the quality of that data by performing QC filtering. Let's start by looking at some particulate backscattering data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "1ad77717",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.004452, 0.004286, 0.004157, 0.004542, 0.005183, 0.004568, 0.004465, 0.005632, 0.005568, 0.00426]\n",
      "[1, 1, 1, 1, 1, 1, 4, 1, 1, 4]\n"
     ]
    }
   ],
   "source": [
    "bbpQuery = {\n",
    "    'id': '2902857_001',\n",
    "    'data': 'bbp700,bbp700_argoqc'\n",
    "}\n",
    "\n",
    "bbp = avh.query('argo', options=bbpQuery, apikey=API_KEY, apiroot=API_ROOT)\n",
    "bbpindex = bbp[0]['data_info'][0].index('bbp700')\n",
    "bbpQCindex = bbp[0]['data_info'][0].index('bbp700_argoqc')\n",
    "print(bbp[0]['data'][bbpindex][0:10])\n",
    "print(bbp[0]['data'][bbpQCindex][0:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3155a8b4",
   "metadata": {},
   "source": [
    "We request both the measurement and its corresponding QC flags, for reference. Recall that for Argo:\n",
    "\n",
    " - QC 1 == definitely good data\n",
    " - QC 2 == probably good data\n",
    " - QC 3 == probably bad data\n",
    " - QC 4 == definitely bad data\n",
    " \n",
    "If we didn't look at the QC flags for our particulate backscatter data, we could easily have missed that some of the measurements shown above (and many more in the profile not printed) have been marked as bad data by the upstream data distributor, and therefore might not be appropriate for your purposes. We can suppress measurements based on a list of allowed QC values by modifying what we pass to the `data` query parameter:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "c9a8848a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.004452, 0.004286, 0.004157, 0.004542, 0.005183, 0.004568, None, 0.005632, 0.005568, None]\n",
      "[1, 1, 1, 1, 1, 1, 4, 1, 1, 4]\n"
     ]
    }
   ],
   "source": [
    "bbpQCfilteredQuery = {\n",
    "    'id': '2902857_001',\n",
    "    'data': 'bbp700,1,bbp700_argoqc'\n",
    "}\n",
    "\n",
    "bbpFiltered = avh.query('argo', options=bbpQCfilteredQuery, apikey=API_KEY, apiroot=API_ROOT)\n",
    "bbpindex = bbpFiltered[0]['data_info'][0].index('bbp700')\n",
    "bbpQCindex = bbpFiltered[0]['data_info'][0].index('bbp700_argoqc')\n",
    "print(bbpFiltered[0]['data'][bbpindex][0:10])\n",
    "print(bbpFiltered[0]['data'][bbpQCindex][0:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbfb9cc8",
   "metadata": {},
   "source": [
    "In our `data` query parameter, we listed which QC flags we find tolerable for each measurement parameter; in this case `bbp700,1` indicates we only want `bbp700` data if it has a corresponding QC flag of 1. Some things implied by this example that are worth highlighting:\n",
    "\n",
    " - QC flags listed after a variable name only apply to that variable name. Try printing the `pressure` record for the profile found above, and you'll see none of its levels were suppressed.\n",
    " - The list of QC flags is an explicit-allow list and can contain as many flags as you want. For example, you might change the above data query to `bbp700,1,2` to get both 1- and 2-flagged `bbp700` measurements back.\n",
    " - We include the explicit QC flag in this example for illustrative purposes, but it's not required when doing QC filtering in this way. Try the above query while omitting `bbp700_argoqc`, and you'll get the same non-`None` values for `bbp700`.\n",
    " - Note however, as with all data requests, if all explicitly requested data variables are `None` for a level, that level is dropped. In the case where you omitted `bbp700_argoqc` and only requested `bbp700`, the levels where the QC filtration set the `bbp700` value to `None` are dropped.\n",
    " - Similarly, if *all* levels of a requested variable are set to `None` by QC filtration, the entire profile will be dropped from the returns, on the grounds that it doesn't contain any of the data you requested at a level of quality you marked as acceptable.\n",
    " \n",
    "Note that QC flags are currently only available for the argo and cchdo datasets, and furthermore that these datasets assign different meanings to their QC flags. Be sure to check the docs for each project to make sure you understand how to interpret that project's QC flags."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d93b313b",
   "metadata": {},
   "source": [
    "### Minimal data responses\n",
    "\n",
    "Sometimes, we might want to use the `data` filter as we've seen to confine our attention to only profiles that have data of interest, but we're only interested in general or metadata about those measurements, and don't want to download the actual measurements; for this, we can add the `except-data-values` token:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "878c435f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'_id': '5905107_001', 'geolocation': {'type': 'Point', 'coordinates': [-154.974, -32.415]}, 'basin': 2, 'timestamp': '2017-08-11T11:57:19.001Z', 'date_updated_argovis': '2024-07-02T09:20:19.141Z', 'source': [{'source': ['argo_bgc'], 'url': 'ftp://ftp.ifremer.fr/ifremer/argo/dac/aoml/5905107/profiles/SD5905107_001.nc', 'date_updated': '2024-07-01T21:54:27.000Z'}, {'source': ['argo_core'], 'url': 'ftp://ftp.ifremer.fr/ifremer/argo/dac/aoml/5905107/profiles/D5905107_001.nc', 'date_updated': '2019-06-24T15:29:23.000Z'}], 'cycle_number': 1, 'geolocation_argoqc': 1, 'profile_direction': 'A', 'timestamp_argoqc': 1, 'vertical_sampling_scheme': 'Primary sampling: mixed [deeper than nominal 985dbar: discrete; nominal 985dbar to surface: 2dbar-bin averaged]', 'data_info': [['doxy', 'pressure'], ['units', 'data_keys_mode'], [['micromole/kg', 'D'], ['decibar', 'D']]], 'metadata': ['5905107_m0']}]\n"
     ]
    }
   ],
   "source": [
    "dataQuery = {\n",
    "    'startDate': '2017-08-01T00:00:00Z',\n",
    "    'endDate': '2017-09-01T00:00:00Z',\n",
    "    'polygon': [[-150,-30],[-155,-30],[-155,-35],[-150,-35],[-150,-30]],\n",
    "    'data': 'doxy,except-data-values'\n",
    "}\n",
    "\n",
    "profiles = avh.query('argo', options=dataQuery, apikey=API_KEY, apiroot=API_ROOT)\n",
    "print(profiles)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b74415b",
   "metadata": {},
   "source": [
    "Note that specifying only `'data': 'except-data-values'` is the same as just leaving the `data` query key off completely; the purpose of this option is to allow you to filter by data, but then only get back the lightweight non-measurement values. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da23231f",
   "metadata": {},
   "source": [
    "If we want an even more minimal response, we can use the `compression=minimal` option:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "fb598d04",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['5905107_001', -154.974, -32.415, '2017-08-11T11:57:19.001Z', ['argo_bgc', 'argo_core'], ['5905107_m0']]]\n"
     ]
    }
   ],
   "source": [
    "dataQuery = {\n",
    "    'startDate': '2017-08-01T00:00:00Z',\n",
    "    'endDate': '2017-09-01T00:00:00Z',\n",
    "    'polygon': [[-150,-30],[-155,-30],[-155,-35],[-150,-35],[-150,-30]],\n",
    "    'data': 'doxy',\n",
    "    'compression': 'minimal'\n",
    "}\n",
    "\n",
    "profiles = avh.query('argo', options=dataQuery, apikey=API_KEY, apiroot=API_ROOT)\n",
    "print(profiles)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd443e09",
   "metadata": {},
   "source": [
    "With `compression: minimal`, for each data document we get only a minimal amount of information describing it; each data product has a slightly different minimal representation tailored to suit."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02be4a93",
   "metadata": {},
   "source": [
    "### Temporospatial request details\n",
    "\n",
    "You have seen in examples above that requests can be temporally limited by `startDate` and `endDate`, and confined to a geographic region with `polygon`. There are a few more features and facts about temporospatial requests in Argovis that are worth exploring."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be3e685a",
   "metadata": {},
   "source": [
    "#### Box regions\n",
    "\n",
    "The `polygon` region definitions you've seen so far define regions on the globe by connecting vertexes with geodesic edges. If instead we want a region bounded by lines of constant latitude and longitude, there is the `box` query string parameter. Compare two similar but different searches, first with `polygon`, similar to the above, tracing geodesics between four corners of a region:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "44764e1f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "70.78334480553657\n",
      "72.868775\n"
     ]
    }
   ],
   "source": [
    "qs = {\n",
    "    'startDate': '2017-08-01T00:00:00Z',\n",
    "    'endDate': '2017-09-01T00:00:00Z',\n",
    "    'polygon': [[-20,70],[20,70],[20,72],[-20,72],[-20,70]],\n",
    "}\n",
    "\n",
    "profiles = avh.query('argo', options=qs, apikey=API_KEY, apiroot=API_ROOT)\n",
    "latitudes = [x['geolocation']['coordinates'][1] for x in profiles]\n",
    "print(min(latitudes))\n",
    "print(max(latitudes))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a95a4d11",
   "metadata": {},
   "source": [
    "Now try something similar, but with a `box` region defined instead across the four corners:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "ddae44ac",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "70.017\n",
      "71.957\n"
     ]
    }
   ],
   "source": [
    "qs = {\n",
    "    'startDate': '2017-08-01T00:00:00Z',\n",
    "    'endDate': '2017-09-01T00:00:00Z',\n",
    "    'box': [[-20,70],[20,72]]\n",
    "}\n",
    "\n",
    "profiles = avh.query('argo', options=qs, apikey=API_KEY, apiroot=API_ROOT)\n",
    "latitudes = [x['geolocation']['coordinates'][1] for x in profiles]\n",
    "print(min(latitudes))\n",
    "print(max(latitudes))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ee9258d",
   "metadata": {},
   "source": [
    "Notice that while both regions share the same corners, the polygon search actually returns profiles with latitudes higher than the region's northermost corners since geodesics between two points sharing a latitude deflect north in this far-north search region. Meanwhile, the latitudes of profiles in the box region are confined between the lines of constant latitude connecting the vertexes and defining the top and bottom of the box.\n",
    "\n",
    "> **Box mode notation**: note that box mode expects exactly two vertexes: the most southern and western corner first, followed by the most northern and eastern corner."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}