Skip to content

Commit

Permalink
Merge pull request #127 from implydata/202411-notebook-fixes
Browse files Browse the repository at this point in the history
Pauses
  • Loading branch information
petermarshallio authored Oct 31, 2024
2 parents 8b86086 + 85ade12 commit 750c1bb
Show file tree
Hide file tree
Showing 5 changed files with 136 additions and 86 deletions.
43 changes: 32 additions & 11 deletions notebooks/02-ingestion/12-spatial-dimensions.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,27 @@
"status_client.version"
]
},
{
"cell_type": "markdown",
"id": "4fe649dc-a6c6-4dde-afcc-782d2cb73f0a",
"metadata": {},
"source": [
"### Import additional modules\n",
"\n",
"Run the following cell to import additional Python modules used by helper functions, and to insert a pause for those API calls that need time to complete."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9e2ad64e-ed63-4b30-adb1-d18844515456",
"metadata": {},
"outputs": [],
"source": [
"import json\n",
"import time"
]
},
{
"cell_type": "markdown",
"id": "70aabd26-6615-48ce-8011-adcc83e7d621",
Expand All @@ -113,7 +134,6 @@
"outputs": [],
"source": [
"def wait_for_datagen( job_name:str):\n",
" import time\n",
" from IPython.display import clear_output\n",
" # wait for the messages to be fully published \n",
" done = False\n",
Expand All @@ -135,7 +155,6 @@
"outputs": [],
"source": [
"def wait_for_task( task_id):\n",
" import time\n",
" from IPython.display import clear_output\n",
" # wait for the messages to be fully published \n",
" done = False\n",
Expand Down Expand Up @@ -235,7 +254,7 @@
"metadata": {},
"outputs": [],
"source": [
"import json\n",
"table_name = 'example-clickstream-spatial'\n",
"\n",
"spatial_index_spec = {\n",
" \"type\": \"index_parallel\",\n",
Expand All @@ -259,7 +278,7 @@
" \"queryGranularity\": \"none\",\n",
" \"rollup\": False\n",
" },\n",
" \"dataSource\": \"example-spatial-index\",\n",
" \"dataSource\": table_name,\n",
" \"timestampSpec\": {\n",
" \"column\": \"time\",\n",
" \"format\": \"iso\"\n",
Expand Down Expand Up @@ -310,7 +329,9 @@
"\n",
"task = rest_client.post(\"/druid/indexer/v1/task\", json.dumps(spatial_index_spec), headers=headers)\n",
"task_id = json.loads(task.text)['task']\n",
"wait_for_task(task_id)"
"wait_for_task(task_id)\n",
"\n",
"time.sleep(25) # wait for the data to be available for query"
]
},
{
Expand Down Expand Up @@ -369,7 +390,7 @@
" \"queryType\": \"topN\",\n",
" \"dataSource\": {\n",
" \"type\": \"table\",\n",
" \"name\": \"example-spatial-index\"\n",
" \"name\": table_name\n",
" },\n",
" \"dimension\": {\n",
" \"type\": \"default\",\n",
Expand Down Expand Up @@ -430,7 +451,7 @@
"\n",
"result = rest_client.post(\"/druid/v2\", json.dumps(rectangular_filter_query), headers=headers)\n",
"\n",
"json.loads(result.text)[0]['result']\n"
"json.loads(result.text)[0]['result']"
]
},
{
Expand Down Expand Up @@ -466,7 +487,7 @@
" \"queryType\": \"topN\",\n",
" \"dataSource\": {\n",
" \"type\": \"table\",\n",
" \"name\": \"example-spatial-index\"\n",
" \"name\": table_name\n",
" },\n",
" \"dimension\": {\n",
" \"type\": \"default\",\n",
Expand Down Expand Up @@ -570,7 +591,7 @@
" \"queryType\": \"topN\",\n",
" \"dataSource\": {\n",
" \"type\": \"table\",\n",
" \"name\": \"example-spatial-index\"\n",
" \"name\": table_name\n",
" },\n",
" \"dimension\": {\n",
" \"type\": \"default\",\n",
Expand Down Expand Up @@ -631,7 +652,7 @@
"\n",
"result = rest_client.post(\"/druid/v2\", json.dumps(polygon_filter_query), headers=headers)\n",
"\n",
"json.loads(result.text)[0]['result']\n"
"json.loads(result.text)[0]['result']"
]
},
{
Expand All @@ -650,7 +671,7 @@
"metadata": {},
"outputs": [],
"source": [
"print(f\"Drop datasource: [{druid.datasources.drop('example-spatial-index')}]\")\n"
"print(f\"Drop datasource: [{druid.datasources.drop(table_name)}]\")"
]
},
{
Expand Down
25 changes: 24 additions & 1 deletion notebooks/02-ingestion/13-native-transforms.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,26 @@
"datagen_headers = {'Content-Type': 'application/json'}"
]
},
{
"cell_type": "markdown",
"id": "f99415f4-b239-49ed-8844-859d3c928c1a",
"metadata": {},
"source": [
"### Import additional modules\n",
"\n",
"Run the following cell to import additional Python modules that you will use to force a pause in the notebook."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "fc410394-48e9-4c6d-bcd6-0ee59c494ed9",
"metadata": {},
"outputs": [],
"source": [
"import time"
]
},
{
"cell_type": "markdown",
"id": "2a7b7439-ad21-4808-96b1-8e3c992fa51e",
Expand Down Expand Up @@ -430,6 +450,7 @@
"source": [
"requests.post(f\"{druid_host}/druid/indexer/v1/supervisor\", json.dumps(ingestion_spec), headers=druid_headers)\n",
"druid.sql.wait_until_ready(table_name, verify_load_status=False)\n",
"time.sleep(60) # wait a moment for the data to become available\n",
"display.table(f'{table_name}')"
]
},
Expand Down Expand Up @@ -590,6 +611,7 @@
"source": [
"requests.post(f\"{druid_host}/druid/indexer/v1/supervisor\", json.dumps(ingestion_spec), headers=druid_headers)\n",
"druid.sql.wait_until_ready(table_name, verify_load_status=False)\n",
"time.sleep(60) # wait a moment for the data to become available\n",
"display.table(f'{table_name}')"
]
},
Expand Down Expand Up @@ -729,7 +751,8 @@
"outputs": [],
"source": [
"requests.post(f\"{druid_host}/druid/indexer/v1/supervisor\", json.dumps(ingestion_spec), headers=druid_headers)\n",
"druid.sql.wait_until_ready(table_name, verify_load_status=False)"
"druid.sql.wait_until_ready(table_name, verify_load_status=False)\n",
"time.sleep(60) # wait a moment for the data to become available"
]
},
{
Expand Down
14 changes: 10 additions & 4 deletions notebooks/03-query/07-functions-datetime.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,9 @@
"id": "9c3d6b39-6551-4b2a-bdfb-9606aa92c853",
"metadata": {},
"source": [
"Finally, run the following cell to import additional Python modules that you will use to create visuals as part of the notebook."
"### Import additional modules\n",
"\n",
"Finally, run the following cell to import additional Python modules that you will use to create visuals and to force a pause in the notebooks."
]
},
{
Expand All @@ -148,7 +150,8 @@
"source": [
"import matplotlib\n",
"import matplotlib.pyplot as plt\n",
"import pandas as pd"
"import pandas as pd\n",
"import time"
]
},
{
Expand Down Expand Up @@ -377,7 +380,9 @@
"* Reading secondary dimensions from a table using `MILLIS_TO_TIMESTAMP`.\n",
"* Filtering data using secondary dimensions.\n",
"\n",
"Run the cell below to create a new dimension - a secondary timestamp - that represents the end timestamp of each session."
"Run the cell below to create a new dimension - a secondary timestamp - that represents the end timestamp of each session.\n",
"\n",
"Notice the addition of the \"ended_at\" column, which is calculated using a number of time functions."
]
},
{
Expand Down Expand Up @@ -416,7 +421,8 @@
"\n",
"display.run_task(sql)\n",
"sql_client.wait_until_ready('example-koalas-fndatetime')\n",
"display.table('example-koalas-fndatetime')"
"display.table('example-koalas-fndatetime')\n",
"time.sleep(60) # wait a moment for the data to become available"
]
},
{
Expand Down
8 changes: 5 additions & 3 deletions notebooks/03-query/08-functions-strings.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,8 @@
"source": [
"import matplotlib\n",
"import matplotlib.pyplot as plt\n",
"import pandas as pd"
"import pandas as pd\n",
"import time"
]
},
{
Expand Down Expand Up @@ -531,7 +532,8 @@
"\n",
"display.run_task(sql)\n",
"sql_client.wait_until_ready('example-koalas-strings')\n",
"display.table('example-koalas-strings')"
"display.table('example-koalas-strings')\n",
"time.sleep(60) # wait a moment for the data to become available"
]
},
{
Expand Down Expand Up @@ -744,7 +746,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.3"
"version": "3.11.6"
}
},
"nbformat": 4,
Expand Down
Loading

0 comments on commit 750c1bb

Please sign in to comment.