From fdce24442ea462213e87229d70bb9ae6dc658027 Mon Sep 17 00:00:00 2001 From: Tyler Kvochick Date: Wed, 24 Feb 2021 15:18:59 -0800 Subject: [PATCH] fix(guides): update recipe example (#271) # What - Update recipe example to be helpful # Why - To fix #198 - To be good OSS maintainers --- docs/guides/recipe.rst | 304 +++++++++++++++++++++++++++++++++++------ 1 file changed, 259 insertions(+), 45 deletions(-) diff --git a/docs/guides/recipe.rst b/docs/guides/recipe.rst index 96b5c72b..16d0fd31 100644 --- a/docs/guides/recipe.rst +++ b/docs/guides/recipe.rst @@ -3,9 +3,14 @@ Create A New Recipe In this section we will walk you through how to create a new Recipe. -.. warning:: - This section is not complete yet. Please refer to the - `Recipe Schema `_ section for now. +It should be noted though, that the recommended way to create a recipe from +scratch is using +`pollination-dsl `_. The +``pollination dsl translate`` command can create a ``queenbee`` recipe like the +one that we will see here. However, creating the recipes by hand is tedious and +error-prone. This example should primarily be a reference for learning more +about how the schema works and seeing a use case than about the most practical +way to author the YAML files. Getting Started --------------- @@ -67,42 +72,70 @@ non-executable fields of ``package.json`` such as ``name``, ``version``, * **tag**: The tag of the recipe. * **app_version**: The version of the application. * **keywords**: A list of strings that are related to the package. -* **maintainers**: A list of ``Maintainer`` objects that describe the contact +* **maintainers**: A list of ``Maintainer`` `objects `_ that describe the contact information of the people responsible for the recipe (you, in this case). -* **home**: ? -* **sources**: ? -* **icon**: A publicly-accessible URI to give the recipe a friendly face. +* **home**: The homepage for this Recipe. +* **sources**: A list of necessary container image URIs, relevant issue trackers, source code, etc. +* **icon**: A publicly-accessible image URI to give the recipe a friendly face. * **deprecated**: Whether or not the recipe is in active development. * **description**: A textual description, analogous to a README. -* **license**: A valid `SPDX Identifier `_ which +* **license**: A valid ``License`` `object `_ which corresponds to the text in the ``LICENSE`` file. +Into the ``package.yaml`` file you can paste the following snippet: + +.. code-block:: yaml + + type: MetaData + name: annual-daylight + tag: 0.4.0 + app_version: null + keywords: + - honeybee + - radiance + - daylight + - annual-daylight + maintainers: + - type: Maintainer + name: Your Name + email: your@email.com + sources: + - https://hub.docker.com/r/ladybugtools/honeybee-radiance + icon: null + deprecated: null + description: Annual daylight recipe for Pollination. + license: + type: License + name: MIT + url: https://spdx.org/licenses/MIT.html + +Naturally, ``Your Name`` and ``your@email.com`` should be replaced with the +correct values. And, as with any software, the license should be one that +comports with the nature of what you are developing. + Dependencies ------------ -The dependencies are specified in the ``dependencies.yaml`` file. Each entry in -the ``dependencies`` array should point to a valid, packaged Plugin at a local -path (such as the result from the ``Create a New Recipe`` documentation) or -hosted on a publicly-available package server. +The dependencies are specified in the ``dependencies.yaml`` file. We can add a dependency that will be used in the Recipe by pulling from a -public repo by replacing the content of the autogenerated ``dependencies.yaml`` +public repo by replacing the content of the auto-generated ``dependencies.yaml`` with the following snippet: .. code-block:: yaml dependencies: - type: Dependency - annotations: {} kind: plugin name: honeybee-radiance hash: null alias: null - tag: 0.4.1 - source: file:///energy-plus-0.1.0.tgz + tag: 0.5.0 + source: https://api.pollination.cloud/registries/ladybug-tools + This will add a reference to the ``honeybee-radiance`` Plugin, version -``0.4.1`` which is cloud-hosted on Pollination's registry server. This will +``0.5.0`` which is hosted on Pollination's registry server. This will allow us to use Functions which are defined in this Plugin as the implementation that does the work inside Tasks which are stitched together in the Recipe. @@ -110,50 +143,158 @@ the Recipe. Flow ---- -The ``flow/`` directory contains the YAML definitions of the DAGs of the recipe -as YAML files. At a minimum, the directory must contain a ``main.yaml`` which -is, surprisingly, the main graph. +The ``flow`` directory is analogous to the ``src`` directory of a ``python`` +package. It is where the actual code of the Recipe is kept. + +Because the recipe files are verbose, we will simply link to them and examine +some snippets. These should be saved with the name of the link inside the +``flow`` directory. + +`main.yaml `_ + +`annual-daylight-ray-tracing.yaml `_ + +After downloading these, the directory should now look like this:: + + my-recipe + ├── flow + │ ├── main.yaml + │ └── annual-daylight-ray-tracing.yaml + ├── dependencies.yaml + ├── LICENSE + ├── package.yaml + └── README.md DAG Tasks --------- -As mentioned above, a Task can itself be a DAG. This both allows complex graphs -to be reused inside other graphs and allows isolating related Tasks into a -logical unit. - -Add a new file at the path ``flow/dag.yaml``. Inside that file, place the -following snippet: +The first snippet that we will examine is the ``tasks`` key from ``main.yaml``. +In the linked file that was saved in the previous section, it should begin like +this: .. code-block:: yaml - type: DAG - -This file ties three key concepts together. The files itself defines that it is -of ``type: DAG`` and has a ``tasks`` array of Task objects. The single Task -that we define uses the function ``daylight-coefficient`` from the -``honeybee-radiance`` Plugin that is defined in ``dependencies.yaml``. + tasks: + - type: DAGTask + name: annual-daylight-raytracing + template: annual-daylight-ray-tracing + needs: + - create-sky-dome + - create-octree-with-suns + - create-octree + - generate-sunpath + - create-total-sky + - create-direct-sky + - create-rad-folder + arguments: + - type: TaskArgument + name: sensor-count + from: + type: InputReference + variable: sensor-count + - type: TaskArgument + name: radiance-parameters + from: + type: InputReference + variable: radiance-parameters + - type: TaskPathArgument + name: octree-file-with-suns + from: + type: TaskFileReference + name: create-octree-with-suns + variable: scene-file + sub_path: null + - type: TaskPathArgument + name: octree-file + from: + type: TaskFileReference + name: create-octree + variable: scene-file + sub_path: null + - type: TaskArgument + name: grid-name + from: + type: ValueReference + value: '{{item.full_id}}' + - type: TaskPathArgument + name: sensor-grid + from: + type: TaskFolderReference + name: create-rad-folder + variable: model-folder + sub_path: grid/{{item.full_id}}.pts + - type: TaskPathArgument + name: sky-matrix + from: + type: TaskFileReference + name: create-total-sky + variable: sky-matrix + sub_path: null + - type: TaskPathArgument + name: sky-dome + from: + type: TaskFileReference + name: create-sky-dome + variable: sky-dome + sub_path: null + - type: TaskPathArgument + name: sky-matrix-direct + from: + type: TaskFileReference + name: create-direct-sky + variable: sky-matrix + sub_path: null + - type: TaskPathArgument + name: sunpath + from: + type: TaskFileReference + name: generate-sunpath + variable: sunpath + sub_path: null + - type: TaskPathArgument + name: sun-modifiers + from: + type: TaskFileReference + name: generate-sunpath + variable: sun-modifiers + sub_path: null + loop: + type: DAGTaskLoop + from: + type: TaskReference + name: create-rad-folder + variable: sensor-grids + sub_folder: initial_results/{{item.name}} + returns: [] + +This key points to an array of Task objects, with the specific type here being +a ``DAGTask``. As mentioned above, the entire Recipe forms a directed acyclic +graph. This type of task allows the nesting of DAGs inside the Recipe, allowing +complex workflows to be isolated into units of related functionality like a +subroutine in a structured programming language. This particular Task +references the neighboring file ``annual-daylight-ray-tracing`` which declares +itself to be of type ``DAG``. Referencing Outputs to Inputs ----------------------------- In order to pass outputs of one Task as inputs to another Task, it is necessary -to create an edge in the DAG that represents this connection. +to create an edge in the DAG that represents this connection. The second +element from ``main.yaml``'s ``tasks`` array is another ``DAGTask`` that looks +like this: .. code-block:: yaml - type: DAGTask - annotations: {} name: create-octree template: honeybee-radiance/create-octree needs: - create-rad-folder arguments: - type: TaskPathArgument - annotations: {} name: model from: type: TaskFolderReference - annotations: {} name: create-rad-folder variable: model-folder sub_path: null @@ -161,7 +302,6 @@ to create an edge in the DAG that represents this connection. sub_folder: null returns: - type: TaskPathReturn - annotations: {} name: scene-file description: null path: resources/scene.oct @@ -175,35 +315,31 @@ another Task. The ``TaskPathArgument`` object specifies a ``from`` field that looks for a Task named ``create-rad-folder`` and acquires the value of its output that is named ``model-folder``. -The Task that supplies this source as an output can be defined by this snippet: +The Task that supplies this source as an output can be defined by the third +element in the ``tasks`` array which looks like this: .. code-block:: yaml - type: DAGTask - annotations: {} name: create-rad-folder template: honeybee-radiance/create-radiance-folder needs: [] arguments: - type: TaskPathArgument - annotations: {} name: input-model from: type: InputFileReference - annotations: {} variable: model sub_path: null loop: null sub_folder: null returns: - type: TaskPathReturn - annotations: {} name: model-folder description: null path: model required: true - type: TaskReturn - annotations: {} name: sensor-grids description: Sensor grids information. @@ -220,9 +356,87 @@ to the first element of ``create-rad-folder``'s ``returns`` array. Artifact Path Context Resolution -------------------------------- -? +It should be noted that the ``model-folder`` return object is not linked to a +specific path on your local system, a path in a remote resource, nor even a +path in a known interface like the Linux filesystem hierarchy. Rather, it names +a path relative to the Task itself. When this task is run on an execution +engine, locally with Luigi, or in the cloud with Pollination, the execution +engine is free to locate the outputs from this task as it sees fit. The +referenced paths are simply relative to the execution context where the task is +actually executed which allows the same Recipe to be used locally for +convenience or in the cloud for enabling massive scale without changes. + +Luigi, for instance, will create a temporary folder on your local drive unique +to each task which will become the context for resolving the path while +Pollination will run the task inside a container and that container's +filesystem will become the context for path resolution. Working With Loops ------------------ -? +While the ``queenbee`` Recipe schema is meant to be declarative, it does +include a primitive for an iterative loop in any Task. This is the key +``loop``. An example usage can be seen in the +``annual-daylight-ray-tracing.yaml`` file. The first element in the ``tasks`` +array has a non-null ``loop`` key that looks like this: + +.. code-block:: yaml + + loop: + type: DAGTaskLoop + from: + type: TaskReference + name: split-grid + variable: grids-list + +This instructs the execution engine to execute this task once for each item +that results from the ``grids-list`` output of the ``split-grid`` Task. + +This task the fifth element in the array and looks like this: + +.. code-block:: yaml + + - type: DAGTask + name: split-grid + template: honeybee-radiance/split-grid + needs: [] + arguments: + - type: TaskArgument + name: sensor-count + from: + type: InputReference + variable: sensor-count + - type: TaskPathArgument + name: input-grid + from: + type: InputFileReference + variable: sensor-grid + sub_path: null + loop: null + sub_folder: null + returns: + - type: TaskReturn + name: grids-list + description: null + - type: TaskPathReturn + name: output-folder + description: null + path: sub_grids + required: true + +This particular function, ``honeybee-radiance/split-grid`` results in a list of +files, the length of which can vary based on the physical dimensions of the +modeled geometry that is split and the parameters used in the splitting +function. Thus, it cannot be known until this task completes how many items +there are for the referencing ``loop`` key to touch. The ``loop`` construct +allows the Recipe to vary across parameters like this that cannot be known +until runtime and allows it to discover inputs as it executes without require +imperative instructions from the author (you). + +Conclusion +------------------ + +Hopefully this gives a deeper understanding of the Recipe schema and how it +allows workflows to be flexible and reused across execution environments. If +you have questions, always feel free to open an issue or reach out on the +`forum `_. Thank you!