Merge pull request #9 from Tauffer-Consulting/quickstart

quickstart

docs/domino_components/_category_.json

@@ -1,6 +1,6 @@
 {
   "label": "Domino Components",
-  "position": 4,
+  "position": 5,
   "link": {
     "type": "generated-index",
     "description": "Domino platform components are the building blocks of Domino applications.",

docs/pieces/create_pieces.md

@@ -6,7 +6,7 @@ sidebar_position: 3
 
 Domino defines standards for writing and distributing modular Pieces, which guarantees their reusability and consistency across Workflows. Those Pieces can be included in any Workflow by drag-and-drop, and Domino will take care of running them according to user choices.
 
-It is very easy to turn your Python code into a Domino Piece, in this session we'll guide you through it, step by step: 
+It is very easy to turn your Python code into a Domino Piece, in this session we'll guide you through it, step by step:
 
 1. write your Python function inside the [piece.py](./create_pieces#piecepy) file
 2. define its data types inside the [models.py](./create_pieces#modelspy) file
@@ -16,7 +16,7 @@ It is very easy to turn your Python code into a Domino Piece, in this session we
 6. publish it in a [git repository](./create_pieces#add-the-piece-to-a-repository) (public or private)
 
 
-Let's create a new Piece, named `MyNewPiece`. The Piece's folder should have the same name as the Piece (in this case `MyNewPiece`) and follow a standard organization: 
+Let's create a new Piece, named `MyNewPiece`. The Piece's folder should have the same name as the Piece (in this case `MyNewPiece`) and follow a standard organization:
 
 ```bash title="Example folder and files structure for MyNewPiece"
 /MyNewPiece
@@ -43,7 +43,7 @@ class MyNewPiece(BasePiece):
 
     # Your custom function code comes in here
     def piece_function(self, input_data: InputModel, secrets_data: SecretsModel):
-        
+
         # The Piece's input arguments are passed in the 'input_data' argument
         print(f"Inpu argument 1: {input_data.in_argument_1}")
         print(f"Inpu argument 2: {input_data.in_argument_2}")
@@ -58,14 +58,14 @@ class MyNewPiece(BasePiece):
         file_path = str(Path(self.results_path)/"msg.txt")
         with open(file_path, "w") as f:
             f.write(msg)
-        
+
         # If you want to display results directly in the Domino GUI,
         # you should set the attribute self.display_result
         self.display_result = {
             "file_type": "txt",
             "file_path": file_path
         }
-        
+
         # You should return the results using the Output model
         return OutputModel(
             out_argument_1="a string result",
@@ -75,10 +75,10 @@ class MyNewPiece(BasePiece):
 
 <details>
 <summary>Save files in a shared storage</summary>
-
 
-Pieces can save files in a shared storage, to be used as input to downstream Pieces, by saving them under `self.results_path`. The `self.results_path` points to a shared storage path specific for that Piece, and it is automatically created when the Piece is executed.  
-
+
+Pieces can save files in a shared storage, to be used as input to downstream Pieces, by saving them under `self.results_path`. The `self.results_path` points to a shared storage path specific for that Piece, and it is automatically created when the Piece is executed.
+
 :::note
 Important: To use this feature your workflow should be configured to use a shared storage.
 :::
@@ -100,7 +100,7 @@ return OutputModel(
     out_file_path=file_path
 )
 ```
-  
+
 </details>
 
 
@@ -131,11 +131,11 @@ self.display_result = {
 }
 ```
 
-In either way, the `file_type` should always be provided. Currently, the supported file types are: 
+In either way, the `file_type` should always be provided. Currently, the supported file types are:
 
-- `txt` 
+- `txt`
 - `json`
-- `png` 
+- `png`
 - `jpeg`
 - `bmp`
 - `tiff`
@@ -144,6 +144,7 @@ In either way, the `file_type` should always be provided. Currently, the support
 - `md`
 - `pdf`
 - `html`
+- `plotly_json`
 
 </details>
 
@@ -205,29 +206,29 @@ Based on the `InputModel`, the Domino GUI will appropriately display input field
 <details>
 <summary>Integer</summary>
 
-```python     
+```python
 integer_arg: int = Field(
     default=2,
     description="Example of integer input argument"
 )
 ```
 
-![Form integer field](/img/pieces/create_pieces/input_int.gif) 
+![Form integer field](/img/pieces/create_pieces/input_int.gif)
 
 </details>
 
 
 <details>
 <summary>Float</summary>
 
-```python   
+```python
 float_arg: float = Field(
     default=1.3,
     description="Example of float input argument"
 )
 ```
 
-![Form float field](/img/pieces/create_pieces/input_float.gif) 
+![Form float field](/img/pieces/create_pieces/input_float.gif)
 
 </details>
 
@@ -240,7 +241,7 @@ string_arg: str = Field(
     default="text value",
     description="Example of string input argument"
 )
-``` 
+```
 
 ![Form text field](/img/pieces/create_pieces/input_text.gif)
 
@@ -339,7 +340,7 @@ input_args: List[InputModel] = Field(
 </details>
 
 
-The `OutputModel` defines the output data types of the Piece and allows for connected downstream Pieces to use this output data correctly. 
+The `OutputModel` defines the output data types of the Piece and allows for connected downstream Pieces to use this output data correctly.
 
 The `SecretsModel` defines the secret variables that should be available to the Piece function. It is important to note that Secrets arguments names should be unique within the same Pieces repository. If the same name is used for more than one Secret argument in the same repository, Domino presumes that both Pieces are using the same secret value (e.g. the same access token to an external service).
 
@@ -354,20 +355,20 @@ The simplest `metadata.json` file should contain basic metadata related to the P
     "description": "This Piece runs my awesome Python function, in any Workflow!",
     "dependency": {
         "requirements_file": "requirements_0.txt"
-    
+
 }
 ```
 
-The `name` field is the official Piece's name and it should match the name of the folder and of the class. 
-The `description` field should contain a short and useful description of your Piece. 
+The `name` field is the official Piece's name and it should match the name of the folder and of the class.
+The `description` field should contain a short and useful description of your Piece.
 The `dependency` field contains the reference to a dependencies file that is required to run your custom code. It can contain either:
 
 - `requirements_file`, with the name of a Python requirements file.
 - `dockerfile`, with the name of a Dockerfile with instructions to build the Docker image serving this Piece
 
 The dependency files are stored in a separate folder inside the [Pieces repository](./pieces_repository).
 
-Optionally, you can also include in the metadata: 
+Optionally, you can also include in the metadata:
 
 - style configurations for the visual node representing `MyNewPiece` in the Domino GUI, such as label and icon. Currently the available icons are the free options from [Font Awesome v5](https://fontawesome.com/v5/search?m=free).
 - minimal and limit resources required to run the Piece, when running Domino in Kubernetes
@@ -436,4 +437,4 @@ All tests files will be automatically run by the repository's actions before the
 
 ## Add the Piece to a repository
 
-Now that you have your new Piece ready, you need to add it to a [Pieces repository](./pieces_repository) so it could be installed in a Domino workspace. 
+Now that you have your new Piece ready, you need to add it to a [Pieces repository](./pieces_repository) so it could be installed in a Domino workspace.

docs/pieces/index.mdx

@@ -1,5 +1,5 @@
 ---
-sidebar_position: 3
+sidebar_position: 4
 sidebar_label: Pieces
 sidebar_class_name: green
 description: "Domino Pieces are functional units that can be distributed and reused in multiple Workflows."
@@ -13,11 +13,11 @@ import { Icon } from '@iconify/react'
 
 **Pieces** are the secret sauce of Domino, they are functional units that can be distributed and reused in multiple Workflows. Domino Pieces are special because they:
 
-<p 
-    style={{ 
-        display: 'flex', 
-        alignItems: 'center', 
-        textDecoration: 'none', 
+<p
+    style={{
+        display: 'flex',
+        alignItems: 'center',
+        textDecoration: 'none',
         fontSize: '16px',
         textDecoration: 'none',
         margin: '0px 0px 10px 20px'
@@ -26,11 +26,11 @@ import { Icon } from '@iconify/react'
     <Icon icon="simple-icons:python" width="20" style={{ marginRight: '10px' }} />
     can execute anything written in Python, heavy-weight (e.g. Machine Learning) as well as light-weight (e.g. sending emails) tasks
 </p>
-<p 
-    style={{ 
-        display: 'flex', 
-        alignItems: 'center', 
-        textDecoration: 'none', 
+<p
+    style={{
+        display: 'flex',
+        alignItems: 'center',
+        textDecoration: 'none',
         fontSize: '16px',
         textDecoration: 'none',
         margin: '0px 0px 10px 20px'
@@ -39,11 +39,11 @@ import { Icon } from '@iconify/react'
     <Icon icon="ic:round-input" width="20" style={{ marginRight: '10px' }} />
     have well defined data models for inputs, outputs and secrets (using Pydantic)
 </p>
-<p 
-    style={{ 
-        display: 'flex', 
-        alignItems: 'center', 
-        textDecoration: 'none', 
+<p
+    style={{
+        display: 'flex',
+        alignItems: 'center',
+        textDecoration: 'none',
         fontSize: '16px',
         textDecoration: 'none',
         margin: '0px 0px 10px 20px'
@@ -52,11 +52,11 @@ import { Icon } from '@iconify/react'
     <Icon icon="octicon:container-16" width="20" style={{ marginRight: '10px' }} />
     run in self-contained and isolated execution environments (Docker containers)
 </p>
-<p 
-    style={{ 
-        display: 'flex', 
-        alignItems: 'center', 
-        textDecoration: 'none', 
+<p
+    style={{
+        display: 'flex',
+        alignItems: 'center',
+        textDecoration: 'none',
         fontSize: '16px',
         textDecoration: 'none',
         margin: '0px 0px 10px 20px'
@@ -65,11 +65,11 @@ import { Icon } from '@iconify/react'
     <Icon icon="teenyicons:box-solid" width="20" style={{ marginRight: '10px' }} />
     are immutable, guaranteeing reproducibility of your workflows.
 </p>
-<p 
-    style={{ 
-        display: 'flex', 
-        alignItems: 'center', 
-        textDecoration: 'none', 
+<p
+    style={{
+        display: 'flex',
+        alignItems: 'center',
+        textDecoration: 'none',
         fontSize: '16px',
         textDecoration: 'none',
         margin: '0px 0px 10px 20px'
@@ -78,11 +78,11 @@ import { Icon } from '@iconify/react'
     <Icon icon="teenyicons:git-solid" width="20" style={{ marginRight: '10px' }} />
     are organized in git repositories, for easy packaging, distribution and installation.
 </p>
-<p 
-    style={{ 
-        display: 'flex', 
-        alignItems: 'center', 
-        textDecoration: 'none', 
+<p
+    style={{
+        display: 'flex',
+        alignItems: 'center',
+        textDecoration: 'none',
         fontSize: '16px',
         textDecoration: 'none',
         margin: '0px 0px 10px 20px'
@@ -91,11 +91,11 @@ import { Icon } from '@iconify/react'
     <Icon icon="iconoir:google-docs" width="20" style={{ marginRight: '10px' }} />
     are properly versioned, tested and documented.
 </p>
-<p 
-    style={{ 
-        display: 'flex', 
-        alignItems: 'center', 
-        textDecoration: 'none', 
+<p
+    style={{
+        display: 'flex',
+        alignItems: 'center',
+        textDecoration: 'none',
         fontSize: '16px',
         textDecoration: 'none',
         margin: '0px 0px 10px 20px'

docs/quickstart/_category_.json

@@ -0,0 +1,8 @@
+{
+    "label": "Get Started",
+    "position": 2,
+    "link": {
+        "type": "generated-index",
+        "description": "Quickstart to domino platform."
+    }
+}

docs/quickstart/advanced_concepts/computational_resources.mdx

@@ -0,0 +1,26 @@
+---
+sidebar_position: 4
+description: "Computational Resources"
+keywords: ["domino", "workflow", "airflow", "data", "machine learning", "ml", "data science", "data engineering", "dataops", "mlops", "devops", "kubernetes", "k8s", "helm", "python", "react", "typescript", "fastapi", "gitops", "docker", "github", "artifacthub", "pypi", "pip", "kind", "open source", "oss", "low code", "no code", "automation",
+"versio control", "collaboration", "monitoring", "orchestration", "data pipeline", "computational", "resources"]
+---
+
+
+# Computational Resources
+
+When creating a workflow each Piece can be considered a task that needs to be executed. As some tasks require more computational resources than others, Domino allows you to specify the resources needed for executing each one. For example, a Piece that trains a model can be executed on a GPU, while a Piece that performs a simple data cleaning can be executed on a CPU.
+Editing the computational resources of a Piece is done by double clicking on the Piece, going to Advanced Options and selecting the desired options.
+
+The available options are:
+
+- **CPU**: The number of CPUs, in milicores(m), to allocate to the Piece.
+- **Memory**: The amount of memory, in mebibyte(Mi) to allocate to the Piece.
+- **GPU**: Define if should give access to a GPU to the Piece. To use a GPU, your cluster, if running in Kubernetes mode, or your machine, if running in compose mode, must have a GPU available. Also the piece must be built with a docker image that has the necessary drivers to use the GPU, you can find a base image with the necessary drivers [here](https://github.com/Tauffer-Consulting/domino/pkgs/container/domino-base-piece/147257355?tag=gpu)
+
+:::info
+**CPU** and **Memory** are considered by Domino only when running in Kubernetes.
+:::
+
+<p style={{display: 'flex', justifyContent: 'center'}}>
+<img src='/img/advanced_workflows/computational_resources.png' alt='Computational Resources'/>
+</p>

docs/quickstart/advanced_concepts/github_token.md

@@ -0,0 +1,24 @@
+---
+sidebar_position: 1
+---
+
+# Authorizing Domino to access Github repositories
+
+This session will guide you through the steps necessary to authorize Domino to access Pieces repositories on Github.
+
+## Create a Github token
+Domino uses a platform level Github token to access repositories. If no token is provided, Domino will still be able to run, but some features will not be available. For the platform level token, we recommend you to create a Github token with read access to public repositories.
+
+Besides the platform level token, Domino also provides a way to use a different token for each Workspace, so you can control which repositories each Workspace can access.
+
+To create a Github token, follow the steps below or check the [Github documentation](https://docs.github.com/en/github/authenticating-to-github/keeping-your-account-and-data-secure/creating-a-personal-access-token).
+
+1. You can generate either a [classic](https://github.com/settings/tokens/new) or [fine-grained](https://github.com/settings/personal-access-tokens/new) **Personal access tokens**.
+2. Enter a name, an expiration date and a description for your token.
+3. Select the **Scope** of your token. For most cases a least priviledged token that is only allowed to read from public repositories will be good enough.
+4. [Optional for private repositories] Under **Resource owner**, select the organization or user that owns the repositories you want to access.
+5. Click **Generate token**.
+6. Copy the token and **save it** in a safe place. You will need it to run Domino with all features.
+
+
+

docs/quickstart/advanced_concepts/index.md

@@ -0,0 +1,11 @@
+---
+sidebar_position: 2
+description: "Advanced Workflows concepts and examples"
+keywords: ["domino", "workflow", "airflow", "data", "machine learning", "ml", "data science", "data engineering", "dataops", "mlops", "devops", "kubernetes", "k8s", "helm", "python", "react", "typescript", "fastapi", "gitops", "docker", "github", "artifacthub", "pypi", "pip", "kind", "open source", "oss", "low code", "no code", "automation",
+"versio control", "collaboration", "monitoring", "orchestration", "data pipeline"]
+---
+
+# Advanced Concepts
+
+When creating advanced workflows you may face some challenges that are not covered in the [Quickstart](/docs/quickstart) section. This section will cover some of those challenges and how to solve them. If you are new to Domino, we recommend you start with [Quickstart](/docs/quickstart).
+This section contains technical details about how Domino works behind the scenes.