diff --git a/docs/hello_nextflow/10_hello_seqera.md b/docs/hello_nextflow/10_hello_seqera.md index 0d2d20a9..6c033d5b 100644 --- a/docs/hello_nextflow/10_hello_seqera.md +++ b/docs/hello_nextflow/10_hello_seqera.md @@ -1,5 +1,5 @@ --- -title: Hello Seqera +title: "Part 9: Hello Seqera" description: Get started with Seqera Platform --- @@ -16,9 +16,17 @@ You'll learn how to use Seqera Platform within Nextflow CLI, the Seqera Platform You can use Seqera Platform via either the **CLI**, through the **online GUI** or through the **API**. -## 1. Use Seqera Platform to capture and monitor Nextflow jobs launched from the CLI +--8<-- "docs/hello_nextflow/seqera/01_run_with_cli.md" ---8<-- "./seqera/01_run_with_cli.md" +--8<-- "docs/hello_nextflow/seqera/02_run_with_launchpad.md" + +--8<-- "docs/hello_nextflow/seqera/03_add_compute_environment.md" + +--8<-- "docs/hello_nextflow/seqera/04_setup_launchpad.md" + +!!! Warning WORK-IN-PROGRESS + + Everything below this note is content that was copied from the original guide and needs to be moved or rewritten into the above subpages. ## Online GUI @@ -28,27 +36,6 @@ To run using the GUI, there are three main steps: 2. Create and configure a new [compute environment](https://docs.seqera.io/platform/24.1/compute-envs/overview). 3. Start [launching pipelines](https://docs.seqera.io/platform/24.1/launch/launchpad#launchpad). -### Configuring your compute environment - -Seqera Platform uses the concept of **Compute Environments** to define the execution platform where a workflow will run. - -It supports the launching of workflows into a growing number of **cloud** and **on-premise** infrastructures. - -![Compute environments](img/compute_env_platforms.png) - -Each compute environment must be pre-configured to enable Seqera Platform to submit tasks. You can read more on how to set up each environment using the links below. - -!!! tip "The following guides describe how to configure each of these compute environments." - - * [AWS Batch](https://docs.seqera.io/platform/24.1/compute-envs/aws-batch) - * [Azure Batch](https://docs.seqera.io/platform/24.1/compute-envs/azure-batch) - * [Google Batch](https://docs.seqera.io/platform/24.1/compute-envs/google-cloud-batch) - * [Google Life Sciences](https://docs.seqera.io/platform/24.1/compute-envs/google-cloud-lifesciences) - * [HPC (LSF, Slurm, Grid Engine, Altair PBS Pro)](https://docs.seqera.io/platform/24.1/compute-envs/hpc) - * [Amazon Kubernetes (EKS)](https://docs.seqera.io/platform/24.1/compute-envs/eks) - * [Google Kubernetes (GKE)](https://docs.seqera.io/platform/24.1/compute-envs/gke) - * [Hosted Kubernetes](https://docs.seqera.io/platform/24.1/compute-envs/k8s) - ### Selecting a default compute environment If you have more than one **Compute Environment**, you can select which one will be used by default when launching a pipeline. diff --git a/docs/hello_nextflow/seqera/01_run_with_cli.md b/docs/hello_nextflow/seqera/01_run_with_cli.md index 6ebfb8d4..687c4604 100644 --- a/docs/hello_nextflow/seqera/01_run_with_cli.md +++ b/docs/hello_nextflow/seqera/01_run_with_cli.md @@ -1,9 +1,12 @@ +## 1. Use Seqera Platform to capture and monitor Nextflow jobs launched from the CLI + We'll start by using the Nextflow CLI to launch a pipeline and monitor it in Seqera Platform. Start by logging into the [Seqera Platform](https://cloud.seqera.io/). !!! info "Nextflow Tower" -Seqera Platform was previously known as Nextflow Tower. -You'll still see references to the previous name in environment variable and cli option names. + + Seqera Platform was previously known as Nextflow Tower. + You'll still see references to the previous name in environment variable and cli option names. ### 1.1. Set up your Seqera Platform token by exporting it to your environment @@ -11,15 +14,15 @@ Follow these steps to set up your token: 1. Create a new token by clicking on the **Settings** drop-down menu: - ![Create a token](img/usage_create_token.png) + ![Create a token](seqera/img/usage_create_token.png) 2. Name your token: - ![Name your token](img/usage_name_token.png) + ![Name your token](seqera/img/usage_name_token.png) 3. Save your token safely: - ![Save token](img/usage_token.png) + ![Save token](seqera/img/usage_token.png) !!! note @@ -66,6 +69,94 @@ Hello world! Use ++ctrl+click++ or ++cmd+click++ on the link to open it in your browser. You'll see the Seqera Platform interface with the job finished and the logs captured. -![Seqera Platform](img/run_with_tower.png) +![Seqera Platform](seqera/img/run_with_tower.png) You will see and be able to monitor your **Nextflow jobs** in Seqera Platform. + +### 1.3. Set up Seqera Platform logging by default + +To set up Seqera Platform logging by default, we first want to store our token more securely as a [Nextflow secret](https://www.nextflow.io/docs/latest/secrets.html). + +```bash +nextflow secrets set tower_access_token "eyxxxxxxxxxxxxxxxQ1ZTE=" +``` + +You can set up Seqera Platform logging by default by adding the following line to your `nextflow.config` file. + +```groovy title="nextflow.config" +tower { + enabled = true + endpoint = "https://api.cloud.seqera.io" + accessToken = secrets.tower_access_token +} +``` + +Instead of putting this in the `nextflow.config` file of a specific project, you can also put it in the global file located at `$HOME/.nextflow/config`. + +```bash +cat <> $HOME/.nextflow/config +tower { + enabled = true + endpoint = "https://api.cloud.seqera.io" + accessToken = secrets.tower_access_token +} +EOF +``` + +Run your Nextflow workflows as usual: + +```bash +nextflow run nextflow-io/hello +``` + +You will see the following output: + +```console title="Output" + N E X T F L O W ~ version 24.04.4 + +Launching `https://github.com/nextflow-io/hello` [fabulous_euclid] DSL2 - revision: afff16a9b4 [master] + +Monitor the execution with Seqera Platform using this URL: https://cloud.seqera.io/user/kenbrewer/watch/KYjRktIlOuxrh +executor > local (4) +[71/eaa915] process > sayHello (3) [100%] 4 of 4 ✔ +Ciao world! + +Bonjour world! + +Hola world! + +Hello world! +``` + +Note that we are logging to Seqera Platform even though we did not use the `-with-tower` command! + +### 1.4. Use Seqera Platform to explore the resolved configuration of a Nextflow pipeline + +Click on the link provided in the output to open the Seqera Platform for your run, then click on the `Configuration` tab. +If you ran your pipeline from the `hello_nextflow` directory, you'll see something like this: + +![Seqera Platform Configuration](seqera/img/resolved_configuration.png) + +Notice that configuration for our pipeline run is being run pulled from three separate files: + +- `/home/gitpod/.nextflow/config` - This is the global configuration file we just added. +- `/home/gitpod/.nextflow/assets/nextflow-io/hello/nextflow.config` - This is the `nextflow.config` file from the `nextflow-io/hello` repository. +- `/workspace/gitpod/nf-training/hello-nextflow/nextflow.config` - This is the `nextflow.config` file from our current working directory. + +Nextflow resolves these configurations at runtime with a [specific order of precedence](https://www.nextflow.io/docs/latest/config.html#configuration-file). +The general rule, however, is that more specific configurations override less specific ones, and config/params specified on the CLI will override defaults in the config files. + +Helpfully, Seqera Platform shows us the final output of this configuration resolution process which can be very useful for debugging! + +### Takeaway + +You have learned how to: + +- Set up your Seqera Platform token by exporting it to your environment. +- Run Nextflow CLI with Seqera Platform visualizing and capturing logs. +- Set up Seqera Platform logging by default. +- Use Seqera Platform to explore the resolved configuration of a Nextflow pipeline. + +### What's next? + +Learn how to launch Nextflow pipelines from Seqera Platform using the Launchpad feature. diff --git a/docs/hello_nextflow/seqera/02_run_with_launchpad.md b/docs/hello_nextflow/seqera/02_run_with_launchpad.md new file mode 100644 index 00000000..94057308 --- /dev/null +++ b/docs/hello_nextflow/seqera/02_run_with_launchpad.md @@ -0,0 +1,5 @@ +## 2 Using Seqera Platform Launchpad to run Nextflow workflows + +### 2.1. Navigate to the community/showcase workspace and launch a pipeline + +### 2.2. Monitor the execution of a test rna-seq pipeline diff --git a/docs/hello_nextflow/seqera/03_add_compute_environment.md b/docs/hello_nextflow/seqera/03_add_compute_environment.md new file mode 100644 index 00000000..ecc0d9ee --- /dev/null +++ b/docs/hello_nextflow/seqera/03_add_compute_environment.md @@ -0,0 +1,26 @@ +## 3. Add a compute environment + +In this section, we will simulate setting up a new compute environment to run our pipeline in Seqera Platform Launchpad. +Seqera Platform uses the concept of **Compute Environments** to define the execution platform where a workflow will run. +It supports the launching of workflows into a growing number of **cloud** and **on-premise** infrastructures. + +![Compute environments](seqera/img/compute_env_platforms.png) + +Each compute environment must be pre-configured to enable Seqera Platform to submit tasks. You can read more on how to set up each environment using the links below. + +!!! tip "The following guides describe how to configure each of these compute environments." + + * [AWS Batch](https://docs.seqera.io/platform/24.1/compute-envs/aws-batch) + * [Azure Batch](https://docs.seqera.io/platform/24.1/compute-envs/azure-batch) + * [Google Batch](https://docs.seqera.io/platform/24.1/compute-envs/google-cloud-batch) + * [Google Life Sciences](https://docs.seqera.io/platform/24.1/compute-envs/google-cloud-lifesciences) + * [HPC (LSF, Slurm, Grid Engine, Altair PBS Pro)](https://docs.seqera.io/platform/24.1/compute-envs/hpc) + * [Amazon Kubernetes (EKS)](https://docs.seqera.io/platform/24.1/compute-envs/eks) + * [Google Kubernetes (GKE)](https://docs.seqera.io/platform/24.1/compute-envs/gke) + * [Hosted Kubernetes](https://docs.seqera.io/platform/24.1/compute-envs/k8s) + +To practice this process, we will simulate setting up a new slurm compute environment on our gitpod using Tower Agent. + +### 3.1. Setup Tower Agent + +### 3.2. Add a new Slurm compute environment diff --git a/docs/hello_nextflow/seqera/04_setup_launchpad.md b/docs/hello_nextflow/seqera/04_setup_launchpad.md new file mode 100644 index 00000000..73cc3f8f --- /dev/null +++ b/docs/hello_nextflow/seqera/04_setup_launchpad.md @@ -0,0 +1,11 @@ +## 4. Setup a new pipeline to run in Seqera Platform Launchpad + +In this section, we will set up a new pipeline to run in Seqera Platform Launchpad. + +### 4.1. Add your Github credentials to your personal workspace in Seqera Platform + +### 4.2. Add the pipeline seqeralabs/nf-hello-gatk to your launchpad + +### 4.3. Change the branch of the pipeline to nextflow_schema + +### 4.4. _Unsuccessfully_ launch a pipeline on our Tower Agent diff --git a/docs/hello_nextflow/seqera/img/compute_env_platforms.png b/docs/hello_nextflow/seqera/img/compute_env_platforms.png new file mode 100644 index 00000000..c76eb417 Binary files /dev/null and b/docs/hello_nextflow/seqera/img/compute_env_platforms.png differ diff --git a/docs/hello_nextflow/seqera/img/resolved_configuration.png b/docs/hello_nextflow/seqera/img/resolved_configuration.png new file mode 100644 index 00000000..01ae3e54 Binary files /dev/null and b/docs/hello_nextflow/seqera/img/resolved_configuration.png differ