Skip to content

Commit

Permalink
🚧 wip(hello_seqera): initial outline of content
Browse files Browse the repository at this point in the history
  • Loading branch information
kenibrewer committed Oct 25, 2024
1 parent 6fc59eb commit 7ef3682
Show file tree
Hide file tree
Showing 7 changed files with 150 additions and 30 deletions.
35 changes: 11 additions & 24 deletions docs/hello_nextflow/10_hello_seqera.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
title: Hello Seqera
title: "Part 9: Hello Seqera"
description: Get started with Seqera Platform
---

Expand All @@ -16,9 +16,17 @@ You'll learn how to use Seqera Platform within Nextflow CLI, the Seqera Platform

You can use Seqera Platform via either the **CLI**, through the **online GUI** or through the **API**.

## 1. Use Seqera Platform to capture and monitor Nextflow jobs launched from the CLI
--8<-- "docs/hello_nextflow/seqera/01_run_with_cli.md"

--8<-- "./seqera/01_run_with_cli.md"
--8<-- "docs/hello_nextflow/seqera/02_run_with_launchpad.md"

--8<-- "docs/hello_nextflow/seqera/03_add_compute_environment.md"

--8<-- "docs/hello_nextflow/seqera/04_setup_launchpad.md"

!!! Warning WORK-IN-PROGRESS

Everything below this note is content that was copied from the original guide and needs to be moved or rewritten into the above subpages.

## Online GUI

Expand All @@ -28,27 +36,6 @@ To run using the GUI, there are three main steps:
2. Create and configure a new [compute environment](https://docs.seqera.io/platform/24.1/compute-envs/overview).
3. Start [launching pipelines](https://docs.seqera.io/platform/24.1/launch/launchpad#launchpad).

### Configuring your compute environment

Seqera Platform uses the concept of **Compute Environments** to define the execution platform where a workflow will run.

It supports the launching of workflows into a growing number of **cloud** and **on-premise** infrastructures.

![Compute environments](img/compute_env_platforms.png)

Each compute environment must be pre-configured to enable Seqera Platform to submit tasks. You can read more on how to set up each environment using the links below.

!!! tip "The following guides describe how to configure each of these compute environments."

* [AWS Batch](https://docs.seqera.io/platform/24.1/compute-envs/aws-batch)
* [Azure Batch](https://docs.seqera.io/platform/24.1/compute-envs/azure-batch)
* [Google Batch](https://docs.seqera.io/platform/24.1/compute-envs/google-cloud-batch)
* [Google Life Sciences](https://docs.seqera.io/platform/24.1/compute-envs/google-cloud-lifesciences)
* [HPC (LSF, Slurm, Grid Engine, Altair PBS Pro)](https://docs.seqera.io/platform/24.1/compute-envs/hpc)
* [Amazon Kubernetes (EKS)](https://docs.seqera.io/platform/24.1/compute-envs/eks)
* [Google Kubernetes (GKE)](https://docs.seqera.io/platform/24.1/compute-envs/gke)
* [Hosted Kubernetes](https://docs.seqera.io/platform/24.1/compute-envs/k8s)

### Selecting a default compute environment

If you have more than one **Compute Environment**, you can select which one will be used by default when launching a pipeline.
Expand Down
103 changes: 97 additions & 6 deletions docs/hello_nextflow/seqera/01_run_with_cli.md
Original file line number Diff line number Diff line change
@@ -1,25 +1,28 @@
## 1. Use Seqera Platform to capture and monitor Nextflow jobs launched from the CLI

We'll start by using the Nextflow CLI to launch a pipeline and monitor it in Seqera Platform.
Start by logging into the [Seqera Platform](https://cloud.seqera.io/).

!!! info "Nextflow Tower"
Seqera Platform was previously known as Nextflow Tower.
You'll still see references to the previous name in environment variable and cli option names.

Seqera Platform was previously known as Nextflow Tower.
You'll still see references to the previous name in environment variable and cli option names.

### 1.1. Set up your Seqera Platform token by exporting it to your environment

Follow these steps to set up your token:

1. Create a new token by clicking on the **Settings** drop-down menu:

![Create a token](img/usage_create_token.png)
![Create a token](seqera/img/usage_create_token.png)

2. Name your token:

![Name your token](img/usage_name_token.png)
![Name your token](seqera/img/usage_name_token.png)

3. Save your token safely:

![Save token](img/usage_token.png)
![Save token](seqera/img/usage_token.png)

!!! note

Expand Down Expand Up @@ -66,6 +69,94 @@ Hello world!
Use ++ctrl+click++ or ++cmd+click++ on the link to open it in your browser.
You'll see the Seqera Platform interface with the job finished and the logs captured.
![Seqera Platform](img/run_with_tower.png)
![Seqera Platform](seqera/img/run_with_tower.png)
You will see and be able to monitor your **Nextflow jobs** in Seqera Platform.
### 1.3. Set up Seqera Platform logging by default
To set up Seqera Platform logging by default, we first want to store our token more securely as a [Nextflow secret](https://www.nextflow.io/docs/latest/secrets.html).
```bash
nextflow secrets set tower_access_token "eyxxxxxxxxxxxxxxxQ1ZTE="
```
You can set up Seqera Platform logging by default by adding the following line to your `nextflow.config` file.
```groovy title="nextflow.config"
tower {
enabled = true
endpoint = "https://api.cloud.seqera.io"
accessToken = secrets.tower_access_token
}
```
Instead of putting this in the `nextflow.config` file of a specific project, you can also put it in the global file located at `$HOME/.nextflow/config`.
```bash
cat <<EOF >> $HOME/.nextflow/config
tower {
enabled = true
endpoint = "https://api.cloud.seqera.io"
accessToken = secrets.tower_access_token
}
EOF
```
Run your Nextflow workflows as usual:
```bash
nextflow run nextflow-io/hello
```
You will see the following output:
```console title="Output"
N E X T F L O W ~ version 24.04.4
Launching `https://github.com/nextflow-io/hello` [fabulous_euclid] DSL2 - revision: afff16a9b4 [master]
Monitor the execution with Seqera Platform using this URL: https://cloud.seqera.io/user/kenbrewer/watch/KYjRktIlOuxrh
executor > local (4)
[71/eaa915] process > sayHello (3) [100%] 4 of 4 ✔
Ciao world!
Bonjour world!
Hola world!
Hello world!
```
Note that we are logging to Seqera Platform even though we did not use the `-with-tower` command!
### 1.4. Use Seqera Platform to explore the resolved configuration of a Nextflow pipeline
Click on the link provided in the output to open the Seqera Platform for your run, then click on the `Configuration` tab.
If you ran your pipeline from the `hello_nextflow` directory, you'll see something like this:

![Seqera Platform Configuration](seqera/img/resolved_configuration.png)

Notice that configuration for our pipeline run is being run pulled from three separate files:

- `/home/gitpod/.nextflow/config` - This is the global configuration file we just added.
- `/home/gitpod/.nextflow/assets/nextflow-io/hello/nextflow.config` - This is the `nextflow.config` file from the `nextflow-io/hello` repository.
- `/workspace/gitpod/nf-training/hello-nextflow/nextflow.config` - This is the `nextflow.config` file from our current working directory.

Nextflow resolves these configurations at runtime with a [specific order of precedence](https://www.nextflow.io/docs/latest/config.html#configuration-file).
The general rule, however, is that more specific configurations override less specific ones, and config/params specified on the CLI will override defaults in the config files.

Helpfully, Seqera Platform shows us the final output of this configuration resolution process which can be very useful for debugging!

### Takeaway

You have learned how to:

- Set up your Seqera Platform token by exporting it to your environment.
- Run Nextflow CLI with Seqera Platform visualizing and capturing logs.
- Set up Seqera Platform logging by default.
- Use Seqera Platform to explore the resolved configuration of a Nextflow pipeline.

### What's next?

Learn how to launch Nextflow pipelines from Seqera Platform using the Launchpad feature.
5 changes: 5 additions & 0 deletions docs/hello_nextflow/seqera/02_run_with_launchpad.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
## 2 Using Seqera Platform Launchpad to run Nextflow workflows

### 2.1. Navigate to the community/showcase workspace and launch a pipeline

### 2.2. Monitor the execution of a test rna-seq pipeline
26 changes: 26 additions & 0 deletions docs/hello_nextflow/seqera/03_add_compute_environment.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
## 3. Add a compute environment

In this section, we will simulate setting up a new compute environment to run our pipeline in Seqera Platform Launchpad.
Seqera Platform uses the concept of **Compute Environments** to define the execution platform where a workflow will run.
It supports the launching of workflows into a growing number of **cloud** and **on-premise** infrastructures.

![Compute environments](seqera/img/compute_env_platforms.png)

Each compute environment must be pre-configured to enable Seqera Platform to submit tasks. You can read more on how to set up each environment using the links below.

!!! tip "The following guides describe how to configure each of these compute environments."

* [AWS Batch](https://docs.seqera.io/platform/24.1/compute-envs/aws-batch)
* [Azure Batch](https://docs.seqera.io/platform/24.1/compute-envs/azure-batch)
* [Google Batch](https://docs.seqera.io/platform/24.1/compute-envs/google-cloud-batch)
* [Google Life Sciences](https://docs.seqera.io/platform/24.1/compute-envs/google-cloud-lifesciences)
* [HPC (LSF, Slurm, Grid Engine, Altair PBS Pro)](https://docs.seqera.io/platform/24.1/compute-envs/hpc)
* [Amazon Kubernetes (EKS)](https://docs.seqera.io/platform/24.1/compute-envs/eks)
* [Google Kubernetes (GKE)](https://docs.seqera.io/platform/24.1/compute-envs/gke)
* [Hosted Kubernetes](https://docs.seqera.io/platform/24.1/compute-envs/k8s)

To practice this process, we will simulate setting up a new slurm compute environment on our gitpod using Tower Agent.

### 3.1. Setup Tower Agent

### 3.2. Add a new Slurm compute environment
11 changes: 11 additions & 0 deletions docs/hello_nextflow/seqera/04_setup_launchpad.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
## 4. Setup a new pipeline to run in Seqera Platform Launchpad

In this section, we will set up a new pipeline to run in Seqera Platform Launchpad.

### 4.1. Add your Github credentials to your personal workspace in Seqera Platform

### 4.2. Add the pipeline seqeralabs/nf-hello-gatk to your launchpad

### 4.3. Change the branch of the pipeline to nextflow_schema

### 4.4. _Unsuccessfully_ launch a pipeline on our Tower Agent
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 7ef3682

Please sign in to comment.