ci: documentation (fluent#7530)

* workflows: add release process Signed-off-by: Patrick Stephens <[email protected]> * packaging: update build info Signed-off-by: Patrick Stephens <[email protected]> * packaging: update scripts and remove unnecessary stuff Signed-off-by: Patrick Stephens <[email protected]> * workflows: add info for all CI Signed-off-by: Patrick Stephens <[email protected]> --------- Signed-off-by: Patrick Stephens <[email protected]>
nikomauno · Jun 8, 2023 · 4895b64 · 4895b64
1 parent e22bdcb
commit 4895b64
Show file tree

Hide file tree

Showing 24 changed files with 225 additions and 695 deletions.
diff --git a/.github/workflows/README.md b/.github/workflows/README.md
@@ -21,7 +21,7 @@
 | Label name | Description |
 | :----------|-------------|
 | docs-required| default tag used to request documentation, has to be removed before merge |
-| ok-package-test | run all package tests |
+| ok-package-test | Build for all possible targets |
 | ok-to-test | run all integration tests |
 | ok-to-merge | run mergebot and merge (rebase) current PR |
 | ci/integration-docker-ok | integration test is able to build docker image |
@@ -66,14 +66,162 @@ For some reason this is not automatically done via permission inheritance or sim
 
 Each major version (e.g. 1.8 & 1.9) supports different targets to build for, e.g. 1.9 includes a CentOS 8 target and 1.8 has some other legacy targets.
 
-This is all handled by the [build matrix generation composite action](../actions/generate-package-build-matrix/action.yaml) so make sure to update appropriately.
-The build matrix is then fed into the reusable job that builds packages which will then fire for the appropriate targets.
+This is all handled by the [build matrix generation composite action](../actions/generate-package-build-matrix/action.yaml).
+This uses a [JSON file](../../packaging/build-config.json) to specify the targets so ensure this is updated.
+The build matrix is then fed into the [reusable job](./call-build-linux-packages.yaml) that builds packages which will then fire for the appropriate targets.
+The reusable job is used for all package builds including unstable/nightly and the PR `ok-package-test` triggered ones.
 
 ## Releases
 
-Currently the process is as follows:
+The process at a high level is as follows:
 
-1. Tag the source with whatever tag you like on master.
-2. The [`Deploy to staging`](./staging-build.yaml) workflow will then kick in to build everything and upload it either to the S3 staging bucket (packages) or ghcr.io (containers).
-3. Once this completes, the [`Test staging`](./staging-test.yaml) workflow will then run to carry out smoke tests on these packages and containers.
-4. The [`Release from staging`](./staging-release.yaml) workflow can then be manually initiated to promote staging to release.
+1. Tag created with `v` prefix.
+2. [Deploy to staging](https://github.com/fluent/fluent-bit/actions/workflows/staging-build.yaml) workflow runs.
+3. [Test staging](https://github.com/fluent/fluent-bit/actions/workflows/staging-test.yaml) workflow runs.
+4. Manually initiate [release from staging](https://github.com/fluent/fluent-bit/actions/workflows/staging-release.yaml) workflow.
+5. A PR is auto-created to increment the minor version now for Fluent Bit using the [`update_version.sh`](../../update_version.sh) script.
+6. Create PRs for doc updates - Windows & container versions. (WIP to automate).
+
+Breaking the steps down.
+
+### Deploy to staging and test
+
+This should run automatically when a tag is created matching the `v*` regex.
+It currently copes with 1.8+ builds although automation is only exercised for 1.9+ releases.
+
+Once this is completed successfully the staging tests should also run automatically.
+
+![Workflows for staging and test example](./resources/auto-build-test-workflow.png "Example of workflows for build and test")
+
+If both complete successfully then we are good to go.
+
+Occasional failures are seen with package builds not downloading dependencies (CentOS 7 in particular seems bad for this).
+A re-run of failed jobs should resolve this.
+
+The workflow builds all Linux, macOS and Windows targets to a staging S3 bucket plus the container images to ghcr.io.
+
+### Release from staging workflow
+
+This is a manually initiated workflow, the intention is multiple staging builds can happen but we only release one.
+Note that currently we do not support parallel staging builds of different versions, e.g. master and 1.9 branches.
+**We can only release the previous staging build and there is a check to confirm version.**
+
+Ensure AppVeyor build for the tag has completed successfully as well.
+
+To trigger: <https://github.com/fluent/fluent-bit/actions/workflows/staging-release.yaml>
+
+All this job does is copy the various artefacts from staging locations to release ones, it does not rebuild them.
+
+![Workflow for release example](./resources/release-from-staging-workflow-incorrect-version.png "Example of workflow for release")
+
+With this example you can see we used the wrong `version` as it requires it without the `v` prefix (it is used for container tag, etc.) and so it fails.
+
+![Workflow for release failure example](./resources/release-version-failure.png "Example of failing workflow for release")
+
+Make sure to provide without the `v` prefix.
+
+![Workflow for release example](./resources/release-from-staging-workflow.png "Example of successful workflow for release")
+
+Once this workflow is initiated you then also need to have it approved by the designated "release team" otherwise it will not progress.
+
+![Release approval example](./resources/release-approval.png "Release approval example")
+
+They will be notified for approval by Github.
+Unfortunately it has to be approved for each job in the sequence rather than a global approval for the whole workflow although that can be useful to check between jobs.
+
+![Release approval per-job required](./resources/release-approval-per-job.png "Release approval per-job required")
+
+This is quite useful to delay the final smoke test of packages until after the manual steps are done as it will then verify them all for you.
+
+#### Packages server sync
+
+The workflow above ensures all release artefacts are pushed to the appropriate container registry and S3 bucket for official releases.
+The packages server then periodically syncs from this bucket to pull down and serve the new packages so there may be a delay (up to 1 hour) before it serves the new versions.
+The syncs happen hourly.
+See <https://github.com/fluent/fluent-bit-infra/blob/main/terraform/provision/package-server-provision.sh.tftpl> for details of the dedicated packages server.
+
+The main reason for a separate server is to accurately track download statistics.
+Container images are handled by ghcr.io and Docker Hub, not this server.
+
+#### Transient container publishing failures
+
+The parallel publishing of multiple container tags for the same image seems to fail occasionally with network errors, particularly more for ghcr.io than DockerHub.
+This can be resolved by just re-running the failed jobs.
+
+#### Windows builds from AppVeyor
+
+This is automated, however confirm that the actual build is successful for the tag: <https://ci.appveyor.com/project/fluent/fluent-bit-2e87g/history>
+If not then ask a maintainer to retrigger.
+
+It can take a while to find the one for the specific tag...
+
+#### ARM builds
+
+All builds are carried out in containers and intended to be run on a valid Ubuntu host to match a standard Github Actions runner.
+This can take some time for ARM as we have to emulate the architecture via QEMU.
+
+<https://github.com/fluent/fluent-bit/pull/7527> introduces support to run ARM builds on a dedicated <actuated.dev> ephemeral VM runner.
+A self-hosted ARM runner is set up and provisioned for this per the [documentation](https://docs.actuated.dev/provision-server/).
+For forks, this should all be skipped and run on a normal Ubuntu Github hosted runner but be aware this may take some time.
+
+### Manual release
+
+As long as it is built to staging we can manually publish packages as well via the script here: <https://github.com/fluent/fluent-bit/blob/master/packaging/update-repos.sh>
+
+Containers can be promoted manually too, ensure to include all architectures and signatures.
+
+### Create PRs
+
+Once releases are published we need to provide PRs for the following documentation updates:
+
+1. Windows checksums: <https://docs.fluentbit.io/manual/installation/windows#installation-packages>
+2. Container versions: <https://docs.fluentbit.io/manual/installation/docker#tags-and-versions>
+
+<https://github.com/fluent/fluent-bit-docs> is the repo for updates to docs.
+
+Take the checksums from the release process above, the AppVeyor stage provides them all and we attempt to auto-create the PR with it.
+
+## Unstable/nightly builds
+
+These happen every 24 hours and [reuse the same workflow](./cron-unstable-build.yaml) as the staging build so are identical except they skip the upload to S3 step.
+This means all targets are built nightly for `master` and `2.0` branches including container images and Linux, macOS and Windows packages.
+
+The container images are available here (the tag refers to the branch):
+
+* [ghcr.io/fluent/fluent-bit/unstable:2.0](ghcr.io/fluent/fluent-bit/unstable:2.0)
+* [ghcr.io/fluent/fluent-bit/unstable:master](ghcr.io/fluent/fluent-bit/unstable:master)
+* [ghcr.io/fluent/fluent-bit/unstable:windows-2019-2.0](ghcr.io/fluent/fluent-bit/unstable:windows-2019-2.0)
+* [ghcr.io/fluent/fluent-bit/unstable:windows-2019-master](ghcr.io/fluent/fluent-bit/unstable:windows-2019-master)
+
+The Linux, macOS and Windows packages are available to download from the specific workflow run.
+
+## Integration tests
+
+On every commit to `master` we rebuild the [packages](./build-master-packages.yaml) and [container images](./master-integration-test.yaml).
+The container images are then used to [run the integration tests](./master-integration-test.yaml) from the <https://github.com/fluent/fluent-bit-ci> repository.
+The container images are available as:
+
+* [ghcr.io/fluent/fluent-bit/master:x86_64](ghcr.io/fluent/fluent-bit/master:x86_64)
+
+## PR checks
+
+Various workflows are run for PRs automatically:
+
+* [Unit tests](./unit-tests.yaml)
+* [Compile checks on CentOS 7 compilers](./pr-compile-check.yaml)
+* [Linting](./pr-lint.yaml)
+* [Windows builds](./pr-windows-build.yaml)
+* [Fuzzing](./pr-fuzz.yaml)
+* [Container image builds](./pr-image-tests.yaml)
+* [Install script checks](./pr-install-script.yaml)
+
+We try to guard these to only trigger when relevant files are changed to reduce any delays or resources used.
+**All should be able to be triggered manually for explicit branches as well.**
+
+The following workflows can be triggered manually for specific PRs too:
+
+* [Integration tests](./pr-integration-test.yaml): Build a container image and run the integration tests as per commits to `master`.
+* [Performance tests](./pr-perf-test.yaml): WIP to trigger a performance test on a dedicated VM and collect the results as a PR comment.
+* [Full package build](./pr-package-tests.yaml): builds all Linux, macOs and Windows packages as well as container images.
+
+To trigger these, apply the relevant label.
diff --git a/.github/workflows/remove-release.yaml b/.github/workflows/remove-release.yaml
diff --git a/.github/workflows/resources/auto-build-test-workflow.png b/.github/workflows/resources/auto-build-test-workflow.png
diff --git a/.github/workflows/resources/release-approval-per-job.png b/.github/workflows/resources/release-approval-per-job.png
diff --git a/.github/workflows/resources/release-approval.png b/.github/workflows/resources/release-approval.png
diff --git a/.github/workflows/resources/release-from-staging-workflow-incorrect-version.png b/.github/workflows/resources/release-from-staging-workflow-incorrect-version.png
diff --git a/.github/workflows/resources/release-from-staging-workflow.png b/.github/workflows/resources/release-from-staging-workflow.png
diff --git a/.github/workflows/resources/release-version-failure.png b/.github/workflows/resources/release-version-failure.png
diff --git a/packaging/README.md b/packaging/README.md
@@ -1,52 +1,64 @@
 # Fluent Bit Packaging
 
-This directory contains Docker files used to build [Fluent Bit](http://fluentbit.io) Linux packages for different distros, the following table describe the supported targets:
-
-| Distro       |   Version / Code Name     | Arch    | Target Option           |
-|--------------|---------------------------|---------|-------------------------|
-| AmazonLinux  |   2                       | x86_64  | amazonlinux/2           |
-| AmazonLinux  |   2                       | arm64v8 | amazonlinux/2.arm64v8   |
-| CentOS       |   8                       | x86_64  | centos/8                |
-| CentOS       |   8                       | arm64v8 | centos/8.arm64v8        |
-| CentOS       |   7                       | x86_64  | centos/7                |
-| CentOS       |   7                       | arm64v8 | centos/7.arm64v8        |
-| Debian       |   12                      | x86_64  | debian/bookworm         |
-| Debian       |   12                      | arm64v8 | debian/bookworm.arm64v8 |
-| Debian       |   11                      | x86_64  | debian/bullseye         |
-| Debian       |   11                      | arm64v8 | debian/bullseye.arm64v8 |
-| Debian       |   10                      | x86_64  | debian/buster           |
-| Debian       |   10                      | arm64v8 | debian/buster.arm64v8   |
-| Ubuntu       |   22.04 / Jammy Jellyfish | x86_64  | ubuntu/22.04            |
-| Ubuntu       |   22.04 / Jammy Jellyfish | arm64v8 | ubuntu/22.04.arm64v8    |
-| Ubuntu       |   20.04 / Focal Fossa     | x86_64  | ubuntu/20.04            |
-| Ubuntu       |   20.04 / Focal Fossa     | arm64v8 | ubuntu/20.04.arm64v8    |
-| Ubuntu       |   18.04 / Bionic Beaver   | x86_64  | ubuntu/18.04            |
-| Ubuntu       |   18.04 / Bionic Beaver   | arm64v8 | ubuntu/18.04.arm64v8    |
-| Ubuntu       |   16.04 / Xenial Xerus    | x86_64  | ubuntu/16.04            |
-| Raspbian     |   11 / Bullseye           | arm32v7 | raspbian/bullseye       |
-| Raspbian     |   10 / Buster             | arm32v7 | raspbian/buster         |
-
-## Usage
+This directory contains files to support building and releasing Fluent Bit.
+
+For PRs, add the `ok-package-test` label to trigger an automated build of all supported Linux, macOS, Windows and container image targets to verify a PR correctly builds for all supported platforms.
+This can take some time to complete so is only triggered via the label on-demand.
+
+## Linux
+
+The [`distros`](./distros/) directory contains OCI container definitions used to build [Fluent Bit](http://fluentbit.io) Linux packages for different distros, the following table describe the supported targets:
+
+| Distro        |   Version / Code Name     | Arch    | Target Option            |
+|---------------|---------------------------|---------|--------------------------|
+| AmazonLinux   |   2                       | x86_64  | amazonlinux/2            |
+| AmazonLinux   |   2                       | arm64v8 | amazonlinux/2.arm64v8    |
+| AmazonLinux   |   2023                    | x86_64  | amazonlinux/2023         |
+| AmazonLinux   |   2023                    | arm64v8 | amazonlinux/2023.arm64v8 |
+| CentOS Stream |   9                       | x86_64  | centos/9                 |
+| CentOS Stream |   9                       | arm64v8 | centos/9.arm64v8         |
+| CentOS        |   8                       | x86_64  | centos/8                 |
+| CentOS        |   8                       | arm64v8 | centos/8.arm64v8         |
+| CentOS        |   7                       | x86_64  | centos/7                 |
+| CentOS        |   7                       | arm64v8 | centos/7.arm64v8         |
+| Debian        |   12                      | x86_64  | debian/bookworm          |
+| Debian        |   12                      | arm64v8 | debian/bookworm.arm64v8  |
+| Debian        |   11                      | x86_64  | debian/bullseye          |
+| Debian        |   11                      | arm64v8 | debian/bullseye.arm64v8  |
+| Debian        |   10                      | x86_64  | debian/buster            |
+| Debian        |   10                      | arm64v8 | debian/buster.arm64v8    |
+| Ubuntu        |   22.04 / Jammy Jellyfish | x86_64  | ubuntu/22.04             |
+| Ubuntu        |   22.04 / Jammy Jellyfish | arm64v8 | ubuntu/22.04.arm64v8     |
+| Ubuntu        |   20.04 / Focal Fossa     | x86_64  | ubuntu/20.04             |
+| Ubuntu        |   20.04 / Focal Fossa     | arm64v8 | ubuntu/20.04.arm64v8     |
+| Ubuntu        |   18.04 / Bionic Beaver   | x86_64  | ubuntu/18.04             |
+| Ubuntu        |   18.04 / Bionic Beaver   | arm64v8 | ubuntu/18.04.arm64v8     |
+| Ubuntu        |   16.04 / Xenial Xerus    | x86_64  | ubuntu/16.04             |
+| Raspbian      |   11 / Bullseye           | arm32v7 | raspbian/bullseye        |
+| Raspbian      |   10 / Buster             | arm32v7 | raspbian/buster          |
+
+These container images are intended to be built from the root of this repo to build the locally checked out/updated version of the source easily for any target.
+
+### Usage
 
 The _build.sh_ script can be used to build packages for a specific target, the command understand the following format:
 
-```
-$ ./build.sh -v VERSION -d DISTRO [-b BRANCH_NAME] [-t TARBALL]
+```shell
+./build.sh -d DISTRO
 ```
 
-Details about the script parameters:
+Replace `DISTRO` with the `Target option` column above.
 
-| Name        | Description                      | Example                |
-|-------------|----------------------------------|------------------------|
-| VERSION     | Github Tag or version number     | 1.3.x                  |
-| TARGET      | Target platform for the packages | ubuntu/18.04           |
+All Linux builds happen in a container so can be run on any supported platform with QEMU installed and a container runtime.
 
-Optionally the script supports the option __-b__ to specify a custom branch, this is useful to package and test _master_ or specific branches.
+## Windows
 
-### Build examples
+Windows builds are carried out by the [dedicated workflow](../.github/workflows/call-build-windows.yaml) in CI.
+This builds using the standard CMake process on a dedicated Windows runner within Github actions.
+The steps involved and additional requirements can all be found there.
 
-#### Package version 1.3.1 for Ubuntu 18.04:
+## macOS
 
-```
-$ ./build.sh -v 1.3.1 -d ubuntu/18.04
-```
+Windows builds are carried out by the [dedicated workflow](../.github/workflows/call-build-macos.yaml) in CI.
+This builds using the standard CMake process on a dedicated macOS runner within Github actions.
+The steps involved and additional requirements can all be found there.
diff --git a/packaging/appveyor-download.sh b/packaging/appveyor-download.sh
@@ -1,4 +1,5 @@
 #!/bin/bash
+# Used during the release process to automatically pull the tagged build from AppVeyor
 set -eux
 
 TAG=${TAG:?}

diff --git a/packaging/build.sh b/packaging/build.sh
@@ -1,4 +1,5 @@
 #!/bin/bash
+# Build a specific Linux target using the local source code via a container image
 set -eux
 
 # Never rely on PWD so we can invoke from anywhere