Skip to content

Commit

Permalink
CI: Automatically create tags for releases (#2570)
Browse files Browse the repository at this point in the history
Whenever we push a tag, this will:

 * Fetch these assets from the cache
   ```
   motoko-linux64-$VERSION.tar.gz
   motoko-macos-$VERSION.tar.gz
   moc-interpreter-$VERSION.js
   moc-$VERSION.js
   ```

 * Extracts the changes from the changelog, failing hard if it doesn’t _precisely_
   look as expected. This also means that nothing happens if you use some tag
   like `some-revision-I-wanted-to-remember`

 * Creates a Github Release, with the body of the changelog entry as the body,
   and the above files as assets. This means one can download motoko from
   ```
   https://github.com/dfinity/motoko/releases/download/0.1.2/motoko-linux64-0.1.2.tar.gz
   ```

Note that the `nix-build` step needs stuff from both linux and darwin. This
will fail if the nix cache was not warm. This could happen if you push the tag
while there were still jobs runnig. In that case, just restart the github
action manually.

With this we can stop deploying via DFINITY’s internal infrastructure.

This also removes the hydra-related infrastructure for releases, and updates `CI.md`.

Fixes #2566
  • Loading branch information
nomeata authored Jun 10, 2021
1 parent 35e7ec2 commit d3b7b67
Show file tree
Hide file tree
Showing 7 changed files with 117 additions and 134 deletions.
51 changes: 51 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
name: release

# We trigger this on all tags. The job will fail for tags that don’t have a
# changelog entry, so that seems good enough
on:
push:
tags:
- '*'

jobs:
# this assumes that the nix cache is warm
# In particular, we assume we can fetch the darwin build products
# May require restarting the job otherwise
release:
runs-on: 'ubuntu-latest'
steps:
- uses: actions/checkout@v2
- uses: cachix/install-nix-action@v12
- uses: cachix/cachix-action@v10
with:
name: ic-hs-test
# NB: No auth token, we don’t expect to push new stuff here

# from https://github.community/t/how-to-get-just-the-tag-name/16241/7
- name: Get the version
id: get_version
run: echo ::set-output name=VERSION::${GITHUB_REF/refs\/tags\//}

- run: nix-build --max-jobs 1 release-files.nix --argstr releaseVersion '${{ steps.get_version.outputs.VERSION }}'

- name: Extract changelog
id: read_release
run: |
export VERSION='${{ steps.get_version.outputs.VERSION }}'
perl -0777 -ne '/^# Motoko compiler changelog\n\n== (??{quotemeta($ENV{VERSION})}) \(\d\d\d\d-\d\d-\d\d\)\n\n(.*?)^==/sm or die "Changelog does not look right for this version\n" ; print $1' Changelog.md > changelog-extract.md
cat changelog-extract.md
# need to mangle to use with set-output, see https://github.com/svenstaro/upload-release-action/pull/49/files
r="$(cat changelog-extract.md)"
r="${r//'%'/'%25'}"
r="${r//$'\n'/'%0A'}"
r="${r//$'\r'/'%0D'}"
echo "::set-output name=RELEASE_BODY::$r"
- name: Upload Release Asset
uses: svenstaro/upload-release-action@v2
with:
repo_token: ${{ secrets.GITHUB_TOKEN }}
tag: ${{ github.ref }}
file: result/*
file_glob: true
body: ${{ steps.read_release.outputs.RELEASE_BODY }}
6 changes: 6 additions & 0 deletions Building.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,12 @@ We make frequent releases, at least weekly. The steps to make a release (say, ve

The `release` branch should thus always reference the latest release commit.

Pushing the tag should cause Github Actions to create a “Release” on the github
project. This will fail if the changelog is not in order (in this case, fix and
force-push the tag). It will also fail if the nix cache did not yet contain
the build artifacts for this revision. In this case, restart the Github Action
on Github’s UI.

After releasing the compiler you can update `motoko-base`'s `master`
branch to the `next-moc` branch.

Expand Down
35 changes: 21 additions & 14 deletions CI.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,12 @@ This document distinguishes use-cases (requirements, goals) from
implementations, to allow evaluating alternative implementations.

The implementation is currently a careful choreography between Github, Github
Actions, Hydra, the nix cache and mergify.
Actions, Hydra, the cachix nix cache, the internal nix cache and mergify.

This file describes both the pre-open-source setup (**internal**) which will
gradually be updated or removed as we move away from the internal CI setup
towards the open setup (**external**).

This file describes the pre-open-source setup, and will gradually be updated or
removed as we move away from the internal CI setup.

Knowing if it is broken
-----------------------
Expand All @@ -21,7 +23,10 @@ Knowing if it is broken
Everything is built and tested upon every push to a branch of the repository,
and the resulting status is visible (at derivation granularity) to developers.

**Implementation:**
**Implementation (external):**
All pushes to any branch are built by a Github Action job, on Linux and Darwin.

**Implementation (internal):**
All pushes to `master`, as well as to all branches with open PRs cause hydra to
build the jobs described in `ci.nix` resp. `ci-pr.nix`.

Expand All @@ -37,7 +42,7 @@ Preventing `master` from breaking
**Use-case:**
A PR that breaks requires jobs (`all-systems-go`) cannot be merged into `master`.

**Implementation:**
**Implementation (internal):**
Github branch protection is enabled for `master`, and requires the
`all-systems-go` job from hydra to succeed.

Expand All @@ -58,19 +63,21 @@ Warm cache
Developers can rely on all build artifacts (espcially, but not exclusively, the
dependencies of the nix shell) being present in a nix cache.

**Implementation:**
**Implementation (external):**
Github Actions pushes all builds to the public cachix.org-based nix cache.

**Implementation (internal):**
Hydra pushes all builds to the internal nix cache.

Push releases
-------------

**Use-case:**
Tagged versions cause a tarball with a Motoko release to be pushed to https://download.dfinity.systems/
Tagged versions cause a tarball with a Motoko release to be pushed to
https://github.com/dfinity/motoko/releases

**Implementation:**
The jobs in `release.nix` detect tagged versions. In these cases, the `release`
jobs defined in `ci.nix`, with help from `nix/publish.nix`, will cause our
Hydra instance to publish files via S3.
**Implementation (external):**
A github action creates Github releases and includes the build artifacts there.

Automatically merge when ready
------------------------------
Expand All @@ -92,15 +99,15 @@ Render and show generated base docs
**Use-case:**
To be able to see and share the effects on the generated documentation for the base library, caused by a change to `mo-doc`, without needing the reviewer to generate the documentation locally.

**Implementation:**
**Implementation (internal):**
Hydra hosts the build product of the `base-doc` CI job. This can be found via the Hydra job status page, and the there is a stable link for the latest build of `master` and of each PR.

Render and show slides
----------------------
**Use-case:**
To be able to see and share the “overview slides”, the rendered version should be hosted somewhere.

**Implementation:**
**Implementation (internal):**
Hydra hosts the build product of the `overview-slides` CI job. This can be found via the Hydra job status page, and the there is a stable link for the latest build of `master` and of each PR.


Expand All @@ -111,7 +118,7 @@ Performance changes are known
For every PR, the developer is told about performance changes relative to the
merge point, via an continuously updated comment on the PR.

**Implementation:**
**Implementation (internal):**
* Hydra calculates the correct merge base using `git-merge-base` (_not_ the
latest version of the target branch) and passes a checkout of that revision
as `src.mergeBase` to `ci-pr.nix`.
Expand Down
39 changes: 0 additions & 39 deletions ci.nix
Original file line number Diff line number Diff line change
Expand Up @@ -7,55 +7,16 @@ let
linux = removeRecurseForDerivations (import ./default.nix { system = "x86_64-linux"; internal = true; });
darwin = removeRecurseForDerivations (import ./default.nix { system = "x86_64-darwin"; internal = true; });

as_tarball = dir: derivations:
nixpkgs.runCommandNoCC "motoko-${releaseVersion}.tar.gz" {
allowedRequisites = [];
meta = {
path = "${dir}/motoko-${releaseVersion}.tar.gz";
content-type = "application/gzip";
};
} ''
tmp=$(mktemp -d)
${nixpkgs.lib.concatMapStringsSep "\n" (d: "cp -v ${d}/bin/* $tmp") derivations}
chmod 0755 $tmp/*
tar -czf "$out" -C $tmp/ .
'';

as_js = name: derivation:
nixpkgs.runCommandNoCC "${name}-${releaseVersion}.js" {
allowedRequisites = [];
meta = {
path = "js/${name}-${releaseVersion}.js";
content-type = "application/javascript";
};
} ''
cp -v ${derivation}/bin/* $out
'';

release = import ./nix/publish.nix
{ pkgs = nixpkgs;
inherit releaseVersion;
derivations = [
(as_tarball "x86_64-linux" (with linux; [ mo-ide mo-doc moc ]))
(as_tarball "x86_64-darwin"(with darwin; [ mo-ide mo-doc moc ]))
(as_js "moc" linux.js.moc)
(as_js "moc-interpreter" linux.js.moc_interpreter)
];
};


all-systems-go =
nixpkgs.releaseTools.aggregate {
name = "all-systems-go";
constituents = [
release.motoko
linux.all-systems-go
darwin.all-systems-go
];
};
in
linux // {
inherit release;
darwin = darwin.all-systems-go;
all-systems-go = inject-rev all-systems-go;
}
66 changes: 0 additions & 66 deletions nix/publish.nix

This file was deleted.

39 changes: 39 additions & 0 deletions release-files.nix
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# This file is used by .github/workflows/release.yaml
# to create the files to be uploaded

# It imports default.nix both for linux and darwin, thus it cannot be part of
# it.

{ releaseVersion ? "latest" }:
let
nixpkgs = import ./nix { };
linux = import ./default.nix { system = "x86_64-linux"; internal = true; };
darwin = import ./default.nix { system = "x86_64-darwin"; internal = true; };

as_tarball = dir: derivations:
nixpkgs.runCommandNoCC "motoko-${releaseVersion}.tar.gz" {
allowedRequisites = [];
} ''
tmp=$(mktemp -d)
${nixpkgs.lib.concatMapStringsSep "\n" (d: "cp -v ${d}/bin/* $tmp") derivations}
chmod 0755 $tmp/*
tar -czf "$out" -C $tmp/ .
'';

as_js = name: derivation:
nixpkgs.runCommandNoCC "${name}-${releaseVersion}.js" {
allowedRequisites = [];
} ''
cp -v ${derivation}/bin/* $out
'';

release =
nixpkgs.runCommandNoCC "motoko-release-${releaseVersion}" {} ''
mkdir $out
cp ${as_tarball "x86_64-linux" (with linux; [ mo-ide mo-doc moc ])} $out/motoko-linux64-${releaseVersion}.tar.gz
cp ${as_tarball "x86_64-darwin" (with linux; [ mo-ide mo-doc moc ])} $out/motoko-macos-${releaseVersion}.tar.gz
cp ${as_js "moc" linux.js.moc} $out/moc-${releaseVersion}.js
cp ${as_js "moc-interpreter" linux.js.moc_interpreter} $out/moc-interpreter-${releaseVersion}.js
'';
in
release
15 changes: 0 additions & 15 deletions release.nix

This file was deleted.

0 comments on commit d3b7b67

Please sign in to comment.