Skip to content

Commit

Permalink
Fix race condition if a workflow job gets canceled before the runner …
Browse files Browse the repository at this point in the history
…VM accepted the job leading to an idling VM, always default to latest GitHub runner release if 'github_runner_download_url' is left empty
  • Loading branch information
Tereius committed Jan 9, 2025
1 parent 5001a86 commit 69ff087
Show file tree
Hide file tree
Showing 3 changed files with 19 additions and 9 deletions.
7 changes: 4 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,12 +149,13 @@ Error applying IAM policy for cloudrun service "v1/projects/github-spot-runner/l

2. Solution: Override the Organization Policy "Domain Restricted Sharing" in the project, by setting it to "Allow all".

#### The VM Instance immediately stops after it was created without processing a workflow job
#### The VM instance stops shortly after it was created without processing a workflow task

The VM will shoutdown itself if the registration at the GitHub runner group fails. This can be caused by:
* An invalid jit-config.
The VM will stop itself if the registration at the GitHub runner group fails. This can be caused by:
* A typo in the GitHub Enterprise, Organization, Repository name. Check the Terraform variables `github_enterprise`, `github_organization`, `github_repositories` for typos.
* A not existing GitHub runner group within the Enterprise/Organization. Check the Terraform variable `github_runner_group` for typos.
* The GitHub runner version is [deprecated](https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/autoscaling-with-self-hosted-runners#controlling-runner-software-updates-on-self-hosted-runners). The GitHub runner won't accept any Workflow job. Check the Terraform variable `github_runner_download_url` and update to latest GitHub runner version.
* An invalid jit-config.

You can observer the runner registration process by connecting to the VM instance via SSH and running:
```
Expand Down
17 changes: 13 additions & 4 deletions compute.tf
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,13 @@ apt-get update && apt-get -y install docker.io docker-buildx curl jq ${local.git
useradd -d /home/agent -u ${var.github_runner_uid} agent
usermod -aG docker agent
newgrp docker
curl -s -o /tmp/agent.tar.gz -L '${var.github_runner_download_url}'
RUNNER_DOWNLOAD_URL='${var.github_runner_download_url}'
if [ -z "$${RUNNER_DOWNLOAD_URL}" ]; then
RUNNER_VERSION=$(curl -s "https://github.com/actions/runner/tags/" | grep -Eo "$Version v[0-9]+.[0-9]+.[0-9]+" | sort -r | head -n1 | tr -d ' ' | tr -d 'v')
echo "Downloading latest runner v$${RUNNER_VERSION}"
RUNNER_DOWNLOAD_URL="https://github.com/actions/runner/releases/download/v$${RUNNER_VERSION}/actions-runner-linux-x64-$${RUNNER_VERSION}.tar.gz"
fi
curl -s -o /tmp/agent.tar.gz -L $${RUNNER_DOWNLOAD_URL}
mkdir -p /home/agent
chown -R agent:agent /home/agent
pushd /home/agent
Expand All @@ -95,14 +101,17 @@ encoded_jit_config=$1
echo -n $encoded_jit_config | base64 -d | jq '.".runner"' -r | base64 -d > .runner
echo -n $encoded_jit_config | base64 -d | jq '.".credentials"' -r | base64 -d > .credentials
echo -n $encoded_jit_config | base64 -d | jq '.".credentials_rsaparams"' -r | base64 -d > .credentials_rsaparams
sed -i 's/{{SvcNameVar}}/actions.runner.${var.github_organization}.$${agent_name}.service/g' bin/systemd.svc.sh.template
sed -i 's/{{SvcDescription}}/GitHub Actions Runner (${var.github_organization}.$${agent_name})/g' bin/systemd.svc.sh.template
sed -i 's/{{SvcNameVar}}/actions.runner.service/g' bin/systemd.svc.sh.template
sed -i 's/{{SvcDescription}}/GitHub Actions Runner/g' bin/systemd.svc.sh.template
cp bin/systemd.svc.sh.template ./svc.sh && chmod +x ./svc.sh
./bin/installdependencies.sh || shutdown now
./svc.sh install agent || shutdown now
./svc.sh start || shutdown now
popd
rm /tmp/agent.tar.gz
echo "Setup finished"
echo "Setup finished - waiting for Workflow Job"
sleep 60
journalctl -u actions.runner.service -o json --no-pager | jq -e '.|.MESSAGE|match("Running job:")' || shutdown now
echo "Accepted Workflow Job - processing"
EOT
}
4 changes: 2 additions & 2 deletions variables.tf
Original file line number Diff line number Diff line change
Expand Up @@ -94,8 +94,8 @@ variable "github_runner_prefix" {

variable "github_runner_download_url" {
type = string
description = "The download link pointing to the gitlab runner package"
default = "https://github.com/actions/runner/releases/download/v2.321.0/actions-runner-linux-x64-2.321.0.tar.gz"
description = "A download link pointing to the gitlab runner package (WARNING: deprecated runner versions won't process jobs). If this variable is empty (by default), the latest runner release will be downloaded."
default = ""
}

variable "github_runner_uid" {
Expand Down

0 comments on commit 69ff087

Please sign in to comment.