Unable to redeploy ARC Runners - Pending Ephemeral Runners Stuck

[gthomson31]

logs_controller.log

We encountered an issue following a recent ARC crash, which necessitated a full platform redeployment. Unfortunately, after the reinstall, the runners are now stuck in a termination loop during deployment.
• Listeners are successfully created for each organization.
• We believe the previous installation was completely removed before the reinstall.
• Broken ephemeral runners were manually cleared, but they keep reappearing with failure states.

looking for any pointers or additional logs that I can check that may point in the direction of some resolution

❯ kubectl describe EphemeralRunner product-hp-65hqr-runner-c8c47 -n runners-product-hp
Name:         product-hp-65hqr-runner-c8c47
Namespace:    runners-product-hp
Labels:       actions.github.com/organization=redacted-product
              actions.github.com/scale-set-name=product-hp
              actions.github.com/scale-set-namespace=runners-product-hp
              app.kubernetes.io/component=runner
              app.kubernetes.io/instance=product-hp
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=product-hp
              app.kubernetes.io/part-of=gha-runner-scale-set
              app.kubernetes.io/version=0.10.1
              helm.sh/chart=gha-rs-0.10.1
Annotations:  actions.github.com/patch-id: 0
              actions.github.com/runner-group-name: product-base
              actions.github.com/runner-scale-set-name: product-hp
              actions.github.com/runner-spec-hash: 79dd7b5f74
API Version:  actions.github.com/v1alpha1
Kind:         EphemeralRunner
Metadata:
  Creation Timestamp:  2025-01-14T18:49:09Z
  Finalizers:
    ephemeralrunner.actions.github.com/finalizer
    ephemeralrunner.actions.github.com/runner-registration-finalizer
  Generate Name:  product-hp-65hqr-runner-
  Generation:     1
  Owner References:
    API Version:           actions.github.com/v1alpha1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  EphemeralRunnerSet
    Name:                  product-hp-65hqr
    UID:                   27b75695-83f0-4cae-a9e6-8fd5051e6937
  Resource Version:        280428129
  UID:                     284a264d-ac10-460f-8e45-46ae6a70278b
Spec:
  Github Config Secret:  github-app-secret
  Github Config URL:     https://github.com/redacted-product
  Metadata:
  Runner Scale Set Id:  7
  Spec:
    Containers:
      Command:
        /home/runner/run.sh
      Env:
        Name:   DOCKER_HOST
        Value:  unix:///var/run/docker.sock
      Image:    redacted/github-actions-runner:base-5.0.0
      Name:     runner
      Resources:
        Limits:
          Cpu:     1500m
          Memory:  10Gi
        Requests:
          Cpu:     1
          Memory:  5Gi
      Volume Mounts:
        Mount Path:  /home/runner/_work
        Name:        work
        Mount Path:  /var/run
        Name:        dind-sock
      Args:
        dockerd
        --host=unix:///var/run/docker.sock
        --group=$(DOCKER_GROUP_GID)
      Env:
        Name:   DOCKER_GROUP_GID
        Value:  123
      Image:    redacted/github-actions-runner:dind-1.0.0
      Name:     dind
      Resources:
        Limits:
          Cpu:     1500m
          Memory:  10Gi
        Requests:
          Cpu:     1
          Memory:  5Gi
      Security Context:
        Privileged:  true
      Volume Mounts:
        Mount Path:  /home/runner/_work
        Name:        work
        Mount Path:  /var/run
        Name:        dind-sock
        Mount Path:  /home/runner/externals
        Name:        dind-externals
    Image Pull Secrets:
      Name:  dockerhub
    Init Containers:
      Command:
        cp
        -r
        /home/runner/externals/.
        /home/runner/tmpDir/
      Image:  redacted/github-actions-runner:base-5.0.0
      Name:   init-dind-externals
      Resources:
      Volume Mounts:
        Mount Path:        /home/runner/tmpDir
        Name:              dind-externals
    Restart Policy:        Never
    Service Account Name:  product-hp-gha-rs-no-permission
    Volumes:
      Empty Dir:
      Name:  work
      Empty Dir:
      Name:  dind-sock
      Empty Dir:
      Name:  dind-externals
Status:
  Failures:
    7921311d-d2cf-4bed-99f5-8393ef32b259:  true
    99d681de-b787-49ab-ac62-6d8a3b37bd9e:  true
    9b2068ec-35fb-4f2e-9cc8-10050a9dc8e1:  true
  Phase:                                   Pending
  Ready:                                   false
  Runner Id:                               3726
  Runner JIT Config:    
  Runner Name:                             product-hp-65hqr-runner-c8c47
Events:                                    <none>

2025-01-14T18:31:43Z	INFO	listener-app	app initialized
2025-01-14T18:31:43Z	INFO	listener-app	Starting listener
2025-01-14T18:31:43Z	INFO	listener-app	refreshing token	{"githubConfigUrl": "https://github.com/redacted-product"}
2025-01-14T18:31:43Z	INFO	listener-app	getting access token for GitHub App auth	{"accessTokenURL": "https://api.github.com/app/installations/53635461/access_tokens"}
2025-01-14T18:31:44Z	INFO	listener-app	getting runner registration token	{"registrationTokenURL": "https://api.github.com/orgs/redacted-product/actions/runners/registration-token"}
2025-01-14T18:31:44Z	INFO	listener-app	getting Actions tenant URL and JWT	{"registrationURL": "https://api.github.com/actions/runner-registration"}
2025-01-14T18:31:45Z	INFO	listener-app.listener	Current runner scale set statistics.	{"statistics": "{\"totalAvailableJobs\":2,\"totalAcquiredJobs\":0,\"totalAssignedJobs\":0,\"totalRunningJobs\":0,\"totalRegisteredRunners\":0,\"totalBusyRunners\":0,\"totalIdleRunners\":0}"}
2025-01-14T18:31:45Z	INFO	listener-app.worker.kubernetesworker	Calculated target runner count	{"assigned job": 0, "decision": 1, "min": 1, "max": 1, "currentRunnerCount": 1, "jobsCompleted": 0}
2025-01-14T18:31:45Z	INFO	listener-app.worker.kubernetesworker	Compare	{"original": "{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"replicas\":-1,\"patchID\":-1,\"ephemeralRunnerSpec\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"containers\":null}}},\"status\":{\"currentReplicas\":0,\"pendingEphemeralRunners\":0,\"runningEphemeralRunners\":0,\"failedEphemeralRunners\":0}}", "patch": "{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"replicas\":1,\"patchID\":0,\"ephemeralRunnerSpec\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"containers\":null}}},\"status\":{\"currentReplicas\":0,\"pendingEphemeralRunners\":0,\"runningEphemeralRunners\":0,\"failedEphemeralRunners\":0}}"}
2025-01-14T18:31:45Z	INFO	listener-app.worker.kubernetesworker	Preparing EphemeralRunnerSet update	{"json": "{\"spec\":{\"patchID\":0,\"replicas\":1}}"}
2025-01-14T18:31:45Z	INFO	listener-app.worker.kubernetesworker	Ephemeral runner set scaled.	{"namespace": "runners-product", "name": "product-base-jddcw", "replicas": 1}
2025-01-14T18:31:45Z	INFO	listener-app.listener	Getting next message	{"lastMessageID": 0}
2025-01-14T18:32:35Z	INFO	listener-app.worker.kubernetesworker	Calculated target runner count	{"assigned job": 0, "decision": 1, "min": 1, "max": 1, "currentRunnerCount": 1, "jobsCompleted": 0}
2025-01-14T18:32:35Z	INFO	listener-app.worker.kubernetesworker	Compare	{"original": "{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"replicas\":-1,\"patchID\":-1,\"ephemeralRunnerSpec\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"containers\":null}}},\"status\":{\"currentReplicas\":0,\"pendingEphemeralRunners\":0,\"runningEphemeralRunners\":0,\"failedEphemeralRunners\":0}}", "patch": "{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"replicas\":1,\"patchID\":0,\"ephemeralRunnerSpec\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"containers\":null}}},\"status\":{\"currentReplicas\":0,\"pendingEphemeralRunners\":0,\"runningEphemeralRunners\":0,\"failedEphemeralRunners\":0}}"}
2025-01-14T18:32:35Z	INFO	listener-app.worker.kubernetesworker	Preparing EphemeralRunnerSet update	{"json": "{\"spec\":{\"patchID\":0,\"replicas\":1}}"}
2025-01-14T18:32:35Z	INFO	listener-app.worker.kubernetesworker	Ephemeral runner set scaled.	{"namespace": "runners-product", "name": "product-base-jddcw", "replicas": 1}
2025-01-14T18:32:35Z	INFO	listener-app.listener	Getting next message	{"lastMessageID": 0}
2025-01-14T18:33:25Z	INFO	listener-app.worker.kubernetesworker	Calculated target runner count	{"assigned job": 0, "decision": 1, "min": 1, "max": 1, "currentRunnerCount": 1, "jobsCompleted": 0}
2025-01-14T18:33:25Z	INFO	listener-app.worker.kubernetesworker	Compare	{"original": "{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"replicas\":-1,\"patchID\":-1,\"ephemeralRunnerSpec\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"containers\":null}}},\"status\":{\"currentReplicas\":0,\"pendingEphemeralRunners\":0,\"runningEphemeralRunners\":0,\"failedEphemeralRunners\":0}}", "patch": "{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"replicas\":1,\"patchID\":0,\"ephemeralRunnerSpec\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"containers\":null}}},\"status\":{\"currentReplicas\":0,\"pendingEphemeralRunners\":0,\"runningEphemeralRunners\":0,\"failedEphemeralRunners\":0}}"}
2025-01-14T18:33:25Z	INFO	listener-app.worker.kubernetesworker	Preparing EphemeralRunnerSet update	{"json": "{\"spec\":{\"patchID\":0,\"replicas\":1}}"}
2025-01-14T18:33:25Z	INFO	listener-app.worker.kubernetesworker	Ephemeral runner set scaled.	{"namespace": "runners-product", "name": "product-base-jddcw", "replicas": 1}
2025-01-14T18:33:25Z	INFO	listener-app.listener	Getting next message	{"lastMessageID": 0}
2025-01-14T18:34:15Z	INFO	listener-app.worker.kubernetesworker	Calculated target runner count	{"assigned job": 0, "decision": 1, "min": 1, "max": 1, "currentRunnerCount": 1, "jobsCompleted": 0}
2025-01-14T18:34:15Z	INFO	listener-app.worker.kubernetesworker	Compare	{"original": "{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"replicas\":-1,\"patchID\":-1,\"ephemeralRunnerSpec\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"containers\":null}}},\"status\":{\"currentReplicas\":0,\"pendingEphemeralRunners\":0,\"runningEphemeralRunners\":0,\"failedEphemeralRunners\":0}}", "patch": "{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"replicas\":1,\"patchID\":0,\"ephemeralRunnerSpec\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"containers\":null}}},\"status\":{\"currentReplicas\":0,\"pendingEphemeralRunners\":0,\"runningEphemeralRunners\":0,\"failedEphemeralRunners\":0}}"}
2025-01-14T18:34:15Z	INFO	listener-app.worker.kubernetesworker	Preparing EphemeralRunnerSet update	{"json": "{\"spec\":{\"patchID\":0,\"replicas\":1}}"}
2025-01-14T18:34:15Z	INFO	listener-app.worker.kubernetesworker	Ephemeral runner set scaled.	{"namespace": "runners-product", "name": "product-base-jddcw", "replicas": 1}
2025-01-14T18:34:15Z	INFO	listener-app.listener	Getting next message	{"lastMessageID": 0}
2025-01-14T18:35:05Z	INFO	listener-app.worker.kubernetesworker	Calculated target runner count	{"assigned job": 0, "decision": 1, "min": 1, "max": 1, "currentRunnerCount": 1, "jobsCompleted": 0}
2025-01-14T18:35:05Z	INFO	listener-app.worker.kubernetesworker	Compare	{"original": "{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"replicas\":-1,\"patchID\":-1,\"ephemeralRunnerSpec\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"containers\":null}}},\"status\":{\"currentReplicas\":0,\"pendingEphemeralRunners\":0,\"runningEphemeralRunners\":0,\"failedEphemeralRunners\":0}}", "patch": "{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"replicas\":1,\"patchID\":0,\"ephemeralRunnerSpec\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"containers\":null}}},\"status\":{\"currentReplicas\":0,\"pendingEphemeralRunners\":0,\"runningEphemeralRunners\":0,\"failedEphemeralRunners\":0}}"}
2025-01-14T18:35:05Z	INFO	listener-app.worker.kubernetesworker	Preparing EphemeralRunnerSet update	{"json": "{\"spec\":{\"patchID\":0,\"replicas\":1}}"}
2025-01-14T18:35:05Z	INFO	listener-app.worker.kubernetesworker	Ephemeral runner set scaled.	{"namespace": "runners-product", "name": "product-base-jddcw", "replicas": 1}
2025-01-14T18:35:05Z	INFO	listener-app.listener	Getting next message	{"lastMessageID": 0}
2025-01-14T18:35:55Z	INFO	listener-app.worker.kubernetesworker	Calculated target runner count	{"assigned job": 0, "decision": 1, "min": 1, "max": 1, "currentRunnerCount": 1, "jobsCompleted": 0}
2025-01-14T18:35:55Z	INFO	listener-app.worker.kubernetesworker	Compare	{"original": "{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"replicas\":-1,\"patchID\":-1,\"ephemeralRunnerSpec\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"containers\":null}}},\"status\":{\"currentReplicas\":0,\"pendingEphemeralRunners\":0,\"runningEphemeralRunners\":0,\"failedEphemeralRunners\":0}}", "patch": "{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"replicas\":1,\"patchID\":0,\"ephemeralRunnerSpec\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"containers\":null}}},\"status\":{\"currentReplicas\":0,\"pendingEphemeralRunners\":0,\"runningEphemeralRunners\":0,\"failedEphemeralRunners\":0}}"}
2025-01-14T18:35:55Z	INFO	listener-app.worker.kubernetesworker	Preparing EphemeralRunnerSet update	{"json": "{\"spec\":{\"patchID\":0,\"replicas\":1}}"}
2025-01-14T18:35:55Z	INFO	listener-app.worker.kubernetesworker	Ephemeral runner set scaled.	{"namespace": "runners-product", "name": "product-base-jddcw", "replicas": 1}
2025-01-14T18:35:55Z	INFO	listener-app.listener	Getting next message	{"lastMessageID": 0}
2025-01-14T18:36:46Z	INFO	listener-app.worker.kubernetesworker	Calculated target runner count	{"assigned job": 0, "decision": 1, "min": 1, "max": 1, "currentRunnerCount": 1, "jobsCompleted": 0}
2025-01-14T18:36:46Z	INFO	listener-app.worker.kubernetesworker	Compare	{"original": "{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"replicas\":-1,\"patchID\":-1,\"ephemeralRunnerSpec\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"containers\":null}}},\"status\":{\"currentReplicas\":0,\"pendingEphemeralRunners\":0,\"runningEphemeralRunners\":0,\"failedEphemeralRunners\":0}}", "patch": "{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"replicas\":1,\"patchID\":0,\"ephemeralRunnerSpec\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"containers\":null}}},\"status\":{\"currentReplicas\":0,\"pendingEphemeralRunners\":0,\"runningEphemeralRunners\":0,\"failedEphemeralRunners\":0}}"}
2025-01-14T18:36:46Z	INFO	listener-app.worker.kubernetesworker	Preparing EphemeralRunnerSet update	{"json": "{\"spec\":{\"patchID\":0,\"replicas\":1}}"}
2025-01-14T18:36:46Z	INFO	listener-app.worker.kubernetesworker	Ephemeral runner set scaled.	{"namespace": "runners-product", "name": "product-base-jddcw", "replicas": 1}
2025-01-14T18:36:46Z	INFO	listener-app.listener	Getting next message	{"lastMessageID": 0}
2025-01-14T18:37:36Z	INFO	listener-app.worker.kubernetesworker	Calculated target runner count	{"assigned job": 0, "decision": 1, "min": 1, "max": 1, "currentRunnerCount": 1, "jobsCompleted": 0}
2025-01-14T18:37:36Z	INFO	listener-app.worker.kubernetesworker	Compare	{"original": "{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"replicas\":-1,\"patchID\":-1,\"ephemeralRunnerSpec\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"containers\":null}}},\"status\":{\"currentReplicas\":0,\"pendingEphemeralRunners\":0,\"runningEphemeralRunners\":0,\"failedEphemeralRunners\":0}}", "patch": "{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"replicas\":1,\"patchID\":0,\"ephemeralRunnerSpec\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"containers\":null}}},\"status\":{\"currentReplicas\":0,\"pendingEphemeralRunners\":0,\"runningEphemeralRunners\":0,\"failedEphemeralRunners\":0}}"}
2025-01-14T18:37:36Z	INFO	listener-app.worker.kubernetesworker	Preparing EphemeralRunnerSet update	{"json": "{\"spec\":{\"patchID\":0,\"replicas\":1}}"}
2025-01-14T18:37:36Z	INFO	listener-app.worker.kubernetesworker	Ephemeral runner set scaled.	{"namespace": "runners-product", "name": "product-base-jddcw", "replicas": 1}
2025-01-14T18:37:36Z	INFO	listener-app.listener	Getting next message	{"lastMessageID": 0}

❯ kubectl get autoscalingrunnerset -A
NAMESPACE             NAME             MINIMUM RUNNERS   MAXIMUM RUNNERS   CURRENT RUNNERS   STATE   PENDING RUNNERS   RUNNING RUNNERS   FINISHED RUNNERS   DELETING RUNNERS
runners-devops-base   devops-base      2                 10                2                         2
runners-devops-hp     devops-hp        1                 4                 1                         1
runners-incubator     incubator-base   1                 1                 1                         1
runners-product-hp    product-hp       4                 6                 4                         4
runners-product       product-base     1                 1                 1                         1
❯ kubectl get pods -n runners-product
NAME                              READY   STATUS        RESTARTS   AGE
product-base-jddcw-runner-zq9nr   1/2     Terminating   0          13s
❯ kubectl get pods -n runners-product
NAME                              READY   STATUS     RESTARTS   AGE
product-base-jddcw-runner-zq9nr   0/2     Init:0/1   0          1s
❯ kubectl get pods -n runners-product
NAME                              READY   STATUS     RESTARTS   AGE
product-base-jddcw-runner-zq9nr   0/2     Init:0/1   0          3s
❯

[YodaKV]

Look like the runner pod init container stuck, so try to fetch pod/ containers state. If init container is it in running state, fetch his logs . Might the issue is in your image.

[gthomson31]

Interestingly I ended up pushing a new runner container image version and all seemed to kick back into life and cleared any cached pending runners

[YodaKV]

Back in the game.