Private registry for an air-gapped install

[carloscarnero]

I'm having trouble with configuring the registries.yaml for my air-gapped setup, that should pull the images from a private registry. I have read several discussions and issues about this subject, but have had no luck. This is what my file looks now:

mirrors:
  docker.io:
    endpoint:
    - https://example.com:5000/some/path
configs:
  example.com:5000:
    auth:
      username: AzureDiamond
      password: hunter2

What I think is different with my setup is that the required Rancher images are to be found in a path, like https://example.com:5000/some/path/rancher/coredns-coredns:1.8.3 instead of being in the root of the registry, which is something I can't change now.

I have tried several combinations for the configs section: with and without the sub path, with and without the port, the scheme, etc. but still I'm getting this in containerd.log:

level=error msg="PullImage \"rancher/coredns-coredns:1.8.3\" failed"
  error="failed to pull and unpack image \"docker.io/rancher/coredns-coredns:1.8.3\":
  failed to resolve reference \"docker.io/rancher/coredns-coredns:1.8.3\":
  unexpected status code [manifests 1.8.3]: 403 Forbidden

Funny thing, I believe that the authentication works as shown above because using the option --pause-image= example.com:5000/rsa/registry/rancher/pause:3.1 did download the image and is listed when I run crictl image.

So, my question is... are sub paths for image mirroring supported?

[ChristianCiach]

UPDATE

This may be the accepted answer, but the solution I proposed is actually not officially supported by K3s. Please see @brandond's answer below: #3616 (comment)

ORIGINAL ANSWER

Try to start/install your k3s server with:

--system-default-registry example.com:5000/some/path

K3s will then download all internal images from this registry. When using the air-gapped bundle, it will even rename the bundled images so it appears like they have been pulled from this registry:

k3s/pkg/agent/containerd/containerd.go

Line 249 in cbfe673

func retagImages(ctx context.Context, client *containerd.Client, images []images.Image, registries []string) error {

Caveats:

• I could have sworn there was some documentation about this option on the k3s website, but I cannot find it anymore.
• This option is not explicitly advertised to support a path segment of a URL (/some/path in your case). But if I remember correctly, I've tried that a few weeks ago and I think it worked for me back then.

Related PR: #3285

Edit: Correction: I think I originally read about it in the release notes of 1.21.1, so it probably just isn't part of the proper docs yet: https://github.com/k3s-io/k3s/releases/tag/v1.21.1%2Bk3s1

[carloscarnero]

Woah! Setting --system-default-registry exactly as you suggested, with the path segment, did the trick.

Thanks a lot!

[ChristianCiach]

Let's hope K3s doesn't break the usage of a custom path-segment for the --system-default-registry, because I need this feature too ;)

@brandond Sorry to ping you personally, but I've seen you the most at these discussions. Is the usage of a system-default-registry that includes a path segment (like example.com:5000/some/path) supported by design or does is just currently work by accident? If K3s would ever use an "official docker image" like nginx:mainline, the current implementation will break this use case, because currently it would be retagged as example.com:5000/some/path/nginx:mainline, even though it must be example.com:5000/some/path/library/nginx:mainline.

@carloscarnero I think you can mark my answer as the "accepted answer", just like at stackoverlow.com.

[ChristianCiach]

@carloscarnero Are you, by any chance, using a Harbor proxy-cache project? If this is the case, you could also solve this issue by putting a simple Nginx reverse-proxy in front of your Harbor proxy-cache project that simply rewrites all requests to emulate the "API" of a single docker registry.

Have a look at my comment here: moby/moby#34319 (comment)

[carloscarnero]

@carloscarnero I think you can mark my answer as the "accepted answer", just like at stackoverlow.com.

@ChristianCiach I wanted to mark your answer since day one, but I don't know how to do that! I have clicked on any clicky thing, in any color or shape... and can't find it. Shame on me. Any pointer?

[carloscarnero]

@carloscarnero Are you, by any chance, using a Harbor proxy-cache project?

No, just a plain GitLab on-premises instance. Thanks a lot, though!

And I just marked your comment as the answer... the "trick" was to change the discussion category to Q&A, which accepts the markings.

[brandond]

--system-default-registry should be just a host and optional port, without a path component.

We don't cover it in the docs at the moment, but you can rewire the paths in registries.yaml. There's an example in rancher/rke2#865 - for example, if you wanted docker.io/library/busybox to be transparently pulled as registry.lan:5000/docker.io/library/busybox you could do:

mirrors:
  docker.io:
    endpoint:
      - https://registry.lan:5000/v2/
    rewrite:
      "(.*)": "docker.io/$1"
configs:
  registry.lan:5000:
    tls:
      insecure_skip_verify: true

[ChristianCiach]

Interesting, I will try that tomorrow. Maybe this is the better approach, but I must say, I still like the way system-default-registry currenty handles path components.

[brandond]

if it handles it properly it's probably accidental lol

[ChristianCiach]

Don't dare to break my workflow!

[carloscarnero]

We don't cover it in the docs at the moment, but you can rewire the paths in registries.yaml.

I have just tested and it did work with rewriting:

mirrors:
  docker.io:
    endpoint:
    - https://example.com:5000/v2/
    rewrite:
      "(.*)": "some/path/$1"
configs:
  example.com:5000:
    auth:
      username: AzureDiamond
      password: hunter2

And with that, in my particular case, there was no further need for --system-default-registry. This was applied on a working cluster, as right now I can't prepare a new one from scratch (will do as soon as possible, though.)

[brandond]

--system-default-registry is most useful when you're also loading airgap tarballs, because (as noted earlier) it also retags the images as they're loaded, to save having to to find them from the registry under the alternative name.