How to start Host DNS?

[matejaputic]

Context

Talos 1.8.0 running on bare metal

From the base install, this is the only patch I've applied:

machine:
  kubelet:
    extraArgs:
      rotate-server-certificates: true
    extraMounts:
      - destination: /var/openebs/local
        type: bind
        source: /var/openebs/local
        options:
          - rbind
          - rshared
          - rw
  features:
    hostDNS:
      enabled: true
      forwardKubeDNSToHost: true
cluster:
  allowSchedulingOnControlPlanes: true

Expected Behavior

After a few moments, I would expect to see an instance of the host-dns service appear in the kube-system namespace.

Actual Behavior

The host-dns service is nowhere to be found. I do see a couple log entries that show that 169.254.116.108 was allocated, though.

In a desperate attempt, I have also tried manually modifying the Corefile in the kube-system/coredns ConfigMap to forward . 169.254.116.108, but that IP can't be resolved.

What am I missing to be able to start the host-dns service? Thanks!

[smira]

You don't need to do anything, everything just works out of the box.

Host DNS runs as part of machined (root Talos process), so you won't see it explicitly. If you want to see logs, you can use talosctl logs dns-resolve-cache.

[matejaputic]

@smira I appreciate your reply. The problem I'm trying to solve is that I have deployed a local container registry service at registry.registry.svc.cluster.local and when I try to pull an image from it, the kubelet reports that it cannot resolve that address from the DNS server at 127.0.0.53:53. I assume that this means that the DNS server did not receive the DNS entry for the registry service because I can see in the logs that 127.0.0.53 is not reachable from the cluster.

I am likely misunderstanding something, but I'd appreciate a second look at resolving this. Thanks!

[smira]

If you see 127.0.0.53:53 it implies, your pod runs with host networking (in the Kubernetes pod spec, unrelated to Talos). Such host-network pods can't resolve service DNS.

[matejaputic]

This isn't a hostNetwork Pod, this is an attempt at an ImagePull by the kubelet from registry.registry.svc.cluster.local. I can successfully push an image to this registry to the same address from within the cluster. However, the ImagePull is failing because the DNS server at 127.0.0.53 has no entry for registry.registry.svc.cluster.local, so it seems that the CoreDNS and the host's DNS server are out of sync.

Could you please help me understand why they might be out of sync?

I have gone so far as to completely reinstall the OS while being careful to use the "out of the box" machine configuration as much as possible, but I'm still seeing this behavior. Thanks for your attention to this.

[smira]

Host DNS is not supposed to resolve Kubernetes addresses, and Kubelet runs on the host, so it is using host DNS.

In order to resolve this issue, you have couple of options, but they all require to pick a fixed IP for the Service registry in Kubernetes (instead of letting Kubernetes allocate one), and then:

• either add host alias for registry to the service IP via https://www.talos.dev/v1.8/reference/configuration/v1alpha1/config/#Config.machine.network.extraHostEntries.
• use that IP directly to pull from it
• set up a registry mirror entry which will point registry/ to that IP: https://www.talos.dev/v1.8/reference/configuration/v1alpha1/config/#Config.machine.registries.mirrors.-

Note: This might change in the future versions of Talos.

[askedrelic]

I just solved this issue yesterday and will share my solution, but was also curious to ask what could go wrong from disabling Talos hostDNS. Or look for a more Talos based way, which I see might exist.

My solution was to

1. disable Talos hostDNS service

machine:
  features:
    hostDNS:
      enabled: false

2. point the host nameservers to the cluster coreDNS.

machine:
  network:
    nameservers:
      # Use cluster CoreDNS for DNS resolution
      - 10.96.0.10
      - 1.1.1.1
      - 8.8.8.8

3. Configure containerd to use the cluster docker-registry

machine:
    registries:
      mirrors:
        docker.internal:
          endpoints:
            - http://docker-registry.docker-registry.svc.cluster.local

I'm trying to move to Talos from a Kubespray based K8S configuration, which does this by default, and where this dependency came from https://kubespray.io/#/docs/advanced/dns-stack?id=resolvconf_mode-host_resolvconf-default

[smira]

This is a planned feature (to resolve K8s services via host DNS), but it's not implemented yet.

[matejaputic]

I got this to work including TLS with the following patch:

machine:
  network:
    extraHostEntries:
      - ip: <REGISTRY_SVC_IP>
        aliases:
          - <REGISTRY_SVC_NAME>
  registries:
    config:
      <REGISTRY_SVC_NAME>:<REGISTRY_SVC_PORT>:
        tls:
          ca: <REGISTRY_CA_CERT>

with the caveat that the registry container is using hostNetwork: true and has an assigned clusterIP. Also using a self-signed cert generated with cfssl like so:

export HOSTNAME=<REGISTRY_SVC_NAME>
cfssl print-defaults csr | \
    cfssl genkey -initca - | \
    cfssljson -bare ca
cfssl print-defaults csr | \
    cfssl gencert -ca ca.pem -ca-key ca-key.pem -hostname=$HOSTNAME - | \
    cfssljson -bare server

then copy-paste the output of openssl base64 -in ca.pem|tr -d '\n' to <REGISTRY_CA_CERT> above.

This way image pulls work without skipping TLS verification.

[smira]

You don't need extra host entries for registry, you can just specify an endpoint as an IP address (given that the certificate mentions the IP address)