Migrating an existing cluster's nodes to Talos

[ethernet-zero]

Hi!

Would it be possible to migrate an entire cluster that's already been set up with Kubespray on Flatcar nodes? I mean, can you join a Talos node to an existing cluster that's not been set up with Talos, or even replace the controlplane nodes one by one?

My guess is that, as long as you provide the right configuration corresponding to the existing cluster, it should probably work, right?

[smira]

It's currently possible to join non-Talos worker nodes to the Talos control plane, but vice versa won't work. Talos API on worker nodes requires a connection to trustd component which only exists on Talos control plane nodes.

[ethernet-zero]

And what about control plane nodes? Would it be possible to create a Talos control plane node but make it join an existing control plane instead of boostrapping the cluster from scratch? If it could be done, we could just replace nodes in the control plane one by one and then do the same with all the workers, since there would be a full Talos control plane by then.

[smira]

This should probably work if all secrets match, but I never tried that, so 🤷‍♂️

[ethernet-zero]

I feel adventurous. 😁

I'll post my findings here when I have the opportunity to test it.

[PhilippKuntschik]

@ethernet-zero did you test adding a talos control plane node to an existing cluster?

[ethernet-zero]

Hi, I'm finally going to test it this week, using the guide posted by @smira. I'll add my comments on the process when I'm done, I'll test with a small cluster: first migrating it to Talos, then waiting for upgrades and upgrading it, and probably also adding a couple of workers to it.

[smira]

We have a way now with 1.2.0 release, a guide https://www.talos.dev/v1.2/advanced/migrating-from-kubeadm/

[ethernet-zero]

Hi,

The guide mentions setting the CNI plugin name to none. My guess is so the installation doesn't affect the CNI plugin currently used by the cluster, is that right? So, it's a matter of letting it join the cluster and it will schedule the Calico nodeplugin normally then?

Same thing with the external cloud controller. If I don't define any in the controlplane.yaml file, it should just schedule the node controllers and pick up the cloud config from the ConfigMap, correct?

[smira]

CNI none means that Talos won't try to deploy any CNI manifests.

As CNI is already deployed to the cluster, that is fine.

[ethernet-zero]

I got the first Talos control plane node to join the etcd cluster and start initializing pods, but now kube-apiserver doesn't become reachable by the controller-runtime because it has trouble trying to contact the metrics-server.svc service, I guess because Calico is not yet running because kube-apiserver is not running yet. Here are some log messages:

talos-anonymized-node-name: 2022-09-21T10:15:26.159069674Z stderr F E0921 10:15:26.159004       1 controller.go:116] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: error trying to reach service: dial tcp <METRICS-SERVER_SVC_IP>:443: i/o timeout
talos-anonymized-node-name: 2022-09-21T10:15:26.159090458Z stderr F , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
talos-anonymized-node-name: 2022-09-21T10:15:26.159142181Z stderr F I0921 10:15:26.159023       1 controller.go:129] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
talos-anonymized-node-name: 2022-09-21T10:15:30.224287246Z stderr F E0921 10:15:30.224203       1 available_controller.go:524] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
talos-anonymized-node-name: 2022-09-21T10:15:35.236581735Z stderr F E0921 10:15:35.236435       1 available_controller.go:524] v1beta1.metrics.k8s.io failed with: failing or missing response from https://<METRICS-SERVER_SVC_IP>:443/apis/metrics.k8s.io/v1beta1: Get "https://<METRICS-SERVER_SVC_IP>:443/apis/metrics.k8s.io/v1beta1": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
talos-anonymized-node-name: 2022-09-21T10:15:40.259410705Z stderr F E0921 10:15:40.259265       1 available_controller.go:524] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again

Does it make any sense that this could have hit a chicken/egg situation?

[smira]

looks like kubelet got issues wrong certificate? the real error might be somewhere above, but I would at least check that join token matches across Talos config and join tokens in the cluster.

[ethernet-zero]

Damn, I think I found it!

My control plane doesn't use encryption for secret data at rest, but the new Talos node does. When generating the configuration files from the secrets, it generated an aescbcEncryptionSecret. After copying the encryption config to the control plane nodes and triggering a re-run of their corresponding kube-apiservers, the Talos node finally showed up in the list.

Now on to the next thing: Calico doesn't initialize because the control plane nodes have a /etc/cni/net.d/calico.conflist.template file and the Talos node doesn't. I'm now looking into that.

[ethernet-zero]

Fixed. I had to add 127.0.0.1 to cluster.apiServer.certSANs or else kube-proxy would be unable to watch for services:

E0923 10:47:52.129067       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://127.0.0.1:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&limit=500&resourceVersion=0": x509: certificate is valid for <ENDPOINT_IPS>, not 127.0.0.1

Also, I made Calico load that conflist template file from a ConfigMap. I'm not sure if that's a good practice for Calico or not, but it works.

That's it, there's my new and shiny Talos control plane node. Now I can send Kubespray to the deepest reaches of the oblivion where it deserves to be. 😆

[smira]

awesome! if you have any additions/fixes to the documentation, please create a pull request!

[ethernet-zero]

Will do. Thanks again!