Flatcar Linux vs Talos Linux installation vmware vsphere

[zainal-abidin-assegaf]

Hi There,

Currently i used flatcar linux for k8s cluster on-prem installation, i created the flatcar linux k8s cluster in vmware workstation on my labtop and sftp to client server via ovf image.
These are my steps to create flatcar linux k8s cluster:

• create self signed cert
• create cloud-config
• convert cloud-config to json dgn CT (content transpiler)
• create manager/worker in VMware workstation
• convet to ovf
• convert ovf to ovf sha256 with VMware tools
• sftp ovf to client server

When client receive the ovf, they only do 1 thing, deploy the ovf in vsphere, that's it flatcar linux k8s cluster will automatically join cluster and client will have flatcar linux k8s cluster prod ready,

Can Talos linux do the same ??

From Talos Linux page, https://www.talos.dev/v0.9/virtualized-platforms/vmware/

Prerequisites
Prior to starting, it is important to have the following infrastructure in place and available:

DHCP server
Load Balancer or DNS address for cluster endpoint
If using a load balancer, the most common setup is to balance tcp/443 across the control plane nodes tcp/6443
If using a DNS address, the A record should return back the addresses of the control plane nodes

For Flatcar linux i don't need DHCP server / Load Balancer / DNS server , just create Flatcar linux one node at a time

[rsmitty]

Hey @4ss3g4f, thanks for joining in!

There's a bit to unpack here, but I'll take a stab. Feel free to ping me on our Slack or join our office hours if talking through this "live" is easier for you. I'll also say that I'm not a VMWare expert, so there's probably some hand waving that I'm doing.

I think that the flow with Talos would look something like this:

• Upload OVA from a Talos release into VMWare Workstation
• Boot the OVA for a control plane node and again for a worker
• Generate a machine config for the control plane and workers with talosctl gen config
• Push the machine config in with talosctl apply-config -f ...
• Once the cluster is up, snapshot and convert to OVF (or something like that)
• Deliver the OVF to clients

Concerns I have:

• Are you planning to have multiple control plane nodes? This is why we recommend the LB or a DNS entry in the docs.
• IPs. Are the IPs going to be the same in the client environment? If the IPs for the nodes change, the kubelet gets grumpy.
• Hostnames. Same as the IP point above essentially.

[zainal-abidin-assegaf]

Concerns I have:

- Are you planning to have multiple control plane nodes?
Yes 3 master node, here are the way i did it with flatcar linux:

• If there are 3 master, there will be 3 etcd master, nodes for k8s master is the same nodes for etcd master

from the Screenshot we can see that node3 192.168.207.223 is the etcd master leader

• no need loadbalancer because i also update /etc/hosts
• etcd and flannel will be orchestrated by systemd not kubernetes

- IPs. Are the IPs going to be the same in the client environment?
All IP config will be static,
I used same IP cidr in vmware workstation also customer DNS which i will not be able to reach from my labtop but i don't care because i add dns 8.8.8.8 for installation purpose

- Hostnames ? will be static from flatcar linux systemd config

• • filesystem: "root"
    path: "/etc/hostname"
    mode: 0644
    contents:
    inline: manager-01.bnpb.go.id

All traffic from worker to api server will goes to haproxy, in all k8s master/worker node there will be haproxy, docker haproxy will be orchestrated by systemd not kubernetes. HAproxy purpose :

• round-robin k8s kubelet and k8s proxy from worker node to any available api server with haproxy health check
• reverse proxy from default api server port 6443 to 6445 as security features
• active-active config, all api server will work to serve k8s worker request

• here is my flatcar linux topology:

Concerns:

• why talos linux v1.0.0 still using flannel v0.15.1 ?? latest flannel is v0.17.0,, i used flannelcni/flannel:v0.16.3-amd64
• I don't think talos linux can do what flatcar linux did with systemd, because talos linux don't use systemd (because i used some docker apps and orchestrated by systemd not kubernetes)
• flatcar linux also provide nfs for rwx apps, i tried with several docker nfs server and it was not working for rwx purpose, but flatcar linux nfs rwx worked properly. Is talos linux provide nfs for rwx in os level ??