Issue with Accessing NodePort via Virtual IP (VIP) on Talos Linux Cluster #9405

AruPersia · 2024-09-30T18:07:03Z

AruPersia
Sep 30, 2024

Hi Talos Community,

I’m running a Talos Linux cluster with KubePrism instead of kube-proxy, and I'm having trouble accessing a NodePort (30443) service (NGINX Ingress Controller) via the Virtual IP (VIP). For over 3 months, accessing the NodePort via the VIP (10.10.30.100:30443) worked fine. However, after an outage on one of the nodes, I’m now getting a Connection refused error when using the VIP.

❯ nc -vz 10.10.30.100 30443
nc: connectx to 10.10.30.100 port 30443 (tcp) failed: Connection refused
❯ nc -vz 10.10.30.101 30443
Connection to 10.10.30.101 port 30443 [tcp/*] succeeded!
❯ nc -vz 10.10.30.102 30443
Connection to 10.10.30.102 port 30443 [tcp/*] succeeded!
❯ nc -vz 10.10.30.103 30443
Connection to 10.10.30.103 port 30443 [tcp/*] succeeded!

Direct access to the NodePort on individual node IPs still works.

Any ideas on how to resolve this?

Thanks!

Nodes

❯ kubectl get nodes -o wide
NAME                       STATUS   ROLES           AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE         KERNEL-VERSION   CONTAINER-RUNTIME
xxxxxxx-controlplane-101   Ready    control-plane   87d   v1.30.1   10.10.30.101   <none>        Talos (v1.7.7)   6.6.52-talos     containerd://1.7.22
xxxxxxx-controlplane-102   Ready    control-plane   87d   v1.30.1   10.10.30.102   <none>        Talos (v1.7.7)   6.6.52-talos     containerd://1.7.22
xxxxxxx-controlplane-103   Ready    control-plane   87d   v1.30.1   10.10.30.103   <none>        Talos (v1.7.7)   6.6.52-talos     containerd://1.7.22
xxxxxxx-worker-201         Ready    <none>          87d   v1.30.1   10.10.30.201   <none>        Talos (v1.7.7)   6.6.52-talos     containerd://1.7.22
xxxxxxx-worker-202         Ready    <none>          87d   v1.30.1   10.10.30.202   <none>        Talos (v1.7.7)   6.6.52-talos     containerd://1.7.22
xxxxxxx-worker-203         Ready    <none>          87d   v1.30.1   10.10.30.203   <none>        Talos (v1.7.7)   6.6.52-talos     containerd://1.7.22

Nginx Ingress Controller

❯ kubectl -n ingress-nginx-core get svc
NAME                                               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-core-external-controller             NodePort    10.107.245.120   <none>        80:30080/TCP,443:30443/TCP   46h
ingress-nginx-core-external-controller-admission   ClusterIP   10.111.184.244   <none>        443/TCP                      46h
ingress-nginx-core-internal-controller             NodePort    10.105.180.252   <none>        80:31080/TCP,443:31443/TCP   46h
ingress-nginx-core-internal-controller-admission   ClusterIP   10.108.38.128    <none>        443/TCP                      46h

Members

❯ talosctl -n 10.10.30.100 get members
NODE           NAMESPACE   TYPE     ID                         VERSION   HOSTNAME                   MACHINE TYPE   OS               ADDRESSES
10.10.30.100   cluster     Member   xxxxxxx-controlplane-101   3         xxxxxxx-controlplane-101   controlplane   Talos (v1.7.7)   ["10.10.30.101"]
10.10.30.100   cluster     Member   xxxxxxx-controlplane-102   3         xxxxxxx-controlplane-102   controlplane   Talos (v1.7.7)   ["10.10.30.102"]
10.10.30.100   cluster     Member   xxxxxxx-controlplane-103   3         xxxxxxx-controlplane-103   controlplane   Talos (v1.7.7)   ["10.10.30.100","10.10.30.103"]
10.10.30.100   cluster     Member   xxxxxxx-worker-201         4         xxxxxxx-worker-201         worker         Talos (v1.7.7)   ["10.10.30.201"]
10.10.30.100   cluster     Member   xxxxxxx-worker-202         4         xxxxxxx-worker-202         worker         Talos (v1.7.7)   ["10.10.30.202"]
10.10.30.100   cluster     Member   xxxxxxx-worker-203         2         xxxxxxx-worker-203         worker         Talos (v1.7.7)   ["10.10.30.203"]

❯ talosctl -n 10.10.30.100 etcd members
NODE           ID                 HOSTNAME                   PEER URLS                   CLIENT URLS                 LEARNER
10.10.30.100   489c0d3d96d436ce   xxxxxxx-controlplane-103   https://10.10.30.103:2380   https://10.10.30.103:2379   false
10.10.30.100   5e0abd023e56c559   xxxxxxx-controlplane-101   https://10.10.30.101:2380   https://10.10.30.101:2379   false
10.10.30.100   821e86300dcfa2c1   xxxxxxx-controlplane-102   https://10.10.30.102:2380   https://10.10.30.102:2379   false

Extensions

❯ talosctl -n 10.10.30.101,10.10.30.102,10.10.30.103 get extensions
NODE           NAMESPACE   TYPE              ID   VERSION   NAME               VERSION
10.10.30.101   runtime     ExtensionStatus   0    1         iscsi-tools        v0.1.4
10.10.30.101   runtime     ExtensionStatus   1    1         qemu-guest-agent   8.2.2
10.10.30.101   runtime     ExtensionStatus   2    1         util-linux-tools   2.39.3
10.10.30.101   runtime     ExtensionStatus   3    1         schematic          88d1f7a5c4f1d3aba7df787c448c1d3d008ed29cfb34af53fa0df4336a56040b
10.10.30.102   runtime     ExtensionStatus   0    1         iscsi-tools        v0.1.4
10.10.30.102   runtime     ExtensionStatus   1    1         qemu-guest-agent   8.2.2
10.10.30.102   runtime     ExtensionStatus   2    1         util-linux-tools   2.39.3
10.10.30.102   runtime     ExtensionStatus   3    1         schematic          88d1f7a5c4f1d3aba7df787c448c1d3d008ed29cfb34af53fa0df4336a56040b
10.10.30.103   runtime     ExtensionStatus   0    1         iscsi-tools        v0.1.4
10.10.30.103   runtime     ExtensionStatus   1    1         qemu-guest-agent   8.2.2
10.10.30.103   runtime     ExtensionStatus   2    1         util-linux-tools   2.39.3
10.10.30.103   runtime     ExtensionStatus   3    1         schematic          88d1f7a5c4f1d3aba7df787c448c1d3d008ed29cfb34af53fa0df4336a56040b

Ping Test

❯ ping -c 1 10.10.30.100
PING 10.10.30.100 (10.10.30.100): 56 data bytes
64 bytes from 10.10.30.100: icmp_seq=0 ttl=63 time=4.705 ms

--- 10.10.30.100 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 4.705/4.705/4.705/0.000 ms

❯ ping -c 1 10.10.30.101
PING 10.10.30.101 (10.10.30.101): 56 data bytes
64 bytes from 10.10.30.101: icmp_seq=0 ttl=63 time=3.499 ms

--- 10.10.30.101 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 3.499/3.499/3.499/0.000 ms

❯ ping -c 1 10.10.30.102
PING 10.10.30.102 (10.10.30.102): 56 data bytes
64 bytes from 10.10.30.102: icmp_seq=0 ttl=63 time=5.076 ms

--- 10.10.30.102 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 5.076/5.076/5.076/nan ms

❯ ping -c 1 10.10.30.103
PING 10.10.30.103 (10.10.30.103): 56 data bytes
64 bytes from 10.10.30.103: icmp_seq=0 ttl=63 time=4.661 ms

--- 10.10.30.103 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 4.661/4.661/4.661/nan ms

Connectivity Test on Port 30443

❯ nc -vz 10.10.30.100 30443
nc: connectx to 10.10.30.100 port 30443 (tcp) failed: Connection refused
❯ nc -vz 10.10.30.101 30443
Connection to 10.10.30.101 port 30443 [tcp/*] succeeded!
❯ nc -vz 10.10.30.102 30443
Connection to 10.10.30.102 port 30443 [tcp/*] succeeded!
❯ nc -vz 10.10.30.103 30443
Connection to 10.10.30.103 port 30443 [tcp/*] succeeded!

Connectivity Test on Port 6443

❯ nc -vz 10.10.30.100 6443
Connection to 10.10.30.100 port 6443 [tcp/sun-sr-https] succeeded!
❯ nc -vz 10.10.30.101 6443
Connection to 10.10.30.101 port 6443 [tcp/sun-sr-https] succeeded!
❯ nc -vz 10.10.30.102 6443
Connection to 10.10.30.102 port 6443 [tcp/sun-sr-https] succeeded!
❯ nc -vz 10.10.30.103 6443
Connection to 10.10.30.103 port 6443 [tcp/sun-sr-https] succeeded!

Connectivity Test on Port 50000

❯ nc -vz 10.10.30.100 50000
Connection to 10.10.30.100 port 50000 [tcp/*] succeeded!
❯ nc -vz 10.10.30.101 50000
Connection to 10.10.30.101 port 50000 [tcp/*] succeeded!
❯ nc -vz 10.10.30.102 50000
Connection to 10.10.30.102 port 50000 [tcp/*] succeeded!
❯ nc -vz 10.10.30.103 50000
Connection to 10.10.30.103 port 50000 [tcp/*] succeeded!

Connectivity Test on Port 50001

❯ nc -vz 10.10.30.100 50001
Connection to 10.10.30.100 port 50001 [tcp/*] succeeded!
❯ nc -vz 10.10.30.101 50001
Connection to 10.10.30.101 port 50001 [tcp/*] succeeded!
❯ nc -vz 10.10.30.102 50001
Connection to 10.10.30.102 port 50001 [tcp/*] succeeded!
❯ nc -vz 10.10.30.103 50001
Connection to 10.10.30.103 port 50001 [tcp/*] succeeded!

Addresses

❯ talosctl -n 10.10.30.101 get addresses
NODE           NAMESPACE   TYPE            ID                                             VERSION   ADDRESS                        LINK
10.10.30.101   network     AddressStatus   cilium_host/10.244.3.193/32                    1         10.244.3.193/32                cilium_host
10.10.30.101   network     AddressStatus   cilium_host/fe80::8da:88ff:feda:abf0/64        1         fe80::8da:88ff:feda:abf0/64    cilium_host
10.10.30.101   network     AddressStatus   cilium_net/fe80::e0f0:a8ff:fe34:94f9/64        2         fe80::e0f0:a8ff:fe34:94f9/64   cilium_net
10.10.30.101   network     AddressStatus   cilium_vxlan/fe80::b8c3:47ff:fe0d:3382/64      2         fe80::b8c3:47ff:fe0d:3382/64   cilium_vxlan
10.10.30.101   network     AddressStatus   eth0/10.10.30.100/32                           1         10.10.30.100/32                eth0
10.10.30.101   network     AddressStatus   eth0/10.10.30.101/24                           1         10.10.30.101/24                eth0
10.10.30.101   network     AddressStatus   eth0/fe80::8489:eaff:fe6d:7cbf/64              2         fe80::8489:eaff:fe6d:7cbf/64   eth0
10.10.30.101   network     AddressStatus   lo/127.0.0.1/8                                 1         127.0.0.1/8                    lo
10.10.30.101   network     AddressStatus   lo/::1/128                                     1         ::1/128                        lo
10.10.30.101   network     AddressStatus   lxc0dccc825abe1/fe80::286d:a1ff:fef3:20f4/64   2         fe80::286d:a1ff:fef3:20f4/64   lxc0dccc825abe1
10.10.30.101   network     AddressStatus   lxc_health/fe80::7834:66ff:fe9c:b48b/64        2         fe80::7834:66ff:fe9c:b48b/64   lxc_health
10.10.30.101   network     AddressStatus   lxcd07f60af1a96/fe80::6c6e:e8ff:fe9a:123c/64   2         fe80::6c6e:e8ff:fe9a:123c/64   lxcd07f60af1a96

❯ talosctl -n 10.10.30.102 get addresses
NODE           NAMESPACE   TYPE            ID                                             VERSION   ADDRESS                        LINK
10.10.30.102   network     AddressStatus   cilium_host/10.244.4.110/32                    1         10.244.4.110/32                cilium_host
10.10.30.102   network     AddressStatus   cilium_host/fe80::7897:51ff:fea8:2e7d/64       2         fe80::7897:51ff:fea8:2e7d/64   cilium_host
10.10.30.102   network     AddressStatus   cilium_net/fe80::50db:e3ff:fe75:fd67/64        2         fe80::50db:e3ff:fe75:fd67/64   cilium_net
10.10.30.102   network     AddressStatus   cilium_vxlan/fe80::64a4:53ff:fe4a:89e0/64      2         fe80::64a4:53ff:fe4a:89e0/64   cilium_vxlan
10.10.30.102   network     AddressStatus   eth0/10.10.30.102/24                           1         10.10.30.102/24                eth0
10.10.30.102   network     AddressStatus   eth0/fe80::64ce:f8ff:fe3f:8699/64              2         fe80::64ce:f8ff:fe3f:8699/64   eth0
10.10.30.102   network     AddressStatus   lo/127.0.0.1/8                                 1         127.0.0.1/8                    lo
10.10.30.102   network     AddressStatus   lo/::1/128                                     1         ::1/128                        lo
10.10.30.102   network     AddressStatus   lxc07e17807ca04/fe80::3082:f7ff:feeb:8b8/64    2         fe80::3082:f7ff:feeb:8b8/64    lxc07e17807ca04
10.10.30.102   network     AddressStatus   lxc7f5d9b65da86/fe80::e0f3:b7ff:fe6e:d75b/64   2         fe80::e0f3:b7ff:fe6e:d75b/64   lxc7f5d9b65da86
10.10.30.102   network     AddressStatus   lxc_health/fe80::58c0:84ff:fed1:bd1b/64        2         fe80::58c0:84ff:fed1:bd1b/64   lxc_health

❯ talosctl -n 10.10.30.103 get addresses
NODE           NAMESPACE   TYPE            ID                                          VERSION   ADDRESS                        LINK
10.10.30.103   network     AddressStatus   cilium_host/10.244.1.110/32                 1         10.244.1.110/32                cilium_host
10.10.30.103   network     AddressStatus   cilium_host/fe80::10af:beff:fe47:4ae/64     1         fe80::10af:beff:fe47:4ae/64    cilium_host
10.10.30.103   network     AddressStatus   cilium_net/fe80::700e:d9ff:fe33:a92d/64     2         fe80::700e:d9ff:fe33:a92d/64   cilium_net
10.10.30.103   network     AddressStatus   cilium_vxlan/fe80::44ea:adff:fe1e:31e0/64   2         fe80::44ea:adff:fe1e:31e0/64   cilium_vxlan
10.10.30.103   network     AddressStatus   eth0/10.10.30.103/24                        1         10.10.30.103/24                eth0
10.10.30.103   network     AddressStatus   eth0/fe80::3050:31ff:fecd:d95b/64           2         fe80::3050:31ff:fecd:d95b/64   eth0
10.10.30.103   network     AddressStatus   lo/127.0.0.1/8                              1         127.0.0.1/8                    lo
10.10.30.103   network     AddressStatus   lo/::1/128                                  1         ::1/128                        lo
10.10.30.103   network     AddressStatus   lxc_health/fe80::4cd8:88ff:fe5c:858e/64     2         fe80::4cd8:88ff:fe5c:858e/64   lxc_health

Health

❯ talosctl -n 10.10.30.101 health
discovered nodes: ["10.10.30.201" "10.10.30.202" "10.10.30.203" "10.10.30.100" "10.10.30.102" "10.10.30.103"]
waiting for etcd to be healthy: ...
waiting for etcd to be healthy: OK
waiting for etcd members to be consistent across nodes: ...
waiting for etcd members to be consistent across nodes: OK
waiting for etcd members to be control plane nodes: ...
waiting for etcd members to be control plane nodes: OK
waiting for apid to be ready: ...
waiting for apid to be ready: OK
waiting for all nodes memory sizes: ...
waiting for all nodes memory sizes: OK
waiting for all nodes disk sizes: ...
waiting for all nodes disk sizes: OK
waiting for kubelet to be healthy: ...
waiting for kubelet to be healthy: OK
waiting for all nodes to finish boot sequence: ...
waiting for all nodes to finish boot sequence: OK
waiting for all k8s nodes to report: ...
waiting for all k8s nodes to report: OK
waiting for all k8s nodes to report ready: ...
waiting for all k8s nodes to report ready: OK
waiting for all control plane static pods to be running: ...
waiting for all control plane static pods to be running: OK
waiting for all control plane components to be ready: ...
waiting for all control plane components to be ready: OK
waiting for kube-proxy to report ready: ...
waiting for kube-proxy to report ready: SKIP
waiting for coredns to report ready: ...
waiting for coredns to report ready: OK
waiting for all k8s nodes to report schedulable: ...
waiting for all k8s nodes to report schedulable: OK

smira · 2024-09-30T18:12:20Z

smira
Sep 30, 2024
Maintainer

First of all, Talos VIP is not supposed to be used for applications, but only for controlplane access on port 6443. Every other use is possible, but not supported.

In your output, the address .100 is assigned to the first machine, and that's all what Talos should do, it doesn't manage anything beyond that.

As a wild guess, you regenerated machine config for Talos v1.8, and you enabled new feature: https://www.talos.dev/v1.8/introduction/what-is-new/#default-node-labels

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue with Accessing NodePort via Virtual IP (VIP) on Talos Linux Cluster #9405

{{title}}

Replies: 1 comment

{{title}}

Select a reply

Issue with Accessing NodePort via Virtual IP (VIP) on Talos Linux Cluster #9405

AruPersia Sep 30, 2024

Nodes

Nginx Ingress Controller

Members

Extensions

Ping Test

Connectivity Test on Port 30443

Connectivity Test on Port 6443

Connectivity Test on Port 50000

Connectivity Test on Port 50001

Addresses

Health

Replies: 1 comment

smira Sep 30, 2024 Maintainer

AruPersia
Sep 30, 2024

smira
Sep 30, 2024
Maintainer