feat: deprecate kube-proxy

[fitbeard]

Partially related: vexxhost/atmosphere#184

TL;DR: EL9, Fedora36+, newer versions of other distributions comes with iptables 1.8.8 (nftable only) when kube-proxy image relies on internal iptables 1.8.4 binaries. In such a situation rules created with older version are not fully compatible with newer version. Actually it’s a total mess.

kubernetes/minikube#15573
kubernetes-sigs/iptables-wrappers#3
kubernetes/kube-proxy#23
https://kubernetes.io/blog/2022/09/07/iptables-chains-not-api/#use-case-iptables-mode
https://twitter.com/thockin/status/1171143481001009152

In my tests on EL9 suggested workaround with iptables-legacy (hacked iptables 1.8.8 package) did not give positive results. Cilium agents on kube worker nodes tries access https://10.96.0.1:443 without success.

We have cilium, eBPF and modern kernel versions - let's deprecate oldie kube-proxy.

https://docs.cilium.io/en/stable/network/kubernetes/kubeproxy-free/#kubeproxy-free

[mnaser]

Hmm.. this is a pretty significant change to implement and I think we'll have to pay more attention to how this affects existing deployments and upgrades.

How are folks running Kubernetes today with newer ip tables? Are they all resorting to moving to dropping kube proxy ? I'll review those links you sent

[fitbeard]

Based on cilium documentation:

$ kubectl -n kube-system delete ds kube-proxy
$ kubectl -n kube-system delete cm kube-proxy

Should be enough. These tasks could be part of the cilium role.

[fitbeard]

https://kubernetes.io/docs/reference/networking/virtual-ips/#proxy-mode-nftables

nftables added as alpha in Kubernetes 1.29

@mnaser i still think that switching fully to cilium is better idea. https://docs.cilium.io/en/stable/network/kubernetes/kubeproxy-free/

I have test environment running for 5 months with atmosphere + rook and this cillium setup on EL9 with zero issues.