[CALICO] Wrong interface in ip route

[antoinetran]

What is wrong
The communication inside pods for DNS fails randomly. For example:

nslookup kubernetes.default.svc

RKE version:
1.4.8
Docker version: (docker version,docker info preferred)

Operating system and kernel: (cat /etc/os-release, uname -r preferred)
Almalinux 8
Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)
Cloud OVH
cluster.yml file:

Steps to Reproduce:

• deploy Rancher with default CNI calico

network:
  plugin: calico
  mtu: 1450

• force network interface

        kubectl --kubeconfig ~/kube_config_rancher-cluster.yaml -n kube-system set env daemonset/calico-node IP_AUTODETECTION_METHOD=interface={{ network_calico_autodetect }}
        kubectl --kubeconfig ~/kube_config_rancher-cluster.yaml -n kube-system delete pod -l k8s-app=calico-node

Results:
calico-node pods are all redeployed with correct network interface, however some routes related to kubernetes pods CIDRs are created with the wrong interface. Some are wit the correct interface.

Workaround

while read -r line ; do
  echo "Deleting $line"
  sudo ip route delete $line
done < <(ip route | grep -v blackhole | grep "10.41." | grep -v 192.168.21)
ip route | grep "10.41." | grep -v 192.168.21

Then immediately , calico-node pods recreates the correct routes related to the correct network interfaces.

[antoinetran]

My analysis is that calico-node pods are first created with default autodetect. I don't know why the first found interface is not the same everywhere, but then it has some time to create the wrong ip routes. Then after forcing the correct interface thanks to env variables (see above), it does not overwrite the already existing route, with wrong interface, even if it differs from the newly found interface.

Proper solution: provide a way from rancher-cluster.yaml to immediately provides the calico mechanism to autodetect network interface. A good configuration value would be "kubernetes-internal-ip" (https://docs.tigera.io/calico/latest/networking/ipam/ip-autodetection#autodetection-methods).

I looked at rke1 code related to CNI calico, there is no existing way to configure the network interface.

[antoinetran]

Related to #711