Tainting worker nodes?

[maulerjan]

Hi all,

I'd like to be able to use:

machine:
  nodeTaints:
    - MyTaint: true:NoSchedule

on worker nodes.

This is currently not possible because of the NodeRestriction admission controller, which is not configurable.

Disabling this restriction is against CIS hardening guidelines and possible other security implications this might introduce.

Is there a way around this issue other than tainting nodes manually?

We have a declarative setup except calling talosctl -n ... bootstrap, which is the only non-declarative step we have in our cluster setup workflow.

Thanks!

[maulerjan]

Also, it doesn't seem like the admission controller can be turned off at all.

cluster:
  apiServer:
    extraArgs:
      enable-admission-plugins: CertificateApproval,CertificateSigning,CertificateSubjectRestriction,DefaultIngressClass,DefaultStorageClass,DefaultTolerationSeconds,LimitRanger,MutatingAdmissionWebhook,NamespaceLifecycle,PersistentVolumeClaimResize,PodSecurity,Priority,ResourceQuota,RuntimeClass,ServiceAccount,StorageObjectInUseProtection,TaintNodesByCondition,ValidatingAdmissionPolicy,ValidatingAdmissionWebhook
      disable-admission-plugins: NodeRestriction

Results in apiserver pods crashing:

"command failed" err="[NodeRestriction] in enable-admission-plugins and disable-admission-plugins overlapped"

As you can see, there is NO overlap in the config options provided, so Talos probably just throws the config option away and use its own (correct me if I'm wrong).

[maulerjan]

Also, the docs are wrong:

Note: In the default Kubernetes configuration, worker nodes are not allowed to modify the taints

But that's not true, NodeRestriction admission controller is not enabled by default:
https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#which-plugins-are-enabled-by-default

[smira]

It is enabled by default in Talos.

[maulerjan]

Oh I see, thanks for pointing that out

[smira]

You can use kubelet's registerWithTaints configuration (and pass it via machine.kubelet.extraConfig to taint at the moment the Node is created. Updating taints on the fly is still denied by default.

[maulerjan]

Works like a charm! Setting this only on initial node registration is enough for our use-case. Thanks!

[maulerjan]

Follow-up question:
You say updating taints on the fly is denied by default. But I can't find a way to disable the NodeRestriction controller. Is there any way?

[smira]

Looks like we don't support that kind of configuration in Talos, and I'd say that probably it should never be disabled.