Prevent Talos from changing EFI boot order

[ai-christianson]

Hi, we're working on getting Talos set up with PXE booting so that we can have a fully-automated infrastructure. Talos is PXE booting and installing itself as expected.

Talos is generally working great for us and we love the project.

The issue we're running into is that once Talos is installed to disk, it also installs itself as a first-priority EFI boot entry. Not all of our systems allow forcing net boot before EFI, or locking EFI boot order, so this ends up taking away our automated control of our systems. E.g., if a machine gets in a completely failed/irrecoverable state, we need to be able to remote reboot the system and have it netboot so that we can recover it. We also need to retain the ability to re-provision the node as something else entirely if need-be, completely out of band of talosctl.

In #7276, @netthier said:

Server providers like Hetzner strongly recommend against changing the boot order as their rescue systems rely on PXE always being the first option, so this should be configurable if implemented.

This sounds like essentially the same issue we're running into.

The concept was also brought up in #2906 by @scheuk:

When you install an OS on a clean disk, the efi boot order is automatically switched to boot from the disk first even if the bios was configured for network boot first. In a traditionally linux OS you can change the boot order from the command line using efibootmgr post installation. By exposing this in the API we can have sidero force the node to pxe boot and wipe/and re-install talos into the disk.

Suggested workaround by @smira at the time:

One way around right now might be to disable machine.install.bootloader in Talos config and enable server.pxeBootAlways in Sidero Server resource. This way Talos won't touch EFI boot manager and it will always PXE boot from Sidero. (not perfect, but workaround for now)

Is there currently any way to prevent Talos from installing itself as an EFI boot entry, or otherwise have some control over the order that it installs itself?

Edit: I see in the legacy Talos docs, there was a bootloader flag for the install yaml. It seems that was removed?

Edit: Removal commit: 423a31a

Deprecate the bootloader installer option. This has not been used in a
long while.

[smira]

I guess you're booting Talos using GRUB, which does install itself into the EFI (otherwise it would never boot from disk).

As a workaround, you can still run efibootmgr in a daemonset/static pod and change the stuff specific to your system.

We will look into this issue as we have some spare cycles.

[ai-christianson]

As a workaround, you can still run efibootmgr in a daemonset/static pod and change the stuff specific to your system.

Good idea, should work in theory. I'm attempting it with this command (doing it manually first to get it right before turning it into a DaemonSet):

kubectl run node-admin --namespace kube-system --privileged -i -t --rm --restart=Never --image=debian:latest --overrides='{"spec":{"nodeName":"cubezero","hostNetwork":true,"hostPID":true,"containers":[{"name":"node-admin","image":"debian:latest","stdin":true,"stdinOnce":true,"tty":true,"volumeMounts":[{"name":"hostfs","mountPath":"/sys/firmware/efi"}]}],"volumes":[{"name":"hostfs","hostPath":{"path":"/sys/firmware/efi","type":"Directory"}}]}}'

I've tried many variations of mounting but can't seem to get anything under /sys to be mounted in the container, let alone rw.

If I run this command, I can get host EFI vars to be readable at /hostfs/sys/firmware/efi/..., but they're read only:

kubectl run node-admin --namespace kube-system --privileged -i -t --rm --restart=Never --image=debian:latest --overrides='{"spec":{"nodeName":"cubezero","hostNetwork":true,"hostPID":true,"containers":[{"name":"node-admin","image":"debian:latest","stdin":true,"stdinOnce":true,"tty":true,"volumeMounts":[{"name":"hostfs","mountPath":"/hostfs"}]}],"volumes":[{"name":"hostfs","hostPath":{"path":"/","type":"Directory"}}]}}'

efibootmgr expects the EFI vars to be under /sys/firmware/efi/... and there seems to be no override.

[smira]

This is more of a Kubernetes question here, but I guess efivars can be mounted separately as needed.