🐛 BUG: "Listener is IPv4, but writing to IPv6 remote" error is usually unhelpful

[johnmaguire]

What version of nebula are you using? (nebula -version)

1.9.4

What operating system are you using?

Linux

Describe the Bug

If a host has IPv6 disabled in the kernel, or listen.host set to 0.0.0.0, and the Lighthouse returns an IPv6 address during the handshake process, this error will occur. This confuses a lot of users into thinking it's the cause of ~(whatever issues they happen to be having). Usually, it just means the host will find a tunnel over IPv4 instead.

Maybe we can make this message warning-level, debug-level, less frequent, more informative, or occur in fewer configurations?

@wadey suggested defaulting local_allow_list and remote_allow_list to to exclude IPv6 when we see listen.host is set to 0.0.0.0 or IPv6 is disabled in the kernel. (This can be done manually as a workaround today.)

Logs from affected hosts

n/a

Config files from affected hosts

n/a

[erikschul]

You meant "if the host has IPv6 disabled" ?

[migs35323]

I have a problem that might be related,

I have setup in all my nodes:

static_map:
  network: ip4
listen:
  host: 0.0.0.0
  port: 4242
static_host_map:
  "10.10.10.10":
    - "1.1.1.1:4242"
    - "lighthouse.example.com:4242"

however it happears they seem to be connecting using ipv6...
error="Listener is IPv4, but writing to IPv6 remote" handshake="map[stage:1 style:ix_psk0]" initiatorIndex=932224779 localIndex=932224779 remoteIndex=0 udpAddr="[2a2a:2a2a:2a2a:2a2a::2]:

theese specific nodes have both ipv4 and ipv6 in the eth interface.
i wanted to only use ipv4 but keep the ipv6 at the system level enabled....

i believe either the message is wrong or nebula is ignoring the static map directive...

[johnmaguire]

@migs35323 It looks like you're experiencing the misleading error this ticket is written to explain.

If a host has IPv6 disabled in the kernel, or listen.host set to 0.0.0.0, and the Lighthouse returns an IPv6 address during the handshake process, this error will occur. This confuses a lot of users into thinking it's the cause of ~(whatever issues they happen to be having).

You mentioned static_map.network, but this is uninvolved here: this simply determines whether IPv4 or IPv6 addresses are returned from DNS lookups of the hosts in static_host_map. This ensures that we don't use an IPv6 address (AAAA record) returned for lighthouse.example.com.

The message you are see likely occurs after you've handshaked to the Lighthouse, when the host attempts to handshake with another host. This is because hosts report all known addresses (e.g. from ip addr) to the Lighthouse: IPv4 and IPv6. The other host you're handshaking with reported an IPv6 address to the LIghthouse. When you go to handshake with a host that has reported IPv6 addresses, this error occurs because you have listen.host set to IPv4-only.

wadey suggested defaulting local_allow_list and remote_allow_list to to exclude IPv6 when we see listen.host is set to 0.0.0.0 or IPv6 is disabled in the kernel. (This can be done manually as a workaround today.)

I suggest giving the work around a try. You can learn more about local_allow_list and remote_allow_list here: https://nebula.defined.net/docs/config/lighthouse/