Shared network with both host to vm anv vm to vm access

[nirs]

I'm trying tart with --net-softnet, and it seems to work for single vm.

I created a linux vm by cloning the ubuntu vm, and starting it using:

tart run --net-softnet --net-softnet-allow 0.0.0.0/0 ubuntu

In the vm I get ip address 192.168.105.12, and I can access it from host, for example:

% iperf3-darwin -c 192.168.105.12      
Connecting to host 192.168.105.12, port 5201
[  5] local 192.168.105.1 port 61944 connected to 192.168.105.12 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd          RTT
[  5]   0.00-1.00   sec   691 MBytes  5.80 Gbits/sec    0   6.10 MBytes   4ms     
[  5]   1.00-2.00   sec   698 MBytes  5.85 Gbits/sec    0   7.56 MBytes   4ms     
[  5]   2.00-3.00   sec   689 MBytes  5.77 Gbits/sec    0   7.59 MBytes   4ms     
[  5]   3.00-4.00   sec   697 MBytes  5.84 Gbits/sec    0   8.00 MBytes   4ms     
[  5]   4.00-5.00   sec   695 MBytes  5.83 Gbits/sec    0   8.00 MBytes   4ms     
[  5]   5.00-6.00   sec   706 MBytes  5.92 Gbits/sec    0   8.00 MBytes   6ms     
[  5]   6.00-7.00   sec   706 MBytes  5.92 Gbits/sec    0   8.00 MBytes   5ms     
[  5]   7.00-8.00   sec   697 MBytes  5.85 Gbits/sec    0   8.00 MBytes   5ms     
[  5]   8.00-9.00   sec   689 MBytes  5.78 Gbits/sec    0   8.00 MBytes   4ms     
[  5]   9.00-10.00  sec   692 MBytes  5.80 Gbits/sec    0   8.00 MBytes   4ms     
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  6.80 GBytes  5.84 Gbits/sec    0             sender
[  5]   0.00-10.01  sec  6.79 GBytes  5.83 Gbits/sec                  receiver

However tart ip reports the vm ip as 192.168.105.13:

% tart ip ubuntu
192.168.105.13

This address is not reachable:

% ping 192.168.105.13
PING 192.168.105.13 (192.168.105.13): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
^C
--- 192.168.105.13 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss

When starting a second vm with the same options, it get exactly the same ip address (192,168.105.12), the first vms is not accessible any more, the second vm is accessible on the same ip address. Also the first vm is not accessible from the second.

The vms get different mac address so it does not make sense they will get the same ip address from the macOS DHCP server.

I can use tart --net-bridged with multiple vms, but this makes the vms accessible in the local network with I'm trying to avoid. The purpose of the vms is testing local k8s clusters, and limiting access the the local host is better.

Looks like this this is a dup of #37 and should be resolved. Maybe I need a newer version?

Tested on latest tart from brew:

% tart --version
2.23.0

[nirs]

The tart/softnet processes - we see unique mac addresses and use of the allow 0.0.0.0/0 which should allow access between vms based on #37.

% ps aux | grep softnet | grep -v grep
nir              94326   0.5  0.2 412435232  65280 s001  S+    5:21PM   0:17.41 /opt/homebrew/Cellar/tart/2.23.0/libexec/tart.app/Contents/MacOS/tart run --net-softnet --net-softnet-allow 0.0.0.0/0 ubuntu
nir              94385   0.3  0.2 412564240  65568 s002  S+    5:29PM   0:13.92 /opt/homebrew/Cellar/tart/2.23.0/libexec/tart.app/Contents/MacOS/tart run --net-softnet --net-softnet-allow=0.0.0.0/0 client
nir              94399   0.0  0.0 410774192   6992 s002  S     5:29PM   0:11.17 /opt/homebrew/bin/softnet --vm-fd 0 --vm-mac-address 6a:55:f0:1f:95:8d --allow 0.0.0.0/0 --sudo-escalation-done --user nir --group staff
root             94388   0.0  0.0 410756624   7392 s002  S     5:29PM   0:00.01 sudo --non-interactive --preserve-env=SENTRY_DSN CIRRUS_SENTRY_TAGS /opt/homebrew/bin/softnet --vm-fd 0 --vm-mac-address 6a:55:f0:1f:95:8d --allow 0.0.0.0/0 --sudo-escalation-done --user nir --group
nir              94340   0.0  0.0 410775216   7136 s001  S     5:21PM   0:40.78 /opt/homebrew/bin/softnet --vm-fd 0 --vm-mac-address f6:75:3f:32:16:3b --allow 0.0.0.0/0 --sudo-escalation-done --user nir --group staff
root             94329   0.0  0.0 410748432   7104 s001  S     5:21PM   0:00.01 sudo --non-interactive --preserve-env=SENTRY_DSN CIRRUS_SENTRY_TAGS /opt/homebrew/bin/softnet --vm-fd 0 --vm-mac-address f6:75:3f:32:16:3b --allow 0.0.0.0/0 --sudo-escalation-done --user nir --group

[edigaryev]

Hi Nir 👋

Let me take a stab in the dark here, the issue here is likely related to one or both of the VMs using the DUID/IAID as a DHCP client identifier instead of the MAC address.

This is problematic, because macOS's DHCP server putting this DUID/IAID into both the hw_address and the identifier fields in /var/db/dhcpd_leases, which makes it impossible for the tart ip to locate the correct entry given a MAC-address of the VM.

As a result:

1. tart ip returns an incorrect entry
2. When you're trying to access the tart ip-returned address from another VM, it doesn't reply to ICMP, because this address does not belong to an active VM

If you're building your VMs yourself, you need to disable the DUID/IAID/non-MAC identifiers to work around that.

If you're using Cirrus Labs VM images, this should be fixed in the latest ghcr.io/cirruslabs/ubuntu:latest. See cirruslabs/linux-image-templates#39 for more details on how we work around that.

You can further debug this by looking at the /var/db/dhcpd_leases. You can also safely delete it, as it will be re-created next time some VM asks for an IP address over DHCP.

[nirs]

If you're using Cirrus Labs VM images, this should be fixed in the latest ghcr.io/cirruslabs/ubuntu:latest. See cirruslabs/linux-image-templates#39 for more details on how we work around that.

I created the vm few days ago using:

tart clone ghcr.io/cirruslabs/ubuntu:latest server
tart clone ghcr.io/cirruslabs/ubuntu:latest client

You can further debug this by looking at the /var/db/dhcpd_leases. You can also safely delete it, as it will be re-created next time some VM asks for an IP address over DHCP.

I'll try to delete /var/db/dhcpd_leases and kickstart bootpd.

[nirs]

The issue was resolved after deleting /var/db/dhcpd_leases, restarting bootpd service, and recreating the vms.

I did:

sudo rm /var/db/dhcpd_leases
sudo launchctl kickstart -kp system/com.apple.bootpd
rm -rf ~/.tart/
tart clone ghcr.io/cirruslabs/ubuntu:latest server
tart run --net-softnet --net-softnet-allow 0.0.0.0/0 server

Server started with the expected 192.168.105.2, creating this record:

% cat /var/db/dhcpd_leases            
{
	name=ubuntu
	ip_address=192.168.105.2
	hw_address=1,72:1e:be:9f:b4:7e
	identifier=1,72:1e:be:9f:b4:7e
	lease=0x678aaeb2
}

name=ubuntu seems wrong - when using vfkit or lima you usually get the same name used in the tool. I think this is the host name.

Then I created and ran another vm:

tart clone ghcr.io/cirruslabs/ubuntu:latest client 
tart run --net-softnet --net-softnet-allow 0.0.0.0/0 client

Which got the expected ip address 192.168.105.3 and added another record:

% cat /var/db/dhcpd_leases
{
	name=ubuntu
	ip_address=192.168.105.3
	hw_address=1,da:b3:7e:2e:58:e9
	identifier=1,da:b3:7e:2e:58:e9
	lease=0x678aaf15
}
{
	name=ubuntu
	ip_address=192.168.105.2
	hw_address=1,72:1e:be:9f:b4:7e
	identifier=1,72:1e:be:9f:b4:7e
	lease=0x678aaeb2
}

Both vms are accessible and I could run iperf3 from host and between the vms.

Previously I had many entries with name=client from other vms created with vfkit and lima, and maybe I had also name=ubuntu for another vm.

I guess this issue should be documented somewhere.

Thanks for the quick response!