Running Docker inside runcvm

[jimaek]

I was wondering if you have advice on what is necessary to make docker run properly inside runcvm.

I am using the standard FROM ubuntu:22.04 image but docker always fails to start

INFO[2024-04-16T18:55:31.377862594Z] containerd successfully booted in 0.028850s  
ERRO[2024-04-16T18:55:31.401038646Z] failed to mount overlay: no such device       storage-driver=overlay2
ERRO[2024-04-16T18:55:31.401552453Z] exec: "fuse-overlayfs": executable file not found in $PATH  storage-driver=fuse-overlayfs
INFO[2024-04-16T18:55:31.404625332Z] Loading containers: start.                   
WARN[2024-04-16T18:55:31.405617629Z] Running modprobe bridge br_netfilter failed with message: , error: exec: "modprobe": executable file not found in $PATH 
INFO[2024-04-16T18:55:31.408205404Z] unable to detect if iptables supports xlock: 'iptables --wait -L -n': `iptables/1.8.7 Failed to initialize nft: Protocol not supported`  error="exit status 1"
INFO[2024-04-16T18:55:31.432359836Z] stopping event stream following graceful shutdown  error="<nil>" module=libcontainerd namespace=moby
INFO[2024-04-16T18:55:31.432619588Z] stopping healthcheck following graceful shutdown  module=libcontainerd
INFO[2024-04-16T18:55:31.432675019Z] stopping event stream following graceful shutdown  error="context canceled" module=libcontainerd namespace=plugins.moby
failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables/1.8.7 Failed to initialize nft: Protocol not supported

Since you already have a small mention of running Docker inside the VM, maybe you could expand on it a bit to better understand how to get it fully operational?

The end goal is to use runcvm to run github runners inside, which in turn run docker containers as part of CI workflows.
Full dockerfile https://github.com/jimaek/github-runner-dockerfile

Thanks!

[struanb]

That's an interesting use-case and it should be a viable one. Running dockerd inside RunCVM is supported.

I've looked over the Dockerfile and on first impressions it looks fine.

What exact docker run command are you running?

[jimaek]

docker run -it  --runtime=runcvm  --entrypoint /bin/bash -env=RUNCVM_DISKS=/disks/docker,/var/lib/docker,ext4,1G ghcr.io/jimaek/github-runner-dockerfile:main 

I fixed the iptables issue by installing kmod, but it still seems somewhat unstable. Maybe there are more recommendations when it comes to running Docker inside RunCVM?

[jimaek]

Something I noticed is that if no resource limits are set, it defaults to some very strict values? I think for RAM its only 512MB. I thought it would do the opposite, no limits/access to all resources.

That was one of the issues as the VM was killing the processes inside due to OOM.

[jimaek]

Weird, even after setting a limit of 32GB of RAM, the container seems to only use up to 6GB RAM and the process inside still gets killed due to OOM. Not sure if its a bug or not.

And another question is what happens if I want to run 2 of these containers? Host has enough RAM to do so, but does the VM reserve the full amount or as long as its not fully used I can run as many VMs with memory set to 32GB?

[struanb]

A VM needs its memory allowance to be specified. If you don't specify with -m the default is indeed 512Mb which may be too small for running all but the smallest images in Docker.

Check out the Docker/Systemd example in the README. It uses 2g memory which has worked fine in testing for running small containers within the VM.

Of course the amount of memory you need to allocate depends on what containers you plan to run inside the VM, and their memory requirements.

Please keep me apprised of your evolving docker run command and I'll be happy to provide further feedback if issues persist.

[jimaek]

So its possible to run 2 containers with -m 8GB on a host with 8GB RAM total, assuming the processes inside the VMs use only 1GB each? Same goes for CPU?

[struanb]

That's a great question but one I can't answer right now.

If you want to run two RunCVM VMs, I would (for now) recommend finding out the minimum memory requirement for one, and specifying that for each one and assume KVM allocates the entire needed host memory for each one at launch time.

That is the conservative assumption.

Of course, feel free to test it the other way.