Skip to content

Commit

Permalink
ensure nomad job containers launch on the outside/VM, not inside HinD…
Browse files Browse the repository at this point in the history
… container
  • Loading branch information
traceypooh committed Mar 14, 2024
1 parent 7ff7973 commit a239015
Show file tree
Hide file tree
Showing 2 changed files with 24 additions and 25 deletions.
5 changes: 2 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Installs `nomad`, `consul`, and `caddyserver` (router) together as a mini cluste

(OK so we are now Hashistck-in-Docker _container_ :)

Nomad jobs will run as `podman` containers on the VM itself, orchestrated by `nomad`, leveraging `/var/lib/containers`.
Nomad jobs will run as `podman` containers on the VM itself, orchestrated by `nomad`, leveraging `/run/podman/podman.sock`.

The _brilliant_ `consul-template` will be used as "glue" between `consul` and `caddyserver` -- turning `caddyserver` into an always up-to-date reverse proxy router from incoming requests' Server Name Indication (SNI) to running containers :)

Expand Down Expand Up @@ -133,8 +133,7 @@ curl -sS https://internetarchive.github.io/hind/install.sh | sudo sh -s -- -e NF
- If you want to use caddy `on_demand_tls`, URL to use to respond with 200/400 status codes.
- @see https://caddy.community/t/11179
- `...`
- other command line arguments to pass on to the main container's `podman run` invocation,
like other `-v` volume mounts or otherwise.
- other command line arguments to pass on to the main container's `podman run` invocation.


## GUI, Monitoring, Interacting
Expand Down
44 changes: 22 additions & 22 deletions install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -8,18 +8,6 @@ export FQDN=$(hostname -f)
podman -v > /dev/null || echo 'please install the podman package first'
podman -v > /dev/null || exit 1

(
# in background, wait for the `bootstrap.sh`, running in the first `podman run` below, to finish
while $(! podman secret ls |grep -q ' BOOTSTRAPPED '); do sleep 1; done
podman commit -q hind-init localhost/hind
podman secret rm BOOTSTRAPPED > /dev/null
) &


# in rare case this is a symlink, ensure we mount the proper source
VLC=$(realpath /var/lib/containers 2>/dev/null || echo /var/lib/containers)


(
set -x
# We need to shared these 2 directories "inside" the running `hind` container, and "outside" on
Expand All @@ -28,39 +16,51 @@ VLC=$(realpath /var/lib/containers 2>/dev/null || echo /var/lib/containers)
mkdir -p -m777 /pv/CERTS
mkdir -p -m777 /opt/nomad/data/alloc

# In rare case this is a symlink, ensure we mount the proper source.
# NOTE: we map in /var/lib/containers here so `podman secret create` inside the `podman run`
# container will effect us, the outside/VM.
VLC=$(realpath /var/lib/containers 2>/dev/null || echo /var/lib/containers)

podman run --net=host --privileged --cgroupns=host \
-v ${VLC}:/var/lib/containers \
-e FQDN -e HOST_UNAME \
--rm --name hind-init --pull=always -q "$@" ghcr.io/internetarchive/hind:main
)


# in background, wait for the `bootstrap.sh`, running in the first `podman run` above, to finish
(
while $(! podman secret ls |grep -q ' BOOTSTRAPPED '); do sleep 1; done
podman commit -q hind-init localhost/hind
podman secret rm BOOTSTRAPPED > /dev/null
) &
wait


if [ "$HOST_UNAME" = Darwin ]; then
ARGS='-p 6000:4646 -p 8000:80 -p 4000:443 -v /sys/fs/cgroup:/sys/fs/cgroup:rw'
else
ARGS='--net=host'
fi

if ( echo "$@" |grep -Fq NFSHOME= ); then
ARGS2='-v /home:/home'
else
ARGS2=''
fi

wait

# now run the new docker image in the background
# Now run the new docker image in the background.
# NOTE: we switch `-v /var/lib/containers` to volume mounting the `podman.sock`, since we want HinD
# container to `podman run` nomad jobs on the outside/VM, not inside itself
(
SOCK=$(podman info |grep -F podman.sock |rev |cut -f1 -d ' ' |rev)
set -x
podman run --privileged --cgroupns=host \
$ARGS $ARGS2 \
-v ${VLC}:/var/lib/containers \
$ARGS \
-v $SOCK:$SOCK \
-v /opt/nomad/data/alloc:/opt/nomad/data/alloc \
-v /pv:/pv \
--secret HIND_C,type=env --secret HIND_N,type=env \
--restart=always --name hind -d -q "$@" localhost/hind >/dev/null
)


export FIRST=${FIRST:-""}
if [ ! $FIRST ]; then
echo '
Congratulations!
Expand Down

0 comments on commit a239015

Please sign in to comment.