Server: Decoupled maintenance procedure

[martastain]

Configuration examples

Default behavior (all in one)

By default, Ayon runs setup, server and mainenance in the single container.
If no AYON_RUN_ environment variable it is an equivalent to

server:
  image: ynput/ayon:latest
  environment:
    - "AYON_RUN_SETUP=true"
    - "AYON_RUN_SERVER=true"
    - "AYUN_RUN_MAINTENANCE=true"

When the container starts, setup is executed to ensure the database is up to date,
then the server is started and Maintenance procedure is scheduled as its background process.

This is perfectly fine for small, non-scaled deployments.

Separate containers

Maintenance procedure can run scheduled in a separate container.

server:
  image: ynput/ayon:latest
  environment:
    - "AYON_RUN_SETUP=true"
    - "AYON_RUN_SERVER=true"
    - "AYON_RUN_MAINTENANCE=false"

maintenance:
  image: ynput/ayon:latest
  depends_on:
    - server
  environment:
    - "AYON_RUN_SETUP=false"
    - "AYON_RUN_SERVER=false"
    - "AYON_RUN_MAINTENANCE=true"

This way an additional container maintenance is started. This container lives indefinitely
and executes maintenance procedure on times specified in the server settings.

This configuration is useful for scaled deployments, where multiple server containers are running.
But since the maintenance container lives indefinitely, it is not recommended for kubernetes-based
deployments where you pay for the resources you use.

Cron-based maintenance

You may also execute the maintenance procedure using cron. This is useful for deployments
where you have a dedicated server or a VM. In this scenario, server settings cannot be used for
scheduling the maintenance period.

Instead you use cron to execute the following command:

docker compose run server python -m maintenance --one-shot

To schedule the task, create /etc/cron.d/ayon with the following content:

# execute ayon maintenance every day at 3:00
0 3 * * * root cd /opt/ayon && docker compose run server python -m maintenance --one-shot

Separating setup

For scaled deployments, it is not desirable to run setup on every server container.
Instead, you can run setup in a separate container and then start the server containers.

This is tricky to do with docker-compose, but it can be achieved in kubernetes or a shell script.
The following script demonstrates an upgrade procedure of a stack with separate containers.

docker compose pull

# start the database and redis (or ensure they are running)
docker compose up redis db --detach

# run the setup in this case, the procedure is blocking
# so server wont start until setup is finished
docker compose run server python -m setup --ensure-installed

# rebuild and start the server and maintenance containers
doccker compose up server maintenance --detach --build