Server maintenance procedure

[martastain]

Decoupling Background Processes

Currently, background processes run within the main server application. To improve flexibility and scalability (e.g., in Kubernetes environments), we should allow background processes, server setup, and the main server itself to run in separate containers (using the same image), controlled by environment variables.

Proposed Environment Variables

• AYON_RUN_SERVER=true - Run the web server providing the API and the frontend
• AYON_RUN_SETUP=true - Run the setup process (create the database, apply migrations, etc.) If used with AYON_RUN_SERVER, the setup process should run before the server starts (as it does now)
• AYON_RUN_MAINTENANCE=true - Run scheduled maintenance procedure (daily, described below)

Background processes

The current BackgroundProcesses should only cover lightweight, production-specific tasks:

• Collecting logs
• Invalidating timed-out webactions
• Background installing/unpacking addons, dependency packages, and so on

Note

While webactions invalidation does not require running on every instance in a scaled environment,
it should run often (every 5 minutes or so) and it has a low impact on the server, so it can be run in the main container.

The logs collector and the background installer, on the other hand, MUST run on every instance.

Maintenance procedure

Maintenance procedure will take care of the following tasks:

• AutoUpdate
• MetricsCollector
• Clean-up thumbnails
• Clean-up logs
• Clean-up links
• Clean-up activities
• Clean-up events
• Clean-up unused files
• Clean-up dead links
• Delete expired sessions from Redis (new)
• Remove old site records (new) Cleanup of Site Records #406
• DB Vacuum (new) - must run at the end of the procedure

The procedure runs all tasks in sequence to reduce the load on the server. No parallelism is needed.
Based on the environment variable AYON_RUN_MAINTENANCE the procedure will be executed in the main container or could be executed in a separate container or as a cron job:

docker compose exec server python -m maintenance

Tip

Running the procedure daily should be enough to keep the server in good shape.

If the server starts the maintenance procedure automatically, it should be possible to schedule the maintenance procedure to run at a specific time using the server configuration. That would allow running potentially blocking tasks (like DB vacuum) during low-traffic hours.
All maintenance tasks should be implemented as separate modules and it should be possible to turn them on/off using the server configuration.

Monitoring

The progress of the maintenance procedure can be tracked using the native Ayon event system. When the maintenance procedure starts. server.maintenance event is emitted with in_progress status. The event is updated to finished state when the procedure is finished. If the procedure fails, the event status is updated to failed.

During each step, the description of the event is updated with the current task name.

Separate setup process

This separation would also allow the setup process to run in a distinct container, useful in scenarios like Kubernetes, where setup could be handled in a start-up pod while the server and task processes run in their containers.