Create MonitorDaemon docker container for ol-www0 to monitor HTTP status codes

[mekarpeles]

Proposal

A general purpose container called something like MonitorDaemon that can be added to any VM and configured with a list of monitoring operations that run on that host.

First, for ol-www0 this entails the IP and status aggregation scripts defined in #8795:

check-node ol-www0 && scripts/nginx_http_status_monitor.py
check-node ol-home0 && scripts/monitoring/solr_updater_lag.py

Justification

Problem

What problem does this proposal address & for what audience(s)?

Currently stats easily interrupted when #8795 scripts run on ol-www0 via tmux are interrupted

Breakdown

Can be closed once #8795 is evolved into a docker container approach that can go into our deploy and, initially, run on ol-www0

monitoring:
    profile: ["ol-www0", "ol-home0"]

Related files

Stakeholders

---

Instructions for Contributors

Please run these commands to ensure your repository is up to date before creating a new branch to work on this issue and each time after pushing code to Github, because the pre-commit bot may add commits to your PRs upstream.

[itsBaivab]

I would love to work on this. Could you please assign this to me?

[mekarpeles]

@itsBaivab I think this one should go to @cdrini on staff for now as he's already built most of the infrastructure

[mekarpeles]

Any new monitoring should use python instead of bash for writing to graphite

[mekarpeles]

This issue requires adding a new container that only runs on production (compose.production.yml) and gets deployed to every host, however the container will only run the jobs relating to the hosts the container is on.

For this issue, the only container with jobs should be ol-www0 and should be those jobs defined by:

• https://github.com/internetarchive/openlibrary/pull/8795/files#diff-fc033adb5105ea5e8cf7814fccf223b018e19852412987d23416dead1431e4c6
• https://github.com/internetarchive/openlibrary/pull/8795/files#diff-9efe6400aaf9cc692b5dd2607a7d3d0ca8d8f0529fd195271fe1c78e31c613a3

This issue can be closed once this new docker instance for prod-only is running these two scripts on ol-www0

We should explore an alternative to watch as the command to run so the container doesn't prematurely die.

@cdrini also needs to stop the legacy tmux flow that's currently on prod for the old approach

[mekarpeles]

@itsBaivab if this is enough to go on, feel free to give it a try and ask questions