database: if all clients fail to connect, emit "disconnect" event

[haraldschilly]

This is related to #5997 … i.e. this records the case where there is no connection to the database, to eventually cause the health check to fail. This is more general than the "forgot the database password"-problem, because it should cause a health check failure in all cases, when it is not possible to establish a connection to the database.

test: I haven't tested this, but I think it should be fine. In the end, the misc.retry_until_success will call this @_connect again and again and the health checks last disconnect timestamp will be set once and kept at the point in time when this failure happened for the first time.

---

This really needed more work. The weird part is, when there is a "disconnect" event, it should be emitted and received by that record-connect-error, but the second time around (connect-disconnect-connect-disconnect), it no longer receives the event. Somewhere, everyone who listens to the event emitter is removed, or just the "disconnect" event, and never re-establishes the connection. I saw there is code in the syncstrings part, which could be relevant – not sure.

In any case, my fix is to call the register connect/disconnect functions directly. A disconnect happens either when the database client is disconnected or if the attempt to connect fails.

My test in my local dev environment was to start/stop the database and check what the health check has to say, i.e.

$ curl http://localhost:5000/healthcheck
healthchecks:
concurrent 0 < undefined – OK
uptime 1m40s – draining in 24m49s – terminating in 25m49s – OK
DB problems for 0.14 < 5 mins – OK

vs.

$ curl http://localhost:5000/healthcheck
healthchecks:
concurrent 0 < undefined – OK
uptime 1m41s – draining in 24m48s – terminating in 25m48s – OK
no DB connection problems – OK

flipping back and forth when starting/stopping my local db.