(PE-40163) automate recovery of failed postgres server

[davidmalloncares]

Summary

This PR aims to automate the recovery of a failed Postgres server node by combining all the steps detailed here into one Bolt Plan. In order to test we need to provision an environment that has a PE server+replica, Postgres server+replica, and at least one compiler. Plan can be executed by running for example:

bolt plan run peadm::replace_failed_postgresql \
  primary_host=nasty-missing.delivery.puppetlabs.net \
  replica_host=alma-drying.delivery.puppetlabs.net \
  working_postgresql_host=frail-streak.delivery.puppetlabs.net \
  failed_postgresql_host=gooey-flatiron.delivery.puppetlabs.net \
  replacement_postgresql_host=crooked-mail.delivery.puppetlabs.net --no-host-key-check

Additional Context

Looking at tests in a separate ticket

Related Issues (if any)

Checklist

• 🟢 Spec tests.
• 🟢 Acceptance tests.

Changes include test coverage?

• Yes
• Not needed

Have you updated the documentation?

• Yes, I've updated the appropriate docs
• Not needed

[davidmalloncares]

@ragingra there seems to be an issue with this line. If I run this plan with this line commented out and then run that line manually it all works ok. Do you see anything obvious in how I am calling it here vs how it is called manually?

bolt plan run peadm::add_database -t <replacement-postgres-server-fqdn> primary_host=<primary-server-fqdn>

[davidmalloncares]

@ragingra I got a fix for the issue I was having and finally got all the tests to go green - could you take a wee look? :)