Add overview of preparing for DR

[asteflova]

What changes are you introducing?

I'm adding a list of recommended disaster recovery plans. Each DR plan description follows the same template to make it easier to compare them and help users decide which one to choose.

Why are you introducing these changes? (Explanation, links to references, issues, etc.)

N/A

Anything else to add? (Considerations, potential downsides, alternative solutions you have explored, etc.)

As a next step, I'd like to provide more detailed instructions for each of the DR methods. Ideally with three procedures for each: 1. Steps to prepare, 2. Steps to test, 3. Steps to recover.

Checklists

• I am okay with my commits getting squashed when you merge this PR.
• I am familiar with the contributing guidelines.

Please cherry-pick my commits into:

• Foreman 3.13/Katello 4.15 (EL9 only)
• Foreman 3.12/Katello 4.14 (Satellite 6.16)
• Foreman 3.11/Katello 4.13 (orcharhino 6.11 on EL8 only; orcharhino 7.0 on EL8+EL9)
• Foreman 3.10/Katello 4.12
• Foreman 3.9/Katello 4.11 (Satellite 6.15; orcharhino 6.8/6.9/6.10)
• Foreman 3.8/Katello 4.10
• Foreman 3.7/Katello 4.9 (Satellite 6.14)
• We do not accept PRs for Foreman older than 3.7.

[github-actions]

The PR preview for 269a7ea is available at theforeman-foreman-documentation-preview-pr-3558.surge.sh

The following output files are affected by this PR:

• Planning_for_Project/index-katello.html
• Planning_for_Project/index-orcharhino.html
• Planning_for_Project/index-satellite.html

show diff

show diff as HTML

[asteflova]

This is just a first draft so things might still change a lot. But I wanted to open it up for early feedback in case anyone notices anything particularly concerning.

[asteflova]

Hi @ehelms, can you please review? This is just a prettified version of the draft you shared but I did fill in a few of my own assumptions that need checking. If this gets merged, I would like to follow up with more detailed instructions for each of the scenarios, see the PR's description.

[maximiliankolb]

two minor suggestions, rest LGTM style-wise.

the overall style is quite different to our documentation but I like it!

[asteflova]

Thanks for the comments @ehelms, I applied your feedback. Can you please re-review and tell me if you have any more?

[ehelms]

I think it's looking good, as the original author of the content, I'd like to get a review from @evgeni.
One area of information I think would be useful is a bit of a pros/cons of each option. From conversations I've had with users, the two key areas of discussion have been what are the methods and what are the trade-offs of each method in order to help them choose.

[asteflova]

One area of information I think would be useful is a bit of a pros/cons of each option. From conversations I've had with users, the two key areas of discussion have been what are the methods and what are the trade-offs of each method in order to help them choose.

It would indeed be great if we could add pros/cons. "What is the expected impact?" was my take on the cons. One of my earliest drafts also included "When should I choose this option?", which would correspond to pros, but I didn't know what to fill in.

[jtruestedt]

Maybe to add some notes from my experience with different customers:

• If you separate at least your pulp-content (/var/lib/pulp) to be on external storage and your backupmechanism is not fully in sync, it is important that the backup of your database is not newer than the pulp-content (if there are packages missing, they usually cannot be recovered, while new packages already synced to the pulp content but lost in the database are no issue).

• In the active passive setup with restore you do not necessarily need to have an existing passive setup but you can create that - in this case it depends how quickly you need to recover. But the restore works as long as you have the same Foreman version and this can be a blank one

• In the active active scenario you, you can also work with content import/export - this has also the advantage that you do not need a direct network connection between both instances

• You can also have a scenario between the two described active - passive instances - you can have the pulp content on network storage (with backups) and create backups using foreman-maintain without pulp-content on your main instance (this is mainly a database dump and needs usually less then 10 min) in case of restoring you just need to make sure that /var/lib/pulp is mounted before you restore and then the whole backup and restore mechanism is much faster as the pulp-content does not need to be extracted

• And the most important one: if you plan this kind of disaster recovery also make sure you test it, that you know what to do if you need it and that you have everything included in your backup - but maybe this should be obvious even if experience shows something different ;)

[asteflova]

I implemented part of the feedback, some of it only as comments.

As stated in the PR's description, I intend to follow up on this PR with others that will provide actual DR procedures with actionable steps for users to take. The lines that are commented out seem like a better fit for such procedures so I want to add them only as internal notes for now.

[asteflova]

Maybe to add some notes from my experience with different customers:

This is great feedback, thank you!

* If you separate at least your pulp-content (/var/lib/pulp) to be on external storage and your backupmechanism is not fully in sync, it is important that the backup of your database is not newer than the pulp-content (if there are packages missing, they usually cannot be recovered, while new packages already synced to the pulp content but lost in the database are no issue).

* In the active passive setup with restore you do not necessarily need to have an existing passive setup but you can create that - in this case it depends how quickly you need to recover. But the restore works as long as you have the same Foreman version and this can be a blank one

* In the active active scenario you, you can also work with content import/export - this has also the advantage that you do not need a direct network connection between both instances

* You can also have a scenario between the two described active - passive instances - you can have the pulp content on network storage (with backups) and create backups using foreman-maintain without pulp-content on your main instance (this is mainly a database dump and needs usually less then 10 min) in case of restoring you just need to make sure that /var/lib/pulp is mounted before you restore and then the whole backup and restore mechanism is much faster as the pulp-content does not need to be extracted

I think all of this will be most useful to document in a subsequent PR where I intend to describe the actual steps/procedures users need to take when implementing their preferred DR scenario. Perhaps as best practices/recommendations? So I won't work this in just yet and will wait until I start working on the procedures.

* And the most important one: if you plan this kind of disaster recovery also make sure you test it, that you know what to do if you need it and that you have everything included in your backup - but maybe this should be obvious even if experience shows something different ;)

Yes!! :) I would very much like to include some testing steps -- not in this PR but in those subsequent PRs I mentioned (see also this PR's description for hints on what I'm planning next). Some sort of verification procedure or a checklist that will help users verify that they are well prepared is certainly a good idea.

[asteflova]

I resolved all comments and hopefully implemented all your feedback @ehelms and @evgeni. Can you please re-review?

Note that some lines are now commented out. I intend to use them in follow-up PRs that will introduce DR procedures. I'd like to keep this PR limited strictly to an overview (= brief descriptions and pros/cons).

I also added sections for Advantages of each DR scenario. They are currently empty because I don't know what these pros/advantages are. Can you please help with that?

[ehelms]

I just have the one comment around virtualization, I think this looks like a good step and intro to disaster recovery especially since I know the plan is the follow up with more details for each scenario.

[asteflova]

I removed the "Advantages" placeholders because we currently don't have any information to provide there.

[asteflova]

Hi @maximiliankolb, you gave this an ack a while ago but some things have changed since then. Do you want to re-review?

[Lennonka]

Suggested change

	Virtualizing your {ProjectServer}::
	.Virtualizing your {ProjectServer}

I would personally prefer informal headings for the plan names but it isn't blocking your PR.