Update backup_restore.md #321
Conversation
| @@ -52,9 +52,13 @@ rke2 server \ | |||
| systemctl start rke2-server | |||
| ``` | |||
|
|
|||
| 4. Remove the rke2 db directory on the other server nodes as follows: | |||
| 4. Move the rke2 db directory on the other server nodes as follows (you want to keep a copy to avoid ending up with only an old or corrupt backup to chose for): | |||
There was a problem hiding this comment.
Having the old DB dir around on the secondary servers doesn't really help with anything. If you run into problems, restoring a snapshot is a better resolution than moving an old db dir back into place.
There was a problem hiding this comment.
The issue is that we currently run rm -rf /var/lib/rancher/rke2/server/db/, which deletes both the etcd data and the snapshots directory. This means we erase the live data along with its backups.
We've encountered cases where customers, not paying close attention, have accidentally executed this command on all three master/etcd nodes, leading to complete data loss.
This change ensures that snapshots are not deleted until the cluster has been fully restored, allowing customers to perform the cleanup on their own afterward.
Can't count how many times we've seen removing backup etcd dbs put a customer in a bad spot.