Backup OpenSearchCluster - How to?

[mame-leadec]

Hello everyone,

i want to backup a OpenSearch cluster running on K8s. Iam running Velero on a Ceph Cluster with Ceph RBD CSI Driver.

some requirements for backup:

• i want to use the snapshotting functionality of OpenSearch (for data consistency)
  • unfortunately OpenSearch doesnt seem to support additional PVC mounts for snapshotting
  • OpenSearch snapshots can be backed up to S3 compatible storage

So the idea was to use a S3 compatible backup POD (e.g. MinIO).

the OpenSearchBackup Hook would look like this:

• spin up MinIO POD
• trigger the OpenSearch snapshot
• velero backup of minio POD (with data movement from PVC)
• spin down MinIO POD
• velero backup of OpenSearch pods (without data movement from PVCs)

Of course, the restore hook would have to do this in reverse order.

A few questions related to this:

1. Is it possible to spin up a POD on demand as describe above? (both for backup and restore)
2. Will the Opensearch pod restore operation, restore also "empty" PVCs? (as there was no data movement)
3. This overall approach is somewhat clumsy due to the limitations mentioned above. Is there a better solution?
4. Ideas, questions, comments?

Thanks a lot!

[blackpiglet]

• You may need to create your plugins to do that: https://velero.io/docs/v1.14/overview-plugins/
• It's possible. Both snapshot and filesystem backup/restore allow that.
• I am unfamiliar with how the OpenSearch works, but from the OpenSearch Helm chart, I found the main workload is a StatefulSet and it has VolumeClaimTemplate. If OpenSearch stores the data in the VolumeClaimTemplate-created PVs, backing up the data in the PVs is a better solution. Since your environment, the CSI driver is available, I suggest considering the CSI snapshot data mover https://velero.io/docs/v1.14/csi-snapshot-data-movement/.

[mame-leadec]

Thanks for your answer.

If OpenSearch stores the data in the VolumeClaimTemplate-created PVs, backing up the data in the PVs is a better solution. Since your environment, the CSI driver is available, I suggest considering the CSI snapshot data mover https://velero.io/docs/v1.14/csi-snapshot-data-movement/.

Yes, OpenSearch stores the data in the VolumeClaimTemplate-created PVs and i could use the data mover / CSI snapshots to back it up. But iam concerned with data integrity. Thas why i wanted to use the OpenSearch snapshots in first place and later the CSI snapshot data mover. But im not sure if this overhead is neccessary.

[mame-leadec]

You may need to create your plugins to do that: https://velero.io/docs/v1.14/overview-plugins/

I dread the effort of writing my own plugin. Is this the only way to dynamically start/stop a POD?

[sseago]

@mame-leadec You could probably start/stop a pod in a backup or restore hook, but if you want to add newly-created resources to the backup you'd probably need a plugin. If any new resources created during the hook are not required to be explicitly added to the backup, the hook may do what you need.

[Lyndon-Li]

Can you explain more about this statement --- unfortunately OpenSearch doesnt seem to support additional PVC mounts for snapshotting?
Since you are using ceph RBD as the storage, the CSI snapshot taken should not be a problem. Do you mean you cannot create a volume(PVC) from the snapshot and attach to OpenSearch cluster? If so, how would the cluster support CSI snapshot restore?

[mame-leadec]

Thanks for your answer and sorry for the confusion:

unfortunately OpenSearch doesnt seem to support additional PVC mounts for snapshotting

What i meant here is the OpenSearch functionality to create snapshots of the current DB state. Its also called "snapshots"/"snapshotting" but its completly unrelated to CSI snapshots. Sorry for the confusion. OpenSearch Snapshots

What i meant with "additional PVC mounts":

The Opensearch snapshots support 2 types of repositories (storage locations for backups)

• a shared filesystem like NFS, which is accessable from all OpenSearch nodes
• an AWS S3 compatible storage

--> but as for now it doesnt support additional volumes via PVC (as storage locations for OpenSearchs snapshots)
opensearch-project/opensearch-k8s-operator#484

[Lyndon-Li]

So you have two directions:

1. Use OpenSearch snapshot functionality to export the data to a NFS location and use Velero to backup the NFS location
2. Use Velero CSI data movement against the ceph volume, create a CSI snapshot and backup the snapshot data, through which you can achieve a crash consistent backup

I recommend 2; for 1, you have more work to do to create a pod and mount the NFS location so that Velero could back it up.

[mame-leadec]

I wanted to use a variation of 1.):

Use OpenSearch snapshot functionality to export the data to an S3 bucket location and use Velero to backup the S3 location.

The reason why i wanted to it this way is because iam worried about the data consistency while backing up a live system. But im not sure if this will be overkill?

[Lyndon-Li]

For 2, CSI snapshot data movement goes from the CSI snapshot instead of the live file system, so it is crash consistent.

[mame-leadec]

So, as i understand you correctly, for 1 it would just introduce unnecessary complexity without adding value?