Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Backup persistence volume is not working in minikube with volumesnapshots and csi-hostpath-driver addons installed #8565

Open
masalinas opened this issue Dec 30, 2024 · 3 comments
Open

Backup persistence volume is not working in minikube with volumesnapshots and csi-hostpath-driver addons installed #8565

masalinas opened this issue Dec 30, 2024 · 3 comments
Assignees
@ywk253100
Labels
Area/CSI Related to Container Storage Interface support

Comments

@masalinas
Copy link

masalinas commented Dec 30, 2024 •
edited
Loading

What steps did you take and what happened:

  1. Install addons minikube:
minikube addons enable volumesnapshots
minikube addons enable csi-hostpath-driver

The addons are running ok

  1. Deploy chart mongodb using the storage class csi-hostpath-sc registered with the csi-hostpath-driver addon like this:
helm install avib-mongodb --set persistence.storageClass="csi-hostpath-sc" oci://registry-1.docker.io/bitnamicharts/mongodb

MongoDB is running ok with the PVC with storageClass with this value csi-hostpath-sc

  1. Deploy velero and minio like this:
    The minio tenant is deploying using the mini operator ok.

The velero is deployed like this:

velero install \
    --provider aws \
    --features=EnableCSI \
    --plugins velero/velero-plugin-for-aws:v1.11.0 \
    --bucket velero \
    --secret-file ./credentials-velero \
    --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=https://avib-minio-hl.default.svc:9000,insecureSkipTLSVerify="true"

velero is running ok

  1. add label csi-hostpath-snapclass velero.io/csi-volumesnapshot-class to my unique VolumeSnapshotClass called csi-hostpath-snapclass using the driver csi-hostpath-driver
kubectl label VolumeSnapshotClass csi-hostpath-snapclass velero.io/csi-volumesnapshot-class="true"

  1. Create some documents in mongo to backup

  2. Create a backup like this:

velero backup create avib-mongodb --or-selector "app.kubernetes.io/name=mongodb or name=avib-mongodb"
  1. Uninstall mongodb to be restored
helm delete avib-mongodb
velero restore create --from-backup avib-mongodb

The restore fail. All resources are created but the PV not.

Where is the issue??**

What did you expect to happen:
Backup the kubernetes resources and restore all resources and PV.

The following information will help us better understand what's going on:

Some velero-avib-mongodb-spmdc logs inside the backup and restore logs related to the VolumeSnapshot, but I don't see anything wrong to be honest.

If I deploy the snapshot by me it works ok, but the snapshot created by velero not:

// volumesnapshot-avib-mongodb.yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: snapshot-avib-mongodb
spec:
  volumeSnapshotClassName: csi-hostpath-snapclass
  source:
    persistentVolumeClaimName: avib-mongodb

kubectl apply -f volumesnapshot-avib-mongodb.yaml

list the snapshots. You see the velero-avib-mongodb-spmdc snapshot created by velero failed:

kubectl get volumesnapshot
NAME                        READYTOUSE   SOURCEPVC      SOURCESNAPSHOTCONTENT             RESTORESIZE   SNAPSHOTCLASS            SNAPSHOTCONTENT                                    CREATIONTIME   AGE
snapshot-avib-mongodb       true         avib-mongodb                                     8Gi           csi-hostpath-snapclass   snapcontent-fd5fc291-82b8-4f30-830a-f96d5f4d66ee   15s            15s
velero-avib-mongodb-spmdc   false                       velero-avib-mongodb-spmdc-v6wj7                 csi-hostpath-snapclass   velero-avib-mongodb-spmdc-v6wj7

Environment:

  • Helm version: v.3.16.4
  • Velero version (use velero version): 1.15.0
  • Velero features (use velero client config get features): Plugin AWS 1.11.0 compatible with velero 1.15.0
  • Kubernetes version (use kubectl version): 1.27.4
  • Kubernetes installer & version: minikube 1.31.2 with volumesnapshots and csi-hostpath-driver installed. Use VirtualBox driver (7.0)
  • Cloud provider or hardware configuration: localhost
  • OS (e.g. from /etc/os-release): ubuntu 24.04 LTS

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

  • 👍 for "I would like to see this bug fixed as soon as possible"
  • 👎 for "There are more important bugs to focus on right now"
@masalinas masalinas changed the title Backup persistence volumne not works in minikube Backup persistence volume is not works in minikube with volumesnapshots and csi-hostpath-driver addons installed Dec 30, 2024
@masalinas masalinas changed the title Backup persistence volume is not works in minikube with volumesnapshots and csi-hostpath-driver addons installed Backup persistence volume is not working in minikube with volumesnapshots and csi-hostpath-driver addons installed Dec 30, 2024
@ywk253100
Copy link
Contributor

ywk253100 commented Dec 31, 2024 •
edited
Loading

The VolumeSnapshotSource of VolumeSnapshot created by Velero when enabling the data movement is VolumeSnapshotContent, you can see the difference from the output of your command:

kubectl get volumesnapshot
NAME                        READYTOUSE   SOURCEPVC      SOURCESNAPSHOTCONTENT             RESTORESIZE   SNAPSHOTCLASS            SNAPSHOTCONTENT                                    CREATIONTIME   AGE
snapshot-avib-mongodb       true         avib-mongodb                                     8Gi           csi-hostpath-snapclass   snapcontent-fd5fc291-82b8-4f30-830a-f96d5f4d66ee   15s            15s
velero-avib-mongodb-spmdc   false                       velero-avib-mongodb-spmdc-v6wj7                 csi-hostpath-snapclass   velero-avib-mongodb-spmdc-v6wj7

I'm not sure whether host path CSI driver support that or not, could you double check whether host path CSI driver supporting creating the VolumeSnapshot from a VolumeSnapshotContent?

@masalinas
Copy link
Author

masalinas commented Jan 2, 2025 •
edited
Loading

The problem is this: when create the backup, velero not create the volume snaphot from backup when execute the backup command like this:

velero backup create avib-mongodb --or-selector "app.kubernetes.io/name=mongodb or name=avib-mongodb"

Executing this command any volume snapshot exist:

kubectl get volumesnapshot

So later, when try to restore the backup, velero restore the kubernetes resources ok, but when try to restore the persistence volume from the volume snapshot content called velero-avib-mongodb-spmdc-v6wj7 (where is this volume snapshot content?, velero not create any volume snapshot as I explain), this not exist because never was created from velero in the backup process.

My CSI driver called csi-hostpath-driver support create volume snapshots because you see that executing this volume snapshot from kubectl works ok, but from velero not. I followed your documentation https://velero.io/docs/v1.15/csi/ configuring my VolumeSnapshotClass called csi-hostpath-sc as docs says with the label velero.io/csi-volumesnapshot-class: "true".

These are backup logs related to the volume snapshot called default/velero-avib-mongodb-4j96s never created. Do you watch some special?

time="2025-01-02T11:50:30Z" level=info msg="Created VolumeSnapshot default/velero-avib-mongodb-4j96s" backup=velero/avib-mongodb cmd=/velero logSource="pkg/backup/actions/csi/pvc_action.go:187" pluginName=velero
time="2025-01-02T11:50:30Z" level=info msg="Returning from PVCBackupItemAction with 1 additionalItems to backup" backup=velero/avib-mongodb cmd=/velero logSource="pkg/backup/actions/csi/pvc_action.go:354" pluginName=velero
time="2025-01-02T11:50:30Z" level=info msg="Untrack the PVC default/avib-mongodb, because it's backed up by CSI BIA." backup=velero/avib-mongodb logSource="pkg/backup/item_backupper.go:407" name=avib-mongodb namespace=default resource=persistentvolumeclaims
time="2025-01-02T11:50:30Z" level=info msg="Additional Item volumesnapshots.snapshot.storage.k8s.io default/velero-avib-mongodb-4j96s not found in ItemBlock, getting from cluster" backup=velero/avib-mongodb logSource="pkg/backup/item_backupper.go:458" name=avib-mongodb namespace=default resource=persistentvolumeclaims
time="2025-01-02T11:50:30Z" level=info msg="Skipping the exclusion checks for this resource" backup=velero/avib-mongodb logSource="pkg/backup/item_backupper.go:112" name=velero-avib-mongodb-4j96s namespace=default resource=volumesnapshots.snapshot.storage.k8s.io
time="2025-01-02T11:50:30Z" level=info msg="Backing up item" backup=velero/avib-mongodb logSource="pkg/backup/item_backupper.go:184" name=velero-avib-mongodb-4j96s namespace=default resource=volumesnapshots.snapshot.storage.k8s.io
time="2025-01-02T11:50:30Z" level=info msg="Executing custom action" backup=velero/avib-mongodb logSource="pkg/backup/item_backupper.go:357" name=velero-avib-mongodb-4j96s namespace=default resource=volumesnapshots.snapshot.storage.k8s.io
time="2025-01-02T11:50:30Z" level=info msg="Executing VolumeSnapshotBackupItemAction" backup=velero/avib-mongodb cmd=/velero logSource="pkg/backup/actions/csi/volumesnapshot_action.go:79" pluginName=velero
time="2025-01-02T11:50:30Z" level=info msg="Getting VolumesnapshotContent for Volumesnapshot default/velero-avib-mongodb-4j96s" backup=velero/avib-mongodb cmd=/velero logSource="pkg/backup/actions/csi/volumesnapshot_action.go:115" pluginName=velero
time="2025-01-02T11:50:30Z" level=info msg="Waiting for CSI driver to reconcile volumesnapshot default/velero-avib-mongodb-4j96s. Retrying in 5s" backup=velero/avib-mongodb cmd=/velero logSource="pkg/util/csi/volume_snapshot.go:712" pluginName=velero
time="2025-01-02T11:50:35Z" level=info msg="Patching VolumeSnapshotContent snapcontent-0e130691-ec32-4da3-8af2-0945da17ae37 with velero BackupNameLabel" backup=velero/avib-mongodb cmd=/velero logSource="pkg/backup/actions/csi/volumesnapshot_action.go:174" pluginName=velero
time="2025-01-02T11:50:35Z" level=info msg="Returning from VolumeSnapshotBackupItemAction with 2 additionalItems to backup" backup=velero/avib-mongodb cmd=/velero logSource="pkg/backup/actions/csi/volumesnapshot_action.go:236" pluginName=velero
time="2025-01-02T11:50:35Z" level=info msg="Additional Item volumesnapshotclasses.snapshot.storage.k8s.io /csi-hostpath-snapclass not found in ItemBlock, getting from cluster" backup=velero/avib-mongodb logSource="pkg/backup/item_backupper.go:458" name=velero-avib-mongodb-4j96s namespace=default resource=volumesnapshots.snapshot.storage.k8s.io
time="2025-01-02T11:50:35Z" level=info msg="Skipping the exclusion checks for this resource" backup=velero/avib-mongodb logSource="pkg/backup/item_backupper.go:112" name=csi-hostpath-snapclass namespace= resource=volumesnapshotclasses.snapshot.storage.k8s.io
time="2025-01-02T11:50:35Z" level=info msg="Backing up item" backup=velero/avib-mongodb logSource="pkg/backup/item_backupper.go:184" name=csi-hostpath-snapclass namespace= resource=volumesnapshotclasses.snapshot.storage.k8s.io
time="2025-01-02T11:50:35Z" level=info msg="Executing custom action" backup=velero/avib-mongodb logSource="pkg/backup/item_backupper.go:357" name=csi-hostpath-snapclass namespace= resource=volumesnapshotclasses.snapshot.storage.k8s.io
time="2025-01-02T11:50:35Z" level=info msg="Executing VolumeSnapshotClassBackupItemAction" backup=velero/avib-mongodb cmd=/velero logSource="pkg/backup/actions/csi/volumesnapshotclass_action.go:64" pluginName=velero
time="2025-01-02T11:50:35Z" level=info msg="Returning from VolumeSnapshotClassBackupItemAction with 0 additionalItems to backup" backup=velero/avib-mongodb cmd=/velero logSource="pkg/backup/actions/csi/volumesnapshotclass_action.go:92" pluginName=velero
time="2025-01-02T11:50:35Z" level=info msg="Additional Item volumesnapshotcontents.snapshot.storage.k8s.io /snapcontent-0e130691-ec32-4da3-8af2-0945da17ae37 not found in ItemBlock, getting from cluster" backup=velero/avib-mongodb logSource="pkg/backup/item_backupper.go:458" name=velero-avib-mongodb-4j96s namespace=default resource=volumesnapshots.snapshot.storage.k8s.io
time="2025-01-02T11:50:35Z" level=info msg="Skipping the exclusion checks for this resource" backup=velero/avib-mongodb logSource="pkg/backup/item_backupper.go:112" name=snapcontent-0e130691-ec32-4da3-8af2-0945da17ae37 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io

So what is the problem?, why from kubectl I can create volume snaphots and from velero is not possible using the same storage class?, is a bad configuration or is a bug?

@ywk253100
Copy link
Contributor

Is the backup completed successfully?
Could you run velero debug --backup <backup-name> --restore <restore-name> command to collect the debug bundle which contains more information?
Also run velero backup describe <backup-name> --details and show us the output.

@reasonerjt reasonerjt added the Area/CSI Related to Container Storage Interface support label Jan 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ywk253100 ywk253100

Labels
Area/CSI Related to Container Storage Interface support
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@masalinas @reasonerjt @ywk253100 and others