Restore PartiallyFailed #3266

kfirfer · 2020-12-12T11:36:23Z

kfirfer
Dec 12, 2020

Hello,

I Couldn't restore a backup, What am I missing?

kubernetes version
1.18.9

velero version

Client:
        Version: v1.5.2
        Git commit: e115e5a191b1fdb5d379b62a35916115e77124a4
Server:
        Version: v1.5.2

velero get backups

NAME                                         STATUS            ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
velero-cluster-backup-daily-20201212024509   Completed         0        0          2020-12-12 04:45:09 +0200 IST   4d        default            <none>

velero get restores

NAME                                                        BACKUP                                       STATUS            STARTED                         COMPLETED                       ERRORS   WARNINGS   CREATED                         SELECTOR
velero-cluster-backup-daily-20201212024509-20201212132326   velero-cluster-backup-daily-20201212024509   PartiallyFailed   2020-12-12 13:23:26 +0200 IST   2020-12-12 13:25:16 +0200 IST   5        0          2020-12-12 13:23:26 +0200 IST   <none>

velero describe restore velero-cluster-backup-daily-20201212024509-20201212132326

Name:         velero-cluster-backup-daily-20201212024509-20201212132326
Namespace:    velero
Labels:       <none>
Annotations:  <none>

Phase:  PartiallyFailed (run 'velero restore logs velero-cluster-backup-daily-20201212024509-20201212132326' for more information)

Started:    2020-12-12 13:23:26 +0200 IST
Completed:  2020-12-12 13:25:16 +0200 IST

Errors:
  Velero:  pod volume restore failed: error restoring volume: error running restic restore, cmd=restic restore --repo=gs:zigi-backups:/restic/harbor --password-file=/tmp/velero-restic-credentials-harbor886982018 --cache-dir=/scratch/.cache/restic 80ab39a7 --target=., stdout=, stderr=: error getting snapshot size: error running command, stderr=Fatal: error loading snapshot: no matching ID found
: exit status 1
    pod volume restore failed: error restoring volume: error running restic restore, cmd=restic restore --repo=gs:zigi-backups:/restic/harbor --password-file=/tmp/velero-restic-credentials-harbor787387056 --cache-dir=/scratch/.cache/restic 8a3a9963 --target=., stdout=, stderr=: error getting snapshot size: error running command, stderr=Fatal: error loading snapshot: no matching ID found
: exit status 1
    pod volume restore failed: error restoring volume: error running restic restore, cmd=restic restore --repo=gs:zigi-backups:/restic/harbor --password-file=/tmp/velero-restic-credentials-harbor183472473 --cache-dir=/scratch/.cache/restic 7fafb274 --target=., stdout=, stderr=: error getting snapshot size: error running command, stderr=Fatal: error loading snapshot: no matching ID found
: exit status 1
    pod volume restore failed: error restoring volume: error running restic restore, cmd=restic restore --repo=gs:zigi-backups:/restic/harbor --password-file=/tmp/velero-restic-credentials-harbor025557042 --cache-dir=/scratch/.cache/restic 47fa80b7 --target=., stdout=, stderr=: error getting snapshot size: error running command, stderr=Fatal: error loading snapshot: no matching ID found
: exit status 1
    pod volume restore failed: error restoring volume: error running restic restore, cmd=restic restore --repo=gs:zigi-backups:/restic/harbor --password-file=/tmp/velero-restic-credentials-harbor741207529 --cache-dir=/scratch/.cache/restic bfc81fbf --target=., stdout=, stderr=: error getting snapshot size: error running command, stderr=Fatal: error loading snapshot: no matching ID found
: exit status 1
  Cluster:    <none>
  Namespaces: <none>

Backup:  velero-cluster-backup-daily-20201212024509

Namespaces:
  Included:  harbor
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io
  Cluster-scoped:  auto

Namespace mappings:  <none>

Label selector:  <none>

Restore PVs:  auto

Restic Restores (specify --details for more information):
  Completed:  7
  Failed:     5

velero restore logs velero-cluster-backup-daily-20201212024509-20201212132326

{"level":"info","logSource":"pkg/restore/restore.go:1107","msg":"Attempting to restore StatefulSet: minio","restore":"velero/velero-cluster-backup-daily-20201212024509-20201212132326","time":"2020-12-12T11:23:47Z"}
{"level":"info","logSource":"pkg/restore/restore.go:1107","msg":"Attempting to restore StatefulSet: postgresql-ha-postgresql","restore":"velero/velero-cluster-backup-daily-20201212024509-20201212132326","time":"2020-12-12T11:23:47Z"}
{"level":"info","logSource":"pkg/restore/restore.go:1107","msg":"Attempting to restore StatefulSet: redis-ha-harbor-server","restore":"velero/velero-cluster-backup-daily-20201212024509-20201212132326","time":"2020-12-12T11:23:47Z"}
{"level":"info","logSource":"pkg/restore/restore.go:437","msg":"Skipping namespace jenkins","restore":"velero/velero-cluster-backup-daily-20201212024509-20201212132326","time":"2020-12-12T11:23:47Z"}
{"level":"info","logSource":"pkg/restore/restore.go:437","msg":"Skipping namespace keycloak","restore":"velero/velero-cluster-backup-daily-20201212024509-20201212132326","time":"2020-12-12T11:23:47Z"}
{"level":"info","logSource":"pkg/restore/restore.go:437","msg":"Skipping namespace nextcloud","restore":"velero/velero-cluster-backup-daily-20201212024509-20201212132326","time":"2020-12-12T11:23:47Z"}
{"level":"info","logSource":"pkg/restore/restore.go:437","msg":"Skipping namespace statping","restore":"velero/velero-cluster-backup-daily-20201212024509-20201212132326","time":"2020-12-12T11:23:47Z"}
{"level":"info","logSource":"pkg/restore/restore.go:437","msg":"Skipping namespace zigi-dev","restore":"velero/velero-cluster-backup-daily-20201212024509-20201212132326","time":"2020-12-12T11:23:47Z"}
{"level":"info","logSource":"pkg/restore/restore.go:437","msg":"Skipping namespace apps","restore":"velero/velero-cluster-backup-daily-20201212024509-20201212132326","time":"2020-12-12T11:23:47Z"}
{"level":"info","logSource":"pkg/restore/restore.go:719","msg":"Skipping resource validatingwebhookconfigurations.admissionregistration.k8s.io because it's cluster-scoped and only specific namespaces are included in the restore","restore":"velero/velero-cluster-backup-daily-20201212024509-20201212132326","time":"2020-12-12T11:23:47Z"}
{"level":"info","logSource":"pkg/restore/restore.go:719","msg":"Skipping resource volumeattachments.storage.k8s.io because it's cluster-scoped and only specific namespaces are included in the restore","restore":"velero/velero-cluster-backup-daily-20201212024509-20201212132326","time":"2020-12-12T11:23:47Z"}
{"level":"info","logSource":"pkg/restore/restore.go:437","msg":"Skipping namespace velero","restore":"velero/velero-cluster-backup-daily-20201212024509-20201212132326","time":"2020-12-12T11:23:47Z"}
{"level":"info","logSource":"pkg/restore/restore.go:488","msg":"Waiting for all restic restores to complete","restore":"velero/velero-cluster-backup-daily-20201212024509-20201212132326","time":"2020-12-12T11:23:47Z"}
{"error.message":"pod volume restore failed: error restoring volume: error running restic restore, cmd=restic restore --repo=gs:zigi-backups:/restic/harbor --password-file=/tmp/velero-restic-credentials-harbor886982018 --cache-dir=/scratch/.cache/restic 80ab39a7 --target=., stdout=, stderr=: error getting snapshot size: error running command, stderr=Fatal: error loading snapshot: no matching ID found\n: exit status 1","level":"error","logSource":"pkg/restore/restore.go:1296","msg":"unable to successfully complete restic restores of pod's volumes","restore":"velero/velero-cluster-backup-daily-20201212024509-20201212132326","time":"2020-12-12T11:23:56Z"}
{"error.message":"pod volume restore failed: error restoring volume: error running restic restore, cmd=restic restore --repo=gs:zigi-backups:/restic/harbor --password-file=/tmp/velero-restic-credentials-harbor787387056 --cache-dir=/scratch/.cache/restic 8a3a9963 --target=., stdout=, stderr=: error getting snapshot size: error running command, stderr=Fatal: error loading snapshot: no matching ID found\n: exit status 1","level":"error","logSource":"pkg/restore/restore.go:1296","msg":"unable to successfully complete restic restores of pod's volumes","restore":"velero/velero-cluster-backup-daily-20201212024509-20201212132326","time":"2020-12-12T11:23:57Z"}
{"error.message":"pod volume restore failed: error restoring volume: error running restic restore, cmd=restic restore --repo=gs:zigi-backups:/restic/harbor --password-file=/tmp/velero-restic-credentials-harbor183472473 --cache-dir=/scratch/.cache/restic 7fafb274 --target=., stdout=, stderr=: error getting snapshot size: error running command, stderr=Fatal: error loading snapshot: no matching ID found\n: exit status 1","level":"error","logSource":"pkg/restore/restore.go:1296","msg":"unable to successfully complete restic restores of pod's volumes","restore":"velero/velero-cluster-backup-daily-20201212024509-20201212132326","time":"2020-12-12T11:24:07Z"}
{"error.message":"[pod volume restore failed: error restoring volume: error running restic restore, cmd=restic restore --repo=gs:zigi-backups:/restic/harbor --password-file=/tmp/velero-restic-credentials-harbor025557042 --cache-dir=/scratch/.cache/restic 47fa80b7 --target=., stdout=, stderr=: error getting snapshot size: error running command, stderr=Fatal: error loading snapshot: no matching ID found\n: exit status 1, pod volume restore failed: error restoring volume: error running restic restore, cmd=restic restore --repo=gs:zigi-backups:/restic/harbor --password-file=/tmp/velero-restic-credentials-harbor741207529 --cache-dir=/scratch/.cache/restic bfc81fbf --target=., stdout=, stderr=: error getting snapshot size: error running command, stderr=Fatal: error loading snapshot: no matching ID found\n: exit status 1]","level":"error","logSource":"pkg/restore/restore.go:1296","msg":"unable to successfully complete restic restores of pod's volumes","restore":"velero/velero-cluster-backup-daily-20201212024509-20201212132326","time":"2020-12-12T11:24:08Z"}
{"level":"info","logSource":"pkg/restore/restore.go:504","msg":"Done waiting for all restic restores to complete","restore":"velero/velero-cluster-backup-daily-20201212024509-20201212132326","time":"2020-12-12T11:25:14Z"}
{"level":"info","logSource":"pkg/restore/restore.go:508","msg":"Waiting for all post-restore-exec hooks to complete","restore":"velero/velero-cluster-backup-daily-20201212024509-20201212132326","time":"2020-12-12T11:25:14Z"}
{"level":"info","logSource":"pkg/restore/restore.go:516","msg":"Done waiting for all post-restore exec hooks to complete","restore":"velero/velero-cluster-backup-daily-20201212024509-20201212132326","time":"2020-12-12T11:25:14Z"}
{"level":"info","logSource":"pkg/controller/restore_controller.go:482","msg":"restore completed","restore":"velero/velero-cluster-backup-daily-20201212024509-20201212132326","time":"2020-12-12T11:25:14Z"}

Answered by carlisia

Jan 25, 2021

This discussion has been moved to an issue: #3343.

View full answer

kfirfer · 2020-12-12T12:17:44Z

kfirfer
Dec 12, 2020
Author

The restoration is restore about 5 stateful application, only 1 has errors (minio)

Im not sure it is only because of Minio, But eitherway I will provide more information:

I have 3 Minio pods (clustered) that does not want to get restored properly

vostro@dev101:~/code/kfirfer/helm-charts$ kubectl get po | grep minio     
minio-0                                        0/1     Init:0/1   0          16m
minio-1                                        0/1     Init:0/1   0          16m
minio-2                                        0/1     Init:0/1   0          16m

kubectl describe pod minio-0

Volumes:
  export-0:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  export-0-minio-0
    ReadOnly:   false
  export-1:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  export-1-minio-0
    ReadOnly:   false

On each Minio pod I have this annotation:

Annotations:  backup.velero.io/backup-volumes: export-0,export-1

0 replies

kfirfer · 2020-12-12T21:54:42Z

kfirfer
Dec 12, 2020
Author

For some reason I couldnt reproduce
I upgraded kubernetes 1.18.13 and reinstalled velero

If the problem will come up again I will reopen

Closing

0 replies

kfirfer · 2020-12-28T03:21:47Z

kfirfer
Dec 28, 2020
Author

I reopening the issue, Since it is reproduces
It is reproduce also on k8s 1.20.0

I assume it is something relates to the volumes and replicas of the statefulset

In the above example I have statefulset with 3 replicas, and each one have 2 identical volume names:

First replica:
export-0-0
export-0-1

Second replica
export-0-0
export-0-1

Third replica:
export-0-0
export-0-1

0 replies

kfirfer · 2021-01-13T20:54:56Z

kfirfer
Jan 13, 2021
Author

Why this issue went to "Community support Q&A" "discussions" ? this is a bug =X

0 replies

carlisia · 2021-01-13T22:28:45Z

carlisia
Jan 13, 2021

If it's confirmed it is a bug, an issue will be opened for it.

0 replies

carlisia · 2021-01-25T21:17:58Z

carlisia
Jan 25, 2021

This discussion has been moved to an issue: #3343.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Restore PartiallyFailed #3266

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 6 comments

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

Restore PartiallyFailed #3266

kfirfer Dec 12, 2020

Replies: 6 comments

kfirfer Dec 12, 2020 Author

kfirfer Dec 12, 2020 Author

kfirfer Dec 28, 2020 Author

kfirfer Jan 13, 2021 Author

carlisia Jan 13, 2021

carlisia Jan 25, 2021

kfirfer
Dec 12, 2020

kfirfer
Dec 12, 2020
Author

kfirfer
Dec 12, 2020
Author

kfirfer
Dec 28, 2020
Author

kfirfer
Jan 13, 2021
Author

carlisia
Jan 13, 2021

carlisia
Jan 25, 2021