PV stuck in terminating state

[GunaKKIBM]

/kind bug

What happened?
Deleting a pv that is not in use - though the volume is actually deleted, the pv status is in terminating state and says volume not found when described. From the powervs-plugin container logs, it is seen that volume is deleted

% kubectl get pv                 
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS        CLAIM                   STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
pvc-4734800e-0af4-479d-ba5f-ff28aa45599f   4Gi        RWO            Delete           Terminating   default/powervs-claim   powervs-sc     <unset>                          99m
pvc-d1b41101-aad5-49ad-bb17-4e7287f4967b   4Gi        RWO            Delete           Terminating   default/powervs-claim   powervs-sc     <unset>                          3h15m

{"description":"deletion of volumeID '<volumeID>' is unknown or not allowed","error":"Bad Request: Volume not found"

I0924 08:17:21.902769       1 controller.go:215] DeleteVolume: called with args: {VolumeId:4080a577-6ece-4d8c-91b8-6d9eba7fc548 Secrets:map[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
GET /pcloud/v1/cloud-instances/08d3459f-41ca-4831-ad33-ce63adee8d1b/volumes/4080a577-6ece-4d8c-91b8-6d9eba7fc548 HTTP/1.1
Host: wdc.power-iaas.cloud.ibm.com
User-Agent: Go-http-client/1.1
Accept: application/json
Authorization: [PRIVATE DATA HIDDEN]
Crn: <crn>
Accept-Encoding: gzip


HTTP/2.0 200 OK

What you expected to happen?
PV shouldn't be listed

How to reproduce it (as minimally and precisely as possible)?
Follow dynamic provisioning in the in READme. And then delete the pod and pvc. PV will be seen in terminating state.

Anything else we need to know?:
Looks like, retry should not be done, once 200 OK response is received, and the same response should be published to the pv.

Environment

• Kubernetes version (use kubectl version):
• Driver version:

[irapandey]

/assign

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[irapandey]

@GunaKKIBM - were you able to replicate the issue?