Kubelet spams NodeUnstageVolume calls

[yurkov-anton]

Describe the bug
If deleted volume by force, without success NodeUnstageVolume, then log contain follow spam:

Dec 13 23:17:28.354 [INFO] [CRHelper] [GetVolumeByID] [pvc-6211b936-656c-494a-be7a-28c9846c4108] Volume CR not found
Dec 13 23:17:28.354 [ERRO] [CSINodeService] [NodeUnstageVolume] [pvc-6211b936-656c-494a-be7a-28c9846c4108] Unable to find volume with ID pvc-6211b936-656c-494a-be7a-28c9846c4108
Dec 13 23:17:28.376 [INFO] [CSINodeService] [NodeUnstageVolume] [pvc-ce222eeb-3fc4-4241-8baa-379b2c9fce52] locking volume on request: volume_id:"pvc-ce222eeb-3fc4-4241-8baa-379b2c9fce52" staging_target_path:"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-ce222eeb-3fc4-4241-8baa-379b2c9fce52/globalmount"
Dec 13 23:17:28.387 [INFO] [CSINodeService] [NodeUnstageVolume] [pvc-e99a3945-b77b-4cb5-bb23-6607227dc497] locking volume on request: volume_id:"pvc-e99a3945-b77b-4cb5-bb23-6607227dc497" staging_target_path:"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e99a3945-b77b-4cb5-bb23-6607227dc497/globalmount"
Dec 13 23:17:28.390 [INFO] [CSINodeService] [NodeUnstageVolume] [pvc-5968be9f-9eb8-416c-9fa3-837a125684da] locking volume on request: volume_id:"pvc-5968be9f-9eb8-416c-9fa3-837a125684da" staging_target_path:"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-5968be9f-9eb8-416c-9fa3-837a125684da/globalmount"
Dec 13 23:17:28.390 [INFO] [CSINodeService] [NodeUnstageVolume] [pvc-5968be9f-9eb8-416c-9fa3-837a125684da] Processing request
Dec 13 23:17:28.404 [INFO] [CRHelper] [GetVolumeByID] [pvc-91c12620-baf4-4c99-81ad-ea26fe2fc770] Volume CR not found
Dec 13 23:17:28.404 [ERRO] [CSINodeService] [NodeUnstageVolume] [pvc-91c12620-baf4-4c99-81ad-ea26fe2fc770] Unable to find volume with ID pvc-91c12620-baf4-4c99-81ad-ea26fe2fc770
Dec 13 23:17:28.453 [INFO] [CRHelper] [GetVolumeByID] [pvc-16bc8535-b667-4a58-9b3f-8cf74a22e6a6] Volume CR not found
Dec 13 23:17:28.453 [ERRO] [CSINodeService] [NodeUnstageVolume] [pvc-16bc8535-b667-4a58-9b3f-8cf74a22e6a6] Unable to find volume with ID pvc-16bc8535-b667-4a58-9b3f-8cf74a22e6a6
Dec 13 23:17:28.453 [INFO] [CSINodeService] [NodeUnstageVolume] [pvc-18b02415-6640-4aad-8605-f9ea92e7487f] Processing request
Dec 13 23:17:28.504 [INFO] [CRHelper] [GetVolumeByID] [pvc-78aedd46-6bc1-4a1b-a4c7-62c254d3597e] Volume CR not found
Dec 13 23:17:28.504 [ERRO] [CSINodeService] [NodeUnstageVolume] [pvc-78aedd46-6bc1-4a1b-a4c7-62c254d3597e] Unable to find volume with ID pvc-78aedd46-6bc1-4a1b-a4c7-62c254d3597e
Dec 13 23:17:28.504 [INFO] [CSINodeService] [NodeUnstageVolume] [pvc-4ce6408d-b637-4c04-af99-9f6ede767e74] Processing request
Dec 13 23:17:28.549 [INFO] [CSINodeService] [NodeUnstageVolume] [pvc-e8529962-4836-4874-a9bd-289fad2311c3] locking volume on request: volume_id:"pvc-e8529962-4836-4874-a9bd-289fad2311c3" staging_target_path:"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e8529962-4836-4874-a9bd-289fad2311c3/globalmount"

This spam leads to an increase in the size of the csi and kubelet logs. (10mb for 10 minutes for ~40 error volumes)

If volume was removed without successful NodeUnstageVolume, kubelet will try to call NodeUnstageVolume, and CSI return NOT_FOUND.

Environment (please complete the following information):

• Version: 1.1.0

To Reproduce

1. Create volume.
2. Delete volume by patch.
3. Delete sts, pod and pv

Expected behavior
kubelet log and csi node log will contain spam with NodeUnstageVolume calls.

Additional context
Possible hotfix spam is restart kubelet.

[yurkov-anton]

Need to cherry-pick to release-1.1 branch