Fix watch event on case of delete

[mhmxs]

This change targets two small but significant issues. I discovered this by running wide variety of Kubernetes e2e tests and dag into cache how it works.

• Kubernetes doesn't like nil Kv or PreKv, except for deletion where it is mandatory to be nil
• In case of deletion it is looking for PrevKv, but log structured sends Kv. I know it is possible to fix this at logstructured.go but in this way Kine should guarantee the right behavior without depending on plugin correctness.

WDYT?

[brandond]

Can you open an issue describing what specific issues you saw and how to reproduce them?

[mhmxs]

Can you open an issue describing what specific issues you saw and how to reproduce them?

@brandond sorry my memories are ... so the problem is not in logstructured.go.

I created a driver implementation based on

kine/pkg/drivers/nats/backend.go

Line 52 in edd8f35

type Backend struct {

kubernetes e2e tests failed, and after some debugging, i found this:
https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/storage/etcd3/watcher.go#L693

So current object is not necessary, old object is mandatory, otherwise it fails to detect event type here:
https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/storage/etcd3/watcher.go#L626

Two things came into my mind:

• save the planet by not sending current object like nats/backend.go and possibly logstructured.go do

  • kine/pkg/drivers/nats/backend.go

    Line 409 in edd8f35

    KV: nd.KV,

  • kine/pkg/logstructured/logstructured.go

    Line 156 in edd8f35

    KV: event.KV,

• ensure old object is always filled to fix/workaround nats/backend.go and every other future plugins working incorrectly. Nats sends only if there was previous version:

  kine/pkg/drivers/nats/backend.go

  Line 416 in edd8f35

  event.PrevKV = pnd.KV