Ability to fill KeyValue (KV) bucket from stream

[jalaziz]

Since the KV feature is built off of streams (if I'm not mistaken), it would amazing if we could have a KV bucket automatically populated and updated from a stream.

The use case here is actually quite simple. We have a stream composed of a series of subjects that are all pushing regular updates. We have consumers that wish to receive all updates in relative real-time, while we have other consumers that simply want to be able to lookup the latest value for one or more subjects.

While we could support both using DeliverLastPerSubject and MaxMsgsPerSubject to support both uses, the KV interface can be much simpler for looking up the latest values for known subjects.

If this is something that is worthy of implementation, it would also be great if we could define a mapping between the subject and key name within the bucket.

[derekcollison]

NATS JetStream has had the ability to source and mirror other streams since inception. And you can allow direct get() operations by subject on a stream without the intervention of consumer state. We have also allowed subject mappings and transforms at the account or server level.

With a change that has landed on the dev branch, @jnmoyne has added the ability to transform subjects on ingest from another stream.

Meaning, on the dev branch, you can source messages into a KV from another stream by transforming the subjects on ingest. So I think we are already there ;)

Will let @jnmoyne expand more.

[ripienaar]

Why not just retrieve the last message per subject using the API? Duplicating all the data and coming up with solid remapping etc seems quite excessive when you could instead just get the message using the get API where it is already

[jalaziz]

I'm sure I'm missing something obvious, but which API would that be? I only see get for key-value and get_msg which seems require a sequence number for the Python client docs.

Digging deeper, on the Go client I do see GetLastMsg and I also see get_last_msg for the Python client actually implemented. Looks like it could be what we need to replicate that behavior.

The API is a bit raw and we still need to be aware of the stream name and the the full subject name to use that API, but it's not too bad.

Duplicating all the data and coming up with solid remapping etc seems quite excessive

Well, I was hoping the data wouldn't actually have to be duplicated. I wasn't aware there was a "Get" API on the streams itself. We can design a reasonable abstraction around that. If GetLastMsg does what I think it does, it should be good enough. Thanks for the pointer.

[ripienaar]

Yeah - I guess support for this varies by client - KV is just a simplification to narrow down the huge API set of JetStream into a convention driven simplification, but generally you can do all it does with the API

if you say which language you use I can see if I can find it

[jalaziz]

Thanks 🙏🏼. We primarily use Golang and Python. We're also exploring Rust.

[ripienaar]

In go there is GetLastMsg() to get the last message for a specific subject else GetMsg() for some specific sequence. Your basic KV Get is GetLastMsg().

In python you have get_msg() that supports all the various combos of per sequence, last per subject etc.

[jalaziz]

Thanks 🙏🏼 !