On-Premise streams

[turowicz]

Are there any recommendations for on premise persistent streaming? I can't seem to be able to find any alternatives to AWS / Azure queues in the official code. 3rd party libraries haven't been maintained for over a year now.

cc @ReubenBond @sergeybykov

[ReubenBond]

Which third-party libraries are you referring to? I'm not sure if there are any on-premises streams providers which I could recommend, unfortunately. Would Kafka be suitable for what you need? I wonder how production-ready this Kafka provider is: https://github.com/jonathansant/Orleans.Streams.Kafka

[turowicz]

I will give it a go. The production readiness bit is quite important. It would be very nice if you guys provided an implementation of something for onprem.

[turowicz]

I am also exploring https://github.com/berdon/Orleans.Providers.Redis

[ReubenBond]

I don't imagine we will be able to prioritize developing a new provider for streams any time soon, so it's likely going to be left up to the community, perhaps a company which is using Kafka for streams on Orleans.
It looks like @berdon is actively working on the Redis providers in that repository (last commit was an hour ago). They could potentially be merged with the providers in https://github.com/OrleansContrib/Orleans.Redis.

[berdon]

Yeah, really that's what should happen. That repo is pretty stagnant but actively used in a large product at CentralSquare (of which I'm no longer affiliated).

The last commit was just a tests fix merge which, itself, was also relatively benign.

[berdon]

By that, I mean, we/they've used it for the better part of 3 years and haven't had any major issues. It's unlikely any new issues will pop up with current versions of Redis (and it's pinned for that). It's possible new version of Orleans could break things but...that'd be relatively easy to fix if it's merged into OrleansContrib. @turowicz

[ReubenBond]

Sounds good to me. If you want commit access to OrleansContrib & don't already have it, let me know and I'll add you. Of course, a new provider should be added via a PR. Recently, @suraciii added a reminders provider for Redis.

[berdon]

@ReubenBond I don't think I do, if you want to add me. I'll see about getting a PR put up. Probably not until next week at the earliest.

[oising]

While I got you here, @berdon - have you considered berdon/Orleans.Providers.Redis#33

[turowicz]

@ReubenBond so the issue is that:

• Kafka requires large amount of system resources, that we don't have (https://docs.confluent.io/platform/current/kafka/deployment.html)
• Redis provider by @berdon queues messages up in memory which is the reason why I want to move away from SMS
  (Consider supporting Redis Streams as a true queue source berdon/Orleans.Providers.Redis#33)

It seems that until @berdon implements the Redis 5.0 (as pointed out by @oising) queues it will not be of use to us.

[turowicz]

Another one: Kafka repo is stagnant

[turowicz]

Another one: Kafka provider depends on a commercial library

[nkosi23]

@turowicz You can use RabbitMQ, it is lightweight, feature-rich and scalable. This repository should be a good starting point.

[turowicz]

@nkosi23 that repo has been silent for the last 2 years. Does it work?

[nkosi23]

This is the one we have selected for our project, I haven't tested it personally (I am working on other parts of the system) but the implementation looks quite complete, it even has extension methods to add the provider from an IClientBuilder.

As far as we are concerned this is all we needed (we just wanted to avoid starting from a blank sheet) so we haven't been too concerned about whether it works out of the box or not. You may need to get your hands dirty, but any change needed should be fairly trivial.

Keep in mind that the API of stream providers isn't changing that much so if an implementation works there isn't necessarily a need to keep updating this kind of library (unless the upstream provider - Kafka etc... - keeps changing its API).

[turowicz]

@nkosi23 do you know what is the point of memory cache being used there? Seems like when the process crashes you may lose data.

https://github.com/galamai/Galamai.Orleans.Streaming.RabbitMQ/blob/1274866f91224f8cc45dd79b535dcade0a2dc525/src/Galamai.Orleans.Streaming.RabbitMQ/Caches/ConcurrentQueueCache.cs

[nkosi23]

Apparently this is an implementation of IQueueCache (an interface defined by Orleans). If there is a process crash the items in the IQueueCache will not be marked as processed and therefore the events will be sent again to consumers the next time the cluster starts. Therefore there would not be data loss but possibly duplication of events (which can be prevented by ensuring the client stores a StreamSequenceToken). I had a similar question that got answered here.

[ReubenBond]

@turowicz does this answer the issue for you?

[turowicz]

@ReubenBond yes it does but actually we are looking into making the system so it doesn't need persistent queues. If we do end up persisting messages, it will probably be RabbitMQ but I still have to look into how that product performs.

Reasoning:

• Kafka repo is stagnant, requires a commercial library and I believe Kafka itself is a resource hog.
• Redis would be great but it requires polling and the library created by @berdon queues in-memory just like SMS.
• RabbitMQ repository has no information on how to use it and has been stagnant for 2 years.