EclipseStore storage in HA setup

[zambrovski]

Hi folks,

I'm reasoning about the setup of an application using EclipseStore as a (projection) persistence. In doing so I'll have a component - let call it projector, responsible for receiving some data and creating a in-memory domain model. In addition, there will be some other components - let call them query handlers which will have a read only access to the in-memory domain models and are solely responsible for answering queries.

As far as I understand correctly, there is at most one component with running StorageManager that can have write access to a physical store (independent on the storage technology), so in my scenario, I would configure the projector to be this one component and configure the query handlers to use read-only StoreManagers.

Is my assumption correct?

If yes - I wonder what to do if this setup must run HA - so I'll have two hot nodes running projector components. Even if I can make sure (e.g. by routing) that the projection is executed on one node, can I still start both nodes and how the take over can work?

Very interested in the architectural / framework approach for this use case.

Thanks,

Simon

[zdenek-jonas]

Hello, I apologize for overlooking this query. Queries are not executed directly on the storage but rather on the data in memory. The storage serves primarily as a persistence layer. Essentially, the architecture should be based on a shared data graph with lock-based concurrency control mechanisms, such as reentrant locks, for managing concurrent access.

[rpx99]

Wonder if you have some whitepaper / guides hat describe this more detailed or best practice suggestions. Something like a HA guide.

I understand there exists a SaaS (cloud based) solution using kubernetes behind the scenes. But not all users like to use cloud / cloud storage.

I am thinking about a classic hosted on-prem cluster solution for Java EE / Jakarta, e.g. in a HA wildly cluster setup.

What do you suggest for this kind of setup?

[zambrovski]

Sure, queries are executed on data in memory, but this data need to get into memory from the storage. And the storage manager needs to connect to the storage - and if I only read, I could try to use the read only storage manager for this (is there a comprehensive example for this somewhere?).

I'm speaking about the projector side. I have two nodes (because they are redundant for the purpose of fault tolerance), which is not very uncommon in HA setups running e.g. on K8 / OCP platforms. And both nodes are hot (not cold standby or whatever), because I don't want to have any start-up take-over time in cases of errors. So two fundamental question arise:

1. The node must contain a storage manager connected to the same storage, even if it is not shared, because my load balancer will make sure that all requests are routed to the same node and only that will execute write operations. (I'm explicit in fault-tolerance and not in load-balancing mode). Is it possible using embedded storage managers?
2. If the node is failing (shut down) can the second node start writing right away?

If one of this answers is no, and I have to use the cluster solution and cluster storage manager, how the on-prem installation looks like?

Please provide at least some architectural hints or maybe videos to watch...

Thanks,

Simon

[rpx99]

Thanks, Simon 😃 My question was more directed to the team / Jonas. Basically I think we have the same question. You have a load balancer / reverse proxy and at least 2 servers (cluster). How to sync the object graphs and how to replicate the files so you can failover in case something bad happens.

JTA and a transaction manager like naranya, atomikos or bitronix - maybe combined with deltaspikes transactional annotation - come into my mind to coordinate and sync the eclipsestore data store. Think this should use reentrant locks and avoid conflicts...

For the filesystem high availability there are some possibilities like opensync or whatever?

Anyone tried something like this? Where are pitfalls? Does this make sense at all?

[hg-ms]

1. The node must contain a storage manager connected to the same storage, even if it is not shared, because my load balancer will make sure that all requests are routed to the same node and only that will execute write operations. (I'm explicit in fault-tolerance and not in load-balancing mode). Is it possible using embedded storage managers?

2. If the node is failing (shut down) can the second node start writing right away?

Hello,
Having a second StorageManager as readonly node is not applicable as there is the limitation that it will not recognize updates made by the writing StorageManager.
To “take over” a storage-target you need to ensure that the writing instance is shut down and then start the new one.

[rpx99]

Sounds like a Cluster-singleton is a possibility...

[hrstoyanov]

I asked similar question some while ago. So, if the Microstream team does not have plans/desire/time to include simple HA API in the ES roadmap, I plan on exploring the following soon:

1. Implements HA setup based on a single leader for updates, and one or more query-only replicas. jGroups has a decent RAFT implementation, for when the leader fails and a new one needs to be elected.
2. Your http load-balancer needs to know how to properly route commands to either the read replicas(queries) or the leader(updates) - typically based on patterns in the url and/or HTTP method. It also need to dynamically switch to a potentially newly elected leader as per RAFT.
3. The (query-only) replicas should be easy - they only query the data, but they must also receive updates from the leader and apply them to their local model. Should a replica fail, it will have to be restarted from the leader's data files only, so you may have to temporarily stop the entire cluster to bring a failed node back up.
4. The leader (the updates node) is the only one making changes to your data model ( and maybe talking to external systems?), and needs to dispatch serialized update commands to all replaces, once it executes the update with no exceptions. This is where jGroups will help a lot. The leader can wait for confirmation from all replicas (strong consistency) or assume they will eventually catch up (eventual consistency).
5. You design should be based on Command as defined in CQRS.
6. jGroups will help with fast UDP-based network protocol and RAFT protocol implementation/leader election.
7. Key element of the above setup is the availability of a shared network storage ("share-nothing" is probably an overkill and even impossible with ES)

This may sound easy, but is a lot of work, so I will need collaborators... but will be waste of time if the Microstream team decides to offer something like that as part of ES.

[rpx99]

Don't think they will put it onto their agenda as they already have the cloud based cluster solution. Believe the focus is on adding integrations like Quarkus or things like GigaTables...but correct me if I am wrong.

[zambrovski]

I'm fine to engineer this, if we get more or less the "take-over" protocol explained by the team. I don't want to reverse engineer their solution and their decisions. They had something in mind as they defined the exclusive write mode and they already have a SaaS solution where they addressed the synchronization. So I hope to gain the knowledge from the team and build an easy-small solution (not a competitor to the SaaS solution).

[hrstoyanov]

The Microstream Enterprise product is a complete cloud product with Web UI management, monitoring, logging, security .. the kitchen sink. I do not think that is what we need or that needs to be open-sourced. What we only need is the basic implementation of and HA cluster.

Btw, I put some effort into the Helidon integration - both for microstream 8 and Eclipse store , if anyone is interested.

[zambrovski]

I'll be happy to create the SpringBoot setup.

[jabolina]

On the jgroups-raft side, we are actively working to improve and maintain the project. Don't hesitate to reach out if needed!