Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Equinox.EventStoreDb - gRPC support #196

Merged
merged 29 commits into from
May 12, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
81e2dd3
grpc
bartelink Feb 5, 2020
5e1f73f
WIP up to L184
bartelink Feb 5, 2020
d30ae84
Handle SliceStatus
bartelink Feb 10, 2020
b183c6e
Migrate StreamSlice -> IAsyncEnumerable
bartelink Feb 17, 2020
acd15f6
Package
bartelink Feb 17, 2020
054ac96
Comment out logging until preview 3
bartelink Feb 17, 2020
f923399
Comment version inferences
bartelink Feb 17, 2020
7724851
Preview 3 starting
bartelink Mar 12, 2020
9db7c0c
Target EventStore.Client 21.2
bartelink Jan 1, 2022
235965f
Sycn with breaking chances to source
bartelink Jan 25, 2022
26508ee
Complete EventStoreDb impl
bartelink Feb 28, 2022
af7712a
Add Equinox.EventStoreDb.Integration
bartelink Feb 28, 2022
661cda1
Update to EventStore.Client.Grpc.Streams 22.0
bartelink Feb 28, 2022
dd384f7
Flip samples to use .EventStoreDb
bartelink Feb 28, 2022
7668259
Fixes
bartelink Feb 28, 2022
c0a6efd
Tidy unported
bartelink Feb 28, 2022
82bc99d
Fix all the things
bartelink Mar 6, 2022
d5dd5cd
Reorg test fixtures
bartelink Mar 8, 2022
93b57db
Tidy
bartelink Mar 8, 2022
354c98b
Cleanup
bartelink Mar 8, 2022
e12e84c
Flip to cluster
bartelink Mar 8, 2022
4ddc2a9
Formatting
bartelink Mar 9, 2022
66b5ef7
tmp
bartelink Mar 10, 2022
87950de
Share Cache with other ES-like stores
bartelink Mar 10, 2022
30d51aa
Cleanup ResolvedEvent mapping
bartelink Mar 20, 2022
6cd9639
Relax netstandard requirements
bartelink Mar 17, 2022
bc7379c
Tidy
bartelink Apr 13, 2022
1f49559
Changelog
bartelink Apr 13, 2022
d9ba265
Remove dead parameter
bartelink May 12, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ The `Unreleased` section name is replaced by the expected version of next releas
- `eqx dump`/`Equinox.Tool`: Show payload statistics [#323](https://github.com/jet/equinox/pull/323)
- `eqx dump`/`Equinox.Tool`: Add `-B` option to prevent assuming UTF-8 bodies [#323](https://github.com/jet/equinox/pull/323)
- `Equinox`: `Decider.Transact(interpret : 'state -> Async<'event list>)` [#314](https://github.com/jet/equinox/pull/314)
- `EventStoreDb`: As per `EventStore` module, but using the modern `EventStore.Client.Grpc.Streams` client [#196](https://github.com/jet/equinox/pull/196)

### Changed

Expand All @@ -26,6 +27,7 @@ The `Unreleased` section name is replaced by the expected version of next releas
- `CosmosStore`: Switch to natively using `System.Text.Json` for serialization of all `Microsoft.Azure.Cosmos` round-trips [#305](https://github.com/jet/equinox/pull/305) :pray: [@ylibrach](https://github.com/ylibrach)
- `CosmosStore`: Only log `bytes` when log level is `Debug` [#305](https://github.com/jet/equinox/pull/305)
- `EventStore`: Target `EventStore.Client` v `22.0.0-preview`; rename `Connector` -> `EventStoreConnector` [#317](https://github.com/jet/equinox/pull/317)
- `Equinox.Tool`/`samples/`: switched to use `Equinox.EventStoreDb` [#196](https://github.com/jet/equinox/pull/196)
- Update all non-Client dependencies except `FSharp.Core`, `FSharp.Control.AsyncSeq` [#310](https://github.com/jet/equinox/pull/310)
- Update all Stores to use `FsCodec` v `3.0.0`, with [`EventBody` types switching from `byte[]` to `ReadOnlyMemory<byte>`, see FsCodec#75](https://github.com/jet/FsCodec/pull/75) [#323](https://github.com/jet/equinox/pull/323)

Expand Down
3 changes: 1 addition & 2 deletions DOCUMENTATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -2230,10 +2230,9 @@ raise Issues first though ;) ).
EventStore, and it's Store adapter is the most proven and is pretty feature
rich relative to the need of consumers to date. Some things remain though:

- __Get impl in `master` ported to the modern gRPC client, currently parked in [#196](https://github.com/jet/equinox/pull/196). See also [#232](https://github.com/jet/equinox/issues/232)__
- Provide a low level walking events in F# API akin to
`Equinox.CosmosStore.Core.Events`; this would allow consumers to jump from direct
use of `EventStore.ClientAPI` -> `Equinox.EventStore.Core.Events` ->
use of `EventStore.Client` -> `Equinox.EventStore.Core.Events` ->
`Equinox.Decider` (with the potential to swap stores once one gets to using
`Equinox.Decider`)
- Get conflict handling as efficient and predictable as for `Equinox.CosmosStore`
Expand Down
12 changes: 12 additions & 0 deletions Equinox.sln
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,10 @@ EndProjectSection
EndProject
Project("{F2A71F9B-5D33-465A-A702-920D77279786}") = "Equinox.CosmosStore.Prometheus", "src\Equinox.CosmosStore.Prometheus\Equinox.CosmosStore.Prometheus.fsproj", "{3107BBC1-2BCB-4750-AED0-42B1F4CD1909}"
EndProject
Project("{F2A71F9B-5D33-465A-A702-920D77279786}") = "Equinox.EventStoreDb", "src\Equinox.EventStoreDb\Equinox.EventStoreDb.fsproj", "{C828E360-6FE8-47F9-9415-0A21869D7A93}"
EndProject
Project("{F2A71F9B-5D33-465A-A702-920D77279786}") = "Equinox.EventStoreDb.Integration", "tests\Equinox.EventStoreDb.Integration\Equinox.EventStoreDb.Integration.fsproj", "{BA63048B-3CA3-448D-A4CD-0C772D57B6F8}"
EndProject
Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution
Debug|Any CPU = Debug|Any CPU
Expand Down Expand Up @@ -197,6 +201,14 @@ Global
{3107BBC1-2BCB-4750-AED0-42B1F4CD1909}.Debug|Any CPU.Build.0 = Debug|Any CPU
{3107BBC1-2BCB-4750-AED0-42B1F4CD1909}.Release|Any CPU.ActiveCfg = Release|Any CPU
{3107BBC1-2BCB-4750-AED0-42B1F4CD1909}.Release|Any CPU.Build.0 = Release|Any CPU
{C828E360-6FE8-47F9-9415-0A21869D7A93}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{C828E360-6FE8-47F9-9415-0A21869D7A93}.Debug|Any CPU.Build.0 = Debug|Any CPU
{C828E360-6FE8-47F9-9415-0A21869D7A93}.Release|Any CPU.ActiveCfg = Release|Any CPU
{C828E360-6FE8-47F9-9415-0A21869D7A93}.Release|Any CPU.Build.0 = Release|Any CPU
{BA63048B-3CA3-448D-A4CD-0C772D57B6F8}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{BA63048B-3CA3-448D-A4CD-0C772D57B6F8}.Debug|Any CPU.Build.0 = Debug|Any CPU
{BA63048B-3CA3-448D-A4CD-0C772D57B6F8}.Release|Any CPU.ActiveCfg = Release|Any CPU
{BA63048B-3CA3-448D-A4CD-0C772D57B6F8}.Release|Any CPU.Build.0 = Release|Any CPU
EndGlobalSection
GlobalSection(SolutionProperties) = preSolution
HideSolutionNode = FALSE
Expand Down
7 changes: 4 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ _If you're looking to learn more about and/or discuss Event Sourcing and it's my
# Currently Supported Data Stores

- [Azure Cosmos DB](https://docs.microsoft.com/en-us/azure/cosmos-db): contains some fragments of code dating back to 2016, however [the storage model](DOCUMENTATION.md#Cosmos-Storage-Model) was arrived at based on intensive benchmarking (squash-merged in [#42](https://github.com/jet/equinox/pull/42)). The V2 and V3 release lines are being used in production systems. (The V3 release provides support for significantly more efficient packing of events ([storing events in the 'Tip'](https://github.com/jet/equinox/pull/251))).
- [EventStoreDB](https://eventstore.org/): this codebase itself has been in production since 2017 (see commit history), with key elements dating back to approx 2016.
- [EventStoreDB](https://eventstore.org/): this codebase itself has been in production since 2017 (see commit history), with key elements dating back to approx 2016. Current versions require EventStoreDB Server editions `21.10` or later, and communicate over the modern gRPC interface.
- [SqlStreamStore](https://github.com/SQLStreamStore/SQLStreamStore): bindings for the powerful and widely used SQL-backed Event Storage system, derived from the EventStoreDB adapter. [See SqlStreamStore docs](https://sqlstreamstore.readthedocs.io/en/latest/#introduction). :pray: [@rajivhost](https://github.com/rajivhost)
- `MemoryStore`: In-memory store (volatile, for unit or integration test purposes). Fulfils the full contract Equinox imposes on a store, but without I/O costs [(it's ~100 LOC wrapping a `ConcurrentDictionary`)](https://github.com/jet/equinox/blob/master/src/Equinox.MemoryStore/MemoryStore.fs). Also enables [take serialization/deserialization out of the picture](https://github.com/jet/FsCodec#boxcodec) in tests.

Expand Down Expand Up @@ -137,6 +137,7 @@ The components within this repository are delivered as multi-targeted Nuget pack
- `Equinox.CosmosStore` [![CosmosStore NuGet](https://img.shields.io/nuget/v/Equinox.CosmosStore.svg)](https://www.nuget.org/packages/Equinox.CosmosStore/): Azure CosmosDB Adapter with integrated 'unfolds' feature, facilitating optimal read performance in terms of latency and RU costs, instrumented to meet Jet's production monitoring requirements. ([depends](https://www.fuget.org/packages/Equinox.CosmosStore) on `Equinox.Core`, `Microsoft.Azure.Cosmos >= 3.25`, `FsCodec`, `System.Text.Json`, `FSharp.Control.AsyncSeq >= 2.0.23`)
- `Equinox.CosmosStore.Prometheus` [![CosmosStore.Prometheus NuGet](https://img.shields.io/nuget/v/Equinox.CosmosStore.Prometheus.svg)](https://www.nuget.org/packages/Equinox.CosmosStore.Prometheus/): Integration package providing a `Serilog.Core.ILogEventSink` that extracts detailed metrics information attached to the `LogEvent`s and feeds them to the `prometheus-net`'s `Prometheus.Metrics` static instance. ([depends](https://www.fuget.org/packages/Equinox.CosmosStore.Prometheus) on `Equinox.CosmosStore`, `prometheus-net >= 3.6.0`)
- `Equinox.EventStore` [![EventStore NuGet](https://img.shields.io/nuget/v/Equinox.EventStore.svg)](https://www.nuget.org/packages/Equinox.EventStore/): [EventStoreDB](https://eventstore.org/) Adapter designed to meet Jet's production monitoring requirements. ([depends](https://www.fuget.org/packages/Equinox.EventStore) on `Equinox.Core`, `EventStore.Client >= 22.0.0-preview`, `FSharp.Control.AsyncSeq >= 2.0.23`), EventStore Server version `21.10` or later)
- `Equinox.EventStoreDb` [![EventStoreDb NuGet](https://img.shields.io/nuget/v/Equinox.EventStoreDb.svg)](https://www.nuget.org/packages/Equinox.EventStoreDb/): Production-strength [EventStoreDB](https://eventstore.org/) Adapter. ([depends](https://www.fuget.org/packages/Equinox.EventStoreDb) on `Equinox.Core`, `EventStore.Client.Grpc.Streams` >= `22.0.0, `FSharp.Control.AsyncSeq` v `2.0.23`, EventStore Server version `21.10` or later)
- `Equinox.SqlStreamStore` [![SqlStreamStore NuGet](https://img.shields.io/nuget/v/Equinox.SqlStreamStore.svg)](https://www.nuget.org/packages/Equinox.SqlStreamStore/): [SqlStreamStore](https://github.com/SQLStreamStore/SQLStreamStore) Adapter derived from `Equinox.EventStore` - provides core facilities (but does not connect to a specific database; see sibling `SqlStreamStore`.* packages). ([depends](https://www.fuget.org/packages/Equinox.SqlStreamStore) on `Equinox.Core`, `FsCodec`, `SqlStreamStore >= 1.2.0-beta.8`, `FSharp.Control.AsyncSeq`)
- `Equinox.SqlStreamStore.MsSql` [![MsSql NuGet](https://img.shields.io/nuget/v/Equinox.SqlStreamStore.MsSql.svg)](https://www.nuget.org/packages/Equinox.SqlStreamStore.MsSql/): [SqlStreamStore.MsSql](https://sqlstreamstore.readthedocs.io/en/latest/sqlserver) Sql Server `Connector` implementation for `Equinox.SqlStreamStore` package). ([depends](https://www.fuget.org/packages/Equinox.SqlStreamStore.MsSql) on `Equinox.SqlStreamStore`, `SqlStreamStore.MsSql >= 1.2.0-beta.8`)
- `Equinox.SqlStreamStore.MySql` [![MySql NuGet](https://img.shields.io/nuget/v/Equinox.SqlStreamStore.MySql.svg)](https://www.nuget.org/packages/Equinox.SqlStreamStore.MySql/): `SqlStreamStore.MySql` MySQL `Connector` implementation for `Equinox.SqlStreamStore` package). ([depends](https://www.fuget.org/packages/Equinox.SqlStreamStore.MySql) on `Equinox.SqlStreamStore`, `SqlStreamStore.MySql >= 1.2.0-beta.8`)
Expand All @@ -151,6 +152,7 @@ Equinox does not focus on projection logic - each store brings its own strengths
- `Propulsion.Cosmos` [![Propulsion.Cosmos NuGet](https://img.shields.io/nuget/v/Propulsion.Cosmos.svg)](https://www.nuget.org/packages/Propulsion.Cosmos/): Wraps the [Microsoft .NET `ChangeFeedProcessor` library](https://github.com/Azure/azure-documentdb-changefeedprocessor-dotnet) providing a [processor loop](DOCUMENTATION.md#change-feed-processors) that maintains a continuous query loop per CosmosDB Physical Partition (Range) yielding new or updated documents (optionally unrolling events written by `Equinox.CosmosStore` for processing or forwarding). ([depends](https://www.fuget.org/packages/Propulsion.Cosmos) on `Equinox.Cosmos`, `Microsoft.Azure.DocumentDb.ChangeFeedProcessor >= 2.2.5`)
- `Propulsion.CosmosStore` [![Propulsion.CosmosStore NuGet](https://img.shields.io/nuget/v/Propulsion.CosmosStore.svg)](https://www.nuget.org/packages/Propulsion.CosmosStore/): Wraps the CosmosDB V3 SDK's Change Feed API, providing a [processor loop](DOCUMENTATION.md#change-feed-processors) that maintains a continuous query loop per CosmosDB Physical Partition (Range) yielding new or updated documents (optionally unrolling events written by `Equinox.CosmosStore` for processing or forwarding). Used in the [`propulsion project stats cosmos`](dotnet-tool-provisioning--benchmarking-tool) tool command; see [`dotnet new proProjector` to generate a sample app](#quickstart) using it. ([depends](https://www.fuget.org/packages/Propulsion.CosmosStore) on `Equinox.CosmosStore`)
- `Propulsion.EventStore` [![Propulsion.EventStore NuGet](https://img.shields.io/nuget/v/Propulsion.EventStore.svg)](https://www.nuget.org/packages/Propulsion.EventStore/) Used in the [`propulsion project es`](dotnet-tool-provisioning--benchmarking-tool) tool command; see [`dotnet new proSync` to generate a sample app](#quickstart) using it. ([depends](https://www.fuget.org/packages/Propulsion.EventStore) on `Equinox.EventStore`)
- `Propulsion.EventStoreDb` [![Propulsion.EventStoreDb NuGet](https://img.shields.io/nuget/v/Propulsion.EventStoreDb.svg)](https://www.nuget.org/packages/Propulsion.EventStoreDb/) Consumes from `EventStoreDB` v `21.10` or later using the gRPC interface. ([depends](https://www.fuget.org/packages/Propulsion.EventStoreDb) on `Equinox.EventStoreDb`)
- `Propulsion.Kafka` [![Propulsion.Kafka NuGet](https://img.shields.io/nuget/v/Propulsion.Kafka.svg)](https://www.nuget.org/packages/Propulsion.Kafka/): Provides a canonical `RenderedSpan` that can be used as a default format when projecting events via e.g. the Producer/Consumer pair in `dotnet new proProjector -k; dotnet new proConsumer`. ([depends](https://www.fuget.org/packages/Propulsion.Kafka) on `Newtonsoft.Json >= 11.0.2`, `Propulsion`, `FsKafka`)

## `dotnet tool` provisioning / benchmarking tool
Expand Down Expand Up @@ -537,7 +539,7 @@ The CLI can drive the Store and TodoBackend samples in the `samples/Web` ASP.NET

### Provisioning EventStore (when not using -s or -se)

There's a `docker-compose.yml` file in the root, so installing `docker-compose` and then running `docker-compose up` rigs a local 3-node cluster, which is assumed to be configured for `Equinox.EventStore.Integration`
There's a `docker-compose.yml` file in the root, so installing `docker-compose` and then running `docker-compose up` rigs a local 3-node cluster, which is assumed to be configured for `Equinox.EventStore.Integration` and `Equinox.EventStoreDb.Integration`

For more complete instructions, follow https://developers.eventstore.com/server/v21.10/installation.html#use-docker-compose

Expand Down Expand Up @@ -597,7 +599,6 @@ All non-alpha releases derive from tagged commits on `master`. The tag defines t
- :cry: the Azure Pipelines script does not run the integration tests, so these need to be run manually via the following steps:

- [Provision](#provisioning):
- Start Local EventStore running in simulated cluster mode
- Set environment variables x 4 for a CosmosDB database and container (you might need to `eqx init`)
- Add a `EQUINOX_COSMOS_CONTAINER_ARCHIVE` environment variable referencing a separate (`eqx init` initialized) CosmosDB Container that will be used to house fallback events in the [Fallback mechanism's tests](https://github.com/jet/equinox/pull/247)
- `docker-compose up` to start
Expand Down
1 change: 1 addition & 0 deletions build.proj
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
<Exec Command="dotnet pack src/Equinox.CosmosStore $(Cfg) $(PackOptions)" />
<Exec Command="dotnet pack src/Equinox.CosmosStore.Prometheus $(Cfg) $(PackOptions)" />
<Exec Command="dotnet pack src/Equinox.EventStore $(Cfg) $(PackOptions)" />
<Exec Command="dotnet pack src/Equinox.EventStoreDb $(Cfg) $(PackOptions)" />
<Exec Command="dotnet pack src/Equinox.MemoryStore $(Cfg) $(PackOptions)" />
<Exec Command="dotnet pack src/Equinox.SqlStreamStore $(Cfg) $(PackOptions)" />
<Exec Command="dotnet pack src/Equinox.SqlStreamStore.MsSql $(Cfg) $(PackOptions)" />
Expand Down
4 changes: 2 additions & 2 deletions diagrams/EventStore.puml
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,9 @@ rectangle "Application Consistent Processing" <<External Container>> {
interface IStream <<Component>>
}

rectangle "Equinox.EventStore" <<Container>> {
rectangle "Equinox.EventStoreDb" <<Container>> {
rectangle eqxes <<Component>> [
Equinox.EventStore OR
Equinox.EventStoreDb OR
Equinox.SqlStreamStore
]
database esstore <<Internal>> [
Expand Down
4 changes: 2 additions & 2 deletions diagrams/container.puml
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,9 @@ frame "Consistent Event Stores" as stores <<Expanded>> {
rectangle "Azure.Cosmos" <<ExternalComponent>> as cc
}
frame "EventStore" <<Internal>> {
rectangle "Equinox.EventStore" <<Component>> as es
rectangle "Equinox.EventStoreDb" <<Component>> as es
rectangle "Propulsion.EventStore" <<External Component>> as er
rectangle "EventStore.ClientAPI" <<External Component>> as esc
rectangle "EventStore.Client.Grpc.Streams" <<External Component>> as esc
}
frame "Integration Test Support" <<Internal>> {
rectangle "Equinox.MemoryStore" <<Component>> as ms
Expand Down
2 changes: 1 addition & 1 deletion samples/Infrastructure/Infrastructure.fsproj
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
<ProjectReference Include="..\..\src\Equinox.SqlStreamStore.MySql\Equinox.SqlStreamStore.MySql.fsproj" />
<ProjectReference Include="..\..\src\Equinox.SqlStreamStore.Postgres\Equinox.SqlStreamStore.Postgres.fsproj" />
<ProjectReference Include="..\..\src\Equinox.CosmosStore\Equinox.CosmosStore.fsproj" />
<ProjectReference Include="..\..\src\Equinox.EventStore\Equinox.EventStore.fsproj" />
<ProjectReference Include="..\..\src\Equinox.EventStoreDb\Equinox.EventStoreDb.fsproj" />
<ProjectReference Include="..\..\src\Equinox.MemoryStore\Equinox.MemoryStore.fsproj" />
<ProjectReference Include="..\Store\Domain\Domain.fsproj" />
<ProjectReference Include="..\TodoBackend\TodoBackend.fsproj" />
Expand Down
4 changes: 2 additions & 2 deletions samples/Infrastructure/Services.fs
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@ type StreamResolver(storage) =
let accessStrategy = if unfolds then Equinox.CosmosStore.AccessStrategy.Snapshot snapshot else Equinox.CosmosStore.AccessStrategy.Unoptimized
Equinox.CosmosStore.CosmosStoreCategory<'event,'state,_>(store, codec.ToJsonElementCodec(), fold, initial, caching, accessStrategy).Resolve
| Storage.StorageConfig.Es (context, caching, unfolds) ->
let accessStrategy = if unfolds then Equinox.EventStore.AccessStrategy.RollingSnapshots snapshot |> Some else None
Equinox.EventStore.EventStoreCategory<'event,'state,_>(context, codec, fold, initial, ?caching = caching, ?access = accessStrategy).Resolve
let accessStrategy = if unfolds then Equinox.EventStoreDb.AccessStrategy.RollingSnapshots snapshot |> Some else None
Equinox.EventStoreDb.EventStoreCategory<'event,'state,_>(context, codec, fold, initial, ?caching = caching, ?access = accessStrategy).Resolve
| Storage.StorageConfig.Sql (context, caching, unfolds) ->
let accessStrategy = if unfolds then Equinox.SqlStreamStore.AccessStrategy.RollingSnapshots snapshot |> Some else None
Equinox.SqlStreamStore.SqlStreamStoreCategory<'event,'state,_>(context, codec, fold, initial, ?caching = caching, ?access = accessStrategy).Resolve
Expand Down
Loading