GH-264: Add support for upsert/delete via periodic merge flushes #286
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Addresses #264
Implements upsert/delete functionality as proposed in the collaborative design doc, fixes some bugs in the connector, and adds an embedded integration test framework and tests for upsert/delete.
Some notes:
SchemaRetriever
API has been updated to differentiate between key and value schemas when invokingsetLastSeen
; this is done in a backwards-compatible fashion, so existing implementations of that interface should work with the updated API seamlessly.MemorySchemaRetriever
is updated to use that new API correctly (the existing one didn't differentiate between key and value schemas, which led to some buggy behavior that was unearthed during integration testing)SchemaManager
class is optimized to avoid redundant operations (which is useful in general but also specifically with upsert/delete enabled as create/update requests are much more likely in that case and the likelihood of them being redundant is fairly high)BigQuerySinkTask
is tweaked to avoid creating multipleBigQuery
andSchemaManager
instances; both classes are thread-safe, so this should be fine