Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add configuration settings for PubSub source #380

Merged
merged 1 commit into from
Nov 15, 2024

Conversation

colmsnowplow
Copy link
Collaborator

Adds configuration options to allow us to tune PubSub source better:

max_outstanding_messages sets the upper limit on maximum messages processed at once (each of which spawns a goroutine)
max_outstanding_bytes sets the upper limit on amount of bytes in the queue of messages waiting to be processed
min_extension_period_seconds configures the minimum ack deadline extension to be set when reading messages from the subscription
streaming_pull_goroutines configures the number of concurrent streaming pulls open
grpc_connection_pool_size configures the connection pool size for the GRPC connection used to communicate with the subscription

Previously, the setting that streaming_pull_goroutines now controls was set by the concurrent_writes setting. However this is confusing behaviour as it behaves very differently to the other sources. The max_outstanding_messages is more like what concurrent_writes does for other sources, but the performance profile is wildly different, to the point that this source's configuration is very different from the others.

We want to make a release to resolve issues with pubsub, and it's preferable to avoid a breaking change in doing so. To achieve this, the logic is as follows:

Where streaming_pull_goroutines is configured, it takes precedence and concurrent_writes is ignored.

Where streaming_pull_goroutines is not configured, a warning is logged and we take the value of concurrent_writes

Where neither of those are configured, we log a warning and default to the previous default behaviour - which is an inadvisable 50.

pkg/source/pubsub/pubsub_source.go Show resolved Hide resolved
pkg/source/pubsub/pubsub_source.go Outdated Show resolved Hide resolved
@colmsnowplow colmsnowplow force-pushed the feat/pubsub-source-options branch from d3c4a72 to 865fde5 Compare November 15, 2024 17:39
@colmsnowplow colmsnowplow merged commit d6ba802 into develop Nov 15, 2024
1 check passed
@colmsnowplow colmsnowplow deleted the feat/pubsub-source-options branch November 15, 2024 18:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants