Add configuration settings for PubSub source #380
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds configuration options to allow us to tune PubSub source better:
max_outstanding_messages
sets the upper limit on maximum messages processed at once (each of which spawns a goroutine)max_outstanding_bytes
sets the upper limit on amount of bytes in the queue of messages waiting to be processedmin_extension_period_seconds
configures the minimum ack deadline extension to be set when reading messages from the subscriptionstreaming_pull_goroutines
configures the number of concurrent streaming pulls opengrpc_connection_pool_size
configures the connection pool size for the GRPC connection used to communicate with the subscriptionPreviously, the setting that
streaming_pull_goroutines
now controls was set by theconcurrent_writes
setting. However this is confusing behaviour as it behaves very differently to the other sources. Themax_outstanding_messages
is more like whatconcurrent_writes
does for other sources, but the performance profile is wildly different, to the point that this source's configuration is very different from the others.We want to make a release to resolve issues with pubsub, and it's preferable to avoid a breaking change in doing so. To achieve this, the logic is as follows:
Where
streaming_pull_goroutines
is configured, it takes precedence andconcurrent_writes
is ignored.Where
streaming_pull_goroutines
is not configured, a warning is logged and we take the value ofconcurrent_writes
Where neither of those are configured, we log a warning and default to the previous default behaviour - which is an inadvisable 50.