Release/3.3.0 #437
Merged
Release/3.3.0 #437
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Introduces version updates to third-party components along with refreshed SLULA license 1.1
Previously it was possible to send arbitrarily large payloads at the collector. Now, we add two mechanisms of handling large requests. First, `networking.maxPayloadSize` (defaulting to 1048576 = 1MB) will stream the input and generate SizeViolation bad event. Second, `networking.dropPayloadSize` (defaulting to 2097152 = 2MB) will return HTTP 413 and will not attempt to process the request. The two mechanisms come hand in hand allowing for setting a safe but tolerant configuration. If requests exceed `sink.maxBytes` the collector will attempt to split them as separate events. However, if payload is bigger than `maxPayloadSize`, parsing the body is risky and in case of an attack may result in failures. The collector will not attempt to parse the request and split it up but will generate a SizeViolation bad event. But if payload exceeds `dropPayloadSize` the request will not be handled and no bad event is going to be emitted. Instead HTTP 413 is generated.
In #431 we improved performance of the Kafka sink by calling `producer.send()` on a compute thread not on a blocking thread. This is a great improvement if it is always true that `producer.send()` never blocks. But `producer.send()` does in fact block if the producer needs to re-fetch topic metadata. This happens every 5 minutes by default, and is configured by the kafka setting `metadata.max.age.ms`. If `producer.send()` blocks on a compute thread, then it can potentially cause thread starvation, and negatively impact the collector's responsiveness to requests. This PR changes to executing `send()` on a dedicated thread. From the point of view of the collector, it is ok if `send()` is blocking on the dedicated thread. It is better than running it inside `Sync[F].blocking` (which is what we did before) because that tended to create a huge number of threads under some circumstances. In theory this shouldn't negatively affect performance much, even though it's a single thread. Because most of the time `send()` does not block; and when it does block (i.e. once per 5 minutes) then it is ok for the other events to get enqueued as a backlog of Callables on the thread. Kafka sink use a dedicated thread for potentially blocking send: Amendment 1
stanch
reviewed
Jan 7, 2025
|
|
||
| ### Security | ||
| - Add limit for payload allowed sizes [snowplow/stream-collector-private#1] | ||
| - Security and license updates [snowplow/stream-collector-private#1] |
There was a problem hiding this comment.
We probably don’t need these links?
There was a problem hiding this comment.
I'd like to keep the minimum changes in the content as it will be hard to manage both origins if content is different. If not strictly required I would keep them there.
stanch
approved these changes
Jan 7, 2025
istreeter
approved these changes
Jan 7, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.