Reduce stack usage #191
Closed
Reduce stack usage #191
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…leak in the stack
CorentinB
approved these changes
Jan 30, 2025
…) and archive() functions and track routines and respect the context.Done() at the archive() level
|
TODO :
|
…cessor: use the dumper.PanicWithDump() when preprocessor encounters a non-fresh child
…make archiver skip items ; models.Item: fixed drawWithStatus()
|
Will not merge, splitting the PR in multiple PR with the change we want to keep |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We are observing solid discrepancies between the heap usage as profiled by Pyroscope and what the system reports as used. As Pyroscope only profiles the heap (mem allocated by go runtime) it suggests that the stack is being used too much and is causing the OOMs observed. Most likely causes of an off-heap usage are :
We are only using one CGO library :
goadaand a change was rolled to forcefully free any object allocated via this library (on the stack as the C++ code callsmalloc())Second line of investigation would be to troubleshoot WARC library as it's the top producer of goroutines and the profiler shows a lot of blocking : maybe the routines get stuck indefinitely and increase stack usage