scale allocator member ItemLimit to the relative share of ItemSlots, and refine suspension eligibility #416
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Item replication may not be evenly distributed.
Previously, we were not accounting for this within each allocation
sub-problem, and scaled member ItemLimits by the relative ratio of
items in each sub-problem rather than the relative ratio of item *slots*.
This caused excessive re-assignement of items in solved maximum
assignment flows, as push/relabel must spend more time back-tracking as
it overflows the initial constraints on item limits.
Introduce a new BenchmarkChangingReplication which reproduces this
effect.
Then, solve it by tweaking sparse flow network construction to scale
a member ItemLimit by the relative portion of assignment slots that
must be allocated within each sub-problem (rather than just the
relative number of items).
BenchmarkChangingReplication confirms this heuristic update dramatically
reduces the degree of assignment churn (note `run.ratio`, which is a
measure of average churn per item replication change):
Before:
final metrics adds=142806 packs=90617 removes=83044 run.ratio=16.58809060161935 run.total=9757
After:
final metrics adds=69854 packs=4457 removes=10092 run.ratio=1.6343138259710976 run.total=9757
jgraettinger
force-pushed
the
johnny/release-pipeline
branch
from
3fe9bed to
f510dce
Compare
jgraettinger
changed the title
Previously, a journal was eliglble for suspension as soon as all of its fragments were persisted. Though this is safe and correct, it doesn't account for raced evaluations in the broader context of a busy cluster with other topology changes and journal churn. Specifically, a journal with regular appends may be temporarily eligble for suspension only because its toplogy was recently changed, causing its fragments to be flushed. Worse, the journal's suspension (and soon, re-activation) may trigger churn which causes *other* journals to flush and themselves become suspended. Groups of journals may also have correlated write schedules, such as journals which are written once daily at midnight. We'd like to avoid thundering herds of journal suspensions and re-activations that might impact other journals due to assignment churn. To resolve this, prevent a journal from being suspended until its fragments are fully remote AND are at-least as old as its configured flush interval (or 24h, if not defined).
jgraettinger
force-pushed
the
johnny/release-pipeline
branch
from
5277ddd to
d700583
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
ItemSlots may be unevenly distributed amoung assignment sub-problems.
We need to account for this, by scaling member ItemLimits by the relative portion of assignment slots that must be allocated within each sub-problem, rather than just the relative number of items.
Also, tune journal suspension to apply the journal's configured flush interval when evaluating whether the index is "dirty".
See individual commits.
This change is