Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor lake compaction scheduler to keep scheduling unrelated partitions on fe restart #54883

Open
wxl24life opened this issue Jan 9, 2025 · 0 comments · May be fixed by #54881
Open

Refactor lake compaction scheduler to keep scheduling unrelated partitions on fe restart #54883

wxl24life opened this issue Jan 9, 2025 · 0 comments · May be fixed by #54881
Labels
type/enhancement Make an enhancement to StarRocks

Comments

@wxl24life
Copy link
Contributor

wxl24life commented Jan 9, 2025

Enhancement

In the previous implementation, if some lake compaction transaction was not finished, and FE restarted, Lake compaction scheduler in FE would fail to schedule at all. If that compaction txn takes long time to finish (published), or even worse it was unable to publish at all, this will block the compaction scheduler forever, and will introduce a nightmare as all partition's compaction score continuously to increase.

In my new design, I will remove the global min active txn id checker logic, which is the main reason that blocks all compaction jobs from scheduling. Instead, I will rebuild the running compaction jobs on FE restart, and give the scheduler another chance to control the unfinished transactions.

@wxl24life wxl24life added the type/enhancement Make an enhancement to StarRocks label Jan 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/enhancement Make an enhancement to StarRocks
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant