You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In the previous implementation, if some lake compaction transaction was not finished, and FE restarted, Lake compaction scheduler in FE would fail to schedule at all. If that compaction txn takes long time to finish (published), or even worse it was unable to publish at all, this will block the compaction scheduler forever, and will introduce a nightmare as all partition's compaction score continuously to increase.
In my new design, I will remove the global min active txn id checker logic, which is the main reason that blocks all compaction jobs from scheduling. Instead, I will rebuild the running compaction jobs on FE restart, and give the scheduler another chance to control the unfinished transactions.
The text was updated successfully, but these errors were encountered:
Enhancement
In the previous implementation, if some lake compaction transaction was not finished, and FE restarted, Lake compaction scheduler in FE would fail to schedule at all. If that compaction txn takes long time to finish (published), or even worse it was unable to publish at all, this will block the compaction scheduler forever, and will introduce a nightmare as all partition's compaction score continuously to increase.
In my new design, I will remove the global min active txn id checker logic, which is the main reason that blocks all compaction jobs from scheduling. Instead, I will rebuild the running compaction jobs on FE restart, and give the scheduler another chance to control the unfinished transactions.
The text was updated successfully, but these errors were encountered: