Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

schema migration #1110

Merged
merged 1 commit into from
Mar 4, 2024
Merged

schema migration #1110

merged 1 commit into from
Mar 4, 2024

Conversation

MarinPostma
Copy link
Contributor

@MarinPostma MarinPostma commented Feb 29, 2024

implements the happy path of schema migration. do not use this feature yet.

TODO:

  • move all of the migration related stuff from the meta_store to the schema module; let the scheduler use it's own connection to the meta store.
  • move the schema for the meta store to a sqlite file, and include! that file for clarity
  • perform error handling, check migration success status, use custom QueryResultHandler to collect results
  • ensure that the temp table enqueued_tasks is updated corectly after a task is popped when a tasks status changed (maybe use a trigger?)
  • handle abort; first when a dry run failed, then if a user abort the migration before the dry-run is complete
  • prevent a job from being enqueued if there is still a pending job for that namespaces
  • Add loads of tests, and test load
  • prevent writes while there is are pending jobs in a database
  • sync database before starting migration
  • sync meta store after enqueue job
  • track version schema
  • go through all todos that I left in the code

@MarinPostma MarinPostma force-pushed the schema-migration branch 7 times, most recently from e258feb to 162fbd9 Compare March 4, 2024 09:24
Copy link
Contributor

@Horusiath Horusiath left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since it's still a draft, just few points. The overall design seems pretty solid and aligned with what we were discussing in previous weeks. Please let me know when this is ready to full review.

.await?;
self.connection
.execute_program(pgm, ctx, response_builder, replication_index)
dbg!();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Committed by accident?

3 => Self::Run,
4 => Self::Success,
5 => Self::Failure,
_ => panic!(),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
_ => panic!(),
_ => unreachable!(),

3 => Self::WaitingRun,
4 => Self::RunSuccess,
5 => Self::RunFailure,
_ => panic!(),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
_ => panic!(),
_ => unreachable!(),

builder: B,
) -> (Result<B, Error>, MigrationTaskStatus) {
// todo error handling is sketchy, improve
let mut savepoint = conn.savepoint().unwrap();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess this is also a place for bottomless savepoint (see #1085) in order to guarantee, that in case of crashes we can restore db to a state where the dry run succeeded. However in a current API it operates on the Database type not a Connection.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's discuss that

@MarinPostma MarinPostma marked this pull request as ready for review March 4, 2024 13:00
@MarinPostma MarinPostma enabled auto-merge March 4, 2024 13:00
@MarinPostma MarinPostma added this pull request to the merge queue Mar 4, 2024
Merged via the queue into main with commit 2b06b6b Mar 4, 2024
16 checks passed
@MarinPostma MarinPostma deleted the schema-migration branch March 4, 2024 13:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants