Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat/multi-version-schema-migrations #349

Merged
merged 21 commits into from
Jul 16, 2024
Merged
127 changes: 124 additions & 3 deletions 050-Manage/030-migrations.mdx
eemmiillyy marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
Expand Up @@ -73,10 +73,131 @@ Migration files should not be modified. If externally modified, the record will

Migrations must be ordered. Ordering is guaranteed by the `parent` and the ledger file, which is stored in `.xata/migrations/.ledger`. The ledger file is an append only file that lists all migration files in the correct order.

## UI Migration Editor for pgroll operations
## Multi-version Schema Migrations (Postgres enabled only)

Under the Schema menu, the Web UI provides a Migration Editor to create migrations using [pgroll](https://github.com/xataio/pgroll) operations.
For Postgres enabled branches, the Web UI provides a Migration Editor to create [multi-version schema migrations](https://xata.io/blog/schema-multi-version-launch-week) using [pgroll](https://github.com/xataio/pgroll) operations.
eemmiillyy marked this conversation as resolved.
Show resolved Hide resolved

In the editor you can provide content for the pgroll operations array and start a migration. Once submitted, the migration will enter the Active state and must be manually completed with the "Complete migration" action button before further schema edits can take place. While in Active state, the migration can also be rolled back.
Multi-version schema migrations address the pain point of having to keep your application code in sync with your database schema by allowing you two active database schemas at once. With multi-version schema migrations, running a migration means being able to preview the new version of your schema alongside the old one. Reading and writing data in both schemas continues to work normally. As a result of having two active schema versions, rollbacks become seamless operations which is particularly useful in the case of an unwanted schema change or backfill.

<Alert status="warning">
Currently multi-version schema migrations are only supported via the Migration Editor in the UI. Migrations made via
eemmiillyy marked this conversation as resolved.
Show resolved Hide resolved
the Table View and Schema View will complete automatically.
</Alert>

![The Migration Editor](images/migrations-multiversion/migration-editor.png)

Some common schema operations are:

<TabbedCode tabs={['create_table', 'add_column', 'rename_table', 'alter_column', 'drop_table', 'drop_column']}>
eemmiillyy marked this conversation as resolved.
Show resolved Hide resolved
```jsonc
[{
"create_table": {
"name": "customers",
"columns": [
{
"name": "id",
"type": "integer",
"pk": true
},
]
}
}]
```
```jsonc
[{
"add_column": {
"table": "customers",
"up": "UPPER(name)",
"column": {
"name": "full_name",
"type": "varchar(255)",
"nullable": true
}
}
}]
```
```jsonc
[{
"rename_table": {
"from": "customers",
"to": "clients"
}
}]
```
```jsonc
[{
"alter_column": {
"table": "clients",
"column": "full_name",
"name": "new_full_name",
"up": "(SELECT CASE WHEN full_name IS NULL OR LENGTH(full_name) <= 3 THEN 'placeholder' ELSE full_name END)",
"down": "full_name"
}
}]
```
```jsonc
[{
"drop_table": {
"name": "clients"
}
}]
```
```jsonc
[{
"drop_column": {
"table": "clients",
"column": "full_name",
"down": "''"
}
}]
```
</TabbedCode>

See the [pgroll Operations Reference](https://github.com/xataio/pgroll/blob/main/docs/README.md#operations-reference) for the list of supported migration operations.

## Lifecycle

A multi-version schema migration's life cycle consists of `start`, `complete`, and `rollback` steps.

### Start Step

After a migration is started it is considered `active`. One active migration per branch is possible. This is reflected in disabled buttons and inputs in the UI while a migration is ongoing. From there you can decide whether to `complete` or `rollback` the active migration.
eemmiillyy marked this conversation as resolved.
Show resolved Hide resolved

You can view the submitted migration under the Schema History tab while it is `active`.

![An active migration under the Schema History tab](images/migrations-multiversion/active-migration.png)

As more data is added to the table through either the old or the new version of the schema, the ‘other’ column is automatically updated by `pgroll` by the `up` or `down` triggers. Data in each version of the column remains editable.
eemmiillyy marked this conversation as resolved.
Show resolved Hide resolved

![Adding a new column](images/migrations-multiversion/add-column.png)

Migrations that involve more than one operation may result in a temporary additional column in the UI. This frequently occurs with alter column operations that change multiple properties. This column will disappear after the schema migration is completed or rolled back.
eemmiillyy marked this conversation as resolved.
Show resolved Hide resolved

![Editing an existing column](images/migrations-multiversion/alter-column.png)

![Deleting a column](images/migrations-multiversion/delete-column.png)

### Complete Step

Successfully completed migrations can be viewed under the Schema History tab in the Schema View.
![The Schema History tab](images/migrations-multiversion/schema-history.png)

If a migration fails, the submitted migration and failure reason can be viewed under the Migration Errors tab in the Schema View.
![Viewing a failed migration](images/migrations-multiversion/migration-errors.png)

### Rollback Step
eemmiillyy marked this conversation as resolved.
Show resolved Hide resolved

You can cancel a migration that has not yet been completed by rolling back. This is useful if the backfill or schema changes are unwanted. After a rollback, the schema changes from the migration will not be applied and the new schema will be deleted.

## Connecting via Wire Proxy

If there is no active migration, the database will contain two schemas.

1. The first schema will be the base schema and contain the underlying tables.
2. The second schema will have a `_create_table` suffix and contain views on all the tables inside the first schema.

If there is an active migration there will be an additional third schema. The third schema is for the latest version and will also contain views on all tables in the first schema.

## Limitations

- Currently we only support rolling back the latest migration. Rolling back a migration after it has been "completed" is not currently possible.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading