Create a migration test suite

[Floppy]

Because any old version of the application could be updated to the latest, users sometimes run into problems when migrating that we don't spot (see #2982 and the various linked issues for where this has happened).

To avoid this, we want to create a migration test suite that will test that old database versions can be successfully migrated to the current version. The old databases should also contain representative data, in order to test the data migration process as well as schema changes.

The test should, for each migration step (or released version?):

1. Create a fresh database
2. Migrate to the "start" version
3. Load seed data appropriate to that version
4. Test that migration to "latest" version works without failure

We also need to be able to run this against various database configurations, so the seed data will need to be independent of the database in use; we can't just keep an SQLite file for instance.

Creating that seed data for each version may involve running the released code for that version, creating test data, then saving somehow in a form that can be loaded later during testing.

[julianharty]

It might be worth investing some time to measuring how long each of the various steps take (steps 1 to 4 above) and also to consider whether to have freeze-dried partially processed databases if doing so a) reduces the compute resources (and elapsed time) needed and b) provided that using freeze-dried snapshots doesn't adversely affect the fidelity of the test results.

How about also designing in consistent logging of the test setup and execution so that a history is established of running the various permutations?

[Floppy]

Rough plan is forming...

Create a set of start states by doing the following for each start state:

• Run docker container for a release
• Run a script via "docker exec rails run" to create known test data using application code from the start version
• Export database dump using appropriate database-specific tool (i.e. pgdump)
• Save dump into testing repo

To run a migration test:

• Load database dump for starting release
• Run head container
• Run migration and check for success

[ericherman]

And ideally, the steps to create the start states will themselves be automated in order to make it easier to add more data to the start states in the future.

[julianharty]

Looks good. How about creating a minimal seed dataset (in other words something small and simple) as practical e.g. for a simple tetrahedron and then using that as a pathfinder for the rest of this work?