[Question] Concurrency management

[pedroneves]

Hello everyone 👋

I have a situation where I need to use Umzug fine control migrations during boot time for my app. The problem is that I must run multiple instances of my service, so the migration script will run for every instance before the app starts.

Is there any built-in concurrency management mechanism in Umzug or this is a concern I should take care of?

Also, if this is something I should address in the code, is there an official recommendation on how to proceed (lib I should use or process I should follow for example)? So far, I only found this lib called umzug-sync.

Thank you!

[mmkal]

Hi @pedroneves - currently there isn't in-built locking support, since this best way to lock varies quite a lot between SQL dialects/implementations. If you're using postgres, you could try @slonik/migrator which does support this via pg_advisory_lock (disclaimer - I am the author of that library). In your case, whichever instance stood up second would wait until any existing migrations are finished before trying to run (once the first instance finishes, any subsequent migrations should be a no-op).

If you're not using postgres, you could take a look at the implementation - it overrides the runCommand method in a subclass. So you could try something like:

import {Umzug} from 'umzug'

export class MyMigrator extends Umzug {
  async runCommand(command, cb) {
    if (command === 'up' || command === 'down') {
      return super.runCommand(command, async params => {
        await acquireLock() // implement `acquireLock` however makes sense for your system. You could also pass in `params.context` here.
        try {
          return cb(params)
        } finally {
          await releaseLock()
        }
      })
    }
    return super.runCommand(command, cb)
  }
}

Then instantiate and use MyMigrator in the same way you would use Umzug. This is essentially what @slonik/migrator does, with a couple of extra bits for logging when acquiring the lock takes more than 1s, etc.

[LinusU]

@mmkal

currently there isn't in-built locking support, since this best way to lock varies quite a lot between SQL dialects/implementations.

Couldn't this easily be handled in the src/storage/*.ts files? Each storage "engine" would have the support itself, so different dialects/implementations would be handled.

Without this it's hard to deploy this package in a serverless, or even micro service, environment.

Would you be open to contributions to work on this?

[mmkal]

You're welcome to give it a go. It's a very tricky thing to test realistically in CI so I probably wouldn't be comfortable rushing it in. But broadly I agree that it'd be a nice feature to have as part of this library. Which storage method were you thinking about putting the lock() and unlock() calls?

[pedroneves]

Hey @mmkal ! Thank you so much for your quick reply. I still did not have the chance to try your suggestions, but I'll post here how it went.

Thanks!