Job Launching

[dylanmcreynolds]

The compute api currently uses the docker client to launch jobs as inside docker containers. This has worked well as a quick way to achieve job scheduling. As we move forward into different deployment scenarios, this becomes a little problematic. This pins a strong dependency on rootful docker.

In the time since this was first built, we have gained a lot of experience with Prefect for job scheduling an management. We would like replace this docker launching code with prefect launching code to give maintainers of MLExchange greater visibility and control over jobs.

Ideally, I think we want to create an optional prefect/postgres server environment in the docker-compose infrastructure of MLExchange, making it easy to spin up a new prefect server environment. But we also want to be able to support implementers of MLExchange using their own pre-existing prefect environment, either local or cloud.

As far as the mlex_computing_api is concerned, this I think there are several pieces of configuration that we'll need:

• Prefect API URL (for all jobs)
• Prefect API Key (for all jobs)
• Job-dependent Prefect deployment ID and input parameters. And example of scheduling a prefect job can be found here