[WIP] Distributed computing with celery

[jchodera]

@andrrizzi : Here's the very basic test code I was playing with, in case you find it useful. This doesn't necessarily have to be merged, but might at least illustrate something along the lines of what I was thinking.

I haven't tested this on the cluster yet.

[jchodera]

We'll still need a way to start celery workers on individual GPUs. I bet we could do this with something like

CUDA_VISIBLE_DEVICES=0 celery -A openmmtools.distributed worker -l info --concurrency=1 &
CUDA_VISIBLE_DEVICES=1 celery -A openmmtools.distributed worker -l info --concurrency=1 &
CUDA_VISIBLE_DEVICES=2 celery -A openmmtools.distributed worker -l info --concurrency=1 &
CUDA_VISIBLE_DEVICES=3 celery -A openmmtools.distributed worker -l info --concurrency=1 &

though @pgrinaway may have better ideas for how best to do this with multiple GPUs on a node.

It looks like there's also a way to specify worker queues with the --queues flag, like --queues gpu vs --queues cpu. Documentation on these can be found here.

[andrrizzi]

Thanks! I'll take a look at this tomorrow.

[jchodera]

This is still very much test code for experimenting. I think the next steps are:

• Try to construct some sort of benchmark example that tests local vs remote execution of a replica exchange like operation on a realistic test system (e.g. Src in explicit solvent) to see how well things perform
• Make it easy to try both celery and redis