Beeflow connect - A REST API for BEEflow itself.

[aquan9]

This PR, adds components to BEEflow to support a REST API for the BEEflow client itself.

Using this API users can remotely submit jobs to BEEflow by doing the following:

1. Making a request to /droppoint to get the location of the droppoint on the remote system.
2. Copying a workflow tarball, and workdir to that droppoint using SCP
3. Making a request to /submit_long with the name of the workflow to start the workflow.

Documentation has been added to the sphinx docs directory.

[aquan9]

After meeting with Pat, she mentioned that we should have a way to turn on the remote API without it being on by default. I'm working on this now.

[aquan9]

The other thing was that we needed to do was put an option in the beeconfig to choose what port to use during startup for the remote api.

[pagrubel]

By default could the port be generated similarly to the others that use the user id in some fashion?

[aquan9]

By default could the port be generated similarly to the others that use the user id in some fashion?

@pagrubel I created a separate issue for this particular feature. In this PR I'm just trying to pull in the basic functionality for the remote api. There are many other features for the remote api I want to add that are missing from this PR.

But I think your suggestion is a good idea, and we should do that at some point.

[jtronge]

Maybe this should be a PUT or POST request? I think GET requests are usually just used for extracting information without causing state changes.

[jtronge]

I tried this locally with curl commands. Maybe I'm doing something wrong, but when I try to use the submit URL it doesn't show anything. I also don't see a workflow running (using beeflow list). I just copied a cat-grep-tar package into the droppoint.

[jtronge]

Oh, actually the remote_api is failing with ERROR: [Errno 98] error while attempting to bind on address ('0.0.0.0', 7777): address already in use for me, but it's showing as RUNNING with a beeflow core status. I must be connecting to your instance.

[jtronge]

I changed the port and now it's working for me. I was able to submit cat-grep-tar.

[pagrubel]

Please add unique port for user.

[pagrubel]

Per Jake: if the port is in use it still shows as Running so try to make it fail if this is the case.

[aquan9]

The fix for finding a port dynamically has been added.

Pat requested I write up documentation for these changes so far as part of the PR.

Additionally I think it would be good if there were better notification for the user when the user's selected port has a conflict. Right now there's just a message in the .beeflow/log/remote saying that it picked a different port.