Supporting ability to support --ip-isolate option

[chu11]

Documenting issue per discussion this morning. I think this crosses multiple flux projects, but I'll start here per discussion this morning.

With Slurm on several clusters, there is a special --ip-isolate option. The option sets up ip tables rules so that only nodes within the allocation can communicate with each other, nodes outside of the allocation cannot (with a few exceptions, such as reserved ports so people can ssh in). This is to protect user jobs when users run software that open up additional insecure services.

For example, software like Hadoop, Spark, and Tensorflow launch additional services on ports which any user on the system can get access to if they are not secured in some manner. It ranges from web interfaces to get information about a program that is running to schedulers which non-job owners could submit jobs to.

I believe that --ip-isolate is done via a job prolog which sets up the ip tables rules for that job and an epilog that tears it down at the end. In addition, a plugin generates an environment variable to inform user software that the feature has been enabled and setup (i.e. if it's not setup, bail b/c you don't want to run in an insecure manner). The ip tables rules setup would have to be run via root.

So the script itself to support --ip-tables is outside of flux scope, but to support this we'll need:

• job prolog/epilog support
• ability for prolog/epilog to run as root
• module/plugin support to generate the environment variable
• module/plugin for an optional --ip-isolate option for "flux job submit" (I suppose this part would be in flux-core)

[grondo]

Yeah, this is a good use case for generic extensions provided by system-level extensions to the IMP.

The requirements here are already detailed in the IMP RFC.