add support for flux option flags

[vsoch]

this adds support for fluxOptionFlags to be defined on the level of the cluster. We check if the variable is defined, and if not, we do not attempt to export in the environment, as the base container might already specify a preference. To test this fully I am waiting for the osu benchmarks container to build, and then I can test it here with an example workflow.

Signed-off-by: vsoch [email protected]

[vsoch]

Okay here is what is needed for merge to be acceptable:

• a new working workflow that uses option flags running successfully
• verifying that previous workflows still work
• it's time to figure out testing. it's not acceptable that I haven't, going to look at https://github.com/kubeflow/mpi-operator/blob/5be2a42bf5af1fc9e5d1d96059f1f2dac489061d/Makefile#L58-L67. We need to be able to programmatically test the workflows we expect to work and verify that. Note that this is checked here and will be further handled in Testing Meta Issue #51 - we have basic tests added here for simple workflows (validating output and return codes) but need more substantial testing.
• As a result of adding this flag, I've refactored the entrypoint for a flux runner (a wait.sh) to now be a wait<N>.sh that corresponds to the container index, where theoretically we could have more than one flux runner with slight variance of logic. I did this because with the flux + spack container I found I couldn't get anything working without a source of the spack environment in the entrypoint, and some tweaks to the command to run flux. Thus I need to be able to define these in the CRD for the workflow associated with this container.

[vsoch]

Lots of pushing (sorry for the noise!) but we're getting somewhere! I have a setup that seems to be able to run a named test, get the output after a known time, and then compare it to what is expected.

I'll pick up on writing more of these after dinner - I got a weird GitHub ratelimiting error but hopefully that was ephemeral.