Save rocoto-generated job cards for manual debugging

[mkavulich]

Description

There is a desire by developers and scientists to be able to manually re-run tasks, both for development/experimentation purposes and for debugging (e.g., for allowing sysadmins to replicate an observed issue). While the rocotorun command does output the submitted job card to log files when run via crontab, these are hard to work with in raw text form, and the WE2E tests by default do not save the job cards at all due to performance issues with Python subprocesses (though this can be enabled with the --debug flag).

Solution

A first step towards meeting these developer needs is to have the submitted job cards saved as stand-alone files at run time.

Requirements**

• Developers should be able to re-run tasks manually without having to go through rocoto as a middleman.

Acceptance Criteria (Definition of Done)

• In addition to traditional submission via rocoto, job cards that can be manually submitted for a given platform and experiment should be saved.

Alternative Solutions (optional)

• Ideally, Rocoto should come with a command that can generate the job cards manually. We are in discussion with @christinaholtNOAA about adding this capability.
• This capability may be much easier if the SRW natively supported ecflow as an alternative workflow manager.