Handling failures

[dpwrussell]

• Dead letter queues for failures from all queue based operations (step functions, batch jobs and potentially SQS)
• Handle communication failures (e.g. a Docker container attempting to launch a step function)
• Log everything
• Report failures (e.g. A fileset extract that fails because of Bio-Formats needs to have that registered in the database)
• Facility to activate retries from a given stage of processing
• Periodically attempt to detect inconsistencies (e.g. unregister objects in S3, leftover data on EFS, incomplete imports without a corresponding record of failure)