Draft: Improve subprocess logging

[sbrandstaeter]

Description and Context

This MR proposes reworking job logging within JobscriptDrivers (and its subclasses) to simplify our current "manual" logging approach. The suggested implementation aims to streamline the logging process while maintaining (almost) the same functionality.

Key motivations and changes are as follows:

1. Direct Bash Logging: The MR suggests switching to direct logging through bash commands, which should simplify the code and reduce the need for manual logging checks within Python. By embedding logging directly within bash, we may achieve the same tracking of job status with much less self-written, complex code.
2. Timeout for Zombie Prevention: Our current manual logging was initially implemented partly to detect “zombie” jobs by checking for the termination expression in continuously running jobs. A match would indicate a job that is running but failed -> a zombie. However, this method has proven to be unreliable. Instead, the MR proposes to replace it with a straightforward job timeout mechanism, which would automatically terminate jobs that exceed a set time limit. This timeout has a clear upper limit: the Dask worker wall time.

Benefits:

• Code Simplification: By logging directly with bash commands, we reduce the complexity in our Python code and make logging more robust.
• Improved Zombie Prevention: Using a timeout provides a more reliable approach to handling long-running jobs compared to manual detection.

If the changes are accepted, we can likely remove the existing manual logging, reducing maintenance overhead while retaining the essential functionality.

Related Issues and Pull Requests

• Related to https://gitlab.lrz.de/queens_community/queens/-/merge_requests/769

How Has This Been Tested?

Checklist

• My commit messages mention the appropriate GitLab issue numbers (advised but optional).
• I mention the appropriate GitLab issue numbers in this MR.
• I updated documentation where necessary.
• I updated the README where necessary
• I have added as (few,) small and fast tests as possible to cover my changes.

Additional Information

Interested Parties

Possible reviewers: @gilrrei

Other interested parties:

[sbrandstaeter]

Closes #70 #66

[sbrandstaeter]

We should continue the discussion started on GitLab: https://gitlab.lrz.de/queens_community/queens/-/merge_requests/769#note_3426821

[sbrandstaeter]

In the original MR on GitLab @gilrrei commented:
"I think we can remove the code for run_subprocess_with_logging in run_subprocess.py as well as get_job_logger in logger_settins.py"

Which I fully agree