sdexec: set KillMode=process SendSIGKILL=no for multi-user jobs

[garlick]

Problem: For multi-user jobs spawned via SDEXEC, the systemd user instance running as the flux user does not have permission to kill guest processes, yet it does try and in the process may kill off the only process that does have permission to continue cleanup efforts, the IMP.

When the job is run by the IMP and sdexec, Set KillMode=process so that systemd only delivers signals to the IMP, which it should forward to the shell and/or cgroup per RFC 15.

Also set SendSIGKILL=off so that SIGKILL is never deployed against the IMP.

Fixes #6399

This should be OK even without the forthcoming changes to job-exec and the IMP to forward signals and avoid using SIGKILL, however I still need to give this a sanity check on my test cluster as the conditional use of these properties for IMP jobs is not tested by CI. I think we can still get a review going however so I'll not set WIP.

[grondo]

LGTM!

[garlick]

I ran a little test and did verify that the properties are set as expected when running a job a guest and when running a job as the instance owner (where the IMP is not used):

As a guest:

 garlick@picl0:~$ flux submit -q batch -N1 sleep 3600
ƒ3L563TwyxT
 garlick@picl0:~$ flux jobs
       JOBID QUEUE    USER     NAME       ST NTASKS NNODES     TIME INFO
 ƒ3L563TwyxT batch    garlick  sleep       R      1      1   2.334s picl1

 garlick@picl0:~$ sudo flux exec -r 1 systemctl --user status
● picl1
    State: running
    Units: 100 loaded (incl. loaded aliases)
     Jobs: 0 queued
   Failed: 0 units
    Since: Thu 2024-10-31 11:43:16 PDT; 15min ago
  systemd: 252.30-1~deb12u2
   CGroup: /user.slice/user-500.slice/[email protected]
           ├─app.slice
           │ └─imp-shell-1-f3L563TwyxT.service
           │   ├─869 /usr/lib/aarch64-linux-gnu/flux/flux-imp exec /usr/lib/aarch64-linux-gnu/flux/flux-shell 17297539929735168
           │   ├─873 /usr/lib/aarch64-linux-gnu/flux/flux-shell 17297539929735168
           │   └─874 sleep 3600
           ├─init.scope
           │ ├─803 /lib/systemd/systemd --user
           │ └─804 "(sd-pam)"
           └─session.slice
             └─dbus.service
               └─835 /usr/bin/dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
 garlick@picl0:~$ sudo flux exec -r 1 systemctl --user show --property SendSIGKILL imp-shell-1-f3L563TwyxT
SendSIGKILL=no
 garlick@picl0:~$ sudo flux exec -r 1 systemctl --user show --property KillMode imp-shell-1-f3L563TwyxT
KillMode=process

As instance owner:

 garlick@picl0:~$ sudo -u flux flux submit -q batch -N1 sleep 3600
 garlick@picl0:~$ flux jobs -A
       JOBID QUEUE    USER     NAME       ST NTASKS NNODES     TIME INFO
 ƒ3L6VaRh81d batch    flux     sleep       R      1      1   10.06s picl1

 garlick@picl0:~$ sudo flux exec -r 1 systemctl --user status
● picl1
    State: running
    Units: 100 loaded (incl. loaded aliases)
     Jobs: 0 queued
   Failed: 0 units
    Since: Thu 2024-10-31 11:43:16 PDT; 18min ago
  systemd: 252.30-1~deb12u2
   CGroup: /user.slice/user-500.slice/[email protected]
           ├─app.slice
           │ └─shell-1-f3L6VaRh81d.service
           │   ├─923 /usr/lib/aarch64-linux-gnu/flux/flux-shell 17300643815358464
           │   └─924 sleep 3600
           ├─init.scope
           │ ├─803 /lib/systemd/systemd --user
           │ └─804 "(sd-pam)"
           └─session.slice
             └─dbus.service
               └─835 /usr/bin/dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
 garlick@picl0:~$ sudo flux exec -r 1 systemctl --user show --property KillMode  shell-1-f3L6VaRh81d
KillMode=control-group
 garlick@picl0:~$ sudo flux exec -r 1 systemctl --user show --property SendSIGKILL shell-1-f3L6VaRh81d
SendSIGKILL=yes

[garlick]

I'll go ahead and set MWP on this one.