---
name: detached-fleet-job-launch
description: Launch a long-running remote job on Dank fleet hosts so it survives ssh/logind teardown
---

# Detached long-running jobs on Dank fleet hosts

## Law
`ssh host 'setsid nohup CMD & '` is NOT detached. systemd-logind deactivates
`session-N.scope` when the ssh command returns and SIGKILLs the whole cgroup:
no journal entry, no OOM trace, python block-buffers stdout so the log stays
0 bytes. setsid/nohup do not escape the cgroup.

## Canonical launch
```
ssh host 'export XDG_RUNTIME_DIR=/run/user/$(id -u); \
  systemd-run --user --collect --unit=NAME \
    -p SendSIGKILL=no \
    -p StandardOutput=append:/home/dank/logs/NAME.log \
    -p StandardError=append:/home/dank/logs/NAME.log \
    sh -c "CMD -u; echo \$? > /home/dank/logs/NAME.rc"'
```
- transient user unit = outside every session scope; survives ssh teardown
  (requires `Linger=yes`, true for `dank` on both hosts)
- `echo $? > …rc` inside the unit = completion sentinel the unit itself writes
- `-u` / unbuffered output so the log is live
- verify launch in the same ssh call: `systemctl --user is-active NAME.service`

## Watcher (one async job, never poll)
```
ssh host 'while [ ! -f /home/dank/logs/NAME.rc ]; do sleep 60; done; \
  echo RC=$(cat /home/dank/logs/NAME.rc); tail -25 /home/dank/logs/NAME.log'
```
Run it async with timeout 0; the result is injected on completion.

## Forensics when a detached job died
1. `ps -p PID` — gone.
2. Log 0 bytes → buffered stdout lost at SIGKILL.
3. `journalctl --since … | grep -E "session-[0-9]+.*(Deactivated|Removed session)"` —
   timestamps matching the death window confirm the scope kill (vs OOM, which
   leaves kernel lines).

## Notes
- zsh remote shells: `echo ===X===` fails (`=cmd` expansion); quote sentinels.
- The HF upload of 2026-10-08 (113.7 GiB, hf_xet) died this way ~2 min in;
  relaunch as above succeeded.