79 lines
3.6 KiB
Markdown
79 lines
3.6 KiB
Markdown
---
|
|
name: detached-fleet-job-launch
|
|
description: Launch long-running jobs (fleet hosts or local workstation) so they survive ssh/logind/harness teardown; includes worker takeover law for queue daemons
|
|
---
|
|
|
|
# Detached long-running jobs on Dank fleet hosts
|
|
|
|
## Law
|
|
`ssh host 'setsid nohup CMD & '` is NOT detached. systemd-logind deactivates
|
|
`session-N.scope` when the ssh command returns and SIGKILLs the whole cgroup:
|
|
no journal entry, no OOM trace, python block-buffers stdout so the log stays
|
|
0 bytes. setsid/nohup do not escape the cgroup.
|
|
|
|
## Canonical launch (remote)
|
|
```
|
|
ssh host 'export XDG_RUNTIME_DIR=/run/user/$(id -u); \
|
|
systemd-run --user --collect --unit=NAME \
|
|
-p SendSIGKILL=no \
|
|
-p StandardOutput=append:/home/dank/logs/NAME.log \
|
|
-p StandardError=append:/home/dank/logs/NAME.log \
|
|
sh -c "CMD -u; echo \$? > /home/dank/logs/NAME.rc"'
|
|
```
|
|
- transient user unit = outside every session scope; survives ssh teardown
|
|
(requires `Linger=yes`, true for `dank` on both hosts)
|
|
- `echo $? > …rc` inside the unit = completion sentinel the unit itself writes
|
|
- `-u` / unbuffered output so the log is live
|
|
- verify launch in the same ssh call: `systemctl --user is-active NAME.service`
|
|
|
|
## Local workstation variant (verified 2026-10-10, blankpad)
|
|
Harness background jobs get **tree-killed** at timeout — setsid/nohup children
|
|
die too (survives the process-group kill, not the tree walk). Same cure as
|
|
remote, run locally:
|
|
```
|
|
loginctl enable-linger $(whoami) # default Linger=no kills user units at session end
|
|
XDG_RUNTIME_DIR=/run/user/$(id -u) systemd-run --user --collect --unit=NAME \
|
|
-p StandardOutput=append:/abs/path/job.log \
|
|
-p StandardError=append:/abs/path/job.log \
|
|
/full/path/to/binary args...
|
|
systemctl --user is-active NAME.service
|
|
```
|
|
Gotchas:
|
|
- bash-tool env lacks `XDG_RUNTIME_DIR` → set it inline.
|
|
- `--collect` GCs the unit when it **exits** — after a crash-exit,
|
|
`systemctl --user restart NAME` fails "Unit not found"; relaunch with
|
|
`systemd-run`. Redeploy code = `stop` (not `restart`) + relaunch.
|
|
- point StandardOutput at the job's canonical log file (what the UI/SSE tails),
|
|
not a sidecar — observed drift: worker.log vs per-run log.txt diverged.
|
|
- foreground `(cmd &)` and `(cmd &)` subshells die with the tool call's group.
|
|
|
|
## Worker takeover law (queue daemons)
|
|
A supervisor killed mid-run leaves `queue.json.active` set ⇒ every relaunching
|
|
worker honors "one run at a time" and exits ⇒ silent queue deadlock. Cure:
|
|
- heartbeat: run dir `worker.pid`, rewritten every ~5s while running, removed
|
|
on completion;
|
|
- claim rule: `active` set + pid dead (`process.kill(pid,0)` ESRCH; EPERM =
|
|
alive) ⇒ resume; pid alive ⇒ exit.
|
|
- resume must be append-only + completed-key skip so takeover is cost-free
|
|
(observed: 386 completed samples skipped, zero re-requests).
|
|
|
|
## Watcher (one async job, never poll)
|
|
```
|
|
ssh host 'while [ ! -f /home/dank/logs/NAME.rc ]; do sleep 60; done; \
|
|
echo RC=$(cat /home/dank/logs/NAME.rc); tail -25 /home/dank/logs/NAME.log'
|
|
```
|
|
Run it async with timeout 0; the result is injected on completion.
|
|
|
|
## Forensics when a detached job died
|
|
1. `ps -p PID` — gone.
|
|
2. Log 0 bytes → buffered stdout lost at SIGKILL.
|
|
3. `journalctl --since … | grep -E "session-[0-9]+.*(Deactivated|Removed session)"` —
|
|
timestamps matching the death window confirm the scope kill (vs OOM, which
|
|
leaves kernel lines).
|
|
|
|
## Notes
|
|
- zsh remote shells: `echo ===X===` fails (`=cmd` expansion); quote sentinels.
|
|
- The HF upload of 2026-08 (113.7 GiB, hf_xet) died this way ~2 min in;
|
|
relaunch as above succeeded.
|
|
- Frank's machine: `systemctl --user terminate` is not a verb — use `stop`.
|