command-center/agent/managed-skills/detached-fleet-job-launch/SKILL.md

name, description

name	description
detached-fleet-job-launch	Launch long-running jobs (fleet hosts or local workstation) so they survive ssh/logind/harness teardown; includes worker takeover law for queue daemons

Detached long-running jobs on Dank fleet hosts

Law

ssh host 'setsid nohup CMD & ' is NOT detached. systemd-logind deactivates session-N.scope when the ssh command returns and SIGKILLs the whole cgroup: no journal entry, no OOM trace, python block-buffers stdout so the log stays 0 bytes. setsid/nohup do not escape the cgroup.

Canonical launch (remote)

ssh host 'export XDG_RUNTIME_DIR=/run/user/$(id -u); \
  systemd-run --user --collect --unit=NAME \
    -p SendSIGKILL=no \
    -p StandardOutput=append:/home/dank/logs/NAME.log \
    -p StandardError=append:/home/dank/logs/NAME.log \
    sh -c "CMD -u; echo \$? > /home/dank/logs/NAME.rc"'

• transient user unit = outside every session scope; survives ssh teardown (requires Linger=yes, true for dank on both hosts)
• echo $? > …rc inside the unit = completion sentinel the unit itself writes
• -u / unbuffered output so the log is live
• verify launch in the same ssh call: systemctl --user is-active NAME.service

Local workstation variant (verified 2026-10-10, blankpad)

Harness background jobs get tree-killed at timeout — setsid/nohup children die too (survives the process-group kill, not the tree walk). Same cure as remote, run locally:

loginctl enable-linger $(whoami)   # default Linger=no kills user units at session end
XDG_RUNTIME_DIR=/run/user/$(id -u) systemd-run --user --collect --unit=NAME \
  -p StandardOutput=append:/abs/path/job.log \
  -p StandardError=append:/abs/path/job.log \
  /full/path/to/binary args...
systemctl --user is-active NAME.service

Gotchas:

• bash-tool env lacks XDG_RUNTIME_DIR → set it inline.
• --collect GCs the unit when it exits — after a crash-exit, systemctl --user restart NAME fails "Unit not found"; relaunch with systemd-run. Redeploy code = stop (not restart) + relaunch.
• point StandardOutput at the job's canonical log file (what the UI/SSE tails), not a sidecar — observed drift: worker.log vs per-run log.txt diverged.
• foreground (cmd &) and (cmd &) subshells die with the tool call's group.

Worker takeover law (queue daemons)

A supervisor killed mid-run leaves queue.json.active set ⇒ every relaunching worker honors "one run at a time" and exits ⇒ silent queue deadlock. Cure:

• heartbeat: run dir worker.pid, rewritten every ~5s while running, removed on completion;
• claim rule: active set + pid dead (process.kill(pid,0) ESRCH; EPERM = alive) ⇒ resume; pid alive ⇒ exit.
• resume must be append-only + completed-key skip so takeover is cost-free (observed: 386 completed samples skipped, zero re-requests).

Watcher (one async job, never poll)

ssh host 'while [ ! -f /home/dank/logs/NAME.rc ]; do sleep 60; done; \
  echo RC=$(cat /home/dank/logs/NAME.rc); tail -25 /home/dank/logs/NAME.log'

Run it async with timeout 0; the result is injected on completion.

Forensics when a detached job died

1. ps -p PID — gone.
2. Log 0 bytes → buffered stdout lost at SIGKILL.
3. journalctl --since … | grep -E "session-[0-9]+.*(Deactivated|Removed session)" — timestamps matching the death window confirm the scope kill (vs OOM, which leaves kernel lines).

Notes

• zsh remote shells: echo ===X=== fails (=cmd expansion); quote sentinels.
• The HF upload of 2026-08 (113.7 GiB, hf_xet) died this way ~2 min in; relaunch as above succeeded.
• Frank's machine: systemctl --user terminate is not a verb — use stop.