live: heartbeat sync 2026-10-18T16:47:59-07:00 (Refs #11)

agent/managed-skills/detached-fleet-job-launch/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: detached-fleet-job-launch
-description: Launch a long-running remote job on Dank fleet hosts so it survives ssh/logind teardown
+description: Launch long-running jobs (fleet hosts or local workstation) so they survive ssh/logind/harness teardown; includes worker takeover law for queue daemons
 ---
 
 # Detached long-running jobs on Dank fleet hosts
@@ -11,7 +11,7 @@ description: Launch a long-running remote job on Dank fleet hosts so it survives
 no journal entry, no OOM trace, python block-buffers stdout so the log stays
 0 bytes. setsid/nohup do not escape the cgroup.
 
-## Canonical launch
+## Canonical launch (remote)
 ```
 ssh host 'export XDG_RUNTIME_DIR=/run/user/$(id -u); \
   systemd-run --user --collect --unit=NAME \
@@ -26,6 +26,37 @@ ssh host 'export XDG_RUNTIME_DIR=/run/user/$(id -u); \
 - `-u` / unbuffered output so the log is live
 - verify launch in the same ssh call: `systemctl --user is-active NAME.service`
 
+## Local workstation variant (verified 2026-10-10, blankpad)
+Harness background jobs get **tree-killed** at timeout — setsid/nohup children
+die too (survives the process-group kill, not the tree walk). Same cure as
+remote, run locally:
+```
+loginctl enable-linger $(whoami)   # default Linger=no kills user units at session end
+XDG_RUNTIME_DIR=/run/user/$(id -u) systemd-run --user --collect --unit=NAME \
+  -p StandardOutput=append:/abs/path/job.log \
+  -p StandardError=append:/abs/path/job.log \
+  /full/path/to/binary args...
+systemctl --user is-active NAME.service
+```
+Gotchas:
+- bash-tool env lacks `XDG_RUNTIME_DIR` → set it inline.
+- `--collect` GCs the unit when it **exits** — after a crash-exit,
+  `systemctl --user restart NAME` fails "Unit not found"; relaunch with
+  `systemd-run`. Redeploy code = `stop` (not `restart`) + relaunch.
+- point StandardOutput at the job's canonical log file (what the UI/SSE tails),
+  not a sidecar — observed drift: worker.log vs per-run log.txt diverged.
+- foreground `(cmd &)` and `(cmd &)` subshells die with the tool call's group.
+
+## Worker takeover law (queue daemons)
+A supervisor killed mid-run leaves `queue.json.active` set ⇒ every relaunching
+worker honors "one run at a time" and exits ⇒ silent queue deadlock. Cure:
+- heartbeat: run dir `worker.pid`, rewritten every ~5s while running, removed
+  on completion;
+- claim rule: `active` set + pid dead (`process.kill(pid,0)` ESRCH; EPERM =
+  alive) ⇒ resume; pid alive ⇒ exit.
+- resume must be append-only + completed-key skip so takeover is cost-free
+  (observed: 386 completed samples skipped, zero re-requests).
+
 ## Watcher (one async job, never poll)
 ```
 ssh host 'while [ ! -f /home/dank/logs/NAME.rc ]; do sleep 60; done; \
@@ -42,5 +73,6 @@ Run it async with timeout 0; the result is injected on completion.
 
 ## Notes
 - zsh remote shells: `echo ===X===` fails (`=cmd` expansion); quote sentinels.
-- The HF upload of 2026-10-08 (113.7 GiB, hf_xet) died this way ~2 min in;
+- The HF upload of 2026-08 (113.7 GiB, hf_xet) died this way ~2 min in;
   relaunch as above succeeded.
+- Frank's machine: `systemctl --user terminate` is not a verb — use `stop`.