0 Fork 0

Files

T

Frank a34322aeb0 refactor(skills): consolidate managed skill library 94 -> 25

The managed-skill library was regenerated: 94 narrow per-incident skills
replaced by 25 broader ones (radiance-*, llamacpp-*, homepage-*, fleet/
endpoint ops). Git renders this as mass delete + mass add because whole
directories changed name; old content stays in history at 12c5e88.

Surface check: the 25 on disk are exactly the skills the harness advertises
this session.

2026-10-09 13:17:48 -07:00

1.9 KiB

Raw Blame History

name, description

name	description
detached-fleet-job-launch	Launch a long-running remote job on Dank fleet hosts so it survives ssh/logind teardown

Detached long-running jobs on Dank fleet hosts

Law

ssh host 'setsid nohup CMD & ' is NOT detached. systemd-logind deactivates session-N.scope when the ssh command returns and SIGKILLs the whole cgroup: no journal entry, no OOM trace, python block-buffers stdout so the log stays 0 bytes. setsid/nohup do not escape the cgroup.

Canonical launch

ssh host 'export XDG_RUNTIME_DIR=/run/user/$(id -u); \
  systemd-run --user --collect --unit=NAME \
    -p SendSIGKILL=no \
    -p StandardOutput=append:/home/dank/logs/NAME.log \
    -p StandardError=append:/home/dank/logs/NAME.log \
    sh -c "CMD -u; echo \$? > /home/dank/logs/NAME.rc"'

transient user unit = outside every session scope; survives ssh teardown (requires Linger=yes, true for dank on both hosts)
echo $? > …rc inside the unit = completion sentinel the unit itself writes
-u / unbuffered output so the log is live
verify launch in the same ssh call: systemctl --user is-active NAME.service

Watcher (one async job, never poll)

ssh host 'while [ ! -f /home/dank/logs/NAME.rc ]; do sleep 60; done; \
  echo RC=$(cat /home/dank/logs/NAME.rc); tail -25 /home/dank/logs/NAME.log'

Run it async with timeout 0; the result is injected on completion.

Forensics when a detached job died

ps -p PID — gone.
Log 0 bytes → buffered stdout lost at SIGKILL.
journalctl --since … | grep -E "session-[0-9]+.*(Deactivated|Removed session)" — timestamps matching the death window confirm the scope kill (vs OOM, which leaves kernel lines).

Notes

zsh remote shells: echo ===X=== fails (=cmd expansion); quote sentinels.
The HF upload of 2026-10-08 (113.7 GiB, hf_xet) died this way ~2 min in; relaunch as above succeeded.

1.9 KiB Raw Blame History