Frank
a34322aeb0
The managed-skill library was regenerated: 94 narrow per-incident skills
replaced by 25 broader ones (radiance-*, llamacpp-*, homepage-*, fleet/
endpoint ops). Git renders this as mass delete + mass add because whole
directories changed name; old content stays in history at 12c5e88.
Surface check: the 25 on disk are exactly the skills the harness advertises
this session.
47 lines
1.9 KiB
Markdown
47 lines
1.9 KiB
Markdown
---
|
|
name: detached-fleet-job-launch
|
|
description: Launch a long-running remote job on Dank fleet hosts so it survives ssh/logind teardown
|
|
---
|
|
|
|
# Detached long-running jobs on Dank fleet hosts
|
|
|
|
## Law
|
|
`ssh host 'setsid nohup CMD & '` is NOT detached. systemd-logind deactivates
|
|
`session-N.scope` when the ssh command returns and SIGKILLs the whole cgroup:
|
|
no journal entry, no OOM trace, python block-buffers stdout so the log stays
|
|
0 bytes. setsid/nohup do not escape the cgroup.
|
|
|
|
## Canonical launch
|
|
```
|
|
ssh host 'export XDG_RUNTIME_DIR=/run/user/$(id -u); \
|
|
systemd-run --user --collect --unit=NAME \
|
|
-p SendSIGKILL=no \
|
|
-p StandardOutput=append:/home/dank/logs/NAME.log \
|
|
-p StandardError=append:/home/dank/logs/NAME.log \
|
|
sh -c "CMD -u; echo \$? > /home/dank/logs/NAME.rc"'
|
|
```
|
|
- transient user unit = outside every session scope; survives ssh teardown
|
|
(requires `Linger=yes`, true for `dank` on both hosts)
|
|
- `echo $? > …rc` inside the unit = completion sentinel the unit itself writes
|
|
- `-u` / unbuffered output so the log is live
|
|
- verify launch in the same ssh call: `systemctl --user is-active NAME.service`
|
|
|
|
## Watcher (one async job, never poll)
|
|
```
|
|
ssh host 'while [ ! -f /home/dank/logs/NAME.rc ]; do sleep 60; done; \
|
|
echo RC=$(cat /home/dank/logs/NAME.rc); tail -25 /home/dank/logs/NAME.log'
|
|
```
|
|
Run it async with timeout 0; the result is injected on completion.
|
|
|
|
## Forensics when a detached job died
|
|
1. `ps -p PID` — gone.
|
|
2. Log 0 bytes → buffered stdout lost at SIGKILL.
|
|
3. `journalctl --since … | grep -E "session-[0-9]+.*(Deactivated|Removed session)"` —
|
|
timestamps matching the death window confirm the scope kill (vs OOM, which
|
|
leaves kernel lines).
|
|
|
|
## Notes
|
|
- zsh remote shells: `echo ===X===` fails (`=cmd` expansion); quote sentinels.
|
|
- The HF upload of 2026-10-08 (113.7 GiB, hf_xet) died this way ~2 min in;
|
|
relaunch as above succeeded.
|