--- type: reference title: Fleet Ops Playbook description: Incident-shaped fleet procedures — LLM endpoint down with an abandoned GPU holder, scratch vLLM boot windows (stop-first law), iterating on a fleet service before commit, incident-window reconstruction, and waking a sleeping host. tags: [operations, incidents, gpu, systemd, wol, forensics] timestamp: 2026-10-05T00:00:00-07:00 --- # Fleet Ops Playbook ## Endpoint down, connection refused — GPU holder reclaim Use when `:7770`/`:7771` gives **connection refused** (not a timeout) and the service was not crashed. Applies to `frank-medium` (dankserver, root unit), `frank-small` (dankclaw, root unit), `frank-large` (dankclaw, rootless Quadlet). **1. Classify the stop before touching anything** ```bash ssh dankserver "systemctl is-enabled frank-medium; systemctl is-active frank-medium; \ systemctl show frank-medium -p LoadState,UnitFileState,ActiveState,SubState,NRestarts" ls -l /etc/systemd/system/.service # symlink to repo = unmasked; -> /dev/null = masked ``` `inactive` + `enabled` + unmasked + `NRestarts=0` → **intentional stop**, never restored. No crash; do not look for segfaults. `NRestarts>0` / `activating` → different path; read the journal first. **2. Are the GPUs actually free?** A check, never an assumption: ```bash rocm-smi --showmeminfo vram | grep Used # idle baseline ~60 MiB/card ps -eo pid,comm,etime,args | grep -Ei 'llama|bench' | grep -v grep ``` Cards idle → `systemctl start ` and verify (step 5). Cards busy with no prod unit → get the holder, and compare its bytes against the service's requirement before concluding a start would even fit (frank-medium 35:30 needs ~30.8 GiB ROCm0 / ~28.9 GiB ROCm1; a stray holding 15+17 GiB blocks the restart on allocation, not config). **3. Identify the holder and prove it is abandoned** ```bash ps -o pid,ppid,user,lstart,etime,comm -p sudo readlink /proc//cwd /proc//exe tr '\0' ' ' < /proc//cmdline # port, ctx, split sudo ls -l /proc//fd # where stdout/stderr go sudo ss -tnp | grep awk '{print $14+$15}' /proc//stat; sleep 10; awk '{print $14+$15}' /proc//stat ``` Abandoned **only if all** hold: utime+stime **identical** across the 10 s sample (frozen CPU, not just low); **zero** established connections; bind scope `127.0.0.1` (loopback scratch, invisible to endpoint monitors); log file (fd 1/2 target) last written long ago, last lines showing success not a hang. Active CPU or live connections → someone's running session. Stop and ask. **4. Reclaim** ```bash cp -a .preserved-YYYYMMDD # evidence outlives the kill kill # SIGTERM first; exits <1 s rocm-smi --showmeminfo vram | grep Used # expect ~60 MiB baseline ``` Never `pkill -f` a pattern present in your own command line. **5. Restore and verify — health 200 alone is NOT proof** ```bash sudo systemctl start for i in $(seq 1 60); do curl -s -m 5 -o /dev/null -w '%{http_code}\n' \ http://127.0.0.1:/health; sleep 10; done # 503 while loading -> 200 sudo tr '\0' ' ' < /proc//cmdline | grep -oE '\-\-(tensor-split|split-mode|ctx-size|port|spec-type|parallel|flash-attn)' ``` Prove with a **real completion** from off-host — content, reasoning, spec counters: ```bash curl -sS -m 180 http://:/v1/chat/completions -H 'Content-Type: application/json' \ -d '{"model":"","messages":[{"role":"user","content":"Reply with exactly: OK"}],"max_tokens":64,"temperature":0}' # choices[0].message.content / .reasoning_content / timings.draft_n, draft_n_accepted ``` Record the runtime fingerprint (` --version` → `b, commit `) and confirm it is the pinned runtime. Append a dated entry to `serverAppConfigs-docs/log.md`, run `./tests/run.sh`, commit, push (origin is the local Gitea on Dankclaw — push allowed). **Gotchas seen here:** dankserver's login shell is ZSH — never start an unquoted word with `=` in an ssh one-liner (`echo ===` aborts the chain). A freshly restarted endpoint under live agent-queue backlog may return empty bodies/timeouts for minutes while `/health` is 200 — re-probe with a long timeout. First request after a halogen (`frank-large`) restart can go silent 45–60 s and self-recover; do not restart over it. Rootless `frank-large`: read logs **without** sudo (per-user storage). If the restore never happened, the sudo journal names who and when: `journalctl -b | grep sudo`. ## Scratch vLLM boot on a fleet GPU host — stop-first law Any scratch/experimental vLLM container on a host that also runs a production LLM service (`frank-primary`, `frank-large`, `frank-small`) MUST stop the production unit first, and the window script must verify it. Two independent failure modes if skipped: 1. **Expert-offload planner dies on host RAM.** `vllm/model_executor/offloader/expert_plan.py` reads `MemAvailable` at plan time. A production rank worker holds tens of GiB of unevictable pinned expert rows (frank-primary: ~76 GiB × 2 ranks), so a beside-prod boot sees ~86 GiB instead of ~239 and dies at plan time: `ExpertPlanError: host holds 5113 of 12096 swappable experts per rank`. Looks like a planner bug; is pure RAM contention. Signature: container exits within ~60 s, after `Resolved architecture: …` but before worker init; `free -g` showing `available ≈ 86` while prod runs is the smoking gun. 2. **GPU ownership.** Two vLLM containers cannot share the GPUs regardless (the fleet's benchmarking law). Scratch ports (`:7771`/`:7773`) are fine only once the prod unit is down. Window skeleton (dankserver): ```bash sudo systemctl stop frank-primary.service sudo systemctl mask --runtime frank-primary.service # relinked unit: plain mask refuses; runtime mask self-clears at reboot sleep 3 podman ps --format "{{.Names}}" | grep -q frank-primary-glm && { echo "prod still running - abort"; exit 1; } # ... launch scratch container, watch logs ... podman rm -f # stuck vLLM ignores SIGTERM sudo systemctl unmask --runtime frank-primary.service sudo systemctl reset-failed frank-primary.service sudo systemctl start frank-primary.service # restore oracle is a CONJUNCTION: health 200 AND is-active active AND is-enabled enabled AND no /run/systemd/system mask symlink ``` The deploy restores the unit after bench windows, so stop is sufficient there; `mask --runtime` is for scratch boots that must survive a stray restart. Always end a window by re-checking the conjunction before reporting — if a restore after failure doesn't happen automatically, prod stays DOWN. See the repo's benchmarking policy (AGENTS.md) and [[LLM Bench Windows and Measurement]]. ## Iterating on a fleet service before commit The repo's contract is git → `deploy.sh` → host, but live testing needs the new file on the host first, and the naive way (copy into the host checkout) blocks the very deploy that would install it. The loop: 1. Edit locally; run `./tests/run.sh`. 2. Ship working files **for testing**: `cat local/file | ssh 'cat > /home/dank/serverAppConfigs/'` (then `chmod +x` if executed — `cat >`/scp drop the exec bit and a container entrypoint fails status 126, "found but not executable"). 3. Test against the live service; iterate. 4. **Before committing**, prove the host copy is byte-identical to what you are about to commit (`sha256sum` both ends) — otherwise step 6 loses work. 5. Commit and push. 6. Clean the host checkout so the deploy can pull: untracked hand copies → `mv` to `/tmp/handcopy/`; modified tracked files → `git checkout -- ` (safe only because of step 4); confirm `git status --short` prints nothing. 7. `./scripts/deploy.sh --service `. Traps, in the order they bite: the **pull aborts on hand copies** ("untracked working tree files would be overwritten by merge" — exactly the file you shipped); `git checkout --` silently eats uncommitted work without step 4; a runtime may **mount files straight out of the checkout** (`dankserver/frank-medium/**`, `shared/grafana/provisioning/**`) so a hand copy takes effect at next container restart with no deploy; root-owned files from a container write need `chown` back before git operations; and **a restart is not a verification** — verify the *specific* claim (new code path's own output, file checksum on the host, the API serving the resource). Why not keep a scratch copy on the host? The deployed artifact must be the committed one. Probes may live anywhere; runtime state lives in git. ## Incident window reconstruction Use when the question is **"what was running when it broke?"** and the answer must come from artifacts, not recollection. Three independent clocks, cross-checked: 1. **Per-run duration** — scripts wrap commands in `time` → `real 4m19.129s`. 2. **Log mtime** — last write of the log. 3. **Journal first-failure instant** — first error line from the affected service. `start ≈ mtime − real` for the last run in a log; each run nests inside the previous one's window; the run whose `[start, end]` contains the journal's first-failure instant is the answer. ```bash ssh 'ls -l --time-style=+%m-%d_%H:%M:%S /*.log | awk "{print \$6, \$7}"' ssh 'grep -ah "^real" /*.log' ssh 'grep -aE "^####|^\+ /path/to/binary" /.log | sed "s/\(.\{130\}\)/\1\n /g"' ssh 'journalctl --user --since " " | grep -a "invalid internal status" | head -2' ``` Traps (all hit for real): `/tmp` does not survive a reboot — launch experiment scripts from a persistent path (`/home/dank/models/logs/.log` survived while `/tmp/x4*.sh` died); a log's mtime is not proof a run finished (truncation with the fd still held keeps growing); long command lines are display-truncated — wrap before reading or you report parameters never used; off-by-one filenames (`x4b.log` vs `x4b-experiment.log`) silently skip evidence — list the directory and match exactly; **overlap is a hypothesis, not an observation** unless a sampler proves it; do not invent the cause — reconstruct the *window* and keep the *mechanism* separate (`journalctl -k | grep -i oom`); publish "trigger not established" when the proof is absent. Fleet-specific: container healthchecks shell out to the **podman CLI**, so the first `invalid internal status … could not find any running process` in the user journal is an early, per-minute indicator of rootless-podman state death — it timestamps the incident to within one healthcheck interval (how the 18:58:30 Dankclaw break was pinned to a bench still loading). Worked example (2026-09-16, Dankclaw): log mtimes x4e-run 18:58:29 / x4f-run 18:59:56; durations E1 `real 4m53.946s`, F `real 5m24.785s` (exit 143 = killed); journal first failure 18:58:30 ⇒ **F** was in flight (launched 18:54:31, inside its window). ## Wake-on-LAN for a sleeping fleet host A fleet host is asleep (100% ping loss; `ip neigh show ` shows STALE or FAILED) and you need it up — typically the FrankBook-Pro Mac mini (192.168.0.112, **MAC `de:6b:6d:c1:52:23`**, sleeps and breaks the e2e `11-admin` spec). - **Find the MAC**: `ip neigh show ` — even a STALE entry carries it; fallback Pi-hole/DHCP leases or fleet docs. - **Send the packet**: `6×0xFF + 16×MAC` = 102 bytes, UDP to port **9** (and 7), broadcast. The dev workstation has **two interfaces on 192.168.0.0/24** (Wi-Fi + wired USB-Ethernet, default route metric 100) — a plain `sendto("255.255.255.255", 9)` leaves only the default-route interface. Bind explicitly to each interface's local IP and target the subnet broadcast `192.168.0.255` as well as `255.255.255.255`; burst ~15 packets per interface ~0.2 s apart. `wakeonlan` is not installed; use a raw UDP socket with `SO_BROADCAST=1`: ```python import socket, time MAC = "de:6b:6d:c1:52:23" mac = bytes(int(x,16) for x in MAC.split(":")) pkt = b"\xff"*6 + mac*16 # 102 bytes for local, dest in [("192.168.0.83","192.168.0.255"),("192.168.0.104","192.168.0.255"), ("192.168.0.83","255.255.255.255"),("192.168.0.104","255.255.255.255")]: s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) s.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1) s.bind((local, 0)) for _ in range(15): s.sendto(pkt,(dest,9)); s.sendto(pkt,(dest,7)); time.sleep(0.2) s.close() ``` - **Poll for the wake**: a Mac mini takes **30–90 s**; `ping -c 1 -W 1` every 5 s for ~2–3 min before concluding failure. - **If it does not wake** (the common case): macOS honors WoL only when BOTH are enabled on the mini — **System Settings → Battery → Options → "Wake for network access"** = ON, and the active port's **"Enable Wake for LAN"** (Network → [interface] → Details). If either is off, no packet from any source wakes it, and you cannot change it while asleep. A full power-off (smart plug cycled) ignores WoL entirely. Ask the user to toggle the setting, then re-fire. ## Related - [[LLM Bench Windows and Measurement]] — the bench windows this playbook arms - [[Remote Job Runner]] — supervising the long jobs that create these windows - [[Container Host Reachability]] — the pasta port gate at container create - [[Quadlet Lifecycle]] — the unit layer behind stop/mask/start - [[Model Swap and Rollback]] — swap-time incidents and rollbacks