serverAppConfigs/serverAppConfigs-docs/Operations/Fleet Ops Playbook.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	Fleet Ops Playbook	Incident-shaped fleet procedures — LLM endpoint down with an abandoned GPU holder, scratch vLLM boot windows (stop-first law), iterating on a fleet service before commit, incident-window reconstruction, and waking a sleeping host.	operations incidents gpu systemd wol forensics	2026-10-05T00:00:00-07:00

Fleet Ops Playbook

Endpoint down, connection refused — GPU holder reclaim

Use when :7770/:7771 gives connection refused (not a timeout) and the service was not crashed. Applies to frank-medium (dankserver, root unit), frank-zealot (dankclaw, root unit), frank-dragoon (dankclaw, rootless Quadlet).

1. Classify the stop before touching anything

ssh dankserver "systemctl is-enabled frank-medium; systemctl is-active frank-medium; \
  systemctl show frank-medium -p LoadState,UnitFileState,ActiveState,SubState,NRestarts"
ls -l /etc/systemd/system/<svc>.service      # symlink to repo = unmasked; -> /dev/null = masked

inactive + enabled + unmasked + NRestarts=0 → intentional stop, never restored. No crash; do not look for segfaults. NRestarts>0 / activating → different path; read the journal first.

2. Are the GPUs actually free? A check, never an assumption:

rocm-smi --showmeminfo vram | grep Used      # idle baseline ~60 MiB/card
ps -eo pid,comm,etime,args | grep -Ei 'llama|bench' | grep -v grep

Cards idle → systemctl start <svc> and verify (step 5). Cards busy with no prod unit → get the holder, and compare its bytes against the service's requirement before concluding a start would even fit (frank-medium 35:30 needs ~30.8 GiB ROCm0 / ~28.9 GiB ROCm1; a stray holding 15+17 GiB blocks the restart on allocation, not config).

3. Identify the holder and prove it is abandoned

ps -o pid,ppid,user,lstart,etime,comm -p <pid>
sudo readlink /proc/<pid>/cwd /proc/<pid>/exe
tr '\0' ' ' < /proc/<pid>/cmdline                     # port, ctx, split
sudo ls -l /proc/<pid>/fd                             # where stdout/stderr go
sudo ss -tnp | grep <port>
awk '{print $14+$15}' /proc/<pid>/stat; sleep 10; awk '{print $14+$15}' /proc/<pid>/stat

Abandoned only if all hold: utime+stime identical across the 10 s sample (frozen CPU, not just low); zero established connections; bind scope 127.0.0.1 (loopback scratch, invisible to endpoint monitors); log file (fd 1/2 target) last written long ago, last lines showing success not a hang. Active CPU or live connections → someone's running session. Stop and ask.

4. Reclaim

cp -a <logfile> <logfile>.preserved-YYYYMMDD   # evidence outlives the kill
kill <pid>                                     # SIGTERM first; exits <1 s
rocm-smi --showmeminfo vram | grep Used        # expect ~60 MiB baseline

Never pkill -f a pattern present in your own command line.

5. Restore and verify — health 200 alone is NOT proof

sudo systemctl start <svc>
for i in $(seq 1 60); do curl -s -m 5 -o /dev/null -w '%{http_code}\n' \
  http://127.0.0.1:<port>/health; sleep 10; done   # 503 while loading -> 200
sudo tr '\0' ' ' < /proc/<MainPID>/cmdline | grep -oE '\-\-(tensor-split|split-mode|ctx-size|port|spec-type|parallel|flash-attn)'

Prove with a real completion from off-host — content, reasoning, spec counters:

curl -sS -m 180 http://<host>:<port>/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"<alias>","messages":[{"role":"user","content":"Reply with exactly: OK"}],"max_tokens":64,"temperature":0}'
# choices[0].message.content / .reasoning_content / timings.draft_n, draft_n_accepted

Record the runtime fingerprint (<bin> --version → b<build>, commit <sha>) and confirm it is the pinned runtime. Append a dated entry to serverAppConfigs-docs/log.md, run ./tests/run.sh, commit, push (origin is the local Gitea on Dankclaw — push allowed).

Gotchas seen here: dankserver's login shell is ZSH — never start an unquoted word with = in an ssh one-liner (echo === aborts the chain). A freshly restarted endpoint under live agent-queue backlog may return empty bodies/timeouts for minutes while /health is 200 — re-probe with a long timeout. First request after a halogen (frank-dragoon) restart can go silent 45–60 s and self-recover; do not restart over it. Rootless frank-dragoon: read logs without sudo (per-user storage). If the restore never happened, the sudo journal names who and when: journalctl -b | grep sudo.

Scratch vLLM boot on a fleet GPU host — stop-first law

Any scratch/experimental vLLM container on a host that also runs a production LLM service (frank-archon, frank-dragoon, frank-zealot) MUST stop the production unit first, and the window script must verify it. Two independent failure modes if skipped:

1. Expert-offload planner dies on host RAM. vllm/model_executor/offloader/expert_plan.py reads MemAvailable at plan time. A production rank worker holds tens of GiB of unevictable pinned expert rows (frank-archon: ~76 GiB × 2 ranks), so a beside-prod boot sees ~86 GiB instead of ~239 and dies at plan time: ExpertPlanError: host holds 5113 of 12096 swappable experts per rank. Looks like a planner bug; is pure RAM contention. Signature: container exits within ~60 s, after Resolved architecture: … but before worker init; free -g showing available ≈ 86 while prod runs is the smoking gun.
2. GPU ownership. Two vLLM containers cannot share the GPUs regardless (the fleet's benchmarking law). Scratch ports (:7771/:7773) are fine only once the prod unit is down.

Window skeleton (dankserver):

sudo systemctl stop frank-archon.service
sudo systemctl mask --runtime frank-archon.service  # relinked unit: plain mask refuses; runtime mask self-clears at reboot
sleep 3
podman ps --format "{{.Names}}" | grep -q frank-archon-glm && { echo "prod still running - abort"; exit 1; }
# ... launch scratch container, watch logs ...
podman rm -f <scratch>                                # stuck vLLM ignores SIGTERM
sudo systemctl unmask --runtime frank-archon.service
sudo systemctl reset-failed frank-archon.service
sudo systemctl start frank-archon.service
# restore oracle is a CONJUNCTION: health 200 AND is-active active AND is-enabled enabled AND no /run/systemd/system mask symlink

The deploy restores the unit after bench windows, so stop is sufficient there; mask --runtime is for scratch boots that must survive a stray restart. Always end a window by re-checking the conjunction before reporting — if a restore after failure doesn't happen automatically, prod stays DOWN. See the repo's benchmarking policy (AGENTS.md) and LLM Bench Windows and Measurement.

Iterating on a fleet service before commit

The repo's contract is git → deploy.sh → host, but live testing needs the new file on the host first, and the naive way (copy into the host checkout) blocks the very deploy that would install it. The loop:

1. Edit locally; run ./tests/run.sh.
2. Ship working files for testing: cat local/file | ssh <host> 'cat > /home/dank/serverAppConfigs/<path>' (then chmod +x if executed — cat >/scp drop the exec bit and a container entrypoint fails status 126, "found but not executable").
3. Test against the live service; iterate.
4. Before committing, prove the host copy is byte-identical to what you are about to commit (sha256sum both ends) — otherwise step 6 loses work.
5. Commit and push.
6. Clean the host checkout so the deploy can pull: untracked hand copies → mv to /tmp/handcopy/; modified tracked files → git checkout -- <paths> (safe only because of step 4); confirm git status --short prints nothing.
7. ./scripts/deploy.sh <target> --service <svc>.

Traps, in the order they bite: the pull aborts on hand copies ("untracked working tree files would be overwritten by merge" — exactly the file you shipped); git checkout -- silently eats uncommitted work without step 4; a runtime may mount files straight out of the checkout (dankserver/frank-medium/**, shared/grafana/provisioning/**) so a hand copy takes effect at next container restart with no deploy; root-owned files from a container write need chown back before git operations; and a restart is not a verification — verify the specific claim (new code path's own output, file checksum on the host, the API serving the resource).

Why not keep a scratch copy on the host? The deployed artifact must be the committed one. Probes may live anywhere; runtime state lives in git.

Incident window reconstruction

Use when the question is "what was running when it broke?" and the answer must come from artifacts, not recollection. Three independent clocks, cross-checked:

1. Per-run duration — scripts wrap commands in time → real 4m19.129s.
2. Log mtime — last write of the log.
3. Journal first-failure instant — first error line from the affected service.

start ≈ mtime − real for the last run in a log; each run nests inside the previous one's window; the run whose [start, end] contains the journal's first-failure instant is the answer.

ssh <host> 'ls -l --time-style=+%m-%d_%H:%M:%S <logdir>/*.log | awk "{print \$6, \$7}"'
ssh <host> 'grep -ah "^real" <logdir>/*.log'
ssh <host> 'grep -aE "^####|^\+ /path/to/binary" <logdir>/<name>.log | sed "s/\(.\{130\}\)/\1\n        /g"'
ssh <host> 'journalctl --user --since "<date> <HH:MM>" | grep -a "invalid internal status" | head -2'

Traps (all hit for real): /tmp does not survive a reboot — launch experiment scripts from a persistent path (/home/dank/models/logs/<tag>.log survived while /tmp/x4*.sh died); a log's mtime is not proof a run finished (truncation with the fd still held keeps growing); long command lines are display-truncated — wrap before reading or you report parameters never used; off-by-one filenames (x4b.log vs x4b-experiment.log) silently skip evidence — list the directory and match exactly; overlap is a hypothesis, not an observation unless a sampler proves it; do not invent the cause — reconstruct the window and keep the mechanism separate (journalctl -k | grep -i oom); publish "trigger not established" when the proof is absent.

Fleet-specific: container healthchecks shell out to the podman CLI, so the first invalid internal status … could not find any running process in the user journal is an early, per-minute indicator of rootless-podman state death — it timestamps the incident to within one healthcheck interval (how the 18:58:30 Dankclaw break was pinned to a bench still loading).

Worked example (2026-09-16, Dankclaw): log mtimes x4e-run 18:58:29 / x4f-run 18:59:56; durations E1 real 4m53.946s, F real 5m24.785s (exit 143 = killed); journal first failure 18:58:30 ⇒ F was in flight (launched 18:54:31, inside its window).

Wake-on-LAN for a sleeping fleet host

A fleet host is asleep (100% ping loss; ip neigh show <ip> shows STALE or FAILED) and you need it up — typically the FrankBook-Pro Mac mini (192.168.0.112, MAC de:6b:6d:c1:52:23, sleeps and breaks the e2e 11-admin spec).

• Find the MAC: ip neigh show <ip> — even a STALE entry carries it; fallback Pi-hole/DHCP leases or fleet docs.
• Send the packet: 6×0xFF + 16×MAC = 102 bytes, UDP to port 9 (and 7), broadcast. The dev workstation has two interfaces on 192.168.0.0/24 (Wi-Fi + wired USB-Ethernet, default route metric 100) — a plain sendto("255.255.255.255", 9) leaves only the default-route interface. Bind explicitly to each interface's local IP and target the subnet broadcast 192.168.0.255 as well as 255.255.255.255; burst ~15 packets per interface ~0.2 s apart. wakeonlan is not installed; use a raw UDP socket with SO_BROADCAST=1:

import socket, time
MAC = "de:6b:6d:c1:52:23"
mac = bytes(int(x,16) for x in MAC.split(":"))
pkt = b"\xff"*6 + mac*16          # 102 bytes
for local, dest in [("192.168.0.83","192.168.0.255"),("192.168.0.104","192.168.0.255"),
                    ("192.168.0.83","255.255.255.255"),("192.168.0.104","255.255.255.255")]:
    s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    s.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
    s.bind((local, 0))
    for _ in range(15):
        s.sendto(pkt,(dest,9)); s.sendto(pkt,(dest,7)); time.sleep(0.2)
    s.close()

• Poll for the wake: a Mac mini takes 30–90 s; ping -c 1 -W 1 every 5 s for ~2–3 min before concluding failure.
• If it does not wake (the common case): macOS honors WoL only when BOTH are enabled on the mini — System Settings → Battery → Options → "Wake for network access" = ON, and the active port's "Enable Wake for LAN" (Network → [interface] → Details). If either is off, no packet from any source wakes it, and you cannot change it while asleep. A full power-off (smart plug cycled) ignores WoL entirely. Ask the user to toggle the setting, then re-fire.

Related

• LLM Bench Windows and Measurement — the bench windows this playbook arms
• Remote Job Runner — supervising the long jobs that create these windows
• Container Host Reachability — the pasta port gate at container create
• Quadlet Lifecycle — the unit layer behind stop/mask/start
• Model Swap and Rollback — swap-time incidents and rollbacks