serverAppConfigs/serverAppConfigs-docs/Operations/Fleet Ops Playbook.md

---
type: reference
title: Fleet Ops Playbook
description: Incident-shaped fleet procedures — LLM endpoint down with an abandoned GPU holder, scratch vLLM boot windows (stop-first law), iterating on a fleet service before commit, incident-window reconstruction, and waking a sleeping host.
tags: [operations, incidents, gpu, systemd, wol, forensics]
timestamp: 2026-10-05T00:00:00-07:00
---

# Fleet Ops Playbook

## Endpoint down, connection refused — GPU holder reclaim

Use when `:7770`/`:7771` gives **connection refused** (not a timeout) and the
service was not crashed. Applies to `frank-medium` (dankserver, root unit),
`frank-secondary` (dankclaw, root unit), `frank-large` (dankclaw, rootless
Quadlet).

**1. Classify the stop before touching anything**

```bash
ssh dankserver "systemctl is-enabled frank-medium; systemctl is-active frank-medium; \
  systemctl show frank-medium -p LoadState,UnitFileState,ActiveState,SubState,NRestarts"
ls -l /etc/systemd/system/<svc>.service      # symlink to repo = unmasked; -> /dev/null = masked
```

`inactive` + `enabled` + unmasked + `NRestarts=0` → **intentional stop**,
never restored. No crash; do not look for segfaults. `NRestarts>0` /
`activating` → different path; read the journal first.

**2. Are the GPUs actually free?** A check, never an assumption:

```bash
rocm-smi --showmeminfo vram | grep Used      # idle baseline ~60 MiB/card
ps -eo pid,comm,etime,args | grep -Ei 'llama|bench' | grep -v grep
```

Cards idle → `systemctl start <svc>` and verify (step 5). Cards busy with no
prod unit → get the holder, and compare its bytes against the service's
requirement before concluding a start would even fit (frank-medium 35:30
needs ~30.8 GiB ROCm0 / ~28.9 GiB ROCm1; a stray holding 15+17 GiB blocks the
restart on allocation, not config).

**3. Identify the holder and prove it is abandoned**

```bash
ps -o pid,ppid,user,lstart,etime,comm -p <pid>
sudo readlink /proc/<pid>/cwd /proc/<pid>/exe
tr '\0' ' ' < /proc/<pid>/cmdline                     # port, ctx, split
sudo ls -l /proc/<pid>/fd                             # where stdout/stderr go
sudo ss -tnp | grep <port>
awk '{print $14+$15}' /proc/<pid>/stat; sleep 10; awk '{print $14+$15}' /proc/<pid>/stat
```

Abandoned **only if all** hold: utime+stime **identical** across the 10 s
sample (frozen CPU, not just low); **zero** established connections; bind
scope `127.0.0.1` (loopback scratch, invisible to endpoint monitors); log
file (fd 1/2 target) last written long ago, last lines showing success not a
hang. Active CPU or live connections → someone's running session. Stop and
ask.

**4. Reclaim**

```bash
cp -a <logfile> <logfile>.preserved-YYYYMMDD   # evidence outlives the kill
kill <pid>                                     # SIGTERM first; exits <1 s
rocm-smi --showmeminfo vram | grep Used        # expect ~60 MiB baseline
```

Never `pkill -f` a pattern present in your own command line.

**5. Restore and verify — health 200 alone is NOT proof**

```bash
sudo systemctl start <svc>
for i in $(seq 1 60); do curl -s -m 5 -o /dev/null -w '%{http_code}\n' \
  http://127.0.0.1:<port>/health; sleep 10; done   # 503 while loading -> 200
sudo tr '\0' ' ' < /proc/<MainPID>/cmdline | grep -oE '\-\-(tensor-split|split-mode|ctx-size|port|spec-type|parallel|flash-attn)'
```

Prove with a **real completion** from off-host — content, reasoning, spec
counters:

```bash
curl -sS -m 180 http://<host>:<port>/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"<alias>","messages":[{"role":"user","content":"Reply with exactly: OK"}],"max_tokens":64,"temperature":0}'
# choices[0].message.content / .reasoning_content / timings.draft_n, draft_n_accepted
```

Record the runtime fingerprint (`<bin> --version` → `b<build>, commit <sha>`)
and confirm it is the pinned runtime. Append a dated entry to
`serverAppConfigs-docs/log.md`, run `./tests/run.sh`, commit, push (origin is
the local Gitea on Dankclaw — push allowed).

**Gotchas seen here:** dankserver's login shell is ZSH — never start an
unquoted word with `=` in an ssh one-liner (`echo ===` aborts the chain). A
freshly restarted endpoint under live agent-queue backlog may return empty
bodies/timeouts for minutes while `/health` is 200 — re-probe with a long
timeout. First request after a halogen (`frank-large`) restart can go silent
45–60 s and self-recover; do not restart over it. Rootless `frank-large`:
read logs **without** sudo (per-user storage). If the restore never happened,
the sudo journal names who and when: `journalctl -b | grep sudo`.

## Scratch vLLM boot on a fleet GPU host — stop-first law

Any scratch/experimental vLLM container on a host that also runs a production
LLM service (`frank-primary`, `frank-large`, `frank-secondary`) MUST stop the
production unit first, and the window script must verify it. Two independent
failure modes if skipped:

1. **Expert-offload planner dies on host RAM.**
   `vllm/model_executor/offloader/expert_plan.py` reads `MemAvailable` at
   plan time. A production rank worker holds tens of GiB of unevictable
   pinned expert rows (frank-primary: ~76 GiB × 2 ranks), so a beside-prod
   boot sees ~86 GiB instead of ~239 and dies at plan time:
   `ExpertPlanError: host holds 5113 of 12096 swappable experts per rank`.
   Looks like a planner bug; is pure RAM contention. Signature: container
   exits within ~60 s, after `Resolved architecture: …` but before worker
   init; `free -g` showing `available ≈ 86` while prod runs is the smoking
   gun.
2. **GPU ownership.** Two vLLM containers cannot share the GPUs regardless
   (the fleet's benchmarking law). Scratch ports (`:7771`/`:7773`) are fine
   only once the prod unit is down.

Window skeleton (dankserver):

```bash
sudo systemctl stop frank-primary.service
sudo systemctl mask --runtime frank-primary.service  # relinked unit: plain mask refuses; runtime mask self-clears at reboot
sleep 3
podman ps --format "{{.Names}}" | grep -q frank-primary-glm && { echo "prod still running - abort"; exit 1; }
# ... launch scratch container, watch logs ...
podman rm -f <scratch>                                # stuck vLLM ignores SIGTERM
sudo systemctl unmask --runtime frank-primary.service
sudo systemctl reset-failed frank-primary.service
sudo systemctl start frank-primary.service
# restore oracle is a CONJUNCTION: health 200 AND is-active active AND is-enabled enabled AND no /run/systemd/system mask symlink
```

The deploy restores the unit after bench windows, so stop is sufficient
there; `mask --runtime` is for scratch boots that must survive a stray
restart. Always end a window by re-checking the conjunction before
reporting — if a restore after failure doesn't happen automatically, prod
stays DOWN. See the repo's benchmarking policy (AGENTS.md) and
[[LLM Bench Windows and Measurement]].

## Iterating on a fleet service before commit

The repo's contract is git → `deploy.sh` → host, but live testing needs the
new file on the host first, and the naive way (copy into the host checkout)
blocks the very deploy that would install it. The loop:

1. Edit locally; run `./tests/run.sh`.
2. Ship working files **for testing**:
   `cat local/file | ssh <host> 'cat > /home/dank/serverAppConfigs/<path>'`
   (then `chmod +x` if executed — `cat >`/scp drop the exec bit and a
   container entrypoint fails status 126, "found but not executable").
3. Test against the live service; iterate.
4. **Before committing**, prove the host copy is byte-identical to what you
   are about to commit (`sha256sum` both ends) — otherwise step 6 loses work.
5. Commit and push.
6. Clean the host checkout so the deploy can pull: untracked hand copies →
   `mv` to `/tmp/handcopy/`; modified tracked files → `git checkout -- <paths>`
   (safe only because of step 4); confirm `git status --short` prints nothing.
7. `./scripts/deploy.sh <target> --service <svc>`.

Traps, in the order they bite: the **pull aborts on hand copies** ("untracked
working tree files would be overwritten by merge" — exactly the file you
shipped); `git checkout --` silently eats uncommitted work without step 4; a
runtime may **mount files straight out of the checkout** (`dankserver/frank-medium/**`,
`shared/grafana/provisioning/**`) so a hand copy takes effect at next
container restart with no deploy; root-owned files from a container write
need `chown` back before git operations; and **a restart is not a
verification** — verify the *specific* claim (new code path's own output,
file checksum on the host, the API serving the resource).

Why not keep a scratch copy on the host? The deployed artifact must be the
committed one. Probes may live anywhere; runtime state lives in git.

## Incident window reconstruction

Use when the question is **"what was running when it broke?"** and the answer
must come from artifacts, not recollection. Three independent clocks,
cross-checked:

1. **Per-run duration** — scripts wrap commands in `time` → `real 4m19.129s`.
2. **Log mtime** — last write of the log.
3. **Journal first-failure instant** — first error line from the affected
   service.

`start ≈ mtime − real` for the last run in a log; each run nests inside the
previous one's window; the run whose `[start, end]` contains the journal's
first-failure instant is the answer.

```bash
ssh <host> 'ls -l --time-style=+%m-%d_%H:%M:%S <logdir>/*.log | awk "{print \$6, \$7}"'
ssh <host> 'grep -ah "^real" <logdir>/*.log'
ssh <host> 'grep -aE "^####|^\+ /path/to/binary" <logdir>/<name>.log | sed "s/\(.\{130\}\)/\1\n        /g"'
ssh <host> 'journalctl --user --since "<date> <HH:MM>" | grep -a "invalid internal status" | head -2'
```

Traps (all hit for real): `/tmp` does not survive a reboot — launch
experiment scripts from a persistent path (`/home/dank/models/logs/<tag>.log`
survived while `/tmp/x4*.sh` died); a log's mtime is not proof a run finished
(truncation with the fd still held keeps growing); long command lines are
display-truncated — wrap before reading or you report parameters never used;
off-by-one filenames (`x4b.log` vs `x4b-experiment.log`) silently skip
evidence — list the directory and match exactly; **overlap is a hypothesis,
not an observation** unless a sampler proves it; do not invent the cause —
reconstruct the *window* and keep the *mechanism* separate (`journalctl -k |
grep -i oom`); publish "trigger not established" when the proof is absent.

Fleet-specific: container healthchecks shell out to the **podman CLI**, so
the first `invalid internal status … could not find any running process` in
the user journal is an early, per-minute indicator of rootless-podman state
death — it timestamps the incident to within one healthcheck interval (how
the 18:58:30 Dankclaw break was pinned to a bench still loading).

Worked example (2026-09-16, Dankclaw): log mtimes x4e-run 18:58:29 /
x4f-run 18:59:56; durations E1 `real 4m53.946s`, F `real 5m24.785s` (exit 143
= killed); journal first failure 18:58:30 ⇒ **F** was in flight (launched
18:54:31, inside its window).

## Wake-on-LAN for a sleeping fleet host

A fleet host is asleep (100% ping loss; `ip neigh show <ip>` shows STALE or
FAILED) and you need it up — typically the FrankBook-Pro Mac mini
(192.168.0.112, **MAC `de:6b:6d:c1:52:23`**, sleeps and breaks the e2e
`11-admin` spec).

- **Find the MAC**: `ip neigh show <ip>` — even a STALE entry carries it;
  fallback Pi-hole/DHCP leases or fleet docs.
- **Send the packet**: `6×0xFF + 16×MAC` = 102 bytes, UDP to port **9** (and
  7), broadcast. The dev workstation has **two interfaces on 192.168.0.0/24**
  (Wi-Fi + wired USB-Ethernet, default route metric 100) — a plain
  `sendto("255.255.255.255", 9)` leaves only the default-route interface.
  Bind explicitly to each interface's local IP and target the subnet
  broadcast `192.168.0.255` as well as `255.255.255.255`; burst ~15 packets
  per interface ~0.2 s apart. `wakeonlan` is not installed; use a raw UDP
  socket with `SO_BROADCAST=1`:

```python
import socket, time
MAC = "de:6b:6d:c1:52:23"
mac = bytes(int(x,16) for x in MAC.split(":"))
pkt = b"\xff"*6 + mac*16          # 102 bytes
for local, dest in [("192.168.0.83","192.168.0.255"),("192.168.0.104","192.168.0.255"),
                    ("192.168.0.83","255.255.255.255"),("192.168.0.104","255.255.255.255")]:
    s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    s.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
    s.bind((local, 0))
    for _ in range(15):
        s.sendto(pkt,(dest,9)); s.sendto(pkt,(dest,7)); time.sleep(0.2)
    s.close()
```

- **Poll for the wake**: a Mac mini takes **30–90 s**; `ping -c 1 -W 1` every
  5 s for ~2–3 min before concluding failure.
- **If it does not wake** (the common case): macOS honors WoL only when BOTH
  are enabled on the mini — **System Settings → Battery → Options → "Wake
  for network access"** = ON, and the active port's **"Enable Wake for LAN"**
  (Network → [interface] → Details). If either is off, no packet from any
  source wakes it, and you cannot change it while asleep. A full power-off
  (smart plug cycled) ignores WoL entirely. Ask the user to toggle the
  setting, then re-fire.

## Related

- [[LLM Bench Windows and Measurement]] — the bench windows this playbook arms
- [[Remote Job Runner]] — supervising the long jobs that create these windows
- [[Container Host Reachability]] — the pasta port gate at container create
- [[Quadlet Lifecycle]] — the unit layer behind stop/mask/start
- [[Model Swap and Rollback]] — swap-time incidents and rollbacks