docs(vault): service and infrastructure pages stop restating endpoint facts

18 pages cut their per-endpoint shape, payload, port and window claims and
link [[frank-archon]] / [[frank-dragoon]] / [[frank-zealot]] instead. Several
of the cut claims had already drifted -- the stack page still described the
kingjones ROCmFP4 era on ROCmFPX, OpenWebUI carried the 27B/3-slot bench
narrative, Hermes named a retired max_model_len and the old service name, and
the OMP harness page's live-window block had drifted off frank-zealot's real
262,144 (it is now labelled a dated worked example whose authority is the
endpoint page).

Two facts gain a single owner instead of two: the up{job=~"frank-.*"} scope
lives in Infrastructure/Prometheus.md, and the blackbox/Anubis probe form in
Architecture/Stale Signal Audit.md, each cited from the other page.

Voice Packs and Frank TTS carried live-state claims that contradicted the
engine page's 2026-10-18 deploy record (nothing listening on 7772/8880, model
dir absent); they now point at the pages that own host state.
This commit is contained in:
Frank committed 2026-10-19 00:56:02 -07:00
1 parent 219a097b39
commit 6bdfd2853e
18 files changed
+169 -169

No files matched your search

@@ -11,15 +11,16 @@ timestamp: 2026-10-05T00:00:00-07:00
## Endpoint down, connection refused — GPU holder reclaim
Use when `:7770`/`:7771` gives **connection refused** (not a timeout) and the
service was not crashed. Applies to `frank-medium` (dankserver, root unit),
`frank-zealot` (dankclaw, root unit), `frank-dragoon` (dankclaw, rootless
Quadlet).
service was not crashed. Applies to the root-unit fleet endpoints:
`frank-archon` (dankserver), `frank-dragoon` and `frank-zealot` (dankclaw) —
unit identity per endpoint is on its own page ([[frank-archon]],
[[frank-dragoon]], [[frank-zealot]]).
**1. Classify the stop before touching anything**
```bash
ssh dankserver "systemctl is-enabled frank-medium; systemctl is-active frank-medium; \
systemctl show frank-medium -p LoadState,UnitFileState,ActiveState,SubState,NRestarts"
ssh dankserver "systemctl is-enabled frank-archon; systemctl is-active frank-archon; \
systemctl show frank-archon -p LoadState,UnitFileState,ActiveState,SubState,NRestarts"
ls -l /etc/systemd/system/<svc>.service # symlink to repo = unmasked; -> /dev/null = masked
```
@@ -36,9 +37,11 @@ ps -eo pid,comm,etime,args | grep -Ei 'llama|bench' | grep -v grep
Cards idle → `systemctl start <svc>` and verify (step 5). Cards busy with no
prod unit → get the holder, and compare its bytes against the service's
requirement before concluding a start would even fit (frank-medium 35:30
needs ~30.8 GiB ROCm0 / ~28.9 GiB ROCm1; a stray holding 15+17 GiB blocks the
restart on allocation, not config).
requirement before concluding a start would even fit. The per-boot requirement
is the endpoint's own arithmetic ([[frank-archon]], [[frank-dragoon]],
[[frank-zealot]]; the retired `frank-medium` split is measured in
[[frank-medium-llamacpp-dflash2]]) — a stray holder blocks the restart on
allocation, not config.
**3. Identify the holder and prove it is abandoned**
@@ -95,9 +98,10 @@ the local Gitea on Dankclaw — push allowed).
unquoted word with `=` in an ssh one-liner (`echo ===` aborts the chain). A
freshly restarted endpoint under live agent-queue backlog may return empty
bodies/timeouts for minutes while `/health` is 200 — re-probe with a long
timeout. First request after a halogen (`frank-dragoon`) restart can go silent
45–60 s and self-recover; do not restart over it. Rootless `frank-dragoon`:
read logs **without** sudo (per-user storage). If the restore never happened,
timeout. The first request after a `frank-dragoon` restart is a warmup artifact — see
[[frank-dragoon]] → Contract; do not restart over it. Where an endpoint's own
log lives (and whether sudo is needed to read it) is on its page. If the
restore never happened,
the sudo journal names who and when: `journalctl -b | grep sudo`.
## Scratch vLLM boot on a fleet GPU host — stop-first law
@@ -110,13 +114,14 @@ failure modes if skipped:
1. **Expert-offload planner dies on host RAM.**
`vllm/model_executor/offloader/expert_plan.py` reads `MemAvailable` at
plan time. A production rank worker holds tens of GiB of unevictable
pinned expert rows (frank-archon: ~76 GiB × 2 ranks), so a beside-prod
boot sees ~86 GiB instead of ~239 and dies at plan time:
pinned expert rows — frank-archon's pinned-RAM ledger, and how much host
RAM the loaded endpoint leaves, is on [[frank-archon]] — so a beside-prod
boot sees far less than the bare host and dies at plan time:
`ExpertPlanError: host holds 5113 of 12096 swappable experts per rank`.
Looks like a planner bug; is pure RAM contention. Signature: container
exits within ~60 s, after `Resolved architecture: …` but before worker
init; `free -g` showing `available ≈ 86` while prod runs is the smoking
gun.
init; `free -g` showing an `available` far below the endpoint page's
loaded-host figure while prod runs is the smoking gun.
2. **GPU ownership.** Two vLLM containers cannot share the GPUs regardless
(the fleet's benchmarking law). Scratch ports (`:7771`/`:7773`) are fine
only once the prod unit is down.
@@ -166,7 +171,8 @@ blocks the very deploy that would install it. The loop:
Traps, in the order they bite: the **pull aborts on hand copies** ("untracked
working tree files would be overwritten by merge" — exactly the file you
shipped); `git checkout --` silently eats uncommitted work without step 4; a
runtime may **mount files straight out of the checkout** (`dankserver/frank-medium/**`,
runtime may **mount files straight out of the checkout** (`scripts/**` — the
serve scripts and the chat template mount from there, see [[frank-archon]] — and
`shared/grafana/provisioning/**`) so a hand copy takes effect at next
container restart with no deploy; root-owned files from a container write
need `chown` back before git operations; and **a restart is not a