docs(vault): service and infrastructure pages stop restating endpoint facts

18 pages cut their per-endpoint shape, payload, port and window claims and
link [[frank-archon]] / [[frank-dragoon]] / [[frank-zealot]] instead. Several
of the cut claims had already drifted -- the stack page still described the
kingjones ROCmFP4 era on ROCmFPX, OpenWebUI carried the 27B/3-slot bench
narrative, Hermes named a retired max_model_len and the old service name, and
the OMP harness page's live-window block had drifted off frank-zealot's real
262,144 (it is now labelled a dated worked example whose authority is the
endpoint page).

Two facts gain a single owner instead of two: the up{job=~"frank-.*"} scope
lives in Infrastructure/Prometheus.md, and the blackbox/Anubis probe form in
Architecture/Stale Signal Audit.md, each cited from the other page.

Voice Packs and Frank TTS carried live-state claims that contradicted the
engine page's 2026-10-18 deploy record (nothing listening on 7772/8880, model
dir absent); they now point at the pages that own host state.
This commit is contained in:
Frank
2026-10-19 00:56:02 -07:00
parent 219a097b39
commit 6bdfd2853e
18 changed files with 169 additions and 169 deletions
@@ -94,8 +94,8 @@ earlier can still hold listener/TIME_WAIT state, so the replacement dies
`pasta: ... bind: Address already in use` (conmon exit 126) — then **self-heals
seconds later** under `Restart=on-failure`. The failure looks like it failed,
then looks fine, and the missing gate stays invisible because the retry masks
it. Observed 2026-09-16 cutting `frank-dragoon` from a root unit to a rootless
Quadlet.
it. First seen on a `frank-dragoon` unit cutover; which supervision shape that
era was is in [[frank-dragoon]]'s era table.
Gate procedure: stop the old unit AND the new one (a redeploy's running
replacement is itself the port-holder — without that stop the gate deadlocks
@@ -11,15 +11,16 @@ timestamp: 2026-10-05T00:00:00-07:00
## Endpoint down, connection refused — GPU holder reclaim
Use when `:7770`/`:7771` gives **connection refused** (not a timeout) and the
service was not crashed. Applies to `frank-medium` (dankserver, root unit),
`frank-zealot` (dankclaw, root unit), `frank-dragoon` (dankclaw, rootless
Quadlet).
service was not crashed. Applies to the root-unit fleet endpoints:
`frank-archon` (dankserver), `frank-dragoon` and `frank-zealot` (dankclaw) —
unit identity per endpoint is on its own page ([[frank-archon]],
[[frank-dragoon]], [[frank-zealot]]).
**1. Classify the stop before touching anything**
```bash
ssh dankserver "systemctl is-enabled frank-medium; systemctl is-active frank-medium; \
systemctl show frank-medium -p LoadState,UnitFileState,ActiveState,SubState,NRestarts"
ssh dankserver "systemctl is-enabled frank-archon; systemctl is-active frank-archon; \
systemctl show frank-archon -p LoadState,UnitFileState,ActiveState,SubState,NRestarts"
ls -l /etc/systemd/system/<svc>.service # symlink to repo = unmasked; -> /dev/null = masked
```
@@ -36,9 +37,11 @@ ps -eo pid,comm,etime,args | grep -Ei 'llama|bench' | grep -v grep
Cards idle → `systemctl start <svc>` and verify (step 5). Cards busy with no
prod unit → get the holder, and compare its bytes against the service's
requirement before concluding a start would even fit (frank-medium 35:30
needs ~30.8 GiB ROCm0 / ~28.9 GiB ROCm1; a stray holding 15+17 GiB blocks the
restart on allocation, not config).
requirement before concluding a start would even fit. The per-boot requirement
is the endpoint's own arithmetic ([[frank-archon]], [[frank-dragoon]],
[[frank-zealot]]; the retired `frank-medium` split is measured in
[[frank-medium-llamacpp-dflash2]]) — a stray holder blocks the restart on
allocation, not config.
**3. Identify the holder and prove it is abandoned**
@@ -95,9 +98,10 @@ the local Gitea on Dankclaw — push allowed).
unquoted word with `=` in an ssh one-liner (`echo ===` aborts the chain). A
freshly restarted endpoint under live agent-queue backlog may return empty
bodies/timeouts for minutes while `/health` is 200 — re-probe with a long
timeout. First request after a halogen (`frank-dragoon`) restart can go silent
45–60 s and self-recover; do not restart over it. Rootless `frank-dragoon`:
read logs **without** sudo (per-user storage). If the restore never happened,
timeout. The first request after a `frank-dragoon` restart is a warmup artifact — see
[[frank-dragoon]] → Contract; do not restart over it. Where an endpoint's own
log lives (and whether sudo is needed to read it) is on its page. If the
restore never happened,
the sudo journal names who and when: `journalctl -b | grep sudo`.
## Scratch vLLM boot on a fleet GPU host — stop-first law
@@ -110,13 +114,14 @@ failure modes if skipped:
1. **Expert-offload planner dies on host RAM.**
`vllm/model_executor/offloader/expert_plan.py` reads `MemAvailable` at
plan time. A production rank worker holds tens of GiB of unevictable
pinned expert rows (frank-archon: ~76 GiB × 2 ranks), so a beside-prod
boot sees ~86 GiB instead of ~239 and dies at plan time:
pinned expert rows — frank-archon's pinned-RAM ledger, and how much host
RAM the loaded endpoint leaves, is on [[frank-archon]] — so a beside-prod
boot sees far less than the bare host and dies at plan time:
`ExpertPlanError: host holds 5113 of 12096 swappable experts per rank`.
Looks like a planner bug; is pure RAM contention. Signature: container
exits within ~60 s, after `Resolved architecture: …` but before worker
init; `free -g` showing `available ≈ 86` while prod runs is the smoking
gun.
init; `free -g` showing an `available` far below the endpoint page's
loaded-host figure while prod runs is the smoking gun.
2. **GPU ownership.** Two vLLM containers cannot share the GPUs regardless
(the fleet's benchmarking law). Scratch ports (`:7771`/`:7773`) are fine
only once the prod unit is down.
@@ -166,7 +171,8 @@ blocks the very deploy that would install it. The loop:
Traps, in the order they bite: the **pull aborts on hand copies** ("untracked
working tree files would be overwritten by merge" — exactly the file you
shipped); `git checkout --` silently eats uncommitted work without step 4; a
runtime may **mount files straight out of the checkout** (`dankserver/frank-medium/**`,
runtime may **mount files straight out of the checkout** (`scripts/**` — the
serve scripts and the chat template mount from there, see [[frank-archon]] — and
`shared/grafana/provisioning/**`) so a hand copy takes effect at next
container restart with no deploy; root-owned files from a container write
need `chown` back before git operations; and **a restart is not a
@@ -48,8 +48,9 @@ table. Three buckets:
re-download.
- **retired** — maps to a retired runtime. Safe candidates.
Trap: a directory can hold **both** a dead target and a live drafter
(frank-dragoon's DFlash2 drafter lives beside the previous target). Move the
Trap: a directory can hold **both** a dead target and a live asset — an era's
drafter can sit beside the retired target it came with (per-endpoint era tables:
[[frank-dragoon]], [[frank-zealot]], [[frank-archon]]). Move the
single file, never the directory, and say so in the manifest.
## 3. The live-reference check — before moving anything
@@ -149,13 +150,13 @@ name**, never `endswith('.gguf')` — mradermacher ships large quants as
## Refresh per engine — know the recovery path BEFORE deleting
- **frank-dragoon (halogen)**: the engine fetches its own payload via
`HALOGEN_DOWNLOAD` when the checkpoint is absent. Clear the directory but
leave it **existing and empty**, then start the unit. **Bind-mount trap:**
- **frank-dragoon**: self-fetching — clear the directory but
leave it **existing and empty**, then start the unit; the download knobs and
the payload's byte count (which is what the re-download costs) are on
[[frank-dragoon]]. **Bind-mount trap:**
podman refuses a mount whose source is missing
(`Error: statfs /home/dank/models/halogen: no such file or directory`) and
the unit loops on `Restart=on-failure` — `mkdir -p` the empty dir. ~124 GiB
at 130–240 MB/s ≈ 10–20 min, then load.
the unit loops on `Restart=on-failure` — `mkdir -p` the empty dir.
- **llama.cpp services**: no self-fetch; the serve script refuses to start
without weights. Pull yourself and verify: