docs(vault): service and infrastructure pages stop restating endpoint facts
18 pages cut their per-endpoint shape, payload, port and window claims and
link [[frank-archon]] / [[frank-dragoon]] / [[frank-zealot]] instead. Several
of the cut claims had already drifted -- the stack page still described the
kingjones ROCmFP4 era on ROCmFPX, OpenWebUI carried the 27B/3-slot bench
narrative, Hermes named a retired max_model_len and the old service name, and
the OMP harness page's live-window block had drifted off frank-zealot's real
262,144 (it is now labelled a dated worked example whose authority is the
endpoint page).
Two facts gain a single owner instead of two: the up{job=~"frank-.*"} scope
lives in Infrastructure/Prometheus.md, and the blackbox/Anubis probe form in
Architecture/Stale Signal Audit.md, each cited from the other page.
Voice Packs and Frank TTS carried live-state claims that contradicted the
engine page's 2026-10-18 deploy record (nothing listening on 7772/8880, model
dir absent); they now point at the pages that own host state.
This commit is contained in:
1 parent
219a097b39
commit
6bdfd2853e
18 files changed
+169
-169
No files matched your search
@@ -11,15 +11,16 @@ timestamp: 2026-10-05T00:00:00-07:00
|
||||
## Endpoint down, connection refused — GPU holder reclaim
|
||||
|
||||
Use when `:7770`/`:7771` gives **connection refused** (not a timeout) and the
|
||||
service was not crashed. Applies to `frank-medium` (dankserver, root unit),
|
||||
`frank-zealot` (dankclaw, root unit), `frank-dragoon` (dankclaw, rootless
|
||||
Quadlet).
|
||||
service was not crashed. Applies to the root-unit fleet endpoints:
|
||||
`frank-archon` (dankserver), `frank-dragoon` and `frank-zealot` (dankclaw) —
|
||||
unit identity per endpoint is on its own page ([[frank-archon]],
|
||||
[[frank-dragoon]], [[frank-zealot]]).
|
||||
|
||||
**1. Classify the stop before touching anything**
|
||||
|
||||
```bash
|
||||
ssh dankserver "systemctl is-enabled frank-medium; systemctl is-active frank-medium; \
|
||||
systemctl show frank-medium -p LoadState,UnitFileState,ActiveState,SubState,NRestarts"
|
||||
ssh dankserver "systemctl is-enabled frank-archon; systemctl is-active frank-archon; \
|
||||
systemctl show frank-archon -p LoadState,UnitFileState,ActiveState,SubState,NRestarts"
|
||||
ls -l /etc/systemd/system/<svc>.service # symlink to repo = unmasked; -> /dev/null = masked
|
||||
```
|
||||
|
||||
@@ -36,9 +37,11 @@ ps -eo pid,comm,etime,args | grep -Ei 'llama|bench' | grep -v grep
|
||||
|
||||
Cards idle → `systemctl start <svc>` and verify (step 5). Cards busy with no
|
||||
prod unit → get the holder, and compare its bytes against the service's
|
||||
requirement before concluding a start would even fit (frank-medium 35:30
|
||||
needs ~30.8 GiB ROCm0 / ~28.9 GiB ROCm1; a stray holding 15+17 GiB blocks the
|
||||
restart on allocation, not config).
|
||||
requirement before concluding a start would even fit. The per-boot requirement
|
||||
is the endpoint's own arithmetic ([[frank-archon]], [[frank-dragoon]],
|
||||
[[frank-zealot]]; the retired `frank-medium` split is measured in
|
||||
[[frank-medium-llamacpp-dflash2]]) — a stray holder blocks the restart on
|
||||
allocation, not config.
|
||||
|
||||
**3. Identify the holder and prove it is abandoned**
|
||||
|
||||
@@ -95,9 +98,10 @@ the local Gitea on Dankclaw — push allowed).
|
||||
unquoted word with `=` in an ssh one-liner (`echo ===` aborts the chain). A
|
||||
freshly restarted endpoint under live agent-queue backlog may return empty
|
||||
bodies/timeouts for minutes while `/health` is 200 — re-probe with a long
|
||||
timeout. First request after a halogen (`frank-dragoon`) restart can go silent
|
||||
45–60 s and self-recover; do not restart over it. Rootless `frank-dragoon`:
|
||||
read logs **without** sudo (per-user storage). If the restore never happened,
|
||||
timeout. The first request after a `frank-dragoon` restart is a warmup artifact — see
|
||||
[[frank-dragoon]] → Contract; do not restart over it. Where an endpoint's own
|
||||
log lives (and whether sudo is needed to read it) is on its page. If the
|
||||
restore never happened,
|
||||
the sudo journal names who and when: `journalctl -b | grep sudo`.
|
||||
|
||||
## Scratch vLLM boot on a fleet GPU host — stop-first law
|
||||
@@ -110,13 +114,14 @@ failure modes if skipped:
|
||||
1. **Expert-offload planner dies on host RAM.**
|
||||
`vllm/model_executor/offloader/expert_plan.py` reads `MemAvailable` at
|
||||
plan time. A production rank worker holds tens of GiB of unevictable
|
||||
pinned expert rows (frank-archon: ~76 GiB × 2 ranks), so a beside-prod
|
||||
boot sees ~86 GiB instead of ~239 and dies at plan time:
|
||||
pinned expert rows — frank-archon's pinned-RAM ledger, and how much host
|
||||
RAM the loaded endpoint leaves, is on [[frank-archon]] — so a beside-prod
|
||||
boot sees far less than the bare host and dies at plan time:
|
||||
`ExpertPlanError: host holds 5113 of 12096 swappable experts per rank`.
|
||||
Looks like a planner bug; is pure RAM contention. Signature: container
|
||||
exits within ~60 s, after `Resolved architecture: …` but before worker
|
||||
init; `free -g` showing `available ≈ 86` while prod runs is the smoking
|
||||
gun.
|
||||
init; `free -g` showing an `available` far below the endpoint page's
|
||||
loaded-host figure while prod runs is the smoking gun.
|
||||
2. **GPU ownership.** Two vLLM containers cannot share the GPUs regardless
|
||||
(the fleet's benchmarking law). Scratch ports (`:7771`/`:7773`) are fine
|
||||
only once the prod unit is down.
|
||||
@@ -166,7 +171,8 @@ blocks the very deploy that would install it. The loop:
|
||||
Traps, in the order they bite: the **pull aborts on hand copies** ("untracked
|
||||
working tree files would be overwritten by merge" — exactly the file you
|
||||
shipped); `git checkout --` silently eats uncommitted work without step 4; a
|
||||
runtime may **mount files straight out of the checkout** (`dankserver/frank-medium/**`,
|
||||
runtime may **mount files straight out of the checkout** (`scripts/**` — the
|
||||
serve scripts and the chat template mount from there, see [[frank-archon]] — and
|
||||
`shared/grafana/provisioning/**`) so a hand copy takes effect at next
|
||||
container restart with no deploy; root-owned files from a container write
|
||||
need `chown` back before git operations; and **a restart is not a
|
||||
|
||||
Reference in new issue
Block a user