docs(vault): service and infrastructure pages stop restating endpoint facts
18 pages cut their per-endpoint shape, payload, port and window claims and
link [[frank-archon]] / [[frank-dragoon]] / [[frank-zealot]] instead. Several
of the cut claims had already drifted -- the stack page still described the
kingjones ROCmFP4 era on ROCmFPX, OpenWebUI carried the 27B/3-slot bench
narrative, Hermes named a retired max_model_len and the old service name, and
the OMP harness page's live-window block had drifted off frank-zealot's real
262,144 (it is now labelled a dated worked example whose authority is the
endpoint page).
Two facts gain a single owner instead of two: the up{job=~"frank-.*"} scope
lives in Infrastructure/Prometheus.md, and the blackbox/Anubis probe form in
Architecture/Stale Signal Audit.md, each cited from the other page.
Voice Packs and Frank TTS carried live-state claims that contradicted the
engine page's 2026-10-18 deploy record (nothing listening on 7772/8880, model
dir absent); they now point at the pages that own host state.
This commit is contained in:
@@ -94,8 +94,8 @@ earlier can still hold listener/TIME_WAIT state, so the replacement dies
|
||||
`pasta: ... bind: Address already in use` (conmon exit 126) — then **self-heals
|
||||
seconds later** under `Restart=on-failure`. The failure looks like it failed,
|
||||
then looks fine, and the missing gate stays invisible because the retry masks
|
||||
it. Observed 2026-09-16 cutting `frank-dragoon` from a root unit to a rootless
|
||||
Quadlet.
|
||||
it. First seen on a `frank-dragoon` unit cutover; which supervision shape that
|
||||
era was is in [[frank-dragoon]]'s era table.
|
||||
|
||||
Gate procedure: stop the old unit AND the new one (a redeploy's running
|
||||
replacement is itself the port-holder — without that stop the gate deadlocks
|
||||
|
||||
@@ -11,15 +11,16 @@ timestamp: 2026-10-05T00:00:00-07:00
|
||||
## Endpoint down, connection refused — GPU holder reclaim
|
||||
|
||||
Use when `:7770`/`:7771` gives **connection refused** (not a timeout) and the
|
||||
service was not crashed. Applies to `frank-medium` (dankserver, root unit),
|
||||
`frank-zealot` (dankclaw, root unit), `frank-dragoon` (dankclaw, rootless
|
||||
Quadlet).
|
||||
service was not crashed. Applies to the root-unit fleet endpoints:
|
||||
`frank-archon` (dankserver), `frank-dragoon` and `frank-zealot` (dankclaw) —
|
||||
unit identity per endpoint is on its own page ([[frank-archon]],
|
||||
[[frank-dragoon]], [[frank-zealot]]).
|
||||
|
||||
**1. Classify the stop before touching anything**
|
||||
|
||||
```bash
|
||||
ssh dankserver "systemctl is-enabled frank-medium; systemctl is-active frank-medium; \
|
||||
systemctl show frank-medium -p LoadState,UnitFileState,ActiveState,SubState,NRestarts"
|
||||
ssh dankserver "systemctl is-enabled frank-archon; systemctl is-active frank-archon; \
|
||||
systemctl show frank-archon -p LoadState,UnitFileState,ActiveState,SubState,NRestarts"
|
||||
ls -l /etc/systemd/system/<svc>.service # symlink to repo = unmasked; -> /dev/null = masked
|
||||
```
|
||||
|
||||
@@ -36,9 +37,11 @@ ps -eo pid,comm,etime,args | grep -Ei 'llama|bench' | grep -v grep
|
||||
|
||||
Cards idle → `systemctl start <svc>` and verify (step 5). Cards busy with no
|
||||
prod unit → get the holder, and compare its bytes against the service's
|
||||
requirement before concluding a start would even fit (frank-medium 35:30
|
||||
needs ~30.8 GiB ROCm0 / ~28.9 GiB ROCm1; a stray holding 15+17 GiB blocks the
|
||||
restart on allocation, not config).
|
||||
requirement before concluding a start would even fit. The per-boot requirement
|
||||
is the endpoint's own arithmetic ([[frank-archon]], [[frank-dragoon]],
|
||||
[[frank-zealot]]; the retired `frank-medium` split is measured in
|
||||
[[frank-medium-llamacpp-dflash2]]) — a stray holder blocks the restart on
|
||||
allocation, not config.
|
||||
|
||||
**3. Identify the holder and prove it is abandoned**
|
||||
|
||||
@@ -95,9 +98,10 @@ the local Gitea on Dankclaw — push allowed).
|
||||
unquoted word with `=` in an ssh one-liner (`echo ===` aborts the chain). A
|
||||
freshly restarted endpoint under live agent-queue backlog may return empty
|
||||
bodies/timeouts for minutes while `/health` is 200 — re-probe with a long
|
||||
timeout. First request after a halogen (`frank-dragoon`) restart can go silent
|
||||
45–60 s and self-recover; do not restart over it. Rootless `frank-dragoon`:
|
||||
read logs **without** sudo (per-user storage). If the restore never happened,
|
||||
timeout. The first request after a `frank-dragoon` restart is a warmup artifact — see
|
||||
[[frank-dragoon]] → Contract; do not restart over it. Where an endpoint's own
|
||||
log lives (and whether sudo is needed to read it) is on its page. If the
|
||||
restore never happened,
|
||||
the sudo journal names who and when: `journalctl -b | grep sudo`.
|
||||
|
||||
## Scratch vLLM boot on a fleet GPU host — stop-first law
|
||||
@@ -110,13 +114,14 @@ failure modes if skipped:
|
||||
1. **Expert-offload planner dies on host RAM.**
|
||||
`vllm/model_executor/offloader/expert_plan.py` reads `MemAvailable` at
|
||||
plan time. A production rank worker holds tens of GiB of unevictable
|
||||
pinned expert rows (frank-archon: ~76 GiB × 2 ranks), so a beside-prod
|
||||
boot sees ~86 GiB instead of ~239 and dies at plan time:
|
||||
pinned expert rows — frank-archon's pinned-RAM ledger, and how much host
|
||||
RAM the loaded endpoint leaves, is on [[frank-archon]] — so a beside-prod
|
||||
boot sees far less than the bare host and dies at plan time:
|
||||
`ExpertPlanError: host holds 5113 of 12096 swappable experts per rank`.
|
||||
Looks like a planner bug; is pure RAM contention. Signature: container
|
||||
exits within ~60 s, after `Resolved architecture: …` but before worker
|
||||
init; `free -g` showing `available ≈ 86` while prod runs is the smoking
|
||||
gun.
|
||||
init; `free -g` showing an `available` far below the endpoint page's
|
||||
loaded-host figure while prod runs is the smoking gun.
|
||||
2. **GPU ownership.** Two vLLM containers cannot share the GPUs regardless
|
||||
(the fleet's benchmarking law). Scratch ports (`:7771`/`:7773`) are fine
|
||||
only once the prod unit is down.
|
||||
@@ -166,7 +171,8 @@ blocks the very deploy that would install it. The loop:
|
||||
Traps, in the order they bite: the **pull aborts on hand copies** ("untracked
|
||||
working tree files would be overwritten by merge" — exactly the file you
|
||||
shipped); `git checkout --` silently eats uncommitted work without step 4; a
|
||||
runtime may **mount files straight out of the checkout** (`dankserver/frank-medium/**`,
|
||||
runtime may **mount files straight out of the checkout** (`scripts/**` — the
|
||||
serve scripts and the chat template mount from there, see [[frank-archon]] — and
|
||||
`shared/grafana/provisioning/**`) so a hand copy takes effect at next
|
||||
container restart with no deploy; root-owned files from a container write
|
||||
need `chown` back before git operations; and **a restart is not a
|
||||
|
||||
@@ -48,8 +48,9 @@ table. Three buckets:
|
||||
re-download.
|
||||
- **retired** — maps to a retired runtime. Safe candidates.
|
||||
|
||||
Trap: a directory can hold **both** a dead target and a live drafter
|
||||
(frank-dragoon's DFlash2 drafter lives beside the previous target). Move the
|
||||
Trap: a directory can hold **both** a dead target and a live asset — an era's
|
||||
drafter can sit beside the retired target it came with (per-endpoint era tables:
|
||||
[[frank-dragoon]], [[frank-zealot]], [[frank-archon]]). Move the
|
||||
single file, never the directory, and say so in the manifest.
|
||||
|
||||
## 3. The live-reference check — before moving anything
|
||||
@@ -149,13 +150,13 @@ name**, never `endswith('.gguf')` — mradermacher ships large quants as
|
||||
|
||||
## Refresh per engine — know the recovery path BEFORE deleting
|
||||
|
||||
- **frank-dragoon (halogen)**: the engine fetches its own payload via
|
||||
`HALOGEN_DOWNLOAD` when the checkpoint is absent. Clear the directory but
|
||||
leave it **existing and empty**, then start the unit. **Bind-mount trap:**
|
||||
- **frank-dragoon**: self-fetching — clear the directory but
|
||||
leave it **existing and empty**, then start the unit; the download knobs and
|
||||
the payload's byte count (which is what the re-download costs) are on
|
||||
[[frank-dragoon]]. **Bind-mount trap:**
|
||||
podman refuses a mount whose source is missing
|
||||
(`Error: statfs /home/dank/models/halogen: no such file or directory`) and
|
||||
the unit loops on `Restart=on-failure` — `mkdir -p` the empty dir. ~124 GiB
|
||||
at 130–240 MB/s ≈ 10–20 min, then load.
|
||||
the unit loops on `Restart=on-failure` — `mkdir -p` the empty dir.
|
||||
- **llama.cpp services**: no self-fetch; the serve script refuses to start
|
||||
without weights. Pull yourself and verify:
|
||||
|
||||
|
||||
Reference in New Issue
Block a user