docs(vault): fleet-wide fact re-verification pass (67 pages)

The docs-fact census wave re-read the live hosts and re-keyed every page
it could: runtime/payload identities, ports, sizes, ledger figures, unit
state, and the retirement banners that drifted as the fleet moved. The
wave was interrupted mid-run, so this commit carries only what passed the
integrity gates: every touched page parses (front matter or H1+status
convention intact, balanced fences, no truncated tables, trailing
newline), the docs_completeness and link-integrity seams are green, and
bats is 384/384.

Numbers that this session proved on live hosts are keyed to the measured
value, not the page's earlier claim.
This commit is contained in:
Frank committed 2026-10-18 13:15:43 -07:00
1 parent 466fd477d8
commit e7f0d93c86
67 files changed
+2619 -1152

No files matched your search

@@ -10,6 +10,14 @@ timestamp: 2026-09-03
> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
> See [frank-redo-2026-08-17.md](../frank-redo-2026-08-17.md) and
> [deployment.md](../deployment.md) for the active configuration.
>
> **2026-10-18: nothing pipeline-parallel runs on the fleet.** No RPC listener
> exists on either host (`ss -ltn` shows no :50052), every artifact named below
> is gone (§Files), and the worker hardware left the box: the RTX 3090 was
> removed from Dankserver on 2026-09-15 (it now sits in Dankclaw) and the server
> runs 2× R9700. The live multi-GPU scheme on Dankserver is tensor parallelism
> inside one process ([[Dual-R9700-Cooperation]],
> [[frank-archon-radiance-qwen38-iq4r]]).
# Qwen3.8-27B Pipeline-Parallel Deployment (Dankclaw + Dankserver)
@@ -54,7 +62,10 @@ helper `frank-restart-small.sh` is obsolete (it restarts the removed
`frank-zealot` unit); restart the worker with
`sudo systemctl restart frank-pp-worker`. The Dankclaw helper
`frank-restart-large.sh` (sync + cache drop + restart) is correct for the
head. `deploy.sh dankserver` is broken until retargeted. See
head. The "deploy.sh dankserver is broken until retargeted" note is long fixed:
`./scripts/deploy.sh dankserver --service frank-archon` is the documented
invocation today (`scripts/deploy.sh:38`, dispatch case
`dankserver:frank-archon` at `scripts/deploy.sh:149`). See
[[deployment]] for the full boot-policy table.
## FA-all-quants rebuild (2026-08-17)
@@ -133,10 +144,13 @@ content, degenerate logits). Root cause chain:
A clean cache and a healthy disk produced correct output immediately
(acceptance 0.89 on the first smoke test).
Guard: `scripts/prune-rpc-cache.sh` on Dankserver (cron, 40 GiB cap).
Never let the cache directory grow unchecked; a full disk silently
corrupts it. Signature to recognize: healthy load + degenerate repeated-
token output.
Guard was `scripts/prune-rpc-cache.sh` on Dankserver (cron, 40 GiB cap). That
script is gone from the repo and from the host, yet the crontab line still
runs every 10 minutes against it
(`*/10 * * * * /home/dank/vLLM/scripts/prune-rpc-cache.sh`, `crontab -l`, read
2026-10-18 — the target does not exist, so it fails silently). Never let the
cache directory grow unchecked; a full disk silently corrupts it. Signature to
recognize: healthy load + degenerate repeated-token output.
## Topology
@@ -163,9 +177,9 @@ llama-server b10433 ggml-rpc-server b10433
- **Why the head is on Dankclaw:** the model file is already there; the
worker streams its tensor share at load (no 31.5 GB copy to the server).
- **Both sides must run the identical build** (b10433 / 9b05354ec). The
worker build lives in a worktree `/home/dank/llama-cpp-rpc` on Dankserver
so the production b10121 build stays untouched. A version skew hangs the
RPC handshake with no error.
worker build lived in a worktree `/home/dank/llama-cpp-rpc` on Dankserver so the
production b10121 build stayed untouched (that path no longer exists, read
2026-10-18). A version skew hangs the RPC handshake with no error.
## Files
@@ -179,6 +193,14 @@ llama-server b10433 ggml-rpc-server b10433
| `scripts/pp-test.sh` | any box with ssh to both | test orchestrator (preflight/start/verify/bench/stop-pp/restore) |
| `scripts/pp-bench.py` | claw | API driver; reports server-side `print_timing` |
**Every PP file in that table except `scripts/frank-dragoon.env` no longer
exists** (checkout read 2026-10-18: `frank-pp-worker.env`,
`serve-pp-worker.sh`, `serve-pp-head.sh`, `frank-pp-worker.service`,
`pp-test.sh`, `pp-bench.py` and `prune-rpc-cache.sh` are all absent), and
`scripts/frank-dragoon.env` is today the halogen-flash-server env, not a PP
head env ([[frank-dragoon-flash-server-v2-native]]).
## Gotchas (all verified in b10433 source / live tests, 2026-08-15)
1. **No `--device` on the head.** Setting `params.devices` makes
@@ -188,9 +210,9 @@ llama-server b10433 ggml-rpc-server b10433
2. **The iGPU side is ROCm by default.** Backends register in fixed order
(HIP before Vulkan); the iGPU dedup rule keeps the first backend's view.
Live check showed `[RPC0, ROCm0]`. For a Vulkan side, use the second
head build `/home/dank/llama.cpp/build-vulkan-rpc` (HIP off) — set
`HEAD_BIN` in the env. Trade: ROCm side = faster prefill, ~3% slower
decode (measured on this box).
head build `/home/dank/llama.cpp/build-vulkan-rpc` (HIP off; removed — the
path is gone on 2026-10-18) — set `HEAD_BIN` in the env. Trade: ROCm side =
faster prefill, ~3% slower decode (measured on this box).
3. **Pin `--tensor-split 1,1`.** The default split is proportional to
*reported free memory*: the Strix reports up to 96 GiB (GTT) vs the
3090's 24 GB, so the default lands ~20/80 on the slow device (verified
@@ -207,7 +229,9 @@ llama-server b10433 ggml-rpc-server b10433
## Runbook (morning of 2026-08-16)
Run from any box with ssh access to both (repo path on claw:
`/home/dank/vllmScripts`, on server: `/home/dank/vLLM`).
`/home/dank/vllmScripts`, on server: `/home/dank/vLLM` — both still exist as
stale checkouts; the deployment checkout today is `/home/dank/serverAppConfigs`
on both hosts).
```
# 0. one-time: pull the kit on both boxes
@@ -243,8 +267,9 @@ Verification checklist after each `start`:
1. `verify` shows `using device RPC0 ... using device ROCm0|Vulkan0`.
2. Layer assignment lines (`-lv 10`): worker gets the first ~32 layers at a
1,1 split.
3. `nvidia-smi` on the server shows ~15–16 GB used by the worker
(production small service + worker share).
3. `nvidia-smi` on the server showed ~15–16 GB used by the worker
(production small service + worker share). No NVIDIA card is in that host
today — the equivalent check is `rocm-smi --showmeminfo vram`.
4. Head GTT usage (rocm-smi) well under 96 GiB; model fully on devices
(`--gpu-layers all` offloads everything; CPU holds embeddings/lm_head glue).