docs(vault): fleet-wide fact re-verification pass (67 pages)
The docs-fact census wave re-read the live hosts and re-keyed every page it could: runtime/payload identities, ports, sizes, ledger figures, unit state, and the retirement banners that drifted as the fleet moved. The wave was interrupted mid-run, so this commit carries only what passed the integrity gates: every touched page parses (front matter or H1+status convention intact, balanced fences, no truncated tables, trailing newline), the docs_completeness and link-integrity seams are green, and bats is 384/384. Numbers that this session proved on live hosts are keyed to the measured value, not the page's earlier claim.
This commit is contained in:
1 parent
466fd477d8
commit
e7f0d93c86
67 files changed
+2619
-1152
No files matched your search
@@ -10,6 +10,14 @@ timestamp: 2026-09-03
|
||||
> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
|
||||
> See [frank-redo-2026-08-17.md](../frank-redo-2026-08-17.md) and
|
||||
> [deployment.md](../deployment.md) for the active configuration.
|
||||
>
|
||||
> **2026-10-18: nothing pipeline-parallel runs on the fleet.** No RPC listener
|
||||
> exists on either host (`ss -ltn` shows no :50052), every artifact named below
|
||||
> is gone (§Files), and the worker hardware left the box: the RTX 3090 was
|
||||
> removed from Dankserver on 2026-09-15 (it now sits in Dankclaw) and the server
|
||||
> runs 2× R9700. The live multi-GPU scheme on Dankserver is tensor parallelism
|
||||
> inside one process ([[Dual-R9700-Cooperation]],
|
||||
> [[frank-archon-radiance-qwen38-iq4r]]).
|
||||
|
||||
# Qwen3.8-27B Pipeline-Parallel Deployment (Dankclaw + Dankserver)
|
||||
|
||||
@@ -54,7 +62,10 @@ helper `frank-restart-small.sh` is obsolete (it restarts the removed
|
||||
`frank-zealot` unit); restart the worker with
|
||||
`sudo systemctl restart frank-pp-worker`. The Dankclaw helper
|
||||
`frank-restart-large.sh` (sync + cache drop + restart) is correct for the
|
||||
head. `deploy.sh dankserver` is broken until retargeted. See
|
||||
head. The "deploy.sh dankserver is broken until retargeted" note is long fixed:
|
||||
`./scripts/deploy.sh dankserver --service frank-archon` is the documented
|
||||
invocation today (`scripts/deploy.sh:38`, dispatch case
|
||||
`dankserver:frank-archon` at `scripts/deploy.sh:149`). See
|
||||
[[deployment]] for the full boot-policy table.
|
||||
|
||||
## FA-all-quants rebuild (2026-08-17)
|
||||
@@ -133,10 +144,13 @@ content, degenerate logits). Root cause chain:
|
||||
A clean cache and a healthy disk produced correct output immediately
|
||||
(acceptance 0.89 on the first smoke test).
|
||||
|
||||
Guard: `scripts/prune-rpc-cache.sh` on Dankserver (cron, 40 GiB cap).
|
||||
Never let the cache directory grow unchecked; a full disk silently
|
||||
corrupts it. Signature to recognize: healthy load + degenerate repeated-
|
||||
token output.
|
||||
Guard was `scripts/prune-rpc-cache.sh` on Dankserver (cron, 40 GiB cap). That
|
||||
script is gone from the repo and from the host, yet the crontab line still
|
||||
runs every 10 minutes against it
|
||||
(`*/10 * * * * /home/dank/vLLM/scripts/prune-rpc-cache.sh`, `crontab -l`, read
|
||||
2026-10-18 — the target does not exist, so it fails silently). Never let the
|
||||
cache directory grow unchecked; a full disk silently corrupts it. Signature to
|
||||
recognize: healthy load + degenerate repeated-token output.
|
||||
|
||||
## Topology
|
||||
|
||||
@@ -163,9 +177,9 @@ llama-server b10433 ggml-rpc-server b10433
|
||||
- **Why the head is on Dankclaw:** the model file is already there; the
|
||||
worker streams its tensor share at load (no 31.5 GB copy to the server).
|
||||
- **Both sides must run the identical build** (b10433 / 9b05354ec). The
|
||||
worker build lives in a worktree `/home/dank/llama-cpp-rpc` on Dankserver
|
||||
so the production b10121 build stays untouched. A version skew hangs the
|
||||
RPC handshake with no error.
|
||||
worker build lived in a worktree `/home/dank/llama-cpp-rpc` on Dankserver so the
|
||||
production b10121 build stayed untouched (that path no longer exists, read
|
||||
2026-10-18). A version skew hangs the RPC handshake with no error.
|
||||
|
||||
## Files
|
||||
|
||||
@@ -179,6 +193,14 @@ llama-server b10433 ggml-rpc-server b10433
|
||||
| `scripts/pp-test.sh` | any box with ssh to both | test orchestrator (preflight/start/verify/bench/stop-pp/restore) |
|
||||
| `scripts/pp-bench.py` | claw | API driver; reports server-side `print_timing` |
|
||||
|
||||
|
||||
**Every PP file in that table except `scripts/frank-dragoon.env` no longer
|
||||
exists** (checkout read 2026-10-18: `frank-pp-worker.env`,
|
||||
`serve-pp-worker.sh`, `serve-pp-head.sh`, `frank-pp-worker.service`,
|
||||
`pp-test.sh`, `pp-bench.py` and `prune-rpc-cache.sh` are all absent), and
|
||||
`scripts/frank-dragoon.env` is today the halogen-flash-server env, not a PP
|
||||
head env ([[frank-dragoon-flash-server-v2-native]]).
|
||||
|
||||
## Gotchas (all verified in b10433 source / live tests, 2026-08-15)
|
||||
|
||||
1. **No `--device` on the head.** Setting `params.devices` makes
|
||||
@@ -188,9 +210,9 @@ llama-server b10433 ggml-rpc-server b10433
|
||||
2. **The iGPU side is ROCm by default.** Backends register in fixed order
|
||||
(HIP before Vulkan); the iGPU dedup rule keeps the first backend's view.
|
||||
Live check showed `[RPC0, ROCm0]`. For a Vulkan side, use the second
|
||||
head build `/home/dank/llama.cpp/build-vulkan-rpc` (HIP off) — set
|
||||
`HEAD_BIN` in the env. Trade: ROCm side = faster prefill, ~3% slower
|
||||
decode (measured on this box).
|
||||
head build `/home/dank/llama.cpp/build-vulkan-rpc` (HIP off; removed — the
|
||||
path is gone on 2026-10-18) — set `HEAD_BIN` in the env. Trade: ROCm side =
|
||||
faster prefill, ~3% slower decode (measured on this box).
|
||||
3. **Pin `--tensor-split 1,1`.** The default split is proportional to
|
||||
*reported free memory*: the Strix reports up to 96 GiB (GTT) vs the
|
||||
3090's 24 GB, so the default lands ~20/80 on the slow device (verified
|
||||
@@ -207,7 +229,9 @@ llama-server b10433 ggml-rpc-server b10433
|
||||
## Runbook (morning of 2026-08-16)
|
||||
|
||||
Run from any box with ssh access to both (repo path on claw:
|
||||
`/home/dank/vllmScripts`, on server: `/home/dank/vLLM`).
|
||||
`/home/dank/vllmScripts`, on server: `/home/dank/vLLM` — both still exist as
|
||||
stale checkouts; the deployment checkout today is `/home/dank/serverAppConfigs`
|
||||
on both hosts).
|
||||
|
||||
```
|
||||
# 0. one-time: pull the kit on both boxes
|
||||
@@ -243,8 +267,9 @@ Verification checklist after each `start`:
|
||||
1. `verify` shows `using device RPC0 ... using device ROCm0|Vulkan0`.
|
||||
2. Layer assignment lines (`-lv 10`): worker gets the first ~32 layers at a
|
||||
1,1 split.
|
||||
3. `nvidia-smi` on the server shows ~15–16 GB used by the worker
|
||||
(production small service + worker share).
|
||||
3. `nvidia-smi` on the server showed ~15–16 GB used by the worker
|
||||
(production small service + worker share). No NVIDIA card is in that host
|
||||
today — the equivalent check is `rocm-smi --showmeminfo vram`.
|
||||
4. Head GTT usage (rocm-smi) well under 96 GiB; model fully on devices
|
||||
(`--gpu-layers all` offloads everything; CPU holds embeddings/lm_head glue).
|
||||
|
||||
|
||||
Reference in new issue
Block a user