docs(vault): strip restated fleet facts from 17 technique pages

Enforces "one fact, one owner": technique pages keep mechanism, laws, traps
and verdicts; per-endpoint shapes, digests and era measurements are cited to
the endpoint page that owns them.

Also repairs the four pointers the pass left empty (each verified against the
cited owner before landing):
- EMULATION dequant doubles host RAM (118 GiB -> ~238 GiB) -> [[frank-archon]]
- 12.11 tok/s flat 304 -> 16,095 tokens -> [[Tcclaviger Fork Ops]]
- bundled-head acceptance ladder 0.739/0.554/0.427/0.340 -> [[Qwen3.8 PP Deployment]]
- llama.cpp PR #23897 dedup provenance -> [[qwen3.8-spec-matrix]]

Replaces the retired-shape illustration in Llama Cpp Serving Sizing with the
owned MiMo Q5_K_M row (6.40 GiB payload -> 20,530 MiB used), and unwraps a
hard-wrapped [[Dankserver System Specs]] link that failed link integrity.

Orphan census over the diff: 0 numbers lost (comma-insensitive match against
the whole vault plus AGENTS.md).
This commit is contained in:
Frank committed 2026-10-19 01:33:47 -07:00
1 parent eb21dbc705
commit 04f4e2c6d4
18 files changed
+348 -425

No files matched your search

+38 -100
View File
@@ -60,6 +60,8 @@ TUNABLE=0`, stock gather 8×16 → 26.80/25.15/26.81 tok/s @ 166/1087/4064 ctx.
Decode on this stack is PCIe-gather-bound (~30% GPU duty cycle), so
batch-shaped compute knobs move ~nothing.
Measured with no spec: **12.11 tok/s FLAT from 304 to 16,095 context tokens** — KV
attention is not the cost, so a context-slope hunt on this stack is null by construction.
The `R4D_*/CLAV_*` knob family (QSA_*, MHC, ROUTE, ACT_QUANT, AR_QUANT,
SILU_QUANT) are kill-switches defaulting ON, not perf dials; real levers are
@@ -120,95 +122,36 @@ exactly SEQS×(n+1) (`suboptimal` warning) → try 8192.
## Serving a model larger than VRAM (dankserver recipe)
Dankserver: 2× R9700 (gfx1201), 31.86 GiB each = 63.7 GiB real VRAM, 246 GiB
RAM. `torch.cuda.device_count()` returns 3 and summing `total_memory` gives
a bogus 186.7 GiB (the Raphael iGPU is in the list — see
[[Llama Cpp Serving Sizing]] for the reading discipline).
Dankserver's card count, per-card VRAM and host RAM are in [[Dankserver System Specs]];
`torch.cuda.device_count()` returns 3 on this box and summing `total_memory` overstates it
badly (the Raphael iGPU is in the list — reading discipline: [[Llama Cpp Serving Sizing]]).
**Kernel gate — this decides which checkpoint can work at all.**
`/app/r4dhip/r4d.so` exports `gemm_moe_mxfp4a8_*` and `gemm_moe_dsfp4a8_*`
but NO `gemm_moe_w4a16_*`. Expert offload requires an experts class with
BOTH a compiled kernel AND an `offload_recipe`; the planner aborts startup
otherwise. Usable: MXFP4, DSFP4, FP8-HIP (`R4dMxfp4MoEExperts`,
`R4dDsfp4MoEExperts`, `Fp8HipMoEExperts`). `R4dW4a16MoEExperts` has no
kernel (the class exists, the r4d.so entry does not); RFA has neither. Check
the checkpoint's quant against this list BEFORE downloading anything.
Everything else about running this image — the quant-kernel gate that decides
which checkpoints can serve at all, the entrypoint contract, the known-good
invocation, the `/dev/dri` and memlock traps, the util and max-model-len arena
traps, the `--ipc=host` / `-d` stdout traps and the gated-fetch discipline — is
written once on [[RDNA4 Container Serving]], and the plan arithmetic behind the
arena/KV traps is [[RDNA4 MoE Expert Offload VRAM Budget]]'s. Three findings
stay here because this harness produced them:
Entrypoint is `/app/tools/image_entrypoint.sh`, ending `exec vllm serve
"$@"` — pass the model path as the FIRST argument and do NOT pass `serve`.
Known-good invocation:
- **The KV reserve undercounts hybrid-attention models ~2×** (the planner's
B/token figure against ~18,000 real: the Mamba state is a fixed per-sequence
cost). Fix by pinning the pool: `--kv-cache-memory-bytes`.
- **`--expert-offload-mem` caps the HOST side**, not the VRAM arena —
structurally orthogonal to `--expert-precise-memory` (verified in-container
2026-10-05: precise-memory only swaps the GPU overhead term,
expert_offload_mem only sets the host bound; they rejoin only inside
`place_experts`, and the plan rewrites `expert_offload_mem` back to the cap
the plan met, so plan and allocation cannot drift).
- **Check whether the stack supports the model natively before porting an
overlay.** The image already registered `Glm5NextForConditionalGeneration`,
`Qwen4ExpForCausalLM` and both MTP variants — chasing four missing symbols
through two image tags to bring an overlay up was entirely wasted work.
```bash
sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'
podman run -d --name <ctr> --ipc=host --ulimit memlock=-1:-1 \
--cap-add SYS_PTRACE \
--device /dev/kfd --device /dev/dri --device /dev/mem \
-e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \
-v /home/dank/models/<model>:/models:ro \
docker.io/tcclaviger/vllm:latest /models \
--served-model-name <name> --port 7770 \
--tensor-parallel-size 2 \
--max-model-len 32768 \
--kv-cache-memory-bytes 2147483648 \
--max-num-seqs 4 \
--gpu-memory-utilization 0.95 \
--enable-prefix-caching --trust-remote-code
```
`-p 7770:7770` is not included, so the API is only reachable via
`podman exec` — add `-p` if host-side curl is needed. Load ~5 min;
`/health` answers connection-refused the whole time; watch for
`Application startup complete` in `podman logs <ctr>`. Verified working:
`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (gated; 172.8 GiB, 131
shards, `Qwen4ExpForConditionalGeneration`, 48 layers, 512 experts top-10,
multimodal, native 262,144).
Traps that each cost a boot:
1. **Mount the whole `/dev/dri`, never individual render nodes.** Render
nodes without the primary `card*` nodes → HIP enumerates two agents and
refuses a context on either (`RuntimeError: No CUDA GPUs are available`)
while `device_count()` cheerfully returns 2. Confirmed by direct A/B.
2. **memlock must be raised** (done on Dankserver via
`/etc/security/limits.d/99-memlock.conf`). Host default 8 MiB makes any
large pinned allocation fail as a misleading
`torch.AcceleratorError: CUDA error: out of memory` from `pin_memory()`;
it cannot be raised from inside a container or root's own shell — only
PAM applies it at login. Matters because `--expert-host-mlock` defaults
True.
3. **`--gpu-memory-utilization` does NOT buy KV under expert offload** —
the planner sizes the arena against the budget and leaves KV the
remainder; raising util 0.95→0.98 grew the arena and the KV pool stayed
at 0.64 GiB.
4. **Lowering `--max-model-len` is the WRONG lever and backfires** — the
budget subtracts the KV reserve first, so a smaller context shrinks the
reserve and hands the space to the arena (measured: 65536 → 0.64 GiB KV;
32768 → 0.34 GiB).
5. **The KV reserve undercounts hybrid-attention models ~2×** (9190 B/token
planned vs ~18,000 real: the Mamba state is a fixed per-sequence cost).
Fix by pinning the pool: `--kv-cache-memory-bytes`.
6. **`--expert-offload-mem` caps the HOST side**, not the VRAM arena —
structurally orthogonal to `--expert-precise-memory` (verified
in-container 2026-10-05: precise-memory only swaps the GPU overhead term,
expert_offload_mem only sets the host bound; they rejoin only inside
`place_experts`, and the plan rewrites `expert_offload_mem` back to the
cap the plan met, so plan and allocation cannot drift).
7. `--ipc=host` and `--shm-size` are mutually exclusive in podman; pick one.
8. `podman run -d` without `-a` sends no stdout to a redirect — the log file
only receives the container ID; use `podman logs`.
Status reads: use `scripts/dankstatus.sh`, never hand-rolled
`/proc/meminfo` parsing — `/proc/meminfo` separates key from value with a
TAB, so `grep MemAvailable | tr -s " " | cut -d" " -f2` returns empty and
awk prints `0.0` — a false "host out of memory" that invites an unnecessary
abort. Gated checkpoints: token via a mode-600 `curl -K` config, never argv
(`pgrep -a curl` prints it); always `--fail`; compare each file's byte count
to the Hub's declared size; `-C -` resumes truncated shards.
**Check whether the stack supports the model natively before porting an
overlay.** The image already registered `Glm5NextForConditionalGeneration`,
`Qwen4ExpForCausalLM` and both MTP variants — chasing four missing symbols
through two image tags to bring an overlay up was entirely wasted work.
The checkpoint this recipe was verified on is
`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (gated; 172.8 GiB, 131 shards,
`Qwen4ExpForConditionalGeneration`, 48 layers, 512 experts top-10, multimodal,
native 262,144) — an [[frank-archon]] era payload.
## Compile-cache persistence (survives `podman rm -f`)
@@ -240,20 +183,15 @@ cold, of which 173 s was one graph).
## Plan-line verification checklist (after any fork deploy)
```
ssh dankserver 'sudo journalctl CONTAINER_NAME=frank-archon-qwen -b -o cat --no-pager \
| grep -aoE "(budget \(total x util\) +[0-9.]+|expert cache space +[0-9.]+ GiB +[0-9,]+ slots|Estimated CUDA graph memory.{0,80}|Graph capturing finished.{0,40}|GPU KV cache size.{0,90})"'
```
`grep -oE` with a digit-anchored tail — plain `grep`/journalctl truncate
these columns to `17.`. Verify: new budget/graphs/arena in the plan;
`GPU KV cache size` matches the requested token count; `Estimated CUDA graph
memory` vs the capture line (raise `_PER_GRAPH_MB` toward 52 if capture
exceeds the charge); `rocm-smi --showmeminfo vram` high-water per card = the
real margin; `systemctl show frank-archon -p NRestarts --value` = 0;
`VLLM_CUDAGRAPH_*` present in `podman exec … env`; then a real chat
completion from a PEER — `curl localhost:7770/health` inside an ssh session
on the host can return `000` and looks like an outage that isn't.
The journal one-liner that reads the plan lines (digit-anchored `grep -oE`,
because plain grep truncates those columns to `17.`) and the post-deploy
verification order — plan vs captured graph memory, `GPU KV cache size` vs the
ask, per-card `rocm-smi` high-water, `NRestarts` = 0, `VLLM_CUDAGRAPH_*` present
in the container env, then a real completion from a PEER because
`curl localhost:7770/health` inside an ssh session on the host can return `000`
and look like an outage — are step 3 of
[[RDNA4 MoE Expert Offload VRAM Budget]] §7. Run that; do not re-derive it
here.
## Related