docs(vault): strip restated fleet facts from 17 technique pages

Enforces "one fact, one owner": technique pages keep mechanism, laws, traps
and verdicts; per-endpoint shapes, digests and era measurements are cited to
the endpoint page that owns them.

Also repairs the four pointers the pass left empty (each verified against the
cited owner before landing):
- EMULATION dequant doubles host RAM (118 GiB -> ~238 GiB) -> [[frank-archon]]
- 12.11 tok/s flat 304 -> 16,095 tokens -> [[Tcclaviger Fork Ops]]
- bundled-head acceptance ladder 0.739/0.554/0.427/0.340 -> [[Qwen3.8 PP Deployment]]
- llama.cpp PR #23897 dedup provenance -> [[qwen3.8-spec-matrix]]

Replaces the retired-shape illustration in Llama Cpp Serving Sizing with the
owned MiMo Q5_K_M row (6.40 GiB payload -> 20,530 MiB used), and unwraps a
hard-wrapped [[Dankserver System Specs]] link that failed link integrity.

Orphan census over the diff: 0 numbers lost (comma-insensitive match against
the whole vault plus AGENTS.md).
This commit is contained in:
Frank committed 2026-10-19 01:33:47 -07:00
1 parent eb21dbc705
commit 04f4e2c6d4
18 files changed
+348 -425

No files matched your search

@@ -28,7 +28,7 @@ Related pages:
| 2 | MTP `n_max` explicit 4–5 + Q4_0 draft head | +5–15% on the spec share | restart, 1.56 GB file | test 08-16 |
| 3 | Plain `Q8_0` A/B vs UD-Q8_K_XL (same 8 bits) | dequant ALUs may differ | restart, 29 GB file | test 08-16 |
| 4 | Prefill path: ROCm backend + tuned ubatch/threads-batch | 2–3× prefill for agentic turns | per-workload backend choice | partial data |
| 5 | TTM `pages_limit` raised to 114 GiB | removes the hard VRAM ceiling; the full 262k f16 KV plus the Q8 AGENT payload, MTP draft, and mmproj fit in one slot | modprobe, set live 2026-08-25 | applied (29,883,536 pages) |
| 5 | TTM `pages_limit` raised | removes the hard VRAM ceiling; the full 262k f16 KV plus the Q8 AGENT payload, MTP draft, and mmproj fit in one slot | modprobe, set live 2026-08-25 | applied — value and file: [[Dankclaw System Specs]] (Memory) |
| 6 | Vulkan ICD pinning + loader-layer disable | robustness; small | service env vars | test 08-16 |
Everything else (quantization downgrade, NVFP4, larger drafts) is documented
@@ -62,32 +62,32 @@ Consequences of the hybrid design:
cannot be prefix-cached the way attention KV can. Every agentic turn
re-prefills the whole conversation. **Prefill is the hidden cost of
agentic use**, and it is the dominant cost we have not yet optimized.
3. **MTP is native and strong.** The model ships a nextn head; our matrix
measured 13.3 t/s vs 7.2 t/s (no spec) at 16k on Vulkan.
3. **MTP is native and strong.** The model ships a nextn head; on Vulkan at 16k
it roughly doubled the no-spec rate (the pair of figures is
[[qwen3.8-spec-matrix]]'s realistic-prompt table).
## Current production state (baseline)
## Baseline this page optimized (2026-08-15)
`frank-dragoon` (Dankclaw, Vulkan0, 2026-08-15):
- Model: UD-Q8_K_XL, 31.5 GB, bundled MTP head.
- `--spec-default` (ngram-mod) with draft-mtp first in the chain.
- 2 parallel slots × 262144 ctx, KV q8_0, batch 8192, ubatch 256,
threads 12, `-dio` off, `--fit off`.
- Measured: ~25 t/s at 16k ctx; 5.68 t/s at 262k ctx; prefill 302 t/s @64k
(Vulkan) vs 410 t/s @64k (ROCm, same build).
- GPU: 28 GiB of 96 GiB GTT used. **74 GiB sits idle.**
`frank-dragoon` then ran llama.cpp on the iGPU with the UD-Q8_K_XL target
(bundled MTP head), `--spec-default` ngram-mod under a draft-mtp first link, two
262144 slots, q8_0 KV, batch 8192 / ubatch 256, threads 12. That era's ledger is
[[frank-dragoon]]'s; what this page keeps is the shape of the problem: the
service occupied **28 GiB of the then 96 GiB firmware VRAM carve, ~74 GiB
idle**, decode held ~25 t/s at 16k and collapsed at 262k depth
([[qwen3.8-spec-matrix]] §High-Context Backend Comparison), and prefill was the
unoptimized agentic cost (section 4).
## The levers, in detail
### 1. Pipeline-parallel to the RTX 3090 (biggest win)
The 3090 on Dankserver is idle (serves a 22 GB MoE small-model service with
~2.4 GB free). llama.cpp RPC splits the model by **layers**: each device
runs its layer range, hidden states cross the 2.5 GbE ethernet once per
token. The per-token transfer is ~41 KiB (5120-wide f32 hidden state, both
directions); at ~0.5 ms LAN RTT the bandwidth cost is negligible — the split
is latency-bound, not bandwidth-bound (see
[Ethernet Bandwidth Math](Pipeline%20Parallelism/Ethernet%20Bandwidth%20Math.md)).
The 3090 on Dankserver had headroom that night (it carried the day's small-model
service with a couple of GiB free — that card's eras are [[frank-zealot]]'s).
llama.cpp RPC splits the model by **layers**: each device runs its layer range,
hidden states cross the 2.5 GbE ethernet once per token. The per-position wire
bill is measured in [[Pipeline Parallelism/Ethernet Bandwidth Math]] §0; at
~0.5 ms LAN RTT the bandwidth cost is negligible — the split is latency-bound,
not bandwidth-bound.
Why this is a real speedup and not just "two GPUs": the 3090 does
matmuls roughly 2–3× faster per layer than the 8060S (936 vs 512 GB/s
@@ -95,58 +95,32 @@ memory, 80 vs 40 CUs). A 50/50 layer split balances per-layer latency, so
each token step takes ≈ max(half-on-3090, half-on-Strix) + one RTT instead
of all-64-layers-on-Strix.
Design decisions (verified in b10433 source, 2026-08-15):
- **Head on Dankclaw, worker on Dankserver.** The 31.5 GB model file stays
on the head; the worker receives its tensor share over the wire at load
(with `-c` it caches the tensors locally after the first load).
- **Head must NOT pass `--device`.** When `params.devices` is set,
`llama_prepare_model_devices` builds the device list from exactly those
devices and the RPC device is silently dropped (`--rpc` becomes a no-op).
Default device selection inserts RPC servers **at the front** of the list.
- **The local iGPU side runs on ROCm, not Vulkan.** Default selection walks
registered backends in order; HIP registers before Vulkan, and the iGPU
dedup rule (PR #23897 workaround) keeps the first backend's device.
Verified live 2026-08-15: device list `[RPC0, ROCm0]`, Vulkan0 never
listed. Consequence: the Strix half runs on the ROCm backend — better
prefill, ~3% slower decode than Vulkan0 (measured on this box).
A second head build (`/home/dank/llama.cpp/build-vulkan-rpc`, HIP off,
Vulkan on, RPC on) was built the same night for an A/B; with it the
device list becomes `[RPC0, Vulkan0]`.
- **Default split is proportional to *reported free memory*, which is wrong
here.** The Strix reports up to 96 GiB (GTT ceiling) vs the 3090's 24 GB,
so the default lands ~20/80 in favor of the slow device (verified with a
4B smoke test: 4 of 36 layers on the 3090). We pin `--tensor-split 1,1`
(50/50) explicitly.
- **`-dio` (direct I/O) on the head.** Large model + RPC load hangs without
it (visorcraft Strix finding; matches our 28 GiB load-stall history).
- **Version skew hangs the handshake.** Worker built at the identical
commit (9b05354ec / b10433) in a worktree
`/home/dank/llama-cpp-rpc` on Dankserver — production b10121 build
untouched.
Smoke test (2026-08-15, 4B model, 0.2/0.8 split, production untouched):
worker VRAM rose ~819 MiB (real split), 95 t/s prefill, 58.8 t/s decode on
the tiny model. RTT on the link: ~0.3–0.5 ms.
Full design, scripts, and morning runbook:
[Qwen3.8 PP Deployment](Pipeline%20Parallelism/Qwen3.8%20PP%20Deployment.md).
Every decision this led to is written once, in
[[Pipeline Parallelism/Qwen3.8 PP Deployment]] §Gotchas (with the 4B smoke test
that proved each): head on the box that already holds the model file, **no
`--device` on the head** (setting it silently drops RPC), the fixed
HIP-before-Vulkan device selection, the reported-free-memory split trap, the
identical-build requirement, and `-dio` on the head — the last one visorcraft's
Strix finding, matching our own 28 GiB load-stall history. Scripts and the
morning runbook are on the same page.
### 2. Speculative-decoding tuning (same bits, free tokens)
- **MTP `n_max`.** Production leaves `--spec-draft-n-max` unset. Our matrix
(Vulkan, 16k) found 13.3 t/s at the tested setting vs 7.2 without spec;
the published Strix tuning curve peaks at n_max 3–4 (30.3 t/s at 4 vs
28.9 at 3 vs 29.9 at 5). The PP env sets **4** explicitly; the matrix
re-tests 3/4/5 on the real split.
- **MTP `n_max`.** Production left `--spec-draft-n-max` unset; the fleet matrix
(Vulkan, 16k, realistic prompt) found the depth-5 peak against the no-spec
baseline — both figures are [[qwen3.8-spec-matrix]]'s realistic-prompt table.
The published Strix tuning curve peaks at n_max 3–4 (30.3 t/s at 4 vs 28.9 at
3 vs 29.9 at 5). The PP env set **4** explicitly; the matrix re-tested 3/4/5
on the real split.
- **Q4_0 draft head.** The published Strix card measured a standalone
Q4_0 MTP head ~6–10% faster than Q8_0 for the same acceptance (the draft
head is a 248k-vocab output projection; cheaper dequant, same accuracy).
The bundled head inside UD-Q8_K_XL is Q8-class; the standalone
`ggml-org/mtp-Qwen3.8-27B-Q4_0.gguf` (1.56 GB) is being downloaded for
an A/B via `--spec-draft-model`.
- **Ngram-mod fallback stays.** It is the only spec type that survived
parallel load (DSpark crashed ROCm, DFlash corrupts multi-slot).
- **Ngram-mod fallback stays.** It is the only spec type that survived parallel
load; the DSpark ROCm fault and the DFlash multi-slot corruption are timed and
explained on [[qwen3.8-spec-matrix]].
### 3. Same-bit quant A/B: UD-Q8_K_XL vs plain Q8_0
@@ -162,10 +136,10 @@ zero quality cost.
Every agentic turn re-prefills the whole conversation (no prefix caching for
hybrid state, upstream). Levers:
- **Backend choice per workload.** Measured @64k on this box (b10433):
ROCm 410 t/s vs Vulkan 302 t/s prefill; Vulkan 25.1 vs ROCm 24.3 t/s
decode. For chat-heavy use: Vulkan. For agentic/batch-heavy use: ROCm.
The PP head (ROCm side) gets this for free.
- **Backend choice per workload.** Measured @64k on this box (b10433, the era
[[frank-dragoon]] owns): ROCm 410 t/s vs Vulkan 302 t/s prefill; Vulkan 25.1
vs ROCm 24.3 t/s decode. For chat-heavy use: Vulkan. For agentic/batch-heavy
use: ROCm. The PP head (ROCm side) gets this for free.
- **ubatch/threads-batch.** Production runs ubatch 256. Larger ubatches
(512/1024) fill the iGPU better during prefill; `--threads-batch` for
the CPU-bound tail. Untested at prefill scale on this box — matrix item.
@@ -175,14 +149,14 @@ hybrid state, upstream). Levers:
### 5. OS level (from the Improvements doc, re-verified 2026-08-15)
- **TTM `pages_limit` is now 114 GiB (29,883,536 pages)** — applied 2026-08-25.
Written live to `/sys/module/ttm/parameters/pages_limit` (no reboot required)
and persisted in `/etc/modprobe.d/ttm.conf`. The full 262k f16 KV cache plus
the ~28 GB Q8 AGENT payload, MTP draft, and mmproj now fit in one slot. The
load-stall risk from the old 15 GiB cap is resolved. Note: 96 GiB is the
driver-reported VRAM carve-out (VGM/SVM pool); the TTM pages_limit is the
kernel page map that was the real ceiling.
- `iommu=pt` — verified present on the kernel cmdline. Done.
- **TTM `pages_limit`** — the raise itself (value, live write, modprobe file,
date) is one row on [[Dankclaw System Specs]] (Memory). What matters to this
page is the mechanism: the driver-reported 96 GiB VRAM carve-out (VGM/SVM
pool) was never the ceiling — the TTM kernel page map was, and until it was
raised a 262k f16 KV cache plus a ~28 GB payload could not load in one slot.
- **Boot line** — IOMMU policy on this host is [[dankclaw-iommu-translation-off]]'s
and [[Dankclaw System Specs]]'s; it has changed twice since this lever was
written, so no era-scoped reading of `/proc/cmdline` belongs here.
- **Vulkan ICD pinning** — `radeon_icd.json` exists; no `VK_ICD_FILENAMES`
or loader-layer env is set. Pinning the radeon ICD and disabling loader
layers (`VK_LOADER_LAYERS_DISABLE=all`) is the recommended Strix
@@ -240,8 +214,8 @@ Per config: `pp-test.sh bench 4` (4k prompt / 256 out) and `bench 16`
(16k / 256). Report: decode t/s (server print_timing), prefill t/s, MTP
acceptance, worker VRAM, head GTT usage.
Success bar: **≥ 30 t/s at 4k and ≥ 20 t/s at 16k decode** (vs 25 t/s
production @16k) with the same 8-bit model.
Success bar: **≥ 30 t/s at 4k and ≥ 20 t/s at 16k decode**, against the
production @16k bar in the Baseline section, with the same 8-bit model.
## Sources