docs(vault): strip restated fleet facts from 17 technique pages
Enforces "one fact, one owner": technique pages keep mechanism, laws, traps and verdicts; per-endpoint shapes, digests and era measurements are cited to the endpoint page that owns them. Also repairs the four pointers the pass left empty (each verified against the cited owner before landing): - EMULATION dequant doubles host RAM (118 GiB -> ~238 GiB) -> [[frank-archon]] - 12.11 tok/s flat 304 -> 16,095 tokens -> [[Tcclaviger Fork Ops]] - bundled-head acceptance ladder 0.739/0.554/0.427/0.340 -> [[Qwen3.8 PP Deployment]] - llama.cpp PR #23897 dedup provenance -> [[qwen3.8-spec-matrix]] Replaces the retired-shape illustration in Llama Cpp Serving Sizing with the owned MiMo Q5_K_M row (6.40 GiB payload -> 20,530 MiB used), and unwraps a hard-wrapped [[Dankserver System Specs]] link that failed link integrity. Orphan census over the diff: 0 numbers lost (comma-insensitive match against the whole vault plus AGENTS.md).
This commit is contained in:
1 parent
eb21dbc705
commit
04f4e2c6d4
18 files changed
+348
-425
No files matched your search
@@ -28,7 +28,7 @@ Related pages:
|
||||
| 2 | MTP `n_max` explicit 4–5 + Q4_0 draft head | +5–15% on the spec share | restart, 1.56 GB file | test 08-16 |
|
||||
| 3 | Plain `Q8_0` A/B vs UD-Q8_K_XL (same 8 bits) | dequant ALUs may differ | restart, 29 GB file | test 08-16 |
|
||||
| 4 | Prefill path: ROCm backend + tuned ubatch/threads-batch | 2–3× prefill for agentic turns | per-workload backend choice | partial data |
|
||||
| 5 | TTM `pages_limit` raised to 114 GiB | removes the hard VRAM ceiling; the full 262k f16 KV plus the Q8 AGENT payload, MTP draft, and mmproj fit in one slot | modprobe, set live 2026-08-25 | applied (29,883,536 pages) |
|
||||
| 5 | TTM `pages_limit` raised | removes the hard VRAM ceiling; the full 262k f16 KV plus the Q8 AGENT payload, MTP draft, and mmproj fit in one slot | modprobe, set live 2026-08-25 | applied — value and file: [[Dankclaw System Specs]] (Memory) |
|
||||
| 6 | Vulkan ICD pinning + loader-layer disable | robustness; small | service env vars | test 08-16 |
|
||||
|
||||
Everything else (quantization downgrade, NVFP4, larger drafts) is documented
|
||||
@@ -62,32 +62,32 @@ Consequences of the hybrid design:
|
||||
cannot be prefix-cached the way attention KV can. Every agentic turn
|
||||
re-prefills the whole conversation. **Prefill is the hidden cost of
|
||||
agentic use**, and it is the dominant cost we have not yet optimized.
|
||||
3. **MTP is native and strong.** The model ships a nextn head; our matrix
|
||||
measured 13.3 t/s vs 7.2 t/s (no spec) at 16k on Vulkan.
|
||||
3. **MTP is native and strong.** The model ships a nextn head; on Vulkan at 16k
|
||||
it roughly doubled the no-spec rate (the pair of figures is
|
||||
[[qwen3.8-spec-matrix]]'s realistic-prompt table).
|
||||
|
||||
## Current production state (baseline)
|
||||
## Baseline this page optimized (2026-08-15)
|
||||
|
||||
`frank-dragoon` (Dankclaw, Vulkan0, 2026-08-15):
|
||||
|
||||
- Model: UD-Q8_K_XL, 31.5 GB, bundled MTP head.
|
||||
- `--spec-default` (ngram-mod) with draft-mtp first in the chain.
|
||||
- 2 parallel slots × 262144 ctx, KV q8_0, batch 8192, ubatch 256,
|
||||
threads 12, `-dio` off, `--fit off`.
|
||||
- Measured: ~25 t/s at 16k ctx; 5.68 t/s at 262k ctx; prefill 302 t/s @64k
|
||||
(Vulkan) vs 410 t/s @64k (ROCm, same build).
|
||||
- GPU: 28 GiB of 96 GiB GTT used. **74 GiB sits idle.**
|
||||
`frank-dragoon` then ran llama.cpp on the iGPU with the UD-Q8_K_XL target
|
||||
(bundled MTP head), `--spec-default` ngram-mod under a draft-mtp first link, two
|
||||
262144 slots, q8_0 KV, batch 8192 / ubatch 256, threads 12. That era's ledger is
|
||||
[[frank-dragoon]]'s; what this page keeps is the shape of the problem: the
|
||||
service occupied **28 GiB of the then 96 GiB firmware VRAM carve, ~74 GiB
|
||||
idle**, decode held ~25 t/s at 16k and collapsed at 262k depth
|
||||
([[qwen3.8-spec-matrix]] §High-Context Backend Comparison), and prefill was the
|
||||
unoptimized agentic cost (section 4).
|
||||
|
||||
## The levers, in detail
|
||||
|
||||
### 1. Pipeline-parallel to the RTX 3090 (biggest win)
|
||||
|
||||
The 3090 on Dankserver is idle (serves a 22 GB MoE small-model service with
|
||||
~2.4 GB free). llama.cpp RPC splits the model by **layers**: each device
|
||||
runs its layer range, hidden states cross the 2.5 GbE ethernet once per
|
||||
token. The per-token transfer is ~41 KiB (5120-wide f32 hidden state, both
|
||||
directions); at ~0.5 ms LAN RTT the bandwidth cost is negligible — the split
|
||||
is latency-bound, not bandwidth-bound (see
|
||||
[Ethernet Bandwidth Math](Pipeline%20Parallelism/Ethernet%20Bandwidth%20Math.md)).
|
||||
The 3090 on Dankserver had headroom that night (it carried the day's small-model
|
||||
service with a couple of GiB free — that card's eras are [[frank-zealot]]'s).
|
||||
llama.cpp RPC splits the model by **layers**: each device runs its layer range,
|
||||
hidden states cross the 2.5 GbE ethernet once per token. The per-position wire
|
||||
bill is measured in [[Pipeline Parallelism/Ethernet Bandwidth Math]] §0; at
|
||||
~0.5 ms LAN RTT the bandwidth cost is negligible — the split is latency-bound,
|
||||
not bandwidth-bound.
|
||||
|
||||
Why this is a real speedup and not just "two GPUs": the 3090 does
|
||||
matmuls roughly 2–3× faster per layer than the 8060S (936 vs 512 GB/s
|
||||
@@ -95,58 +95,32 @@ memory, 80 vs 40 CUs). A 50/50 layer split balances per-layer latency, so
|
||||
each token step takes ≈ max(half-on-3090, half-on-Strix) + one RTT instead
|
||||
of all-64-layers-on-Strix.
|
||||
|
||||
Design decisions (verified in b10433 source, 2026-08-15):
|
||||
|
||||
- **Head on Dankclaw, worker on Dankserver.** The 31.5 GB model file stays
|
||||
on the head; the worker receives its tensor share over the wire at load
|
||||
(with `-c` it caches the tensors locally after the first load).
|
||||
- **Head must NOT pass `--device`.** When `params.devices` is set,
|
||||
`llama_prepare_model_devices` builds the device list from exactly those
|
||||
devices and the RPC device is silently dropped (`--rpc` becomes a no-op).
|
||||
Default device selection inserts RPC servers **at the front** of the list.
|
||||
- **The local iGPU side runs on ROCm, not Vulkan.** Default selection walks
|
||||
registered backends in order; HIP registers before Vulkan, and the iGPU
|
||||
dedup rule (PR #23897 workaround) keeps the first backend's device.
|
||||
Verified live 2026-08-15: device list `[RPC0, ROCm0]`, Vulkan0 never
|
||||
listed. Consequence: the Strix half runs on the ROCm backend — better
|
||||
prefill, ~3% slower decode than Vulkan0 (measured on this box).
|
||||
A second head build (`/home/dank/llama.cpp/build-vulkan-rpc`, HIP off,
|
||||
Vulkan on, RPC on) was built the same night for an A/B; with it the
|
||||
device list becomes `[RPC0, Vulkan0]`.
|
||||
- **Default split is proportional to *reported free memory*, which is wrong
|
||||
here.** The Strix reports up to 96 GiB (GTT ceiling) vs the 3090's 24 GB,
|
||||
so the default lands ~20/80 in favor of the slow device (verified with a
|
||||
4B smoke test: 4 of 36 layers on the 3090). We pin `--tensor-split 1,1`
|
||||
(50/50) explicitly.
|
||||
- **`-dio` (direct I/O) on the head.** Large model + RPC load hangs without
|
||||
it (visorcraft Strix finding; matches our 28 GiB load-stall history).
|
||||
- **Version skew hangs the handshake.** Worker built at the identical
|
||||
commit (9b05354ec / b10433) in a worktree
|
||||
`/home/dank/llama-cpp-rpc` on Dankserver — production b10121 build
|
||||
untouched.
|
||||
|
||||
Smoke test (2026-08-15, 4B model, 0.2/0.8 split, production untouched):
|
||||
worker VRAM rose ~819 MiB (real split), 95 t/s prefill, 58.8 t/s decode on
|
||||
the tiny model. RTT on the link: ~0.3–0.5 ms.
|
||||
|
||||
Full design, scripts, and morning runbook:
|
||||
[Qwen3.8 PP Deployment](Pipeline%20Parallelism/Qwen3.8%20PP%20Deployment.md).
|
||||
Every decision this led to is written once, in
|
||||
[[Pipeline Parallelism/Qwen3.8 PP Deployment]] §Gotchas (with the 4B smoke test
|
||||
that proved each): head on the box that already holds the model file, **no
|
||||
`--device` on the head** (setting it silently drops RPC), the fixed
|
||||
HIP-before-Vulkan device selection, the reported-free-memory split trap, the
|
||||
identical-build requirement, and `-dio` on the head — the last one visorcraft's
|
||||
Strix finding, matching our own 28 GiB load-stall history. Scripts and the
|
||||
morning runbook are on the same page.
|
||||
|
||||
### 2. Speculative-decoding tuning (same bits, free tokens)
|
||||
|
||||
- **MTP `n_max`.** Production leaves `--spec-draft-n-max` unset. Our matrix
|
||||
(Vulkan, 16k) found 13.3 t/s at the tested setting vs 7.2 without spec;
|
||||
the published Strix tuning curve peaks at n_max 3–4 (30.3 t/s at 4 vs
|
||||
28.9 at 3 vs 29.9 at 5). The PP env sets **4** explicitly; the matrix
|
||||
re-tests 3/4/5 on the real split.
|
||||
- **MTP `n_max`.** Production left `--spec-draft-n-max` unset; the fleet matrix
|
||||
(Vulkan, 16k, realistic prompt) found the depth-5 peak against the no-spec
|
||||
baseline — both figures are [[qwen3.8-spec-matrix]]'s realistic-prompt table.
|
||||
The published Strix tuning curve peaks at n_max 3–4 (30.3 t/s at 4 vs 28.9 at
|
||||
3 vs 29.9 at 5). The PP env set **4** explicitly; the matrix re-tested 3/4/5
|
||||
on the real split.
|
||||
- **Q4_0 draft head.** The published Strix card measured a standalone
|
||||
Q4_0 MTP head ~6–10% faster than Q8_0 for the same acceptance (the draft
|
||||
head is a 248k-vocab output projection; cheaper dequant, same accuracy).
|
||||
The bundled head inside UD-Q8_K_XL is Q8-class; the standalone
|
||||
`ggml-org/mtp-Qwen3.8-27B-Q4_0.gguf` (1.56 GB) is being downloaded for
|
||||
an A/B via `--spec-draft-model`.
|
||||
- **Ngram-mod fallback stays.** It is the only spec type that survived
|
||||
parallel load (DSpark crashed ROCm, DFlash corrupts multi-slot).
|
||||
- **Ngram-mod fallback stays.** It is the only spec type that survived parallel
|
||||
load; the DSpark ROCm fault and the DFlash multi-slot corruption are timed and
|
||||
explained on [[qwen3.8-spec-matrix]].
|
||||
|
||||
### 3. Same-bit quant A/B: UD-Q8_K_XL vs plain Q8_0
|
||||
|
||||
@@ -162,10 +136,10 @@ zero quality cost.
|
||||
Every agentic turn re-prefills the whole conversation (no prefix caching for
|
||||
hybrid state, upstream). Levers:
|
||||
|
||||
- **Backend choice per workload.** Measured @64k on this box (b10433):
|
||||
ROCm 410 t/s vs Vulkan 302 t/s prefill; Vulkan 25.1 vs ROCm 24.3 t/s
|
||||
decode. For chat-heavy use: Vulkan. For agentic/batch-heavy use: ROCm.
|
||||
The PP head (ROCm side) gets this for free.
|
||||
- **Backend choice per workload.** Measured @64k on this box (b10433, the era
|
||||
[[frank-dragoon]] owns): ROCm 410 t/s vs Vulkan 302 t/s prefill; Vulkan 25.1
|
||||
vs ROCm 24.3 t/s decode. For chat-heavy use: Vulkan. For agentic/batch-heavy
|
||||
use: ROCm. The PP head (ROCm side) gets this for free.
|
||||
- **ubatch/threads-batch.** Production runs ubatch 256. Larger ubatches
|
||||
(512/1024) fill the iGPU better during prefill; `--threads-batch` for
|
||||
the CPU-bound tail. Untested at prefill scale on this box — matrix item.
|
||||
@@ -175,14 +149,14 @@ hybrid state, upstream). Levers:
|
||||
|
||||
### 5. OS level (from the Improvements doc, re-verified 2026-08-15)
|
||||
|
||||
- **TTM `pages_limit` is now 114 GiB (29,883,536 pages)** — applied 2026-08-25.
|
||||
Written live to `/sys/module/ttm/parameters/pages_limit` (no reboot required)
|
||||
and persisted in `/etc/modprobe.d/ttm.conf`. The full 262k f16 KV cache plus
|
||||
the ~28 GB Q8 AGENT payload, MTP draft, and mmproj now fit in one slot. The
|
||||
load-stall risk from the old 15 GiB cap is resolved. Note: 96 GiB is the
|
||||
driver-reported VRAM carve-out (VGM/SVM pool); the TTM pages_limit is the
|
||||
kernel page map that was the real ceiling.
|
||||
- `iommu=pt` — verified present on the kernel cmdline. Done.
|
||||
- **TTM `pages_limit`** — the raise itself (value, live write, modprobe file,
|
||||
date) is one row on [[Dankclaw System Specs]] (Memory). What matters to this
|
||||
page is the mechanism: the driver-reported 96 GiB VRAM carve-out (VGM/SVM
|
||||
pool) was never the ceiling — the TTM kernel page map was, and until it was
|
||||
raised a 262k f16 KV cache plus a ~28 GB payload could not load in one slot.
|
||||
- **Boot line** — IOMMU policy on this host is [[dankclaw-iommu-translation-off]]'s
|
||||
and [[Dankclaw System Specs]]'s; it has changed twice since this lever was
|
||||
written, so no era-scoped reading of `/proc/cmdline` belongs here.
|
||||
- **Vulkan ICD pinning** — `radeon_icd.json` exists; no `VK_ICD_FILENAMES`
|
||||
or loader-layer env is set. Pinning the radeon ICD and disabling loader
|
||||
layers (`VK_LOADER_LAYERS_DISABLE=all`) is the recommended Strix
|
||||
@@ -240,8 +214,8 @@ Per config: `pp-test.sh bench 4` (4k prompt / 256 out) and `bench 16`
|
||||
(16k / 256). Report: decode t/s (server print_timing), prefill t/s, MTP
|
||||
acceptance, worker VRAM, head GTT usage.
|
||||
|
||||
Success bar: **≥ 30 t/s at 4k and ≥ 20 t/s at 16k decode** (vs 25 t/s
|
||||
production @16k) with the same 8-bit model.
|
||||
Success bar: **≥ 30 t/s at 4k and ≥ 20 t/s at 16k decode**, against the
|
||||
production @16k bar in the Baseline section, with the same 8-bit model.
|
||||
|
||||
## Sources
|
||||
|
||||
|
||||
Reference in new issue
Block a user