frank-primary: pin the -ot override set to the real tensor names

Discovery, not guesswork: dump the GGUF v3 headers of the fetched parts
(scripts/dump-gguf.py, added — a dependency-free reader that never
touches tensor data, so a 50 GiB part scans in milliseconds) before
the first load.

What the dump says (parts 2-3, 2026-09-26):

- The 51B-param PLE n-gram table is ONE tensor: per_layer_token_embd.
  weight, dims [160, 320001536], Q8_0, 54.4 GiB — exactly the whole of
  part 3. It is the same tensor the shared MTP head borrows, so the
  =CPU override is required for placement AND for the draft load
  (GPU-resident borrow: "invalid vector subscript").
- The model is qwen4exp, 512 experts, top-10 routing
  (qwen4exp.ple.ngram_size=3, heads_per_ngram=8).
- Part 1 is metadata-only (ntensors=0); parts 2-6 carry tensors.

The placeholder pattern ngram_embeddings matched nothing — dropped from
OT_OVERRIDES (env, serve-script default, seam, AGENTS.md, runbook,
LLM index) before it could masquerade as a working override at load
time. Deployed set: exps=CPU,per_layer_token_embd.weight=CPU.

Suite: 346 green.
This commit is contained in:
Frank
2026-09-26 19:51:57 -07:00
parent 8500600d09
commit a9b94a79eb
7 changed files with 137 additions and 18 deletions
@@ -41,13 +41,21 @@ Lucebox DS4.1 quadlet that held the port for hours earlier the same day
speed.
- **The placement law, not a preference:** the routed experts and the
51B-param n-gram (PLE) embedding table are **computed in host RAM**
(`-ot exps=CPU,ngram_embeddings=CPU`) and never streamed over PCIe. The
(`-ot exps=CPU,per_layer_token_embd.weight=CPU`) and never streamed over PCIe. The
x4-pair measurement on the same model shape: streaming loses (3.7 t/s),
computing in RAM does not care (20+ t/s). The dense weights, attention,
projector and MTP head sit on the two cards (~29 GiB); the ~100 GiB of
expert pages live in mmap page cache — `--no-mmap` is forbidden (it
forces the table into anonymous memory; the cgroup kill is silent) and
pinning the PLE table with a GPU target cost 23 → 13.4 t/s when tried.
The tensor names are verified, not guessed: the GGUF v3 header dump
(`dump-gguf.py`, run on the fetched parts 2026-09-26) shows the PLE
table is one tensor — `per_layer_token_embd.weight`, dims
`[160, 320001536]`, Q8_0, 54.4 GiB, exactly the whole of part 3 — and
the model is `qwen4exp` with 512 experts top-10
(`qwen4exp.ple.ngram_size=3`, `heads_per_ngram=8`). The same tensor is
the MTP head's borrow target, so the override is required even if the
placement were optional.
- **Two cards, layer split** (`-ts 1,1`): both R9700s are on CPU root
ports with full PCIe atomics (the ProArt B850-Creator swap,
2026-09-20), but this deployment uses llama.cpp's layer split —
+1 -1
View File
@@ -62,7 +62,7 @@ frank-small, port 7771]
unsloth llama.cpp (b11030-mix prebuilt, ROCm gfx120X, bundled runtime)
serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 across both R9700s —
routed experts and the 51B-param n-gram table computed in host RAM
(`-ot exps=CPU,ngram_embeddings=CPU`), the shared Q8_0 MTP head
(`-ot exps=CPU,per_layer_token_embd.weight=CPU`), the shared Q8_0 MTP head
(depth 2) chained behind ngram-mod, BF16 vision, one full 262,144-token
conversation, `reasoning_content`, the placement law, the cutover and
rollback