frank-primary: pin the -ot override set to the real tensor names
Discovery, not guesswork: dump the GGUF v3 headers of the fetched parts (scripts/dump-gguf.py, added — a dependency-free reader that never touches tensor data, so a 50 GiB part scans in milliseconds) before the first load. What the dump says (parts 2-3, 2026-09-26): - The 51B-param PLE n-gram table is ONE tensor: per_layer_token_embd. weight, dims [160, 320001536], Q8_0, 54.4 GiB — exactly the whole of part 3. It is the same tensor the shared MTP head borrows, so the =CPU override is required for placement AND for the draft load (GPU-resident borrow: "invalid vector subscript"). - The model is qwen4exp, 512 experts, top-10 routing (qwen4exp.ple.ngram_size=3, heads_per_ngram=8). - Part 1 is metadata-only (ntensors=0); parts 2-6 carry tensors. The placeholder pattern ngram_embeddings matched nothing — dropped from OT_OVERRIDES (env, serve-script default, seam, AGENTS.md, runbook, LLM index) before it could masquerade as a working override at load time. Deployed set: exps=CPU,per_layer_token_embd.weight=CPU. Suite: 346 green.
This commit is contained in:
@@ -41,13 +41,21 @@ Lucebox DS4.1 quadlet that held the port for hours earlier the same day
|
||||
speed.
|
||||
- **The placement law, not a preference:** the routed experts and the
|
||||
51B-param n-gram (PLE) embedding table are **computed in host RAM**
|
||||
(`-ot exps=CPU,ngram_embeddings=CPU`) and never streamed over PCIe. The
|
||||
(`-ot exps=CPU,per_layer_token_embd.weight=CPU`) and never streamed over PCIe. The
|
||||
x4-pair measurement on the same model shape: streaming loses (3.7 t/s),
|
||||
computing in RAM does not care (20+ t/s). The dense weights, attention,
|
||||
projector and MTP head sit on the two cards (~29 GiB); the ~100 GiB of
|
||||
expert pages live in mmap page cache — `--no-mmap` is forbidden (it
|
||||
forces the table into anonymous memory; the cgroup kill is silent) and
|
||||
pinning the PLE table with a GPU target cost 23 → 13.4 t/s when tried.
|
||||
The tensor names are verified, not guessed: the GGUF v3 header dump
|
||||
(`dump-gguf.py`, run on the fetched parts 2026-09-26) shows the PLE
|
||||
table is one tensor — `per_layer_token_embd.weight`, dims
|
||||
`[160, 320001536]`, Q8_0, 54.4 GiB, exactly the whole of part 3 — and
|
||||
the model is `qwen4exp` with 512 experts top-10
|
||||
(`qwen4exp.ple.ngram_size=3`, `heads_per_ngram=8`). The same tensor is
|
||||
the MTP head's borrow target, so the override is required even if the
|
||||
placement were optional.
|
||||
- **Two cards, layer split** (`-ts 1,1`): both R9700s are on CPU root
|
||||
ports with full PCIe atomics (the ProArt B850-Creator swap,
|
||||
2026-09-20), but this deployment uses llama.cpp's layer split —
|
||||
|
||||
@@ -62,7 +62,7 @@ frank-small, port 7771]
|
||||
unsloth llama.cpp (b11030-mix prebuilt, ROCm gfx120X, bundled runtime)
|
||||
serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 across both R9700s —
|
||||
routed experts and the 51B-param n-gram table computed in host RAM
|
||||
(`-ot exps=CPU,ngram_embeddings=CPU`), the shared Q8_0 MTP head
|
||||
(`-ot exps=CPU,per_layer_token_embd.weight=CPU`), the shared Q8_0 MTP head
|
||||
(depth 2) chained behind ngram-mod, BF16 vision, one full 262,144-token
|
||||
conversation, `reasoning_content`, the placement law, the cutover and
|
||||
rollback
|
||||
|
||||
Reference in New Issue
Block a user