frank-primary: pin the -ot override set to the real tensor names
Discovery, not guesswork: dump the GGUF v3 headers of the fetched parts (scripts/dump-gguf.py, added — a dependency-free reader that never touches tensor data, so a 50 GiB part scans in milliseconds) before the first load. What the dump says (parts 2-3, 2026-09-26): - The 51B-param PLE n-gram table is ONE tensor: per_layer_token_embd. weight, dims [160, 320001536], Q8_0, 54.4 GiB — exactly the whole of part 3. It is the same tensor the shared MTP head borrows, so the =CPU override is required for placement AND for the draft load (GPU-resident borrow: "invalid vector subscript"). - The model is qwen4exp, 512 experts, top-10 routing (qwen4exp.ple.ngram_size=3, heads_per_ngram=8). - Part 1 is metadata-only (ntensors=0); parts 2-6 carry tensors. The placeholder pattern ngram_embeddings matched nothing — dropped from OT_OVERRIDES (env, serve-script default, seam, AGENTS.md, runbook, LLM index) before it could masquerade as a working override at load time. Deployed set: exps=CPU,per_layer_token_embd.weight=CPU. Suite: 346 green.
This commit is contained in:
1 parent
8500600d09
commit
a9b94a79eb
7 files changed
+137
-18
No files matched your search
@@ -62,7 +62,7 @@ frank-small, port 7771]
|
||||
unsloth llama.cpp (b11030-mix prebuilt, ROCm gfx120X, bundled runtime)
|
||||
serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 across both R9700s —
|
||||
routed experts and the 51B-param n-gram table computed in host RAM
|
||||
(`-ot exps=CPU,ngram_embeddings=CPU`), the shared Q8_0 MTP head
|
||||
(`-ot exps=CPU,per_layer_token_embd.weight=CPU`), the shared Q8_0 MTP head
|
||||
(depth 2) chained behind ngram-mod, BF16 vision, one full 262,144-token
|
||||
conversation, `reasoning_content`, the placement law, the cutover and
|
||||
rollback
|
||||
|
||||
Reference in new issue
Block a user