frank-primary: pin the -ot override set to the real tensor names
Discovery, not guesswork: dump the GGUF v3 headers of the fetched parts (scripts/dump-gguf.py, added — a dependency-free reader that never touches tensor data, so a 50 GiB part scans in milliseconds) before the first load. What the dump says (parts 2-3, 2026-09-26): - The 51B-param PLE n-gram table is ONE tensor: per_layer_token_embd. weight, dims [160, 320001536], Q8_0, 54.4 GiB — exactly the whole of part 3. It is the same tensor the shared MTP head borrows, so the =CPU override is required for placement AND for the draft load (GPU-resident borrow: "invalid vector subscript"). - The model is qwen4exp, 512 experts, top-10 routing (qwen4exp.ple.ngram_size=3, heads_per_ngram=8). - Part 1 is metadata-only (ntensors=0); parts 2-6 carry tensors. The placeholder pattern ngram_embeddings matched nothing — dropped from OT_OVERRIDES (env, serve-script default, seam, AGENTS.md, runbook, LLM index) before it could masquerade as a working override at load time. Deployed set: exps=CPU,per_layer_token_embd.weight=CPU. Suite: 346 green.
This commit is contained in:
1 parent
8500600d09
commit
a9b94a79eb
7 files changed
+137
-18
No files matched your search
+16
-12
@@ -43,19 +43,23 @@ SPEC_DRAFT_N_MAX=2
|
||||
# RAM (computed there — the x4-pair measurement: streaming over PCIe loses,
|
||||
# computing in RAM does not). OT_OVERRIDES is comma-separated
|
||||
# <tensor-pattern>=<backend> pairs passed as repeated --override-tensor
|
||||
# flags:
|
||||
# * exps=CPU — the routed experts. ~100 GiB at Q8, read once per token;
|
||||
# the CPU backend computes them from page cache.
|
||||
# * the n-gram embedding block (the 51B-param PLE table) — sparse row
|
||||
# lookups, tiny PCIe traffic from CPU RAM; NEVER pin with a GPU target
|
||||
# (the 23 -> 13.4 t/s regression).
|
||||
# * per_layer_token_embd.weight=CPU — REQUIRED by the shared MTP head's
|
||||
# borrow path: GPU-resident, it crashes the draft load with
|
||||
# "invalid vector subscript" (llama.cpp PR #28243).
|
||||
# The exact tensor pattern for the n-gram block was discovered at load time
|
||||
# on 2026-09-26 (see the runbook); this list is the deployed set.
|
||||
# flags. Both overrides are pinned by the header dump of parts 2-3
|
||||
# (2026-09-26, /home/dank/models/dump-gguf.py — GGUF v3, 512 experts
|
||||
# top-10, qwen4exp.ple.ngram_size=3, heads_per_ngram=8):
|
||||
# * exps=CPU — the routed experts (blk.*.ffn_*_exps.weight, ~100 GiB at
|
||||
# Q8_0), read once per token; the CPU backend computes them from page
|
||||
# cache.
|
||||
# * per_layer_token_embd.weight=CPU — the 51.2B-element PLE n-gram table
|
||||
# (dims [160, 320001536], Q8_0, 54.4 GiB — the whole of part 3):
|
||||
# sparse row lookups, tiny PCIe traffic from CPU RAM. It doubles as
|
||||
# the shared MTP head's borrow tensor — GPU-resident, the draft load
|
||||
# crashes with "invalid vector subscript" (llama.cpp PR #28243) — so
|
||||
# this override is REQUIRED, not just a placement choice. NEVER give
|
||||
# it a GPU target (the 23 -> 13.4 t/s regression) and never --no-mmap
|
||||
# (the table must stay page-cache-backed; anonymous memory
|
||||
# cgroup-kills silently).
|
||||
NGL=999
|
||||
OT_OVERRIDES=exps=CPU,ngram_embeddings=CPU,per_layer_token_embd.weight=CPU
|
||||
OT_OVERRIDES=exps=CPU,per_layer_token_embd.weight=CPU
|
||||
TENSOR_SPLIT=1,1
|
||||
|
||||
# The drafter lives on the first R9700 (the card the load log shows holding
|
||||
|
||||
Reference in new issue
Block a user