frank-primary: pin the -ot override set to the real tensor names

Discovery, not guesswork: dump the GGUF v3 headers of the fetched parts
(scripts/dump-gguf.py, added — a dependency-free reader that never
touches tensor data, so a 50 GiB part scans in milliseconds) before
the first load.

What the dump says (parts 2-3, 2026-09-26):

- The 51B-param PLE n-gram table is ONE tensor: per_layer_token_embd.
  weight, dims [160, 320001536], Q8_0, 54.4 GiB — exactly the whole of
  part 3. It is the same tensor the shared MTP head borrows, so the
  =CPU override is required for placement AND for the draft load
  (GPU-resident borrow: "invalid vector subscript").
- The model is qwen4exp, 512 experts, top-10 routing
  (qwen4exp.ple.ngram_size=3, heads_per_ngram=8).
- Part 1 is metadata-only (ntensors=0); parts 2-6 carry tensors.

The placeholder pattern ngram_embeddings matched nothing — dropped from
OT_OVERRIDES (env, serve-script default, seam, AGENTS.md, runbook,
LLM index) before it could masquerade as a working override at load
time. Deployed set: exps=CPU,per_layer_token_embd.weight=CPU.

Suite: 346 green.
This commit is contained in:
Frank committed 2026-09-26 19:51:57 -07:00
1 parent 8500600d09
commit a9b94a79eb
7 files changed
+137 -18

No files matched your search

+1 -1
View File
@@ -59,12 +59,12 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
# computing them in RAM does not care (20+ tok/s on the same model
# shape). exps=CPU plus the n-gram embedding table on CPU; the PLE
# table must NEVER carry a GPU target (the 23 -> 13.4 t/s regression).
grep -q '^OT_OVERRIDES=exps=CPU,ngram_embeddings=CPU,per_layer_token_embd.weight=CPU$' "$ENV_FILE"
grep -q -- '--override-tensor "$ot"' "$SERVE"
# ...and mmap must stay on: --no-mmap forces the 51B-param table into
# anonymous memory and the cgroup kill is silent.
! grep -q -- '--no-mmap' "$SERVE"
! grep -q '^MMPROJ_ON_GPU' "$ENV_FILE"
grep -q '^OT_OVERRIDES=exps=CPU,per_layer_token_embd.weight=CPU$' "$ENV_FILE"
}
@test "the projector is explicit, never autoloaded" {