frank-primary: pin the -ot override set to the real tensor names
Discovery, not guesswork: dump the GGUF v3 headers of the fetched parts (scripts/dump-gguf.py, added — a dependency-free reader that never touches tensor data, so a 50 GiB part scans in milliseconds) before the first load. What the dump says (parts 2-3, 2026-09-26): - The 51B-param PLE n-gram table is ONE tensor: per_layer_token_embd. weight, dims [160, 320001536], Q8_0, 54.4 GiB — exactly the whole of part 3. It is the same tensor the shared MTP head borrows, so the =CPU override is required for placement AND for the draft load (GPU-resident borrow: "invalid vector subscript"). - The model is qwen4exp, 512 experts, top-10 routing (qwen4exp.ple.ngram_size=3, heads_per_ngram=8). - Part 1 is metadata-only (ntensors=0); parts 2-6 carry tensors. The placeholder pattern ngram_embeddings matched nothing — dropped from OT_OVERRIDES (env, serve-script default, seam, AGENTS.md, runbook, LLM index) before it could masquerade as a working override at load time. Deployed set: exps=CPU,per_layer_token_embd.weight=CPU. Suite: 346 green.
This commit is contained in:
Frank
@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
|
||||
| Target | Service | Runtime | Model | Port |
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-primary` | unsloth llama.cpp b11030-mix prebuilt (ROCm gfx120X, bundled runtime, root systemd unit) | unsloth Qwen3.8-Flash-Next-GGUF Q_8 (175.30 GiB, 6 split parts) with the shared Q_8 MTP head (depth 2) chained behind `ngram-mod`, BF16 vision (mmproj), layer split `-ts 1,1` across both R9700s with the routed experts + 51B-param n-gram table computed in host RAM (`-ot exps=CPU,ngram_embeddings=CPU,per_layer_token_embd.weight=CPU` — the shared-head borrow path needs the last), 262144 ctx single stream (1 slot), mmap on, `--cache-ram 16384` | 7770 |
|
||||
| Dankserver | `frank-primary` | unsloth llama.cpp b11030-mix prebuilt (ROCm gfx120X, bundled runtime, root systemd unit) | unsloth Qwen3.8-Flash-Next-GGUF Q_8 (175.30 GiB, 6 split parts) with the shared Q_8 MTP head (depth 2) chained behind `ngram-mod`, BF16 vision (mmproj), layer split `-ts 1,1` across both R9700s with the routed experts + 51B-param n-gram table computed in host RAM (`-ot exps=CPU,per_layer_token_embd.weight=CPU` — the shared-head borrow path needs the last), 262144 ctx single stream (1 slot), mmap on, `--cache-ram 16384` | 7770 |
|
||||
| Dankclaw | `frank-large` | unsloth llama.cpp b11030 prebuilt (ROCm gfx1151, root systemd unit) | unsloth Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 31.5 GiB) with the z-lab DFlash2 Q4_K_M drafter (depth 4) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single stream (1 slot), q8_0 KV, text only | 7770 |
|
||||
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 |
|
||||
|
||||
@@ -355,7 +355,7 @@ old) on 2026-09-26, which had replaced radiance frank-medium the same day.
|
||||
**Placement: compute the experts in RAM, never stream them.** Layer split
|
||||
`-ts 1,1` with `-ngl 999`; the routed experts and the 51B-param n-gram
|
||||
(PLE) table are overridden to CPU (`-ot
|
||||
exps=CPU,ngram_embeddings=CPU,per_layer_token_embd.weight=CPU`) and computed
|
||||
exps=CPU,per_layer_token_embd.weight=CPU`) and computed
|
||||
from the mmap page cache — the x4-pair law (streaming over PCIe loses at
|
||||
3.7 t/s; computing in RAM does not care). `per_layer_token_embd.weight=CPU`
|
||||
is REQUIRED by the shared MTP head's borrow path (a GPU-resident borrow
|
||||
|
||||
Executable
+107
@@ -0,0 +1,107 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Dump GGUF part headers: split metadata, tensor names/dims/types/bytes.
|
||||
|
||||
Reads ONLY the header region (metadata + tensor-info table); tensor data is
|
||||
never touched, so a 50 GiB part scans in milliseconds. Split GGUFs are valid
|
||||
GGUFs per part; part 1 of this payload is metadata-only (ntensors=0) and
|
||||
parts 2-6 carry the tensors. Confirmed layout for this payload: GGUF v3 —
|
||||
magic, u32 version, u64 ntensors, u64 nkv, then kvs (v3: u64-length strings),
|
||||
then tensor infos (v3: u64-length names).
|
||||
"""
|
||||
import struct, sys, traceback
|
||||
|
||||
TYPE_SIZE = {0:4,1:2,2:1,3:1/2,4:4,5:8,6:2,7:1,8:0.4375,9:1.4375,10:1,11:0.5,
|
||||
12:2,13:4/3,14:2,15:2,16:4,17:2,18:2,19:1,20:2,21:1,22:2,23:2,
|
||||
24:2,25:1.0625,26:1,27:1.375,28:2,29:2,30:2,31:4,32:8,33:2.125,
|
||||
34:2.5,35:2,36:2.375,37:1.5625,38:2.3125,39:2.1875,40:0,41:1.75,
|
||||
42:1.75,43:1.8125,44:2,45:2.25,46:1.5,47:2.0625,48:2.625,49:1.0625,
|
||||
50:2.3125,51:2.875,52:2,53:2.3125,54:2.75,55:2.5,56:2.875,57:2.25,
|
||||
58:2,59:3.8125,60:2.8125,61:1.5625,62:1.5,63:0.5}
|
||||
|
||||
class P:
|
||||
def __init__(self, f):
|
||||
self.f = f
|
||||
self.ver = 2
|
||||
def rd(self, n):
|
||||
b = self.f.read(n)
|
||||
if len(b) != n:
|
||||
raise EOFError(f"short read {len(b)}/{n}")
|
||||
return b
|
||||
def u8(self): return self.rd(1)[0]
|
||||
def u32(self): return struct.unpack("<I", self.rd(4))[0]
|
||||
def u64(self): return struct.unpack("<Q", self.rd(8))[0]
|
||||
def f32(self): return struct.unpack("<f", self.rd(4))[0]
|
||||
def f64(self): return struct.unpack("<d", self.rd(8))[0]
|
||||
def s(self):
|
||||
# GGUF v3 strings carry a u64 length; v2 used u32.
|
||||
n = self.u64() if self.ver >= 3 else self.u32()
|
||||
return self.rd(n).decode("utf-8", "replace")
|
||||
def val(self, t):
|
||||
# GGUF kv value types: 0 u8, 1 i8, 2 u16, 3 i16, 4 u32, 5 i32,
|
||||
# 6 f32, 7 bool, 8 string, 9 array, 10 u64, 11 i64, 12 f64.
|
||||
if t == 8: return self.s()
|
||||
if t == 9:
|
||||
et = self.u32(); n = self.u64()
|
||||
return [self.val(et) for _ in range(n)]
|
||||
if t == 0: return self.u8()
|
||||
if t == 1: return struct.unpack("<b", self.rd(1))[0]
|
||||
if t == 2: return struct.unpack("<H", self.rd(2))[0]
|
||||
if t == 3: return struct.unpack("<h", self.rd(2))[0]
|
||||
if t == 4: return self.u32()
|
||||
if t == 5: return struct.unpack("<i", self.rd(4))[0]
|
||||
if t == 6: return self.f32()
|
||||
if t == 7: return self.u8()
|
||||
if t == 10: return self.u64()
|
||||
if t == 11: return struct.unpack("<q", self.rd(8))[0]
|
||||
if t == 12: return self.f64()
|
||||
raise ValueError(f"unknown kv type {t}")
|
||||
|
||||
def main(path):
|
||||
tensors = []
|
||||
kvs = {}
|
||||
with open(path, "rb") as f:
|
||||
p = P(f)
|
||||
magic = p.rd(4)
|
||||
assert magic == b"GGUF", magic
|
||||
p.ver = p.u32()
|
||||
ntens = p.u64()
|
||||
nkv = p.u64()
|
||||
for _ in range(nkv):
|
||||
k = p.s()
|
||||
kvs[k] = p.val(p.u32())
|
||||
for _ in range(ntens):
|
||||
name = p.s()
|
||||
nd = p.u32()
|
||||
dims = [p.u64() for _ in range(nd)]
|
||||
t = p.u32()
|
||||
off = p.u64()
|
||||
numel = 1
|
||||
for d in dims: numel *= d
|
||||
tensors.append((name, dims, t, numel))
|
||||
print(f"== {path} (gguf v{p.ver}, {ntens} tensors, {nkv} kv)")
|
||||
for k in ("general.architecture", "general.name", "split.no", "split.count",
|
||||
"split.tensors", "qwen4exp.expert_count", "qwen4exp.expert_used_count",
|
||||
"qwen4exp.ngram.tensors"):
|
||||
if k in kvs: print(f" kv {k} = {kvs[k]}")
|
||||
ngram_kvs = {k: v for k, v in kvs.items() if "ngram" in k.lower()}
|
||||
if ngram_kvs: print(" ngram kvs:", ngram_kvs)
|
||||
big = []
|
||||
for name, dims, t, numel in tensors:
|
||||
big.append((numel * TYPE_SIZE.get(t, 0), name, dims, t, numel))
|
||||
big.sort(reverse=True)
|
||||
print(f" -- {min(12, len(big))} largest of {len(big)}:")
|
||||
for bytes_, name, dims, t, numel in big[:12]:
|
||||
print(f" {bytes_/2**30:9.3f} GiB {name} dims={dims} type={t} ({numel:,} elems)")
|
||||
named = [(b, n, d, t) for b, n, d, t, e in big
|
||||
if any(s in n.lower() for s in ("ngram", "ple", "token_embd", "exps"))]
|
||||
if named:
|
||||
print(" -- ngram/ple/embed/experts-named:")
|
||||
for bytes_, name, dims, t in named[:20]:
|
||||
print(f" {bytes_/2**30:9.3f} GiB {name} dims={dims} type={t}")
|
||||
|
||||
for pth in sys.argv[1:]:
|
||||
try:
|
||||
main(pth)
|
||||
except Exception:
|
||||
print(f"== {pth}: FAILED")
|
||||
traceback.print_exc()
|
||||
+16
-12
@@ -43,19 +43,23 @@ SPEC_DRAFT_N_MAX=2
|
||||
# RAM (computed there — the x4-pair measurement: streaming over PCIe loses,
|
||||
# computing in RAM does not). OT_OVERRIDES is comma-separated
|
||||
# <tensor-pattern>=<backend> pairs passed as repeated --override-tensor
|
||||
# flags:
|
||||
# * exps=CPU — the routed experts. ~100 GiB at Q8, read once per token;
|
||||
# the CPU backend computes them from page cache.
|
||||
# * the n-gram embedding block (the 51B-param PLE table) — sparse row
|
||||
# lookups, tiny PCIe traffic from CPU RAM; NEVER pin with a GPU target
|
||||
# (the 23 -> 13.4 t/s regression).
|
||||
# * per_layer_token_embd.weight=CPU — REQUIRED by the shared MTP head's
|
||||
# borrow path: GPU-resident, it crashes the draft load with
|
||||
# "invalid vector subscript" (llama.cpp PR #28243).
|
||||
# The exact tensor pattern for the n-gram block was discovered at load time
|
||||
# on 2026-09-26 (see the runbook); this list is the deployed set.
|
||||
# flags. Both overrides are pinned by the header dump of parts 2-3
|
||||
# (2026-09-26, /home/dank/models/dump-gguf.py — GGUF v3, 512 experts
|
||||
# top-10, qwen4exp.ple.ngram_size=3, heads_per_ngram=8):
|
||||
# * exps=CPU — the routed experts (blk.*.ffn_*_exps.weight, ~100 GiB at
|
||||
# Q8_0), read once per token; the CPU backend computes them from page
|
||||
# cache.
|
||||
# * per_layer_token_embd.weight=CPU — the 51.2B-element PLE n-gram table
|
||||
# (dims [160, 320001536], Q8_0, 54.4 GiB — the whole of part 3):
|
||||
# sparse row lookups, tiny PCIe traffic from CPU RAM. It doubles as
|
||||
# the shared MTP head's borrow tensor — GPU-resident, the draft load
|
||||
# crashes with "invalid vector subscript" (llama.cpp PR #28243) — so
|
||||
# this override is REQUIRED, not just a placement choice. NEVER give
|
||||
# it a GPU target (the 23 -> 13.4 t/s regression) and never --no-mmap
|
||||
# (the table must stay page-cache-backed; anonymous memory
|
||||
# cgroup-kills silently).
|
||||
NGL=999
|
||||
OT_OVERRIDES=exps=CPU,ngram_embeddings=CPU,per_layer_token_embd.weight=CPU
|
||||
OT_OVERRIDES=exps=CPU,per_layer_token_embd.weight=CPU
|
||||
TENSOR_SPLIT=1,1
|
||||
|
||||
# The drafter lives on the first R9700 (the card the load log shows holding
|
||||
|
||||
@@ -50,7 +50,7 @@ fi
|
||||
: "${PARALLEL_SLOTS:=1}"
|
||||
: "${SPEC_DRAFT_N_MAX:=2}"
|
||||
: "${NGL:=999}"
|
||||
: "${OT_OVERRIDES:=exps=CPU,ngram_embeddings=CPU,per_layer_token_embd.weight=CPU}"
|
||||
: "${OT_OVERRIDES:=exps=CPU,per_layer_token_embd.weight=CPU}"
|
||||
: "${TENSOR_SPLIT:=1,1}"
|
||||
: "${SPEC_DRAFT_DEVICE:=ROCm0}"
|
||||
: "${BATCH_SIZE:=8192}"
|
||||
|
||||
@@ -41,13 +41,21 @@ Lucebox DS4.1 quadlet that held the port for hours earlier the same day
|
||||
speed.
|
||||
- **The placement law, not a preference:** the routed experts and the
|
||||
51B-param n-gram (PLE) embedding table are **computed in host RAM**
|
||||
(`-ot exps=CPU,ngram_embeddings=CPU`) and never streamed over PCIe. The
|
||||
(`-ot exps=CPU,per_layer_token_embd.weight=CPU`) and never streamed over PCIe. The
|
||||
x4-pair measurement on the same model shape: streaming loses (3.7 t/s),
|
||||
computing in RAM does not care (20+ t/s). The dense weights, attention,
|
||||
projector and MTP head sit on the two cards (~29 GiB); the ~100 GiB of
|
||||
expert pages live in mmap page cache — `--no-mmap` is forbidden (it
|
||||
forces the table into anonymous memory; the cgroup kill is silent) and
|
||||
pinning the PLE table with a GPU target cost 23 → 13.4 t/s when tried.
|
||||
The tensor names are verified, not guessed: the GGUF v3 header dump
|
||||
(`dump-gguf.py`, run on the fetched parts 2026-09-26) shows the PLE
|
||||
table is one tensor — `per_layer_token_embd.weight`, dims
|
||||
`[160, 320001536]`, Q8_0, 54.4 GiB, exactly the whole of part 3 — and
|
||||
the model is `qwen4exp` with 512 experts top-10
|
||||
(`qwen4exp.ple.ngram_size=3`, `heads_per_ngram=8`). The same tensor is
|
||||
the MTP head's borrow target, so the override is required even if the
|
||||
placement were optional.
|
||||
- **Two cards, layer split** (`-ts 1,1`): both R9700s are on CPU root
|
||||
ports with full PCIe atomics (the ProArt B850-Creator swap,
|
||||
2026-09-20), but this deployment uses llama.cpp's layer split —
|
||||
|
||||
@@ -62,7 +62,7 @@ frank-small, port 7771]
|
||||
unsloth llama.cpp (b11030-mix prebuilt, ROCm gfx120X, bundled runtime)
|
||||
serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 across both R9700s —
|
||||
routed experts and the 51B-param n-gram table computed in host RAM
|
||||
(`-ot exps=CPU,ngram_embeddings=CPU`), the shared Q8_0 MTP head
|
||||
(`-ot exps=CPU,per_layer_token_embd.weight=CPU`), the shared Q8_0 MTP head
|
||||
(depth 2) chained behind ngram-mod, BF16 vision, one full 262,144-token
|
||||
conversation, `reasoning_content`, the placement law, the cutover and
|
||||
rollback
|
||||
|
||||
@@ -59,12 +59,12 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
# computing them in RAM does not care (20+ tok/s on the same model
|
||||
# shape). exps=CPU plus the n-gram embedding table on CPU; the PLE
|
||||
# table must NEVER carry a GPU target (the 23 -> 13.4 t/s regression).
|
||||
grep -q '^OT_OVERRIDES=exps=CPU,ngram_embeddings=CPU,per_layer_token_embd.weight=CPU$' "$ENV_FILE"
|
||||
grep -q -- '--override-tensor "$ot"' "$SERVE"
|
||||
# ...and mmap must stay on: --no-mmap forces the 51B-param table into
|
||||
# anonymous memory and the cgroup kill is silent.
|
||||
! grep -q -- '--no-mmap' "$SERVE"
|
||||
! grep -q '^MMPROJ_ON_GPU' "$ENV_FILE"
|
||||
grep -q '^OT_OVERRIDES=exps=CPU,per_layer_token_embd.weight=CPU$' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "the projector is explicit, never autoloaded" {
|
||||
|
||||
Reference in New Issue
Block a user