feat(qwen38): fold the calibration tap's Grams into libquant's imatrix GGUF
rad-convert's GPTQ path reads importance from an imatrix GGUF; the reference container's --kld-record pass emits per-expert and per-hidden Grams as raw recordings. This folds them into one v3 file: a single expert-major f32 entry per MoE layer plus the hidden-width entries, every data offset a multiple of general.alignment as the loader requires. Refusals, each one a silent-worse outcome the swift15 campaign hit: an expert with no gram of its own carries the layer's pooled diagonal and the count is reported (a silent zero is a down-quantized expert nobody priced); expert grams with no pooled gram, a gram short by one float (names the file), the same global expert recorded twice with different numbers, an empty recording directory, and an existing output. --min-rows sets the noise floor below which a recording is dropped. rows is u64 so a corpus that outgrows u32 survives the header instead of wrapping.
This commit is contained in:
Frank committed
1 parent
c13a2568d4
commit
3d8d18a45c
2 files changed
+863
No files matched your search
Executable
+418
@@ -0,0 +1,418 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""grams-to-imatrix.py — the engine's own calibration recording, re-voiced as a
|
||||||
|
llama.cpp importance matrix for `rad-convert --imatrix F`. RUNS ON DANKSERVER
|
||||||
|
beside the recording. Python 3 stdlib only: numpy must NOT be required, because
|
||||||
|
this runs in the plain container next to a 335 GiB checkpoint, not in a venv.
|
||||||
|
|
||||||
|
WHY. `rad-convert` takes its per-column activation second moments and its
|
||||||
|
expert-popularity profile from ONE input, an imatrix (core/format/imatrix.h).
|
||||||
|
Only the GGUF form carries a per-expert dimension — with the legacy flat form,
|
||||||
|
or with none, every expert in a layer is weighted by the layer's aggregate,
|
||||||
|
which is exactly the wrong thing on a 512-expert MoE. The calibration tap has
|
||||||
|
already measured everything; this tool only re-voices it, so the imatrix and the
|
||||||
|
protected-expert census (derive-protected-experts.py) are the SAME recording:
|
||||||
|
one measurement, two consumers, neither able to disagree about which experts are
|
||||||
|
hot.
|
||||||
|
|
||||||
|
INPUT. RADGRAM2 files in a RADIANCE_CALIB_DIR output directory
|
||||||
|
(arch/common/rad_arch.h, calib_write). Header law, same parse as
|
||||||
|
scripts/derive-protected-experts.py — 32 bytes, `<8sIIQI`: magic `RADGRAM2`,
|
||||||
|
u32 k, u32 rank, u64 rows, u32 domain, a pad word, then k*k f32 row-major. Two
|
||||||
|
kinds of name, both GGUF-style, both with the checkpoint's GLOBAL expert
|
||||||
|
numbering (rad_block_moe_fp8.h: ge = expert_base + e):
|
||||||
|
|
||||||
|
gram.r<rank>.blk.L.ffn_down_exps.bin the layer's pooled gram
|
||||||
|
gram.r<rank>.blk.L.ffn_down_exps.<e>.bin expert e's own gram
|
||||||
|
|
||||||
|
`rows` is that file's routing/token count and the matrix is Σ x xᵀ over those
|
||||||
|
rows, so **the diagonal is exactly the per-column activation second moment an
|
||||||
|
imatrix wants**, and `rows` is the count the reader hands back from `count()`.
|
||||||
|
Every off-diagonal term is dropped on purpose: the GGUF format has nowhere to
|
||||||
|
put a full matrix, and the quantiser asks for columns.
|
||||||
|
|
||||||
|
OUTPUT, per layer L (core/format/imatrix.h: names WITHOUT `.weight`, a MoE
|
||||||
|
layer's experts folded into ONE entry, so expert 9 is slice 9 of
|
||||||
|
`blk.L.ffn_down_exps`):
|
||||||
|
|
||||||
|
blk.L.ffn_down_exps in_sum2 [k, n_expert] expert-major, counts [1, n_expert]
|
||||||
|
blk.L.ffn_gate_up_exps the SAME matrices at the SAME offsets. A routed expert's
|
||||||
|
gate_up input is the same hidden-state rows as its down
|
||||||
|
input, so the second moment over them is identical and
|
||||||
|
so is the routing count. Two tensor infos over one
|
||||||
|
payload: the reader maps each half independently
|
||||||
|
(imatrix.cpp joins `<base>.in_sum2` with `<base>.counts`
|
||||||
|
by base name), and copying ~200 MB to state one fact
|
||||||
|
twice buys no correctness.
|
||||||
|
blk.L.ssm_inz [k] and [1], from the pooled gram: that row's input is
|
||||||
|
the same hidden state, so its columns take the pooled
|
||||||
|
diagonal and its count is the layer's pooled rows.
|
||||||
|
|
||||||
|
Then STOP. Nothing whose input width is not the gram's k — no ssm_out, no
|
||||||
|
attn_output, no mtp.*, no ple_ngram. importance_for() answers false when the
|
||||||
|
entry's cols do not equal the weight's width and the convert proceeds without
|
||||||
|
it: an entry the reader silently ignores is a lie in the artifact, worse than
|
||||||
|
an absent one. The corollary to check before trusting the two hidden-width
|
||||||
|
entries: the tap writes the gate_up gram at g.n_embd and the down gram at the
|
||||||
|
expert width, so the gram's k must equal the checkpoint's hidden width (its
|
||||||
|
config.json). Where they differ, gate_up/ssm_inz are inert, not wrong.
|
||||||
|
|
||||||
|
INHERITANCE. An expert with no own gram carries the layer's POOLED diagonal, and
|
||||||
|
its count stays its own routing rows — the rule the engine itself applies when it
|
||||||
|
picks a Hessian (libquant/lq_hess.h, calib_resolve: an expert's own gram only
|
||||||
|
once its rank files hold kOwnRowsPerK * K rows; below that a fit to so few rows
|
||||||
|
is worse than round-to-nearest, and those experts carry a fraction of a percent
|
||||||
|
of the routing). The count never inherits: popularity is `count`'s job. The count
|
||||||
|
of inherited experts is reported on stderr.
|
||||||
|
|
||||||
|
RANK FOLDING. TP=2 writes gram.r0.* and gram.r1.* with GLOBAL expert numbering,
|
||||||
|
so the ranks describe DISJOINT experts and the fold is a UNION into the one
|
||||||
|
[k, n_expert] entry. A pooled gram is the opposite case: every rank summed the
|
||||||
|
same stem over its own rows, so the fold is a SUM, rows included (calib_sum,
|
||||||
|
calib_rows). The same global expert in two per-expert files is legal only when
|
||||||
|
the numbers agree exactly; otherwise the recording is corrupt and the entry would
|
||||||
|
be whichever file sorted last, so refuse.
|
||||||
|
|
||||||
|
GGUF v3 (the only version imatrix.cpp accepts — version != 3 is a refusal):
|
||||||
|
`general.alignment` u32, `imatrix.chunk_size` u32, `imatrix.chunk_count` u32,
|
||||||
|
`imatrix.datasets` array of strings; tensors f32 (ggml type 0) with offsets
|
||||||
|
relative to the aligned data section. Every tensor offset is padded up to
|
||||||
|
`general.alignment` and the data section itself starts aligned. That is the
|
||||||
|
classic hand-rolled-writer bug: an unaligned offset makes the reader's extent
|
||||||
|
arithmetic point into the middle of the previous tensor's floats, and it reads
|
||||||
|
them without complaint.
|
||||||
|
|
||||||
|
chunk_count DERIVATION. The corpus row count is the LARGEST single-layer pooled
|
||||||
|
row total, not a sum over layers: every layer's tap accumulated over the whole
|
||||||
|
corpus, so adding the layers counts each token once per layer and inflates the
|
||||||
|
figure by the layer count. chunk_count = that row total // chunk_size, the
|
||||||
|
"chunks of chunk_size" line imatrix.cpp prints for itself.
|
||||||
|
|
||||||
|
REFUSALS, each exit 2 with a message naming what is missing:
|
||||||
|
NO_GRAMS the directory holds no RADGRAM2 down-projection gram
|
||||||
|
NOT_RADGRAM2 a gram whose magic is not RADGRAM2
|
||||||
|
SHORT a gram whose length is not 32 + k*k*4
|
||||||
|
NO_POOLED a layer with expert grams and no pooled gram to inherit from
|
||||||
|
K_MISMATCH a layer whose files disagree about k (rank pooled grams, or an
|
||||||
|
expert gram against its layer's pooled gram)
|
||||||
|
EXPERT_CONFLICT a global expert recorded twice with different numbers
|
||||||
|
EXISTS the -o path already holds a file (the trash law: move it)
|
||||||
|
CHUNK_COUNT the derived chunk count does not fit the u32 metadata field
|
||||||
|
|
||||||
|
USAGE. grams-to-imatrix.py <calib-dir> -o <out.gguf>
|
||||||
|
[--datasets NAME,...] [--chunk-size N] [--min-rows N] [-v]
|
||||||
|
"""
|
||||||
|
import argparse
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import struct
|
||||||
|
import sys
|
||||||
|
|
||||||
|
MAGIC = b"RADGRAM2"
|
||||||
|
HDR = 32 # 8s I I Q I + pad, rad_arch.h
|
||||||
|
KOWN_ROWS_PER_K = 4 # libquant/lq_hess.h kOwnRowsPerK
|
||||||
|
ALIGNMENT = 32 # general.alignment, and the pad unit
|
||||||
|
|
||||||
|
POOLED_RE = re.compile(r"^gram\.r(\d+)\.blk\.(\d+)\.ffn_down_exps\.bin$")
|
||||||
|
EXPERT_RE = re.compile(r"^gram\.r(\d+)\.blk\.(\d+)\.ffn_down_exps\.(\d+)\.bin$")
|
||||||
|
|
||||||
|
GT_U32, GT_STR, GT_ARR = 4, 8, 9
|
||||||
|
GGML_F32 = 0
|
||||||
|
|
||||||
|
|
||||||
|
def refuse(msg):
|
||||||
|
sys.stderr.write("REFUSED %s\n" % msg)
|
||||||
|
sys.exit(2)
|
||||||
|
|
||||||
|
|
||||||
|
def read_gram(path):
|
||||||
|
"""(k, rank, rows, domain, diagonal) from one RADGRAM2 file."""
|
||||||
|
try:
|
||||||
|
with open(path, "rb") as f:
|
||||||
|
raw = f.read()
|
||||||
|
except OSError as e:
|
||||||
|
refuse("UNREADABLE %s: %s" % (path, e))
|
||||||
|
if len(raw) < HDR:
|
||||||
|
refuse("SHORT %s: %d bytes, under the %d-byte RADGRAM2 header"
|
||||||
|
% (path, len(raw), HDR))
|
||||||
|
magic, k, rank, rows, domain = struct.unpack_from("<8sIIQI", raw, 0)
|
||||||
|
if magic != MAGIC:
|
||||||
|
refuse("NOT_RADGRAM2 %s: magic %r, expected b'RADGRAM2'" % (path, magic))
|
||||||
|
if k <= 0:
|
||||||
|
refuse("SHORT %s: header claims k=%d, which describes no matrix" % (path, k))
|
||||||
|
need = HDR + k * k * 4
|
||||||
|
if len(raw) != need:
|
||||||
|
refuse("SHORT %s: %d bytes, expected %d (32 + k*k*4 for k=%d)"
|
||||||
|
% (path, len(raw), need, k))
|
||||||
|
# The diagonal only, read by stride out of the row-major matrix.
|
||||||
|
diag = [struct.unpack_from("<f", raw, HDR + (i * k + i) * 4)[0]
|
||||||
|
for i in range(k)]
|
||||||
|
return k, rank, rows, domain, diag
|
||||||
|
|
||||||
|
|
||||||
|
def load_dir(d, verbose):
|
||||||
|
"""{layer: {"pooled": gram|None, "experts": {global e: gram}}} + rotated count.
|
||||||
|
|
||||||
|
Files outside the ffn_down_exps stem are not ours to fold: the tap names a
|
||||||
|
pooled gram after every projection it taps, and those carry other widths."""
|
||||||
|
if not os.path.isdir(d):
|
||||||
|
refuse("NO_GRAMS %s: not a directory" % d)
|
||||||
|
layers, rotated = {}, 0
|
||||||
|
for name in sorted(os.listdir(d)):
|
||||||
|
m = POOLED_RE.match(name) or EXPERT_RE.match(name)
|
||||||
|
if not m:
|
||||||
|
continue
|
||||||
|
path = os.path.join(d, name)
|
||||||
|
if not os.path.isfile(path):
|
||||||
|
continue
|
||||||
|
pooled = bool(POOLED_RE.match(name))
|
||||||
|
layer = int(m.group(2))
|
||||||
|
k, rank, rows, domain, diag = read_gram(path)
|
||||||
|
if domain != 0:
|
||||||
|
rotated += 1
|
||||||
|
L = layers.setdefault(layer, {"pooled": None, "experts": {}})
|
||||||
|
if pooled:
|
||||||
|
P = L["pooled"]
|
||||||
|
if P is None:
|
||||||
|
L["pooled"] = {"k": k, "rows": rows, "diag": diag, "paths": [path]}
|
||||||
|
elif P["k"] != k:
|
||||||
|
refuse("K_MISMATCH layer %d: pooled grams %s (k=%d) and %s (k=%d)"
|
||||||
|
" disagree about the width"
|
||||||
|
% (layer, P["paths"][0], P["k"], path, k))
|
||||||
|
else:
|
||||||
|
# calib_sum: each rank summed the same stem over its own rows.
|
||||||
|
P["rows"] += rows
|
||||||
|
P["paths"].append(path)
|
||||||
|
for i in range(k):
|
||||||
|
P["diag"][i] += diag[i]
|
||||||
|
if verbose:
|
||||||
|
sys.stderr.write("pooled layer %d: %s k=%d rows=%d\n"
|
||||||
|
% (layer, name, k, rows))
|
||||||
|
continue
|
||||||
|
e = int(m.group(3))
|
||||||
|
own = L["experts"].get(e)
|
||||||
|
if own is None:
|
||||||
|
L["experts"][e] = {"k": k, "rows": rows, "diag": diag,
|
||||||
|
"path": path, "ranks": [rank]}
|
||||||
|
if verbose:
|
||||||
|
sys.stderr.write("expert layer %d r%d e%d: rows=%d\n"
|
||||||
|
% (layer, rank, e, rows))
|
||||||
|
continue
|
||||||
|
if own["rows"] != rows or own["diag"] != diag:
|
||||||
|
refuse("EXPERT_CONFLICT layer %d expert %d: %s (rows=%d) and %s"
|
||||||
|
" (rows=%d) record the same global expert with different"
|
||||||
|
" numbers" % (layer, e, own["path"], own["rows"], path, rows))
|
||||||
|
own["ranks"].append(rank)
|
||||||
|
sys.stderr.write("note: layer %d expert %d recorded identically by rank %s"
|
||||||
|
" and %s; folded once\n"
|
||||||
|
% (layer, e, own["path"], name))
|
||||||
|
if not layers:
|
||||||
|
refuse("NO_GRAMS %s: no RADGRAM2 gram.r<rank>.blk.<L>.ffn_down_exps"
|
||||||
|
"[.<expert>].bin files in the directory" % d)
|
||||||
|
return layers, rotated
|
||||||
|
|
||||||
|
|
||||||
|
def build_entries(layers, min_rows, verbose):
|
||||||
|
"""([ (name, dims, payload|None, key) ], census, layers emitted)."""
|
||||||
|
entries = []
|
||||||
|
census = {"inherited": 0, "experts": 0, "routing": 0, "rows": 0, "layers": 0}
|
||||||
|
for layer in sorted(layers):
|
||||||
|
P, E = layers[layer]["pooled"], layers[layer]["experts"]
|
||||||
|
if E and P is None:
|
||||||
|
refuse("NO_POOLED layer %d: %d expert gram(s) but no pooled gram (an"
|
||||||
|
" expected gram.r<rank>.blk.%d.ffn_down_exps.bin), so an expert"
|
||||||
|
" under the row threshold would have nothing to inherit"
|
||||||
|
% (layer, len(E), layer))
|
||||||
|
if P is None:
|
||||||
|
continue
|
||||||
|
k = P["k"]
|
||||||
|
census["layers"] += 1
|
||||||
|
census["rows"] = max(census["rows"], P["rows"])
|
||||||
|
dn = "blk.%d.ffn_down_exps" % layer
|
||||||
|
# A staging key names ONE payload. A gate_up entry reuses the down
|
||||||
|
# layer's bytes by carrying the same key with no payload of its own;
|
||||||
|
# in_sum2 and counts are different bytes and must never share a key.
|
||||||
|
ksum, kcnt = (layer, "sum2"), (layer, "counts")
|
||||||
|
kpgm, kpcnt = (layer, "pooled_sum2"), (layer, "pooled_counts")
|
||||||
|
# One entry holds one width: an expert gram recorded at another k would
|
||||||
|
# inherit (or be inherited by) a diagonal of the wrong length, and the
|
||||||
|
# reader would then answer `cols` from whichever file set dims[0].
|
||||||
|
for e in sorted(E):
|
||||||
|
if E[e]["k"] != k:
|
||||||
|
refuse("K_MISMATCH layer %d expert %d: %s holds k=%d and this"
|
||||||
|
" layer's pooled gram is k=%d" % (layer, e, E[e]["path"],
|
||||||
|
E[e]["k"], k))
|
||||||
|
|
||||||
|
if not E:
|
||||||
|
# No per-expert recording for this layer: emit the dense form
|
||||||
|
# ([k], [1]) rather than 512 identical slices, which would claim a
|
||||||
|
# per-expert measurement that was never taken.
|
||||||
|
entries.append((dn + ".in_sum2", [k],
|
||||||
|
struct.pack("<%df" % k, *P["diag"]), ksum))
|
||||||
|
entries.append((dn + ".counts", [1],
|
||||||
|
struct.pack("<f", float(P["rows"])), kcnt))
|
||||||
|
sys.stderr.write("note: layer %d has no per-expert grams; %s is"
|
||||||
|
" dense over the pooled gram\n" % (layer, dn))
|
||||||
|
else:
|
||||||
|
floor = min_rows if min_rows > 0 else KOWN_ROWS_PER_K * k
|
||||||
|
n_expert = max(E) + 1
|
||||||
|
sum2, counts = bytearray(), bytearray()
|
||||||
|
for e in range(n_expert):
|
||||||
|
own = E.get(e)
|
||||||
|
if own is not None and own["rows"] >= floor:
|
||||||
|
src = own["diag"]
|
||||||
|
rows = own["rows"]
|
||||||
|
else:
|
||||||
|
src = P["diag"]
|
||||||
|
rows = own["rows"] if own is not None else 0
|
||||||
|
census["inherited"] += 1
|
||||||
|
sum2 += struct.pack("<%df" % k, *src)
|
||||||
|
counts += struct.pack("<f", float(rows))
|
||||||
|
census["experts"] += 1
|
||||||
|
census["routing"] += rows
|
||||||
|
if min(E) != 0:
|
||||||
|
sys.stderr.write("note: layer %d has no gram for experts 0..%d;"
|
||||||
|
" those slices carry the pooled gram at count 0\n"
|
||||||
|
% (layer, min(E) - 1))
|
||||||
|
entries.append((dn + ".in_sum2", [k, n_expert], bytes(sum2), ksum))
|
||||||
|
entries.append((dn + ".counts", [1, n_expert], bytes(counts), kcnt))
|
||||||
|
gu = "blk.%d.ffn_gate_up_exps" % layer
|
||||||
|
entries.append((gu + ".in_sum2", [k, n_expert], None, ksum))
|
||||||
|
entries.append((gu + ".counts", [1, n_expert], None, kcnt))
|
||||||
|
if verbose:
|
||||||
|
sys.stderr.write("layer %d: k=%d n_expert=%d pooled_rows=%d"
|
||||||
|
" own-gram floor=%d\n"
|
||||||
|
% (layer, k, n_expert, P["rows"], floor))
|
||||||
|
|
||||||
|
inz = "blk.%d.ssm_inz" % layer
|
||||||
|
entries.append((inz + ".in_sum2", [k],
|
||||||
|
struct.pack("<%df" % k, *P["diag"]), kpgm))
|
||||||
|
entries.append((inz + ".counts", [1],
|
||||||
|
struct.pack("<f", float(P["rows"])), kpcnt))
|
||||||
|
return entries, census
|
||||||
|
|
||||||
|
|
||||||
|
def enc_str(v):
|
||||||
|
b = v.encode("utf-8")
|
||||||
|
return struct.pack("<Q", len(b)) + b
|
||||||
|
|
||||||
|
|
||||||
|
def write_gguf(path, entries, datasets, chunk_size, chunk_count):
|
||||||
|
"""Write GGUF v3 and return its size in bytes.
|
||||||
|
|
||||||
|
One payload per key: entries that share a key share the bytes and the
|
||||||
|
offset, so a second tensor info states the same fact without copying it."""
|
||||||
|
kv = bytearray()
|
||||||
|
kv += enc_str("general.alignment") + struct.pack("<I", GT_U32) \
|
||||||
|
+ struct.pack("<I", ALIGNMENT)
|
||||||
|
kv += enc_str("imatrix.chunk_size") + struct.pack("<I", GT_U32) \
|
||||||
|
+ struct.pack("<I", chunk_size)
|
||||||
|
kv += enc_str("imatrix.chunk_count") + struct.pack("<I", GT_U32) \
|
||||||
|
+ struct.pack("<I", chunk_count)
|
||||||
|
kv += enc_str("imatrix.datasets") + struct.pack("<I", GT_ARR) \
|
||||||
|
+ struct.pack("<I", GT_STR) + struct.pack("<Q", len(datasets))
|
||||||
|
for d in datasets:
|
||||||
|
kv += enc_str(d)
|
||||||
|
|
||||||
|
head = bytearray()
|
||||||
|
head += b"GGUF" + struct.pack("<I", 3)
|
||||||
|
head += struct.pack("<Q", len(entries)) + struct.pack("<Q", 4)
|
||||||
|
head += kv
|
||||||
|
|
||||||
|
payloads = {}
|
||||||
|
for name, _dims, payload, key in entries:
|
||||||
|
if payload is None:
|
||||||
|
continue
|
||||||
|
prev = payloads.get(key)
|
||||||
|
if prev is not None and prev != payload:
|
||||||
|
# Two different payloads on one key would ship whichever was
|
||||||
|
# appended last while BOTH tensor infos pointed at it: the counts
|
||||||
|
# payload landing over the in_sum2 payload is exactly how a
|
||||||
|
# hand-rolled GGUF writer loses half its data without an error.
|
||||||
|
refuse("INTERNAL %s: two different payloads share one staging key"
|
||||||
|
% name)
|
||||||
|
payloads.setdefault(key, payload)
|
||||||
|
infos = bytearray()
|
||||||
|
offsets, order, size = {}, [], 0
|
||||||
|
for name, dims, payload, key in entries:
|
||||||
|
if key not in payloads:
|
||||||
|
refuse("INTERNAL %s: no payload staged for its key" % name)
|
||||||
|
if key not in offsets:
|
||||||
|
offsets[key] = (size + ALIGNMENT - 1) // ALIGNMENT * ALIGNMENT
|
||||||
|
size = offsets[key] + len(payloads[key])
|
||||||
|
order.append(key)
|
||||||
|
infos += enc_str(name) + struct.pack("<I", len(dims))
|
||||||
|
for d in dims:
|
||||||
|
infos += struct.pack("<Q", d)
|
||||||
|
infos += struct.pack("<I", GGML_F32) + struct.pack("<Q", offsets[key])
|
||||||
|
|
||||||
|
body = bytes(head) + bytes(infos)
|
||||||
|
data_start = (len(body) + ALIGNMENT - 1) // ALIGNMENT * ALIGNMENT
|
||||||
|
with open(path, "wb") as f:
|
||||||
|
f.write(body)
|
||||||
|
f.write(b"\0" * (data_start - len(body)))
|
||||||
|
# The offsets in the tensor infos are the file's law, not a hint: each
|
||||||
|
# payload must land at its own padded offset. Writing them back to back
|
||||||
|
# instead leaves every reader that trusts the info — imatrix.cpp does —
|
||||||
|
# reading its neighbour's floats: plausible values, no error, wrong
|
||||||
|
# quantisation.
|
||||||
|
here = data_start
|
||||||
|
for key in order:
|
||||||
|
f.write(b"\0" * (data_start + offsets[key] - here))
|
||||||
|
f.write(payloads[key])
|
||||||
|
here = data_start + offsets[key] + len(payloads[key])
|
||||||
|
return f.tell()
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser(
|
||||||
|
description="Fold a radiance calibration recording into a GGUF v3 imatrix.")
|
||||||
|
ap.add_argument("calib_dir")
|
||||||
|
ap.add_argument("-o", "--out", required=True)
|
||||||
|
ap.add_argument("--datasets", default="",
|
||||||
|
help="comma-separated names for imatrix.datasets")
|
||||||
|
ap.add_argument("--chunk-size", type=int, default=4096,
|
||||||
|
help="rows per chunk for imatrix.chunk_size (default 4096)")
|
||||||
|
ap.add_argument("--min-rows", type=int, default=0,
|
||||||
|
help="own-gram row threshold (default 4*k, the engine's"
|
||||||
|
" kOwnRowsPerK law)")
|
||||||
|
ap.add_argument("-v", "--verbose", action="store_true")
|
||||||
|
a = ap.parse_args()
|
||||||
|
|
||||||
|
if os.path.exists(a.out):
|
||||||
|
# The repo's deletion law: an artifact that exists is moved to trash by
|
||||||
|
# a human, never silently overwritten by a re-run of the tool that made
|
||||||
|
# it, so a second convert cannot destroy the imatrix the first one fed.
|
||||||
|
refuse("EXISTS %s: an imatrix is already there; move it to"
|
||||||
|
" .AGENTTRASH/ or name another -o" % a.out)
|
||||||
|
if a.chunk_size <= 0:
|
||||||
|
refuse("BAD_CHUNK_SIZE --chunk-size %d is not positive" % a.chunk_size)
|
||||||
|
layers, rotated = load_dir(a.calib_dir, a.verbose)
|
||||||
|
entries, census = build_entries(layers, a.min_rows, a.verbose)
|
||||||
|
if not entries:
|
||||||
|
refuse("NO_GRAMS %s: no layer holds a pooled gram to fold" % a.calib_dir)
|
||||||
|
if rotated:
|
||||||
|
sys.stderr.write("note: %d gram(s) recorded in a rotated domain: their"
|
||||||
|
" diagonal is the rotated activation's second moment,"
|
||||||
|
" not the model's own\n" % rotated)
|
||||||
|
|
||||||
|
chunk_count = census["rows"] // a.chunk_size
|
||||||
|
if chunk_count > 0xFFFFFFFF:
|
||||||
|
# imatrix.chunk_count is a u32 in the reader's grammar, so there is no
|
||||||
|
# legal way to record a bigger one. Reaching this means either a chunk
|
||||||
|
# size picked for a far larger run or a row total read out of the wrong
|
||||||
|
# header offset — rows is a u64 and a u32 parse is off by 2**32 — and
|
||||||
|
# the reader would trust whatever number landed in the field.
|
||||||
|
refuse("CHUNK_COUNT %d = %d rows // %d does not fit the u32"
|
||||||
|
" imatrix.chunk_count field; raise --chunk-size"
|
||||||
|
% (chunk_count, census["rows"], a.chunk_size))
|
||||||
|
nbytes = write_gguf(a.out, entries, [d for d in a.datasets.split(",") if d],
|
||||||
|
a.chunk_size, chunk_count)
|
||||||
|
sys.stderr.write("IMATRIX_OK entries=%d layers=%d experts=%d inherited=%d"
|
||||||
|
" rows=%d routing=%d chunks=%d/%d bytes=%d out=%s\n"
|
||||||
|
% (len(entries), census["layers"], census["experts"],
|
||||||
|
census["inherited"], census["rows"], census["routing"],
|
||||||
|
chunk_count, a.chunk_size, nbytes, a.out))
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
Executable
+445
@@ -0,0 +1,445 @@
|
|||||||
|
#!/usr/bin/env bats
|
||||||
|
# Seam: scripts/grams-to-imatrix.py — the calibration recording (RADGRAM2 grams)
|
||||||
|
# folded into the GGUF v3 imatrix `rad-convert --imatrix F` reads.
|
||||||
|
#
|
||||||
|
# The only contract that matters is the reader's: core/format/imatrix.cpp joins
|
||||||
|
# `<base>.in_sum2` with `<base>.counts`, takes cols from in_sum2 dims[0] and
|
||||||
|
# n_expert from dims[1], demands ggml f32 tensors and GGUF version 3, and
|
||||||
|
# silently IGNORES an entry whose width does not match the weight it is asked
|
||||||
|
# about. So every test here parses the artifact back with its own tiny GGUF
|
||||||
|
# reader and asserts bytes — a file that merely exists and exits 0 proves
|
||||||
|
# nothing about a quantiser that will read it hours later on another machine.
|
||||||
|
#
|
||||||
|
# Fixtures are synthetic: k=4 grams with real RADGRAM2 headers (rows is a u64
|
||||||
|
# at offset 16 — pack it as u32 and 100 rows read back as 4294967296000) and
|
||||||
|
# real f32 payloads whose OFF-DIAGONAL is filled with 99.0, so an entry built
|
||||||
|
# from anything but the diagonal is instantly visible. Expected numbers are
|
||||||
|
# worked examples from the fixture tables below, never recomputed the way the
|
||||||
|
# tool computes them.
|
||||||
|
|
||||||
|
load helpers
|
||||||
|
|
||||||
|
TOOL="$REPO_ROOT/scripts/grams-to-imatrix.py"
|
||||||
|
|
||||||
|
# make_gram <path> <k> <rank> <rows> <domain> <diag,csv>
|
||||||
|
# A RADGRAM2 file: 32-byte header <8sIIQI (magic, k, rank, rows, domain, pad)
|
||||||
|
# then k*k f32 row-major. Off-diagonals are 99.0 — a sentinel, not data.
|
||||||
|
make_gram() {
|
||||||
|
python3 - "$@" <<'PY'
|
||||||
|
import struct, sys
|
||||||
|
path, k, rank, rows, domain = sys.argv[1], int(sys.argv[2]), int(sys.argv[3]), \
|
||||||
|
int(sys.argv[4]), int(sys.argv[5])
|
||||||
|
diag = [float(x) for x in sys.argv[6].split(",")]
|
||||||
|
assert len(diag) == k, "diagonal must hold k values"
|
||||||
|
vals = [99.0] * (k * k)
|
||||||
|
for i, d in enumerate(diag):
|
||||||
|
vals[i * k + i] = d
|
||||||
|
hdr = struct.pack("<8sIIQI", b"RADGRAM2", k, rank, rows, domain) + struct.pack("<I", 0)
|
||||||
|
with open(path, "wb") as f:
|
||||||
|
f.write(hdr + struct.pack("<%df" % (k * k), *vals))
|
||||||
|
PY
|
||||||
|
}
|
||||||
|
|
||||||
|
# full_recording <dir> — the fixture every happy-path test reads.
|
||||||
|
#
|
||||||
|
# layer 0 (k=4, own-gram floor 4*k = 16 rows):
|
||||||
|
# pooled r0 rows 40 diag 1,2,3,4 + r1 rows 8 diag .5,.5,.5,.5
|
||||||
|
# -> summed gram (the engine's calib_sum law): diag 1.5,2.5,3.5,4.5,
|
||||||
|
# pooled rows 48
|
||||||
|
# expert 0 r0 rows 100 diag 10,20,30,40 (>= 16: own gram kept)
|
||||||
|
# expert 1 r0 rows 3 diag 7,7,7,7 (< 16: inherits pooled)
|
||||||
|
# expert 2 r1 rows 64 diag 1,1,1,2 (>= 16: own gram kept)
|
||||||
|
# expert 3 no file at all (inherits pooled, count 0)
|
||||||
|
# expert 4 r1 rows 100 diag 5,6,7,8 (>= 16: own gram kept)
|
||||||
|
# -> n_expert 5, counts 100 3 64 0 100, inherited 2, routing 267
|
||||||
|
# layer 1: pooled r0 rows 48 diag 2,2,2,2 ; expert 0 on r1 rows 48 diag 1,1,1,1
|
||||||
|
# -> n_expert 1, counts 48, inherited 0, routing 48
|
||||||
|
# Corpus rows = the largest single-layer pooled rows = 48.
|
||||||
|
full_recording() {
|
||||||
|
local d="$1"
|
||||||
|
mkdir -p "$d"
|
||||||
|
make_gram "$d/gram.r0.blk.0.ffn_down_exps.bin" 4 0 40 0 "1,2,3,4"
|
||||||
|
make_gram "$d/gram.r1.blk.0.ffn_down_exps.bin" 4 1 8 0 "0.5,0.5,0.5,0.5"
|
||||||
|
make_gram "$d/gram.r0.blk.0.ffn_down_exps.0.bin" 4 0 100 0 "10,20,30,40"
|
||||||
|
make_gram "$d/gram.r0.blk.0.ffn_down_exps.1.bin" 4 0 3 0 "7,7,7,7"
|
||||||
|
make_gram "$d/gram.r1.blk.0.ffn_down_exps.2.bin" 4 1 64 0 "1,1,1,2"
|
||||||
|
make_gram "$d/gram.r1.blk.0.ffn_down_exps.4.bin" 4 1 100 0 "5,6,7,8"
|
||||||
|
make_gram "$d/gram.r0.blk.1.ffn_down_exps.bin" 4 0 48 0 "2,2,2,2"
|
||||||
|
make_gram "$d/gram.r1.blk.1.ffn_down_exps.0.bin" 4 1 48 0 "1,1,1,1"
|
||||||
|
}
|
||||||
|
|
||||||
|
# gguf_read <file> [vals <tensor> <float-index> <count>] — an independent GGUF
|
||||||
|
# v3 reader: KV block, tensor infos, and (on request) the f32 values behind a
|
||||||
|
# tensor's own declared offset. Prints KV/HDR/TENSOR lines. Nothing here shares
|
||||||
|
# code with the writer, which is the only reason the byte assertions mean
|
||||||
|
# anything: imatrix.cpp walks the same three structures.
|
||||||
|
gguf_read() {
|
||||||
|
python3 - "$@" <<'PY'
|
||||||
|
import struct, sys
|
||||||
|
d = open(sys.argv[1], "rb").read()
|
||||||
|
def u32(o): return struct.unpack_from("<I", d, o)[0]
|
||||||
|
def u64(o): return struct.unpack_from("<Q", d, o)[0]
|
||||||
|
def st(o):
|
||||||
|
n = u64(o)
|
||||||
|
return d[o + 8:o + 8 + n].decode(), o + 8 + n
|
||||||
|
assert d[:4] == b"GGUF", "magic is %r" % d[:4]
|
||||||
|
ver, n_t, n_kv = u32(4), u64(8), u64(16)
|
||||||
|
p, align, out = 24, 32, []
|
||||||
|
for _ in range(n_kv):
|
||||||
|
key, p = st(p)
|
||||||
|
ty = u32(p); p += 4
|
||||||
|
if ty == 4: # uint32
|
||||||
|
v = u32(p); p += 4
|
||||||
|
if key == "general.alignment":
|
||||||
|
align = v
|
||||||
|
elif ty == 8: # string
|
||||||
|
v, p = st(p)
|
||||||
|
elif ty == 9: # array
|
||||||
|
et = u32(p); n = u64(p + 4); p += 12
|
||||||
|
got = []
|
||||||
|
for _i in range(n):
|
||||||
|
if et != 8:
|
||||||
|
raise SystemExit("kv array element type %d" % et)
|
||||||
|
s, p = st(p)
|
||||||
|
got.append(s)
|
||||||
|
v = ",".join(got)
|
||||||
|
else:
|
||||||
|
raise SystemExit("kv type %d is not steppable" % ty)
|
||||||
|
out.append("KV %s %s" % (key, v))
|
||||||
|
tens = []
|
||||||
|
for _ in range(n_t):
|
||||||
|
name, p = st(p)
|
||||||
|
nd = u32(p); p += 4
|
||||||
|
dims = [u64(p + 8 * i) for i in range(nd)]; p += 8 * nd
|
||||||
|
ty = u32(p); off = u64(p + 4); p += 12
|
||||||
|
tens.append((name, dims, ty, off))
|
||||||
|
ds = (p + align - 1) // align * align
|
||||||
|
out.append("HDR version=%d n_tensors=%d n_kv=%d alignment=%d data_start=%d"
|
||||||
|
% (ver, n_t, n_kv, align, ds))
|
||||||
|
for name, dims, ty, off in tens:
|
||||||
|
out.append("TENSOR %s %s %d %d" % (name, "x".join(str(x) for x in dims), ty, off))
|
||||||
|
if len(sys.argv) > 2 and sys.argv[2] == "vals":
|
||||||
|
name, i0, n = sys.argv[3], int(sys.argv[4]), int(sys.argv[5])
|
||||||
|
for nm, dims, ty, off in tens:
|
||||||
|
if nm == name:
|
||||||
|
base = ds + off
|
||||||
|
vals = [("%d" % v) if float(v).is_integer() else repr(v)
|
||||||
|
for v in (struct.unpack_from("<f", d, base + 4 * (i0 + i))[0]
|
||||||
|
for i in range(n))]
|
||||||
|
out.append("VALS %s %s" % (name, " ".join(vals)))
|
||||||
|
print("\n".join(out))
|
||||||
|
PY
|
||||||
|
}
|
||||||
|
|
||||||
|
# aligned_offsets <file> — exit 0 iff the data section starts aligned AND every
|
||||||
|
# tensor's absolute data offset (data_start + t.off) is a multiple of
|
||||||
|
# general.alignment. This is the writer bug that no value assertion can see: an
|
||||||
|
# unaligned offset makes the reader's extent arithmetic land mid-float in the
|
||||||
|
# previous tensor, and it reads the result without complaining.
|
||||||
|
aligned_offsets() {
|
||||||
|
python3 - "$1" <<'PY'
|
||||||
|
import struct, sys
|
||||||
|
d = open(sys.argv[1], "rb").read()
|
||||||
|
def u32(o): return struct.unpack_from("<I", d, o)[0]
|
||||||
|
def u64(o): return struct.unpack_from("<Q", d, o)[0]
|
||||||
|
def st(o):
|
||||||
|
n = u64(o)
|
||||||
|
return d[o + 8:o + 8 + n].decode(), o + 8 + n
|
||||||
|
ver, n_t, n_kv = u32(4), u64(8), u64(16)
|
||||||
|
p, align = 24, 32
|
||||||
|
for _ in range(n_kv):
|
||||||
|
key, p = st(p)
|
||||||
|
ty = u32(p); p += 4
|
||||||
|
if ty == 4:
|
||||||
|
v = u32(p); p += 4
|
||||||
|
if key == "general.alignment":
|
||||||
|
align = v
|
||||||
|
if ty == 8:
|
||||||
|
_v, p = st(p)
|
||||||
|
elif ty == 9:
|
||||||
|
et = u32(p); n = u64(p + 4); p += 12
|
||||||
|
for _i in range(n):
|
||||||
|
if et == 8:
|
||||||
|
_v, p = st(p)
|
||||||
|
elif et == 4:
|
||||||
|
p += 4
|
||||||
|
else:
|
||||||
|
raise SystemExit("kv array element type %d" % et)
|
||||||
|
elif ty in (0, 1):
|
||||||
|
p += 1
|
||||||
|
elif ty in (2, 3):
|
||||||
|
p += 2
|
||||||
|
elif ty in (5, 6):
|
||||||
|
p += 4
|
||||||
|
elif ty in (10, 11, 12):
|
||||||
|
p += 8
|
||||||
|
tens = []
|
||||||
|
for _ in range(n_t):
|
||||||
|
name, p = st(p)
|
||||||
|
nd = u32(p); p += 4
|
||||||
|
p += 8 * nd
|
||||||
|
ty = u32(p); off = u64(p + 4); p += 12
|
||||||
|
tens.append((name, ty, off))
|
||||||
|
ds = (p + align - 1) // align * align
|
||||||
|
bad = [] if ds % align == 0 else ["data_start=%d" % ds]
|
||||||
|
bad += ["%s=%d" % (nm, ds + off) for nm, _ty, off in tens if (ds + off) % align]
|
||||||
|
if bad:
|
||||||
|
print("UNALIGNED", ";".join(bad))
|
||||||
|
sys.exit(1)
|
||||||
|
print("ALIGNED %d tensor(s), data_start=%d, alignment=%d" % (len(tens), ds, align))
|
||||||
|
PY
|
||||||
|
}
|
||||||
|
|
||||||
|
# --- the artifact -----------------------------------------------------------
|
||||||
|
|
||||||
|
@test "a complete recording becomes a GGUF v3 file and the run reports itself" {
|
||||||
|
C="$BATS_TEST_TMPDIR/full"
|
||||||
|
full_recording "$C"
|
||||||
|
run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/im.gguf" \
|
||||||
|
--datasets calib,chat --chunk-size 16
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[[ "$output" == *"IMATRIX_OK"* ]]
|
||||||
|
head -c 4 "$BATS_TEST_TMPDIR/im.gguf" | grep -q "GGUF"
|
||||||
|
run gguf_read "$BATS_TEST_TMPDIR/im.gguf"
|
||||||
|
[[ "$output" == *"HDR version=3"* ]]
|
||||||
|
# The reader refuses anything but version 3, so this is the whole gate.
|
||||||
|
[[ "$output" != *"version=2"* ]]
|
||||||
|
# Everything the reader looks for by key, and the honesty of the chunk math:
|
||||||
|
# 48 corpus rows over 16-row chunks is 3 chunks.
|
||||||
|
[[ "$output" == *"KV general.alignment 32"* ]]
|
||||||
|
[[ "$output" == *"KV imatrix.chunk_size 16"* ]]
|
||||||
|
[[ "$output" == *"KV imatrix.chunk_count 3"* ]]
|
||||||
|
[[ "$output" == *"KV imatrix.datasets calib,chat"* ]]
|
||||||
|
[[ "$output" == *"n_tensors=12"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the reader finds the folded MoE entry: one per layer, expert-major, f32" {
|
||||||
|
C="$BATS_TEST_TMPDIR/full"
|
||||||
|
full_recording "$C"
|
||||||
|
run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/im.gguf"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
run gguf_read "$BATS_TEST_TMPDIR/im.gguf"
|
||||||
|
# dims (k, n_expert) with k fastest, ggml type 0 (f32) or the entry is
|
||||||
|
# warned away, and 5 experts because the fixture's largest global id is 4.
|
||||||
|
[[ "$output" == *"TENSOR blk.0.ffn_down_exps.in_sum2 4x5 0 "* ]]
|
||||||
|
[[ "$output" == *"TENSOR blk.0.ffn_down_exps.counts 1x5 0 "* ]]
|
||||||
|
# counts are the routing rows, not the energy trace: 100 3 64 0 100.
|
||||||
|
run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ffn_down_exps.counts 0 5
|
||||||
|
[[ "$output" == *"VALS blk.0.ffn_down_exps.counts 100 3 64 0 100"* ]]
|
||||||
|
# Expert 0 (floats 0..3) and expert 4 (floats 16..19) carry their own
|
||||||
|
# matrices' diagonals; the 99.0 off-diagonal fill appears nowhere.
|
||||||
|
run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ffn_down_exps.in_sum2 0 4
|
||||||
|
[[ "$output" == *"VALS blk.0.ffn_down_exps.in_sum2 10 20 30 40"* ]]
|
||||||
|
run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ffn_down_exps.in_sum2 16 4
|
||||||
|
[[ "$output" == *"VALS blk.0.ffn_down_exps.in_sum2 5 6 7 8"* ]]
|
||||||
|
[[ "$output" != *"99"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "an expert with no own gram carries the pooled diagonal, and the count of them is reported" {
|
||||||
|
C="$BATS_TEST_TMPDIR/full"
|
||||||
|
full_recording "$C"
|
||||||
|
run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/im.gguf"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[[ "$output" == *"inherited=2"* ]]
|
||||||
|
# Expert 1 has a gram (3 rows, under the 4*k floor) and expert 3 has none at
|
||||||
|
# all; both slices must be the POOLED diag 1+0.5, 2+0.5, 3+0.5, 4+0.5.
|
||||||
|
run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ffn_down_exps.in_sum2 4 4
|
||||||
|
[[ "$output" == *"VALS blk.0.ffn_down_exps.in_sum2 1.5 2.5 3.5 4.5"* ]]
|
||||||
|
run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ffn_down_exps.in_sum2 12 4
|
||||||
|
[[ "$output" == *"VALS blk.0.ffn_down_exps.in_sum2 1.5 2.5 3.5 4.5"* ]]
|
||||||
|
# Their counts stay their own routing: 3 for the under-threshold expert, 0
|
||||||
|
# for the unrecorded one — popularity is count's job, never the gram's.
|
||||||
|
run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ffn_down_exps.counts 1 3
|
||||||
|
[[ "$output" == *"VALS blk.0.ffn_down_exps.counts 3 64 0"* ]]
|
||||||
|
# The census is a report for the operator, so it belongs on stderr. `run`
|
||||||
|
# folds both streams into $output, so a second run into a different -o (an
|
||||||
|
# existing output is refused by design) captures stderr alone: without the
|
||||||
|
# stderr write there is no file to grep and this assertion goes red.
|
||||||
|
python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/im2.gguf" 2>"$BATS_TEST_TMPDIR/err"
|
||||||
|
[ -s "$BATS_TEST_TMPDIR/err" ]
|
||||||
|
grep -q "inherited=2" "$BATS_TEST_TMPDIR/err"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the hidden-width entries are the same bytes, and nothing else is emitted" {
|
||||||
|
C="$BATS_TEST_TMPDIR/full"
|
||||||
|
full_recording "$C"
|
||||||
|
run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/im.gguf"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
run gguf_read "$BATS_TEST_TMPDIR/im.gguf"
|
||||||
|
# gate_up shares the down matrices (same hidden-state rows), so its slice 4
|
||||||
|
# must read 5 6 7 8 at its own tensor info's offset.
|
||||||
|
[[ "$output" == *"TENSOR blk.0.ffn_gate_up_exps.in_sum2 4x5 0 "* ]]
|
||||||
|
[[ "$output" == *"TENSOR blk.0.ssm_inz.in_sum2 4 0 "* ]]
|
||||||
|
[[ "$output" == *"TENSOR blk.0.ssm_inz.counts 1 0 "* ]]
|
||||||
|
run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ffn_gate_up_exps.in_sum2 16 4
|
||||||
|
[[ "$output" == *"VALS blk.0.ffn_gate_up_exps.in_sum2 5 6 7 8"* ]]
|
||||||
|
run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ssm_inz.in_sum2 0 4
|
||||||
|
[[ "$output" == *"VALS blk.0.ssm_inz.in_sum2 1.5 2.5 3.5 4.5"* ]]
|
||||||
|
run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ssm_inz.counts 0 1
|
||||||
|
[[ "$output" == *"VALS blk.0.ssm_inz.counts 48"* ]]
|
||||||
|
# The stop rule: importance_for() false-on-a-width-mismatch means an entry
|
||||||
|
# for a projection whose input is NOT the gram's width is a silent lie.
|
||||||
|
[[ "$output" != *"ssm_out"* ]]
|
||||||
|
[[ "$output" != *"attn"* ]]
|
||||||
|
[[ "$output" != *"mtp."* ]]
|
||||||
|
[[ "$output" != *"ple_ngram"* ]]
|
||||||
|
[[ "$output" != *"output.weight"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "rows is a u64: a corpus too big for u32 survives the header" {
|
||||||
|
# The trap this fixture law exists for: pack rows as u32 at offset 16 and a
|
||||||
|
# 100-row file reads back as 4294967296000. 2**32 is exactly representable
|
||||||
|
# in f32, so the count round-trips only if the tool read 8 bytes.
|
||||||
|
C="$BATS_TEST_TMPDIR/wide"
|
||||||
|
mkdir -p "$C"
|
||||||
|
make_gram "$C/gram.r0.blk.0.ffn_down_exps.bin" 4 0 4294967296 0 "1,1,1,1"
|
||||||
|
make_gram "$C/gram.r0.blk.0.ffn_down_exps.0.bin" 4 0 4294967296 0 "3,3,3,3"
|
||||||
|
run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/wide.gguf" --chunk-size 4096
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
run gguf_read "$BATS_TEST_TMPDIR/wide.gguf" vals blk.0.ffn_down_exps.counts 0 1
|
||||||
|
[[ "$output" == *"VALS blk.0.ffn_down_exps.counts 4294967296"* ]]
|
||||||
|
[[ "$output" != *"4294967296000"* ]]
|
||||||
|
run gguf_read "$BATS_TEST_TMPDIR/wide.gguf"
|
||||||
|
[[ "$output" == *"KV imatrix.chunk_count 1048576"* ]]
|
||||||
|
# imatrix.chunk_count is a u32 field, so 2**32 rows in 1-row chunks has no
|
||||||
|
# legal encoding: wrapping it would put a number downstream believes.
|
||||||
|
run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/wide2.gguf" --chunk-size 1
|
||||||
|
[ "$status" -eq 2 ]
|
||||||
|
[[ "$output" == *"REFUSED"*"CHUNK_COUNT"* ]]
|
||||||
|
[ ! -e "$BATS_TEST_TMPDIR/wide2.gguf" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "every tensor's data offset is a multiple of general.alignment" {
|
||||||
|
C="$BATS_TEST_TMPDIR/full"
|
||||||
|
full_recording "$C"
|
||||||
|
run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/im.gguf"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
run aligned_offsets "$BATS_TEST_TMPDIR/im.gguf"
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
[[ "$output" == "ALIGNED 12 tensor(s),"*"alignment=32" ]]
|
||||||
|
|
||||||
|
# Red proof, in the same test so the assertion cannot rot: the same file
|
||||||
|
# with one tensor's declared offset nudged by 4. A counts tensor is 4 bytes,
|
||||||
|
# so a writer that forgets to pad between tensors produces exactly this
|
||||||
|
# shape of file — and every value assertion above still passes on it.
|
||||||
|
python3 - "$BATS_TEST_TMPDIR/im.gguf" "$BATS_TEST_TMPDIR/unaligned.gguf" <<'PY'
|
||||||
|
import struct, sys
|
||||||
|
src, dst = sys.argv[1], sys.argv[2]
|
||||||
|
d = bytearray(open(src, "rb").read())
|
||||||
|
def u32(o): return struct.unpack_from("<I", d, o)[0]
|
||||||
|
def u64(o): return struct.unpack_from("<Q", d, o)[0]
|
||||||
|
def st(o):
|
||||||
|
n = u64(o)
|
||||||
|
return d[o + 8:o + 8 + n].decode(), o + 8 + n
|
||||||
|
p = 24
|
||||||
|
for _ in range(u64(16)):
|
||||||
|
key, p = st(p)
|
||||||
|
ty = u32(p); p += 4
|
||||||
|
if ty == 4:
|
||||||
|
p += 4
|
||||||
|
elif ty == 8:
|
||||||
|
_v, p = st(p)
|
||||||
|
elif ty == 9:
|
||||||
|
et = u32(p); n = u64(p + 4); p += 12
|
||||||
|
for _i in range(n):
|
||||||
|
_v, p = st(p)
|
||||||
|
for _ in range(u64(8)):
|
||||||
|
name, p = st(p)
|
||||||
|
nd = u32(p); p += 4
|
||||||
|
p += 8 * nd
|
||||||
|
off_at = p + 4 # the u64 offset field of this tensor
|
||||||
|
if name == "blk.0.ssm_inz.counts":
|
||||||
|
struct.pack_into("<Q", d, off_at, u64(off_at) + 4)
|
||||||
|
p += 12
|
||||||
|
open(dst, "wb").write(bytes(d))
|
||||||
|
PY
|
||||||
|
run aligned_offsets "$BATS_TEST_TMPDIR/unaligned.gguf"
|
||||||
|
[ "$status" -eq 1 ]
|
||||||
|
[[ "$output" == *"UNALIGNED"*"blk.0.ssm_inz.counts"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
# --- the refusals -----------------------------------------------------------
|
||||||
|
|
||||||
|
@test "a gram short by one float is refused and the file is named" {
|
||||||
|
C="$BATS_TEST_TMPDIR/short"
|
||||||
|
full_recording "$C"
|
||||||
|
# Cut the last 4 bytes of one expert gram: a torn write, which is what a
|
||||||
|
# killed tap actually leaves behind.
|
||||||
|
p="$C/gram.r0.blk.0.ffn_down_exps.0.bin"
|
||||||
|
python3 -c 'import sys
|
||||||
|
p = sys.argv[1]
|
||||||
|
open(p, "wb").write(open(p, "rb").read()[:-4])' "$p"
|
||||||
|
run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/no.gguf"
|
||||||
|
[ "$status" -eq 2 ]
|
||||||
|
[[ "$output" == *"REFUSED"*"SHORT"* ]]
|
||||||
|
[[ "$output" == *"gram.r0.blk.0.ffn_down_exps.0.bin"* ]]
|
||||||
|
[ ! -e "$BATS_TEST_TMPDIR/no.gguf" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "an empty recording directory is refused" {
|
||||||
|
mkdir -p "$BATS_TEST_TMPDIR/empty"
|
||||||
|
run python3 "$TOOL" "$BATS_TEST_TMPDIR/empty" -o "$BATS_TEST_TMPDIR/no.gguf"
|
||||||
|
[ "$status" -eq 2 ]
|
||||||
|
[[ "$output" == *"REFUSED"*"NO_GRAMS"* ]]
|
||||||
|
[[ "$output" == *"$BATS_TEST_TMPDIR/empty"* ]]
|
||||||
|
[ ! -e "$BATS_TEST_TMPDIR/no.gguf" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "expert grams with no pooled gram for the layer are refused" {
|
||||||
|
# Without the pooled gram there is nothing for an under-threshold expert to
|
||||||
|
# inherit, and the tool would quietly invent a zero diagonal for it.
|
||||||
|
C="$BATS_TEST_TMPDIR/nopooled"
|
||||||
|
mkdir -p "$C"
|
||||||
|
make_gram "$C/gram.r0.blk.3.ffn_down_exps.7.bin" 4 0 500 0 "1,2,3,4"
|
||||||
|
run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/no.gguf"
|
||||||
|
[ "$status" -eq 2 ]
|
||||||
|
[[ "$output" == *"REFUSED"*"NO_POOLED"*"layer 3"* ]]
|
||||||
|
[[ "$output" == *"gram.r<rank>.blk.3.ffn_down_exps.bin"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the same global expert recorded twice with different numbers is refused" {
|
||||||
|
# Rank files carry GLOBAL expert numbering and so describe disjoint experts;
|
||||||
|
# a collision means the recording is corrupt, and whichever file sorted last
|
||||||
|
# would win.
|
||||||
|
C="$BATS_TEST_TMPDIR/conflict"
|
||||||
|
full_recording "$C"
|
||||||
|
make_gram "$C/gram.r1.blk.0.ffn_down_exps.0.bin" 4 1 77 0 "10,20,30,40"
|
||||||
|
run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/no.gguf"
|
||||||
|
[ "$status" -eq 2 ]
|
||||||
|
[[ "$output" == *"REFUSED"*"EXPERT_CONFLICT"*"expert 0"* ]]
|
||||||
|
[[ "$output" == *"gram.r0.blk.0.ffn_down_exps.0.bin"* ]]
|
||||||
|
[[ "$output" == *"gram.r1.blk.0.ffn_down_exps.0.bin"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "an existing output is refused by name rather than overwritten" {
|
||||||
|
# The trash law: a re-run must not destroy the imatrix a convert is reading.
|
||||||
|
C="$BATS_TEST_TMPDIR/full"
|
||||||
|
full_recording "$C"
|
||||||
|
: > "$BATS_TEST_TMPDIR/keep.gguf"
|
||||||
|
printf 'do not clobber me\n' > "$BATS_TEST_TMPDIR/keep.gguf"
|
||||||
|
run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/keep.gguf"
|
||||||
|
[ "$status" -eq 2 ]
|
||||||
|
[[ "$output" == *"REFUSED"*"EXISTS"*"keep.gguf"* ]]
|
||||||
|
run cat "$BATS_TEST_TMPDIR/keep.gguf"
|
||||||
|
[[ "$output" == *"do not clobber me"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "--min-rows moves the threshold and the bytes follow it" {
|
||||||
|
# The default threshold is the engine's own law (4*k rows, lq_hess.h), but
|
||||||
|
# the operator has to be able to raise it when a corpus is thin. Asserted on
|
||||||
|
# the slices, not on a report line: a flag that only changes prose is a
|
||||||
|
# flag that changes nothing.
|
||||||
|
C="$BATS_TEST_TMPDIR/floor"
|
||||||
|
full_recording "$C"
|
||||||
|
run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/far.gguf" --min-rows 200
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
|
# 200 is above every expert's routing (100, 3, 64, 0, 100 and layer 1's 48),
|
||||||
|
# so all six slices now inherit.
|
||||||
|
[[ "$output" == *"inherited=6"* ]]
|
||||||
|
run gguf_read "$BATS_TEST_TMPDIR/far.gguf" vals blk.0.ffn_down_exps.in_sum2 0 4
|
||||||
|
[[ "$output" == *"VALS blk.0.ffn_down_exps.in_sum2 1.5 2.5 3.5 4.5"* ]]
|
||||||
|
# The counts do not move with the threshold: routing is a measurement, not a
|
||||||
|
# judgement about how much data is enough.
|
||||||
|
run gguf_read "$BATS_TEST_TMPDIR/far.gguf" vals blk.0.ffn_down_exps.counts 0 5
|
||||||
|
[[ "$output" == *"VALS blk.0.ffn_down_exps.counts 100 3 64 0 100"* ]]
|
||||||
|
}
|
||||||
Reference in new issue
Block a user