scripts/grams-to-imatrix.py

@@ -0,0 +1,418 @@
+#!/usr/bin/env python3
+"""grams-to-imatrix.py — the engine's own calibration recording, re-voiced as a
+llama.cpp importance matrix for `rad-convert --imatrix F`. RUNS ON DANKSERVER
+beside the recording. Python 3 stdlib only: numpy must NOT be required, because
+this runs in the plain container next to a 335 GiB checkpoint, not in a venv.
+
+WHY. `rad-convert` takes its per-column activation second moments and its
+expert-popularity profile from ONE input, an imatrix (core/format/imatrix.h).
+Only the GGUF form carries a per-expert dimension — with the legacy flat form,
+or with none, every expert in a layer is weighted by the layer's aggregate,
+which is exactly the wrong thing on a 512-expert MoE. The calibration tap has
+already measured everything; this tool only re-voices it, so the imatrix and the
+protected-expert census (derive-protected-experts.py) are the SAME recording:
+one measurement, two consumers, neither able to disagree about which experts are
+hot.
+
+INPUT. RADGRAM2 files in a RADIANCE_CALIB_DIR output directory
+(arch/common/rad_arch.h, calib_write). Header law, same parse as
+scripts/derive-protected-experts.py — 32 bytes, `<8sIIQI`: magic `RADGRAM2`,
+u32 k, u32 rank, u64 rows, u32 domain, a pad word, then k*k f32 row-major. Two
+kinds of name, both GGUF-style, both with the checkpoint's GLOBAL expert
+numbering (rad_block_moe_fp8.h: ge = expert_base + e):
+
+    gram.r<rank>.blk.L.ffn_down_exps.bin        the layer's pooled gram
+    gram.r<rank>.blk.L.ffn_down_exps.<e>.bin    expert e's own gram
+
+`rows` is that file's routing/token count and the matrix is Σ x xᵀ over those
+rows, so **the diagonal is exactly the per-column activation second moment an
+imatrix wants**, and `rows` is the count the reader hands back from `count()`.
+Every off-diagonal term is dropped on purpose: the GGUF format has nowhere to
+put a full matrix, and the quantiser asks for columns.
+
+OUTPUT, per layer L (core/format/imatrix.h: names WITHOUT `.weight`, a MoE
+layer's experts folded into ONE entry, so expert 9 is slice 9 of
+`blk.L.ffn_down_exps`):
+
+  blk.L.ffn_down_exps    in_sum2 [k, n_expert] expert-major, counts [1, n_expert]
+  blk.L.ffn_gate_up_exps the SAME matrices at the SAME offsets. A routed expert's
+                         gate_up input is the same hidden-state rows as its down
+                         input, so the second moment over them is identical and
+                         so is the routing count. Two tensor infos over one
+                         payload: the reader maps each half independently
+                         (imatrix.cpp joins `<base>.in_sum2` with `<base>.counts`
+                         by base name), and copying ~200 MB to state one fact
+                         twice buys no correctness.
+  blk.L.ssm_inz          [k] and [1], from the pooled gram: that row's input is
+                         the same hidden state, so its columns take the pooled
+                         diagonal and its count is the layer's pooled rows.
+
+  Then STOP. Nothing whose input width is not the gram's k — no ssm_out, no
+  attn_output, no mtp.*, no ple_ngram. importance_for() answers false when the
+  entry's cols do not equal the weight's width and the convert proceeds without
+  it: an entry the reader silently ignores is a lie in the artifact, worse than
+  an absent one. The corollary to check before trusting the two hidden-width
+  entries: the tap writes the gate_up gram at g.n_embd and the down gram at the
+  expert width, so the gram's k must equal the checkpoint's hidden width (its
+  config.json). Where they differ, gate_up/ssm_inz are inert, not wrong.
+
+INHERITANCE. An expert with no own gram carries the layer's POOLED diagonal, and
+its count stays its own routing rows — the rule the engine itself applies when it
+picks a Hessian (libquant/lq_hess.h, calib_resolve: an expert's own gram only
+once its rank files hold kOwnRowsPerK * K rows; below that a fit to so few rows
+is worse than round-to-nearest, and those experts carry a fraction of a percent
+of the routing). The count never inherits: popularity is `count`'s job. The count
+of inherited experts is reported on stderr.
+
+RANK FOLDING. TP=2 writes gram.r0.* and gram.r1.* with GLOBAL expert numbering,
+so the ranks describe DISJOINT experts and the fold is a UNION into the one
+[k, n_expert] entry. A pooled gram is the opposite case: every rank summed the
+same stem over its own rows, so the fold is a SUM, rows included (calib_sum,
+calib_rows). The same global expert in two per-expert files is legal only when
+the numbers agree exactly; otherwise the recording is corrupt and the entry would
+be whichever file sorted last, so refuse.
+
+GGUF v3 (the only version imatrix.cpp accepts — version != 3 is a refusal):
+`general.alignment` u32, `imatrix.chunk_size` u32, `imatrix.chunk_count` u32,
+`imatrix.datasets` array of strings; tensors f32 (ggml type 0) with offsets
+relative to the aligned data section. Every tensor offset is padded up to
+`general.alignment` and the data section itself starts aligned. That is the
+classic hand-rolled-writer bug: an unaligned offset makes the reader's extent
+arithmetic point into the middle of the previous tensor's floats, and it reads
+them without complaint.
+
+chunk_count DERIVATION. The corpus row count is the LARGEST single-layer pooled
+row total, not a sum over layers: every layer's tap accumulated over the whole
+corpus, so adding the layers counts each token once per layer and inflates the
+figure by the layer count. chunk_count = that row total // chunk_size, the
+"chunks of chunk_size" line imatrix.cpp prints for itself.
+
+REFUSALS, each exit 2 with a message naming what is missing:
+  NO_GRAMS         the directory holds no RADGRAM2 down-projection gram
+  NOT_RADGRAM2     a gram whose magic is not RADGRAM2
+  SHORT            a gram whose length is not 32 + k*k*4
+  NO_POOLED        a layer with expert grams and no pooled gram to inherit from
+  K_MISMATCH       a layer whose files disagree about k (rank pooled grams, or an
+                   expert gram against its layer's pooled gram)
+  EXPERT_CONFLICT  a global expert recorded twice with different numbers
+  EXISTS           the -o path already holds a file (the trash law: move it)
+  CHUNK_COUNT      the derived chunk count does not fit the u32 metadata field
+
+USAGE. grams-to-imatrix.py <calib-dir> -o <out.gguf>
+       [--datasets NAME,...] [--chunk-size N] [--min-rows N] [-v]
+"""
+import argparse
+import os
+import re
+import struct
+import sys
+
+MAGIC = b"RADGRAM2"
+HDR = 32                       # 8s I I Q I + pad, rad_arch.h
+KOWN_ROWS_PER_K = 4            # libquant/lq_hess.h kOwnRowsPerK
+ALIGNMENT = 32                 # general.alignment, and the pad unit
+
+POOLED_RE = re.compile(r"^gram\.r(\d+)\.blk\.(\d+)\.ffn_down_exps\.bin$")
+EXPERT_RE = re.compile(r"^gram\.r(\d+)\.blk\.(\d+)\.ffn_down_exps\.(\d+)\.bin$")
+
+GT_U32, GT_STR, GT_ARR = 4, 8, 9
+GGML_F32 = 0
+
+
+def refuse(msg):
+    sys.stderr.write("REFUSED %s\n" % msg)
+    sys.exit(2)
+
+
+def read_gram(path):
+    """(k, rank, rows, domain, diagonal) from one RADGRAM2 file."""
+    try:
+        with open(path, "rb") as f:
+            raw = f.read()
+    except OSError as e:
+        refuse("UNREADABLE %s: %s" % (path, e))
+    if len(raw) < HDR:
+        refuse("SHORT %s: %d bytes, under the %d-byte RADGRAM2 header"
+               % (path, len(raw), HDR))
+    magic, k, rank, rows, domain = struct.unpack_from("<8sIIQI", raw, 0)
+    if magic != MAGIC:
+        refuse("NOT_RADGRAM2 %s: magic %r, expected b'RADGRAM2'" % (path, magic))
+    if k <= 0:
+        refuse("SHORT %s: header claims k=%d, which describes no matrix" % (path, k))
+    need = HDR + k * k * 4
+    if len(raw) != need:
+        refuse("SHORT %s: %d bytes, expected %d (32 + k*k*4 for k=%d)"
+               % (path, len(raw), need, k))
+    # The diagonal only, read by stride out of the row-major matrix.
+    diag = [struct.unpack_from("<f", raw, HDR + (i * k + i) * 4)[0]
+            for i in range(k)]
+    return k, rank, rows, domain, diag
+
+
+def load_dir(d, verbose):
+    """{layer: {"pooled": gram|None, "experts": {global e: gram}}} + rotated count.
+
+    Files outside the ffn_down_exps stem are not ours to fold: the tap names a
+    pooled gram after every projection it taps, and those carry other widths."""
+    if not os.path.isdir(d):
+        refuse("NO_GRAMS %s: not a directory" % d)
+    layers, rotated = {}, 0
+    for name in sorted(os.listdir(d)):
+        m = POOLED_RE.match(name) or EXPERT_RE.match(name)
+        if not m:
+            continue
+        path = os.path.join(d, name)
+        if not os.path.isfile(path):
+            continue
+        pooled = bool(POOLED_RE.match(name))
+        layer = int(m.group(2))
+        k, rank, rows, domain, diag = read_gram(path)
+        if domain != 0:
+            rotated += 1
+        L = layers.setdefault(layer, {"pooled": None, "experts": {}})
+        if pooled:
+            P = L["pooled"]
+            if P is None:
+                L["pooled"] = {"k": k, "rows": rows, "diag": diag, "paths": [path]}
+            elif P["k"] != k:
+                refuse("K_MISMATCH layer %d: pooled grams %s (k=%d) and %s (k=%d)"
+                       " disagree about the width"
+                       % (layer, P["paths"][0], P["k"], path, k))
+            else:
+                # calib_sum: each rank summed the same stem over its own rows.
+                P["rows"] += rows
+                P["paths"].append(path)
+                for i in range(k):
+                    P["diag"][i] += diag[i]
+            if verbose:
+                sys.stderr.write("pooled layer %d: %s k=%d rows=%d\n"
+                                 % (layer, name, k, rows))
+            continue
+        e = int(m.group(3))
+        own = L["experts"].get(e)
+        if own is None:
+            L["experts"][e] = {"k": k, "rows": rows, "diag": diag,
+                               "path": path, "ranks": [rank]}
+            if verbose:
+                sys.stderr.write("expert layer %d r%d e%d: rows=%d\n"
+                                 % (layer, rank, e, rows))
+            continue
+        if own["rows"] != rows or own["diag"] != diag:
+            refuse("EXPERT_CONFLICT layer %d expert %d: %s (rows=%d) and %s"
+                   " (rows=%d) record the same global expert with different"
+                   " numbers" % (layer, e, own["path"], own["rows"], path, rows))
+        own["ranks"].append(rank)
+        sys.stderr.write("note: layer %d expert %d recorded identically by rank %s"
+                         " and %s; folded once\n"
+                         % (layer, e, own["path"], name))
+    if not layers:
+        refuse("NO_GRAMS %s: no RADGRAM2 gram.r<rank>.blk.<L>.ffn_down_exps"
+               "[.<expert>].bin files in the directory" % d)
+    return layers, rotated
+
+
+def build_entries(layers, min_rows, verbose):
+    """([ (name, dims, payload|None, key) ], census, layers emitted)."""
+    entries = []
+    census = {"inherited": 0, "experts": 0, "routing": 0, "rows": 0, "layers": 0}
+    for layer in sorted(layers):
+        P, E = layers[layer]["pooled"], layers[layer]["experts"]
+        if E and P is None:
+            refuse("NO_POOLED layer %d: %d expert gram(s) but no pooled gram (an"
+                   " expected gram.r<rank>.blk.%d.ffn_down_exps.bin), so an expert"
+                   " under the row threshold would have nothing to inherit"
+                   % (layer, len(E), layer))
+        if P is None:
+            continue
+        k = P["k"]
+        census["layers"] += 1
+        census["rows"] = max(census["rows"], P["rows"])
+        dn = "blk.%d.ffn_down_exps" % layer
+        # A staging key names ONE payload. A gate_up entry reuses the down
+        # layer's bytes by carrying the same key with no payload of its own;
+        # in_sum2 and counts are different bytes and must never share a key.
+        ksum, kcnt = (layer, "sum2"), (layer, "counts")
+        kpgm, kpcnt = (layer, "pooled_sum2"), (layer, "pooled_counts")
+        # One entry holds one width: an expert gram recorded at another k would
+        # inherit (or be inherited by) a diagonal of the wrong length, and the
+        # reader would then answer `cols` from whichever file set dims[0].
+        for e in sorted(E):
+            if E[e]["k"] != k:
+                refuse("K_MISMATCH layer %d expert %d: %s holds k=%d and this"
+                       " layer's pooled gram is k=%d" % (layer, e, E[e]["path"],
+                                                          E[e]["k"], k))
+
+        if not E:
+            # No per-expert recording for this layer: emit the dense form
+            # ([k], [1]) rather than 512 identical slices, which would claim a
+            # per-expert measurement that was never taken.
+            entries.append((dn + ".in_sum2", [k],
+                            struct.pack("<%df" % k, *P["diag"]), ksum))
+            entries.append((dn + ".counts", [1],
+                            struct.pack("<f", float(P["rows"])), kcnt))
+            sys.stderr.write("note: layer %d has no per-expert grams; %s is"
+                             " dense over the pooled gram\n" % (layer, dn))
+        else:
+            floor = min_rows if min_rows > 0 else KOWN_ROWS_PER_K * k
+            n_expert = max(E) + 1
+            sum2, counts = bytearray(), bytearray()
+            for e in range(n_expert):
+                own = E.get(e)
+                if own is not None and own["rows"] >= floor:
+                    src = own["diag"]
+                    rows = own["rows"]
+                else:
+                    src = P["diag"]
+                    rows = own["rows"] if own is not None else 0
+                    census["inherited"] += 1
+                sum2 += struct.pack("<%df" % k, *src)
+                counts += struct.pack("<f", float(rows))
+                census["experts"] += 1
+                census["routing"] += rows
+            if min(E) != 0:
+                sys.stderr.write("note: layer %d has no gram for experts 0..%d;"
+                                 " those slices carry the pooled gram at count 0\n"
+                                 % (layer, min(E) - 1))
+            entries.append((dn + ".in_sum2", [k, n_expert], bytes(sum2), ksum))
+            entries.append((dn + ".counts", [1, n_expert], bytes(counts), kcnt))
+            gu = "blk.%d.ffn_gate_up_exps" % layer
+            entries.append((gu + ".in_sum2", [k, n_expert], None, ksum))
+            entries.append((gu + ".counts", [1, n_expert], None, kcnt))
+            if verbose:
+                sys.stderr.write("layer %d: k=%d n_expert=%d pooled_rows=%d"
+                                 " own-gram floor=%d\n"
+                                 % (layer, k, n_expert, P["rows"], floor))
+
+        inz = "blk.%d.ssm_inz" % layer
+        entries.append((inz + ".in_sum2", [k],
+                        struct.pack("<%df" % k, *P["diag"]), kpgm))
+        entries.append((inz + ".counts", [1],
+                        struct.pack("<f", float(P["rows"])), kpcnt))
+    return entries, census
+
+
+def enc_str(v):
+    b = v.encode("utf-8")
+    return struct.pack("<Q", len(b)) + b
+
+
+def write_gguf(path, entries, datasets, chunk_size, chunk_count):
+    """Write GGUF v3 and return its size in bytes.
+
+    One payload per key: entries that share a key share the bytes and the
+    offset, so a second tensor info states the same fact without copying it."""
+    kv = bytearray()
+    kv += enc_str("general.alignment") + struct.pack("<I", GT_U32) \
+        + struct.pack("<I", ALIGNMENT)
+    kv += enc_str("imatrix.chunk_size") + struct.pack("<I", GT_U32) \
+        + struct.pack("<I", chunk_size)
+    kv += enc_str("imatrix.chunk_count") + struct.pack("<I", GT_U32) \
+        + struct.pack("<I", chunk_count)
+    kv += enc_str("imatrix.datasets") + struct.pack("<I", GT_ARR) \
+        + struct.pack("<I", GT_STR) + struct.pack("<Q", len(datasets))
+    for d in datasets:
+        kv += enc_str(d)
+
+    head = bytearray()
+    head += b"GGUF" + struct.pack("<I", 3)
+    head += struct.pack("<Q", len(entries)) + struct.pack("<Q", 4)
+    head += kv
+
+    payloads = {}
+    for name, _dims, payload, key in entries:
+        if payload is None:
+            continue
+        prev = payloads.get(key)
+        if prev is not None and prev != payload:
+            # Two different payloads on one key would ship whichever was
+            # appended last while BOTH tensor infos pointed at it: the counts
+            # payload landing over the in_sum2 payload is exactly how a
+            # hand-rolled GGUF writer loses half its data without an error.
+            refuse("INTERNAL %s: two different payloads share one staging key"
+                   % name)
+        payloads.setdefault(key, payload)
+    infos = bytearray()
+    offsets, order, size = {}, [], 0
+    for name, dims, payload, key in entries:
+        if key not in payloads:
+            refuse("INTERNAL %s: no payload staged for its key" % name)
+        if key not in offsets:
+            offsets[key] = (size + ALIGNMENT - 1) // ALIGNMENT * ALIGNMENT
+            size = offsets[key] + len(payloads[key])
+            order.append(key)
+        infos += enc_str(name) + struct.pack("<I", len(dims))
+        for d in dims:
+            infos += struct.pack("<Q", d)
+        infos += struct.pack("<I", GGML_F32) + struct.pack("<Q", offsets[key])
+
+    body = bytes(head) + bytes(infos)
+    data_start = (len(body) + ALIGNMENT - 1) // ALIGNMENT * ALIGNMENT
+    with open(path, "wb") as f:
+        f.write(body)
+        f.write(b"\0" * (data_start - len(body)))
+        # The offsets in the tensor infos are the file's law, not a hint: each
+        # payload must land at its own padded offset. Writing them back to back
+        # instead leaves every reader that trusts the info — imatrix.cpp does —
+        # reading its neighbour's floats: plausible values, no error, wrong
+        # quantisation.
+        here = data_start
+        for key in order:
+            f.write(b"\0" * (data_start + offsets[key] - here))
+            f.write(payloads[key])
+            here = data_start + offsets[key] + len(payloads[key])
+        return f.tell()
+
+
+def main():
+    ap = argparse.ArgumentParser(
+        description="Fold a radiance calibration recording into a GGUF v3 imatrix.")
+    ap.add_argument("calib_dir")
+    ap.add_argument("-o", "--out", required=True)
+    ap.add_argument("--datasets", default="",
+                    help="comma-separated names for imatrix.datasets")
+    ap.add_argument("--chunk-size", type=int, default=4096,
+                    help="rows per chunk for imatrix.chunk_size (default 4096)")
+    ap.add_argument("--min-rows", type=int, default=0,
+                    help="own-gram row threshold (default 4*k, the engine's"
+                         " kOwnRowsPerK law)")
+    ap.add_argument("-v", "--verbose", action="store_true")
+    a = ap.parse_args()
+
+    if os.path.exists(a.out):
+        # The repo's deletion law: an artifact that exists is moved to trash by
+        # a human, never silently overwritten by a re-run of the tool that made
+        # it, so a second convert cannot destroy the imatrix the first one fed.
+        refuse("EXISTS %s: an imatrix is already there; move it to"
+               " .AGENTTRASH/ or name another -o" % a.out)
+    if a.chunk_size <= 0:
+        refuse("BAD_CHUNK_SIZE --chunk-size %d is not positive" % a.chunk_size)
+    layers, rotated = load_dir(a.calib_dir, a.verbose)
+    entries, census = build_entries(layers, a.min_rows, a.verbose)
+    if not entries:
+        refuse("NO_GRAMS %s: no layer holds a pooled gram to fold" % a.calib_dir)
+    if rotated:
+        sys.stderr.write("note: %d gram(s) recorded in a rotated domain: their"
+                         " diagonal is the rotated activation's second moment,"
+                         " not the model's own\n" % rotated)
+
+    chunk_count = census["rows"] // a.chunk_size
+    if chunk_count > 0xFFFFFFFF:
+        # imatrix.chunk_count is a u32 in the reader's grammar, so there is no
+        # legal way to record a bigger one. Reaching this means either a chunk
+        # size picked for a far larger run or a row total read out of the wrong
+        # header offset — rows is a u64 and a u32 parse is off by 2**32 — and
+        # the reader would trust whatever number landed in the field.
+        refuse("CHUNK_COUNT %d = %d rows // %d does not fit the u32"
+               " imatrix.chunk_count field; raise --chunk-size"
+               % (chunk_count, census["rows"], a.chunk_size))
+    nbytes = write_gguf(a.out, entries, [d for d in a.datasets.split(",") if d],
+                        a.chunk_size, chunk_count)
+    sys.stderr.write("IMATRIX_OK entries=%d layers=%d experts=%d inherited=%d"
+                     " rows=%d routing=%d chunks=%d/%d bytes=%d out=%s\n"
+                     % (len(entries), census["layers"], census["experts"],
+                        census["inherited"], census["rows"], census["routing"],
+                        chunk_count, a.chunk_size, nbytes, a.out))
+
+
+if __name__ == "__main__":
+    main()

tests/qwen38_grams.bats

@@ -0,0 +1,445 @@
+#!/usr/bin/env bats
+# Seam: scripts/grams-to-imatrix.py — the calibration recording (RADGRAM2 grams)
+# folded into the GGUF v3 imatrix `rad-convert --imatrix F` reads.
+#
+# The only contract that matters is the reader's: core/format/imatrix.cpp joins
+# `<base>.in_sum2` with `<base>.counts`, takes cols from in_sum2 dims[0] and
+# n_expert from dims[1], demands ggml f32 tensors and GGUF version 3, and
+# silently IGNORES an entry whose width does not match the weight it is asked
+# about. So every test here parses the artifact back with its own tiny GGUF
+# reader and asserts bytes — a file that merely exists and exits 0 proves
+# nothing about a quantiser that will read it hours later on another machine.
+#
+# Fixtures are synthetic: k=4 grams with real RADGRAM2 headers (rows is a u64
+# at offset 16 — pack it as u32 and 100 rows read back as 4294967296000) and
+# real f32 payloads whose OFF-DIAGONAL is filled with 99.0, so an entry built
+# from anything but the diagonal is instantly visible. Expected numbers are
+# worked examples from the fixture tables below, never recomputed the way the
+# tool computes them.
+
+load helpers
+
+TOOL="$REPO_ROOT/scripts/grams-to-imatrix.py"
+
+# make_gram <path> <k> <rank> <rows> <domain> <diag,csv>
+# A RADGRAM2 file: 32-byte header <8sIIQI (magic, k, rank, rows, domain, pad)
+# then k*k f32 row-major. Off-diagonals are 99.0 — a sentinel, not data.
+make_gram() {
+    python3 - "$@" <<'PY'
+import struct, sys
+path, k, rank, rows, domain = sys.argv[1], int(sys.argv[2]), int(sys.argv[3]), \
+    int(sys.argv[4]), int(sys.argv[5])
+diag = [float(x) for x in sys.argv[6].split(",")]
+assert len(diag) == k, "diagonal must hold k values"
+vals = [99.0] * (k * k)
+for i, d in enumerate(diag):
+    vals[i * k + i] = d
+hdr = struct.pack("<8sIIQI", b"RADGRAM2", k, rank, rows, domain) + struct.pack("<I", 0)
+with open(path, "wb") as f:
+    f.write(hdr + struct.pack("<%df" % (k * k), *vals))
+PY
+}
+
+# full_recording <dir> — the fixture every happy-path test reads.
+#
+# layer 0 (k=4, own-gram floor 4*k = 16 rows):
+#   pooled   r0 rows 40 diag 1,2,3,4   + r1 rows 8 diag .5,.5,.5,.5
+#            -> summed gram (the engine's calib_sum law): diag 1.5,2.5,3.5,4.5,
+#               pooled rows 48
+#   expert 0 r0 rows 100 diag 10,20,30,40   (>= 16: own gram kept)
+#   expert 1 r0 rows   3 diag 7,7,7,7       (<  16: inherits pooled)
+#   expert 2 r1 rows  64 diag 1,1,1,2       (>= 16: own gram kept)
+#   expert 3           no file at all       (inherits pooled, count 0)
+#   expert 4 r1 rows 100 diag 5,6,7,8       (>= 16: own gram kept)
+#   -> n_expert 5, counts 100 3 64 0 100, inherited 2, routing 267
+# layer 1: pooled r0 rows 48 diag 2,2,2,2 ; expert 0 on r1 rows 48 diag 1,1,1,1
+#   -> n_expert 1, counts 48, inherited 0, routing 48
+# Corpus rows = the largest single-layer pooled rows = 48.
+full_recording() {
+    local d="$1"
+    mkdir -p "$d"
+    make_gram "$d/gram.r0.blk.0.ffn_down_exps.bin" 4 0 40 0 "1,2,3,4"
+    make_gram "$d/gram.r1.blk.0.ffn_down_exps.bin" 4 1 8 0 "0.5,0.5,0.5,0.5"
+    make_gram "$d/gram.r0.blk.0.ffn_down_exps.0.bin" 4 0 100 0 "10,20,30,40"
+    make_gram "$d/gram.r0.blk.0.ffn_down_exps.1.bin" 4 0 3 0 "7,7,7,7"
+    make_gram "$d/gram.r1.blk.0.ffn_down_exps.2.bin" 4 1 64 0 "1,1,1,2"
+    make_gram "$d/gram.r1.blk.0.ffn_down_exps.4.bin" 4 1 100 0 "5,6,7,8"
+    make_gram "$d/gram.r0.blk.1.ffn_down_exps.bin" 4 0 48 0 "2,2,2,2"
+    make_gram "$d/gram.r1.blk.1.ffn_down_exps.0.bin" 4 1 48 0 "1,1,1,1"
+}
+
+# gguf_read <file> [vals <tensor> <float-index> <count>] — an independent GGUF
+# v3 reader: KV block, tensor infos, and (on request) the f32 values behind a
+# tensor's own declared offset. Prints KV/HDR/TENSOR lines. Nothing here shares
+# code with the writer, which is the only reason the byte assertions mean
+# anything: imatrix.cpp walks the same three structures.
+gguf_read() {
+    python3 - "$@" <<'PY'
+import struct, sys
+d = open(sys.argv[1], "rb").read()
+def u32(o): return struct.unpack_from("<I", d, o)[0]
+def u64(o): return struct.unpack_from("<Q", d, o)[0]
+def st(o):
+    n = u64(o)
+    return d[o + 8:o + 8 + n].decode(), o + 8 + n
+assert d[:4] == b"GGUF", "magic is %r" % d[:4]
+ver, n_t, n_kv = u32(4), u64(8), u64(16)
+p, align, out = 24, 32, []
+for _ in range(n_kv):
+    key, p = st(p)
+    ty = u32(p); p += 4
+    if ty == 4:                                   # uint32
+        v = u32(p); p += 4
+        if key == "general.alignment":
+            align = v
+    elif ty == 8:                                 # string
+        v, p = st(p)
+    elif ty == 9:                                 # array
+        et = u32(p); n = u64(p + 4); p += 12
+        got = []
+        for _i in range(n):
+            if et != 8:
+                raise SystemExit("kv array element type %d" % et)
+            s, p = st(p)
+            got.append(s)
+        v = ",".join(got)
+    else:
+        raise SystemExit("kv type %d is not steppable" % ty)
+    out.append("KV %s %s" % (key, v))
+tens = []
+for _ in range(n_t):
+    name, p = st(p)
+    nd = u32(p); p += 4
+    dims = [u64(p + 8 * i) for i in range(nd)]; p += 8 * nd
+    ty = u32(p); off = u64(p + 4); p += 12
+    tens.append((name, dims, ty, off))
+ds = (p + align - 1) // align * align
+out.append("HDR version=%d n_tensors=%d n_kv=%d alignment=%d data_start=%d"
+           % (ver, n_t, n_kv, align, ds))
+for name, dims, ty, off in tens:
+    out.append("TENSOR %s %s %d %d" % (name, "x".join(str(x) for x in dims), ty, off))
+if len(sys.argv) > 2 and sys.argv[2] == "vals":
+    name, i0, n = sys.argv[3], int(sys.argv[4]), int(sys.argv[5])
+    for nm, dims, ty, off in tens:
+        if nm == name:
+            base = ds + off
+            vals = [("%d" % v) if float(v).is_integer() else repr(v)
+                    for v in (struct.unpack_from("<f", d, base + 4 * (i0 + i))[0]
+                              for i in range(n))]
+            out.append("VALS %s %s" % (name, " ".join(vals)))
+print("\n".join(out))
+PY
+}
+
+# aligned_offsets <file> — exit 0 iff the data section starts aligned AND every
+# tensor's absolute data offset (data_start + t.off) is a multiple of
+# general.alignment. This is the writer bug that no value assertion can see: an
+# unaligned offset makes the reader's extent arithmetic land mid-float in the
+# previous tensor, and it reads the result without complaining.
+aligned_offsets() {
+    python3 - "$1" <<'PY'
+import struct, sys
+d = open(sys.argv[1], "rb").read()
+def u32(o): return struct.unpack_from("<I", d, o)[0]
+def u64(o): return struct.unpack_from("<Q", d, o)[0]
+def st(o):
+    n = u64(o)
+    return d[o + 8:o + 8 + n].decode(), o + 8 + n
+ver, n_t, n_kv = u32(4), u64(8), u64(16)
+p, align = 24, 32
+for _ in range(n_kv):
+    key, p = st(p)
+    ty = u32(p); p += 4
+    if ty == 4:
+        v = u32(p); p += 4
+        if key == "general.alignment":
+            align = v
+    if ty == 8:
+        _v, p = st(p)
+    elif ty == 9:
+        et = u32(p); n = u64(p + 4); p += 12
+        for _i in range(n):
+            if et == 8:
+                _v, p = st(p)
+            elif et == 4:
+                p += 4
+            else:
+                raise SystemExit("kv array element type %d" % et)
+    elif ty in (0, 1):
+        p += 1
+    elif ty in (2, 3):
+        p += 2
+    elif ty in (5, 6):
+        p += 4
+    elif ty in (10, 11, 12):
+        p += 8
+tens = []
+for _ in range(n_t):
+    name, p = st(p)
+    nd = u32(p); p += 4
+    p += 8 * nd
+    ty = u32(p); off = u64(p + 4); p += 12
+    tens.append((name, ty, off))
+ds = (p + align - 1) // align * align
+bad = [] if ds % align == 0 else ["data_start=%d" % ds]
+bad += ["%s=%d" % (nm, ds + off) for nm, _ty, off in tens if (ds + off) % align]
+if bad:
+    print("UNALIGNED", ";".join(bad))
+    sys.exit(1)
+print("ALIGNED %d tensor(s), data_start=%d, alignment=%d" % (len(tens), ds, align))
+PY
+}
+
+# --- the artifact -----------------------------------------------------------
+
+@test "a complete recording becomes a GGUF v3 file and the run reports itself" {
+    C="$BATS_TEST_TMPDIR/full"
+    full_recording "$C"
+    run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/im.gguf" \
+        --datasets calib,chat --chunk-size 16
+    [ "$status" -eq 0 ]
+    [[ "$output" == *"IMATRIX_OK"* ]]
+    head -c 4 "$BATS_TEST_TMPDIR/im.gguf" | grep -q "GGUF"
+    run gguf_read "$BATS_TEST_TMPDIR/im.gguf"
+    [[ "$output" == *"HDR version=3"* ]]
+    # The reader refuses anything but version 3, so this is the whole gate.
+    [[ "$output" != *"version=2"* ]]
+    # Everything the reader looks for by key, and the honesty of the chunk math:
+    # 48 corpus rows over 16-row chunks is 3 chunks.
+    [[ "$output" == *"KV general.alignment 32"* ]]
+    [[ "$output" == *"KV imatrix.chunk_size 16"* ]]
+    [[ "$output" == *"KV imatrix.chunk_count 3"* ]]
+    [[ "$output" == *"KV imatrix.datasets calib,chat"* ]]
+    [[ "$output" == *"n_tensors=12"* ]]
+}
+
+@test "the reader finds the folded MoE entry: one per layer, expert-major, f32" {
+    C="$BATS_TEST_TMPDIR/full"
+    full_recording "$C"
+    run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/im.gguf"
+    [ "$status" -eq 0 ]
+    run gguf_read "$BATS_TEST_TMPDIR/im.gguf"
+    # dims (k, n_expert) with k fastest, ggml type 0 (f32) or the entry is
+    # warned away, and 5 experts because the fixture's largest global id is 4.
+    [[ "$output" == *"TENSOR blk.0.ffn_down_exps.in_sum2 4x5 0 "* ]]
+    [[ "$output" == *"TENSOR blk.0.ffn_down_exps.counts 1x5 0 "* ]]
+    # counts are the routing rows, not the energy trace: 100 3 64 0 100.
+    run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ffn_down_exps.counts 0 5
+    [[ "$output" == *"VALS blk.0.ffn_down_exps.counts 100 3 64 0 100"* ]]
+    # Expert 0 (floats 0..3) and expert 4 (floats 16..19) carry their own
+    # matrices' diagonals; the 99.0 off-diagonal fill appears nowhere.
+    run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ffn_down_exps.in_sum2 0 4
+    [[ "$output" == *"VALS blk.0.ffn_down_exps.in_sum2 10 20 30 40"* ]]
+    run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ffn_down_exps.in_sum2 16 4
+    [[ "$output" == *"VALS blk.0.ffn_down_exps.in_sum2 5 6 7 8"* ]]
+    [[ "$output" != *"99"* ]]
+}
+
+@test "an expert with no own gram carries the pooled diagonal, and the count of them is reported" {
+    C="$BATS_TEST_TMPDIR/full"
+    full_recording "$C"
+    run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/im.gguf"
+    [ "$status" -eq 0 ]
+    [[ "$output" == *"inherited=2"* ]]
+    # Expert 1 has a gram (3 rows, under the 4*k floor) and expert 3 has none at
+    # all; both slices must be the POOLED diag 1+0.5, 2+0.5, 3+0.5, 4+0.5.
+    run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ffn_down_exps.in_sum2 4 4
+    [[ "$output" == *"VALS blk.0.ffn_down_exps.in_sum2 1.5 2.5 3.5 4.5"* ]]
+    run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ffn_down_exps.in_sum2 12 4
+    [[ "$output" == *"VALS blk.0.ffn_down_exps.in_sum2 1.5 2.5 3.5 4.5"* ]]
+    # Their counts stay their own routing: 3 for the under-threshold expert, 0
+    # for the unrecorded one — popularity is count's job, never the gram's.
+    run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ffn_down_exps.counts 1 3
+    [[ "$output" == *"VALS blk.0.ffn_down_exps.counts 3 64 0"* ]]
+    # The census is a report for the operator, so it belongs on stderr. `run`
+    # folds both streams into $output, so a second run into a different -o (an
+    # existing output is refused by design) captures stderr alone: without the
+    # stderr write there is no file to grep and this assertion goes red.
+    python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/im2.gguf" 2>"$BATS_TEST_TMPDIR/err"
+    [ -s "$BATS_TEST_TMPDIR/err" ]
+    grep -q "inherited=2" "$BATS_TEST_TMPDIR/err"
+}
+
+@test "the hidden-width entries are the same bytes, and nothing else is emitted" {
+    C="$BATS_TEST_TMPDIR/full"
+    full_recording "$C"
+    run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/im.gguf"
+    [ "$status" -eq 0 ]
+    run gguf_read "$BATS_TEST_TMPDIR/im.gguf"
+    # gate_up shares the down matrices (same hidden-state rows), so its slice 4
+    # must read 5 6 7 8 at its own tensor info's offset.
+    [[ "$output" == *"TENSOR blk.0.ffn_gate_up_exps.in_sum2 4x5 0 "* ]]
+    [[ "$output" == *"TENSOR blk.0.ssm_inz.in_sum2 4 0 "* ]]
+    [[ "$output" == *"TENSOR blk.0.ssm_inz.counts 1 0 "* ]]
+    run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ffn_gate_up_exps.in_sum2 16 4
+    [[ "$output" == *"VALS blk.0.ffn_gate_up_exps.in_sum2 5 6 7 8"* ]]
+    run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ssm_inz.in_sum2 0 4
+    [[ "$output" == *"VALS blk.0.ssm_inz.in_sum2 1.5 2.5 3.5 4.5"* ]]
+    run gguf_read "$BATS_TEST_TMPDIR/im.gguf" vals blk.0.ssm_inz.counts 0 1
+    [[ "$output" == *"VALS blk.0.ssm_inz.counts 48"* ]]
+    # The stop rule: importance_for() false-on-a-width-mismatch means an entry
+    # for a projection whose input is NOT the gram's width is a silent lie.
+    [[ "$output" != *"ssm_out"* ]]
+    [[ "$output" != *"attn"* ]]
+    [[ "$output" != *"mtp."* ]]
+    [[ "$output" != *"ple_ngram"* ]]
+    [[ "$output" != *"output.weight"* ]]
+}
+
+@test "rows is a u64: a corpus too big for u32 survives the header" {
+    # The trap this fixture law exists for: pack rows as u32 at offset 16 and a
+    # 100-row file reads back as 4294967296000. 2**32 is exactly representable
+    # in f32, so the count round-trips only if the tool read 8 bytes.
+    C="$BATS_TEST_TMPDIR/wide"
+    mkdir -p "$C"
+    make_gram "$C/gram.r0.blk.0.ffn_down_exps.bin" 4 0 4294967296 0 "1,1,1,1"
+    make_gram "$C/gram.r0.blk.0.ffn_down_exps.0.bin" 4 0 4294967296 0 "3,3,3,3"
+    run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/wide.gguf" --chunk-size 4096
+    [ "$status" -eq 0 ]
+    run gguf_read "$BATS_TEST_TMPDIR/wide.gguf" vals blk.0.ffn_down_exps.counts 0 1
+    [[ "$output" == *"VALS blk.0.ffn_down_exps.counts 4294967296"* ]]
+    [[ "$output" != *"4294967296000"* ]]
+    run gguf_read "$BATS_TEST_TMPDIR/wide.gguf"
+    [[ "$output" == *"KV imatrix.chunk_count 1048576"* ]]
+    # imatrix.chunk_count is a u32 field, so 2**32 rows in 1-row chunks has no
+    # legal encoding: wrapping it would put a number downstream believes.
+    run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/wide2.gguf" --chunk-size 1
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"REFUSED"*"CHUNK_COUNT"* ]]
+    [ ! -e "$BATS_TEST_TMPDIR/wide2.gguf" ]
+}
+
+@test "every tensor's data offset is a multiple of general.alignment" {
+    C="$BATS_TEST_TMPDIR/full"
+    full_recording "$C"
+    run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/im.gguf"
+    [ "$status" -eq 0 ]
+    run aligned_offsets "$BATS_TEST_TMPDIR/im.gguf"
+    [ "$status" -eq 0 ]
+    [[ "$output" == "ALIGNED 12 tensor(s),"*"alignment=32" ]]
+
+    # Red proof, in the same test so the assertion cannot rot: the same file
+    # with one tensor's declared offset nudged by 4. A counts tensor is 4 bytes,
+    # so a writer that forgets to pad between tensors produces exactly this
+    # shape of file — and every value assertion above still passes on it.
+    python3 - "$BATS_TEST_TMPDIR/im.gguf" "$BATS_TEST_TMPDIR/unaligned.gguf" <<'PY'
+import struct, sys
+src, dst = sys.argv[1], sys.argv[2]
+d = bytearray(open(src, "rb").read())
+def u32(o): return struct.unpack_from("<I", d, o)[0]
+def u64(o): return struct.unpack_from("<Q", d, o)[0]
+def st(o):
+    n = u64(o)
+    return d[o + 8:o + 8 + n].decode(), o + 8 + n
+p = 24
+for _ in range(u64(16)):
+    key, p = st(p)
+    ty = u32(p); p += 4
+    if ty == 4:
+        p += 4
+    elif ty == 8:
+        _v, p = st(p)
+    elif ty == 9:
+        et = u32(p); n = u64(p + 4); p += 12
+        for _i in range(n):
+            _v, p = st(p)
+for _ in range(u64(8)):
+    name, p = st(p)
+    nd = u32(p); p += 4
+    p += 8 * nd
+    off_at = p + 4                      # the u64 offset field of this tensor
+    if name == "blk.0.ssm_inz.counts":
+        struct.pack_into("<Q", d, off_at, u64(off_at) + 4)
+    p += 12
+open(dst, "wb").write(bytes(d))
+PY
+    run aligned_offsets "$BATS_TEST_TMPDIR/unaligned.gguf"
+    [ "$status" -eq 1 ]
+    [[ "$output" == *"UNALIGNED"*"blk.0.ssm_inz.counts"* ]]
+}
+
+# --- the refusals -----------------------------------------------------------
+
+@test "a gram short by one float is refused and the file is named" {
+    C="$BATS_TEST_TMPDIR/short"
+    full_recording "$C"
+    # Cut the last 4 bytes of one expert gram: a torn write, which is what a
+    # killed tap actually leaves behind.
+    p="$C/gram.r0.blk.0.ffn_down_exps.0.bin"
+    python3 -c 'import sys
+p = sys.argv[1]
+open(p, "wb").write(open(p, "rb").read()[:-4])' "$p"
+    run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/no.gguf"
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"REFUSED"*"SHORT"* ]]
+    [[ "$output" == *"gram.r0.blk.0.ffn_down_exps.0.bin"* ]]
+    [ ! -e "$BATS_TEST_TMPDIR/no.gguf" ]
+}
+
+@test "an empty recording directory is refused" {
+    mkdir -p "$BATS_TEST_TMPDIR/empty"
+    run python3 "$TOOL" "$BATS_TEST_TMPDIR/empty" -o "$BATS_TEST_TMPDIR/no.gguf"
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"REFUSED"*"NO_GRAMS"* ]]
+    [[ "$output" == *"$BATS_TEST_TMPDIR/empty"* ]]
+    [ ! -e "$BATS_TEST_TMPDIR/no.gguf" ]
+}
+
+@test "expert grams with no pooled gram for the layer are refused" {
+    # Without the pooled gram there is nothing for an under-threshold expert to
+    # inherit, and the tool would quietly invent a zero diagonal for it.
+    C="$BATS_TEST_TMPDIR/nopooled"
+    mkdir -p "$C"
+    make_gram "$C/gram.r0.blk.3.ffn_down_exps.7.bin" 4 0 500 0 "1,2,3,4"
+    run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/no.gguf"
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"REFUSED"*"NO_POOLED"*"layer 3"* ]]
+    [[ "$output" == *"gram.r<rank>.blk.3.ffn_down_exps.bin"* ]]
+}
+
+@test "the same global expert recorded twice with different numbers is refused" {
+    # Rank files carry GLOBAL expert numbering and so describe disjoint experts;
+    # a collision means the recording is corrupt, and whichever file sorted last
+    # would win.
+    C="$BATS_TEST_TMPDIR/conflict"
+    full_recording "$C"
+    make_gram "$C/gram.r1.blk.0.ffn_down_exps.0.bin" 4 1 77 0 "10,20,30,40"
+    run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/no.gguf"
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"REFUSED"*"EXPERT_CONFLICT"*"expert 0"* ]]
+    [[ "$output" == *"gram.r0.blk.0.ffn_down_exps.0.bin"* ]]
+    [[ "$output" == *"gram.r1.blk.0.ffn_down_exps.0.bin"* ]]
+}
+
+@test "an existing output is refused by name rather than overwritten" {
+    # The trash law: a re-run must not destroy the imatrix a convert is reading.
+    C="$BATS_TEST_TMPDIR/full"
+    full_recording "$C"
+    : > "$BATS_TEST_TMPDIR/keep.gguf"
+    printf 'do not clobber me\n' > "$BATS_TEST_TMPDIR/keep.gguf"
+    run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/keep.gguf"
+    [ "$status" -eq 2 ]
+    [[ "$output" == *"REFUSED"*"EXISTS"*"keep.gguf"* ]]
+    run cat "$BATS_TEST_TMPDIR/keep.gguf"
+    [[ "$output" == *"do not clobber me"* ]]
+}
+
+@test "--min-rows moves the threshold and the bytes follow it" {
+    # The default threshold is the engine's own law (4*k rows, lq_hess.h), but
+    # the operator has to be able to raise it when a corpus is thin. Asserted on
+    # the slices, not on a report line: a flag that only changes prose is a
+    # flag that changes nothing.
+    C="$BATS_TEST_TMPDIR/floor"
+    full_recording "$C"
+    run python3 "$TOOL" "$C" -o "$BATS_TEST_TMPDIR/far.gguf" --min-rows 200
+    [ "$status" -eq 0 ]
+    # 200 is above every expert's routing (100, 3, 64, 0, 100 and layer 1's 48),
+    # so all six slices now inherit.
+    [[ "$output" == *"inherited=6"* ]]
+    run gguf_read "$BATS_TEST_TMPDIR/far.gguf" vals blk.0.ffn_down_exps.in_sum2 0 4
+    [[ "$output" == *"VALS blk.0.ffn_down_exps.in_sum2 1.5 2.5 3.5 4.5"* ]]
+    # The counts do not move with the threshold: routing is a measurement, not a
+    # judgement about how much data is enough.
+    run gguf_read "$BATS_TEST_TMPDIR/far.gguf" vals blk.0.ffn_down_exps.counts 0 5
+    [[ "$output" == *"VALS blk.0.ffn_down_exps.counts 100 3 64 0 100"* ]]
+}