feat(qwen38): score the inherited corpus at the window the container really has

The reference rig declared a 32,768-token window while the boot ledger hands it
7,436, so 197 of the 198 inherited documents were too long to score. The engine
warns about the real pool; the flag was the only cited source.

Add scripts/rechunk-kld-corpus.py, which moves document boundaries without
moving tokens: each record is re-cut into windows of at most --max-doc-tokens
(7,168 = 7x1024, the largest whole kilopage under 7,436), and each window gets
its own score_from = len - --score-tail. The reference scores the same token
series at more positions and at a shorter context; the calibration corpus is
untouched, so no quant moves. Writes go through a temp file and the token
counts must match before the output is published.

The rig header now says what the pool actually is, and scripts/test-changed.sh
routes the new script to its own seam instead of the generic syntax fallback.
This commit is contained in:
Frank committed 2026-10-28 00:26:28 -07:00
1 parent 1c56dea67a
commit 9c386d4f89
4 files changed
+328 -5

No files matched your search

+5 -5
View File
@@ -39,11 +39,11 @@
# speculation, fp8 KV, a 32,768-token window, the PLE n-gram table pinned to RAM,
# and the production chat template rather than the one baked into the container —
# every scored container is asked the same tokenised questions the reference
# answered. The window is 32,768: kld.jsonl is 197 records with a longest of
# 38,263 characters (longest prompt FIELD 36,855, median record 27,228). Its
# token length — 10,554 for that longest record, counted by the engine itself —
# is in the funding paragraph below; the kldref gate re-checks the file against
# the window before it boots anything.
# answered. The window the flag spells is 32,768; the pool this payload's boot
# actually gets is 7,436 tokens, so kld.jsonl is re-chunked by
# scripts/rechunk-kld-corpus.py into documents of at most 7,168 tokens — the
# widest one the reference can hold. The kldref gate below re-checks the file
# against the declared window, and the re-chunker is what keeps that honest.
#
# NO --vram-kv-mib: it was REMOVED at 1.3.0 and naming it is a boot refusal, not
# a no-op — verified on the pinned image 2026-10-27: "E --vram-kv-mib was