feat(qwen38): score the inherited corpus at the window the container really has
The reference rig declared a 32,768-token window while the boot ledger hands it 7,436, so 197 of the 198 inherited documents were too long to score. The engine warns about the real pool; the flag was the only cited source. Add scripts/rechunk-kld-corpus.py, which moves document boundaries without moving tokens: each record is re-cut into windows of at most --max-doc-tokens (7,168 = 7x1024, the largest whole kilopage under 7,436), and each window gets its own score_from = len - --score-tail. The reference scores the same token series at more positions and at a shorter context; the calibration corpus is untouched, so no quant moves. Writes go through a temp file and the token counts must match before the output is published. The rig header now says what the pool actually is, and scripts/test-changed.sh routes the new script to its own seam instead of the generic syntax fallback.
This commit is contained in:
1 parent
1c56dea67a
commit
9c386d4f89
4 files changed
+328
-5
No files matched your search
@@ -39,11 +39,11 @@
|
||||
# speculation, fp8 KV, a 32,768-token window, the PLE n-gram table pinned to RAM,
|
||||
# and the production chat template rather than the one baked into the container —
|
||||
# every scored container is asked the same tokenised questions the reference
|
||||
# answered. The window is 32,768: kld.jsonl is 197 records with a longest of
|
||||
# 38,263 characters (longest prompt FIELD 36,855, median record 27,228). Its
|
||||
# token length — 10,554 for that longest record, counted by the engine itself —
|
||||
# is in the funding paragraph below; the kldref gate re-checks the file against
|
||||
# the window before it boots anything.
|
||||
# answered. The window the flag spells is 32,768; the pool this payload's boot
|
||||
# actually gets is 7,436 tokens, so kld.jsonl is re-chunked by
|
||||
# scripts/rechunk-kld-corpus.py into documents of at most 7,168 tokens — the
|
||||
# widest one the reference can hold. The kldref gate below re-checks the file
|
||||
# against the declared window, and the re-chunker is what keeps that honest.
|
||||
#
|
||||
# NO --vram-kv-mib: it was REMOVED at 1.3.0 and naming it is a boot refusal, not
|
||||
# a no-op — verified on the pinned image 2026-10-27: "E --vram-kv-mib was
|
||||
|
||||
Reference in new issue
Block a user