feat(qwen38): score the inherited corpus at the window the container really has

The reference rig declared a 32,768-token window while the boot ledger hands it
7,436, so 197 of the 198 inherited documents were too long to score. The engine
warns about the real pool; the flag was the only cited source.

Add scripts/rechunk-kld-corpus.py, which moves document boundaries without
moving tokens: each record is re-cut into windows of at most --max-doc-tokens
(7,168 = 7x1024, the largest whole kilopage under 7,436), and each window gets
its own score_from = len - --score-tail. The reference scores the same token
series at more positions and at a shorter context; the calibration corpus is
untouched, so no quant moves. Writes go through a temp file and the token
counts must match before the output is published.

The rig header now says what the pool actually is, and scripts/test-changed.sh
routes the new script to its own seam instead of the generic syntax fallback.
This commit is contained in:
Frank committed 2026-10-28 00:26:28 -07:00
1 parent 1c56dea67a
commit 9c386d4f89
4 files changed
+328 -5

No files matched your search

+7
View File
@@ -265,6 +265,13 @@ for path in "${CHANGED[@]}"; do
add_test tests/voice_pack_unit.bats
add_test tests/scripts_syntax.bats
;;
# The KLD corpus re-chunker: its seam is its own (the campaign's
# one-token-series invariant is asserted there), so a change to it does
# not land on the generic scripts/* fallback below.
scripts/rechunk-kld-corpus.py)
add_test tests/qwen38_rechunk.bats
add_test tests/scripts_syntax.bats
;;
scripts/*)
add_test tests/scripts_syntax.bats
;;