feat(qwen38): score the inherited corpus at the window the container really has
The reference rig declared a 32,768-token window while the boot ledger hands it 7,436, so 197 of the 198 inherited documents were too long to score. The engine warns about the real pool; the flag was the only cited source. Add scripts/rechunk-kld-corpus.py, which moves document boundaries without moving tokens: each record is re-cut into windows of at most --max-doc-tokens (7,168 = 7x1024, the largest whole kilopage under 7,436), and each window gets its own score_from = len - --score-tail. The reference scores the same token series at more positions and at a shorter context; the calibration corpus is untouched, so no quant moves. Writes go through a temp file and the token counts must match before the output is published. The rig header now says what the pool actually is, and scripts/test-changed.sh routes the new script to its own seam instead of the generic syntax fallback.
This commit is contained in:
1 parent
1c56dea67a
commit
9c386d4f89
4 files changed
+328
-5
No files matched your search
@@ -265,6 +265,13 @@ for path in "${CHANGED[@]}"; do
|
||||
add_test tests/voice_pack_unit.bats
|
||||
add_test tests/scripts_syntax.bats
|
||||
;;
|
||||
# The KLD corpus re-chunker: its seam is its own (the campaign's
|
||||
# one-token-series invariant is asserted there), so a change to it does
|
||||
# not land on the generic scripts/* fallback below.
|
||||
scripts/rechunk-kld-corpus.py)
|
||||
add_test tests/qwen38_rechunk.bats
|
||||
add_test tests/scripts_syntax.bats
|
||||
;;
|
||||
scripts/*)
|
||||
add_test tests/scripts_syntax.bats
|
||||
;;
|
||||
|
||||
Reference in new issue
Block a user