feat(qwen38): score the inherited corpus at the window the container really has #78

Merged
dank merged 1 commits from feat/qwen38-w5nl64 into main 2026-10-28 11:04:59 -07:00
1 Commits
Author SHA1 Message Date
Frank 9c386d4f89 feat(qwen38): score the inherited corpus at the window the container really has
The reference rig declared a 32,768-token window while the boot ledger hands it
7,436, so 197 of the 198 inherited documents were too long to score. The engine
warns about the real pool; the flag was the only cited source.

Add scripts/rechunk-kld-corpus.py, which moves document boundaries without
moving tokens: each record is re-cut into windows of at most --max-doc-tokens
(7,168 = 7x1024, the largest whole kilopage under 7,436), and each window gets
its own score_from = len - --score-tail. The reference scores the same token
series at more positions and at a shorter context; the calibration corpus is
untouched, so no quant moves. Writes go through a temp file and the token
counts must match before the output is published.

The rig header now says what the pool actually is, and scripts/test-changed.sh
routes the new script to its own seam instead of the generic syntax fallback.
2026-10-28 00:26:28 -07:00