The campaign's scoring corpus cannot be scored by the container it has to be scored in, so this adds the tool that moves the document boundaries without moving the tokens.
scripts/rechunk-kld-corpus.py - re-chunks a KLD corpus into documents of at most --max-doc-tokens (default 7,168: 7x1024, the largest whole kilopage under the 7,436-token KV pool this payload's boot gets). Every window is the parent's own token sequence, cut; each window carries its own score_from = len - --score-tail, so the reference measures more positions over the same text instead of fewer. The calibration corpus is not touched, so no quant moves. Refusals exit 2, the argparse convention; the output is written through a temp file and the token-count invariant is checked before it is published.
tests/qwen38_rechunk.bats - its own seam: the token-exactness invariant (children concatenate back to the parent, every window inside the ceiling, every score_from correct), the summary census (documents, tokens, longest document, scored positions, per-source counts), and the refusals (missing corpus, out == corpus, existing output without --force, a ceiling too small to hold the tail). Offline: no engine, no GPU, no network; the tokenizer is a fixture built in the test tmpdir.
scripts/test-changed.sh - maps the new script to its seam ahead of the generic scripts/* fallback, so changing the re-chunker runs its own tests instead of only the syntax check.
scripts/run-qwen38-bf16.sh - the header claimed the corpus window is 32,768 (--max-model-len). The container's real window is 7,436, printed by the boot ledger, not by a flag; --kld-ref re-checks the file against the declared window, so the header now states the real pool and points at the re-chunker that keeps the declaration honest.
Why it was wrong
make-calib-corpus.py caps records at 8,192 tokens and never rewrites them, so the inherited corpus kept 197-token records (8,193-11,970 tokens each) plus a 38,263-token one. The bf16 arm boots with a 7,436-token pool, so 197 of its 198 documents were unscoreable there and the earlier arms proved it the same way. The engine's own warning in the ledger names the true window while the header repeated the flag.
Measured from the file the rig reads, /home/dank/corpora/out/kld.jsonl: 197 records, 5,507,573 chars total, longest 36,855 chars, 3,757,189 scored chars at the recorded score_from offsets. The rig documents that corpus as 197 prompts / 50,432 scored positions (197 x 256). With a 7,168-token ceiling the same text re-chunks into more, shorter documents; the ceiling is the fact that matters, not the row count.
No vault page changes: this repo has no change diary (destroyed 2026-10-19), git holds the chronology, and the rig header above is the point-of-use record.
Verification
bash tests/run.sh - green, reported in the comment thread. The new seam also runs on its own (bats tests/qwen38_rechunk.bats, 4 tests).
## What
The campaign's scoring corpus cannot be scored by the container it has to be scored in, so this adds the tool that moves the document boundaries without moving the tokens.
- `scripts/rechunk-kld-corpus.py` - re-chunks a KLD corpus into documents of at most `--max-doc-tokens` (default 7,168: 7x1024, the largest whole kilopage under the 7,436-token KV pool this payload's boot gets). Every window is the parent's own token sequence, cut; each window carries its own `score_from = len - --score-tail`, so the reference measures more positions over the same text instead of fewer. The calibration corpus is not touched, so no quant moves. Refusals exit 2, the argparse convention; the output is written through a temp file and the token-count invariant is checked before it is published.
- `tests/qwen38_rechunk.bats` - its own seam: the token-exactness invariant (children concatenate back to the parent, every window inside the ceiling, every `score_from` correct), the summary census (documents, tokens, longest document, scored positions, per-source counts), and the refusals (missing corpus, out == corpus, existing output without `--force`, a ceiling too small to hold the tail). Offline: no engine, no GPU, no network; the tokenizer is a fixture built in the test tmpdir.
- `scripts/test-changed.sh` - maps the new script to its seam ahead of the generic `scripts/*` fallback, so changing the re-chunker runs its own tests instead of only the syntax check.
- `scripts/run-qwen38-bf16.sh` - the header claimed the corpus window is 32,768 (`--max-model-len`). The container's real window is 7,436, printed by the boot ledger, not by a flag; `--kld-ref` re-checks the file against the declared window, so the header now states the real pool and points at the re-chunker that keeps the declaration honest.
## Why it was wrong
`make-calib-corpus.py` caps records at 8,192 tokens and never rewrites them, so the inherited corpus kept 197-token records (8,193-11,970 tokens each) plus a 38,263-token one. The bf16 arm boots with a 7,436-token pool, so 197 of its 198 documents were unscoreable there and the earlier arms proved it the same way. The engine's own warning in the ledger names the true window while the header repeated the flag.
Measured from the file the rig reads, `/home/dank/corpora/out/kld.jsonl`: 197 records, 5,507,573 chars total, longest 36,855 chars, 3,757,189 scored chars at the recorded `score_from` offsets. The rig documents that corpus as 197 prompts / 50,432 scored positions (197 x 256). With a 7,168-token ceiling the same text re-chunks into more, shorter documents; the ceiling is the fact that matters, not the row count.
No vault page changes: this repo has no change diary (destroyed 2026-10-19), git holds the chronology, and the rig header above is the point-of-use record.
## Verification
`bash tests/run.sh` - green, reported in the comment thread. The new seam also runs on its own (`bats tests/qwen38_rechunk.bats`, 4 tests).
---
cwd: /home/blank/.omp/wt/wt-20261027-142335-6d1fc1f · session: 6d1fc1f
The reference rig declared a 32,768-token window while the boot ledger hands it
7,436, so 197 of the 198 inherited documents were too long to score. The engine
warns about the real pool; the flag was the only cited source.
Add scripts/rechunk-kld-corpus.py, which moves document boundaries without
moving tokens: each record is re-cut into windows of at most --max-doc-tokens
(7,168 = 7x1024, the largest whole kilopage under 7,436), and each window gets
its own score_from = len - --score-tail. The reference scores the same token
series at more positions and at a shorter context; the calibration corpus is
untouched, so no quant moves. Writes go through a temp file and the token
counts must match before the output is published.
The rig header now says what the pool actually is, and scripts/test-changed.sh
routes the new script to its own seam instead of the generic syntax fallback.
The reference rig declared a 32,768-token window while the boot ledger hands it
7,436, so 197 of the 198 inherited documents were too long to score. The engine
warns about the real pool; the flag was the only cited source.
Add scripts/rechunk-kld-corpus.py, which moves document boundaries without
moving tokens: each record is re-cut into windows of at most --max-doc-tokens
(7,168 = 7x1024, the largest whole kilopage under 7,436), and each window gets
its own score_from = len - --score-tail. The reference scores the same token
series at more positions and at a shorter context; the calibration corpus is
untouched, so no quant moves. Writes go through a temp file and the token
counts must match before the output is published.
The rig header now says what the pool actually is, and scripts/test-changed.sh
routes the new script to its own seam instead of the generic syntax fallback.
Tests: bash tests/run.sh -> green, 76.6 s (502 tests, exit 0). The new seam, from that run:
ok 497 an existing output is refused by name rather than overwritten
ok 499 a record's windows are that record's own tokens, cut at the ceiling the container holds
ok 500 the summary reports exactly the documents and tokens it wrote
ok 501 an existing output is refused without --force and replaced with it
ok 502 a corpus or ceiling it cannot honour is refused before anything is written
Two fixes fell out of the first red run: refuse() now exits 2 (the argparse convention for unusable arguments) instead of 1, and the two small-ceiling cases pass --score-tail 16 so the fixture's 100-300 token records can actually be re-chunked at a 128-token ceiling instead of tripping the ceiling-vs-tail refusal.
Tests: `bash tests/run.sh` -> green, 76.6 s (502 tests, exit 0). The new seam, from that run:
```
ok 497 an existing output is refused by name rather than overwritten
ok 499 a record's windows are that record's own tokens, cut at the ceiling the container holds
ok 500 the summary reports exactly the documents and tokens it wrote
ok 501 an existing output is refused without --force and replaced with it
ok 502 a corpus or ceiling it cannot honour is refused before anything is written
```
Two fixes fell out of the first red run: `refuse()` now exits 2 (the argparse convention for unusable arguments) instead of 1, and the two small-ceiling cases pass `--score-tail 16` so the fixture's 100-300 token records can actually be re-chunked at a 128-token ceiling instead of tripping the ceiling-vs-tail refusal.
dank
merged commit 58dc7a64c1 into main2026-10-28 11:04:59 -07:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
What
The campaign's scoring corpus cannot be scored by the container it has to be scored in, so this adds the tool that moves the document boundaries without moving the tokens.
scripts/rechunk-kld-corpus.py- re-chunks a KLD corpus into documents of at most--max-doc-tokens(default 7,168: 7x1024, the largest whole kilopage under the 7,436-token KV pool this payload's boot gets). Every window is the parent's own token sequence, cut; each window carries its ownscore_from = len - --score-tail, so the reference measures more positions over the same text instead of fewer. The calibration corpus is not touched, so no quant moves. Refusals exit 2, the argparse convention; the output is written through a temp file and the token-count invariant is checked before it is published.tests/qwen38_rechunk.bats- its own seam: the token-exactness invariant (children concatenate back to the parent, every window inside the ceiling, everyscore_fromcorrect), the summary census (documents, tokens, longest document, scored positions, per-source counts), and the refusals (missing corpus, out == corpus, existing output without--force, a ceiling too small to hold the tail). Offline: no engine, no GPU, no network; the tokenizer is a fixture built in the test tmpdir.scripts/test-changed.sh- maps the new script to its seam ahead of the genericscripts/*fallback, so changing the re-chunker runs its own tests instead of only the syntax check.scripts/run-qwen38-bf16.sh- the header claimed the corpus window is 32,768 (--max-model-len). The container's real window is 7,436, printed by the boot ledger, not by a flag;--kld-refre-checks the file against the declared window, so the header now states the real pool and points at the re-chunker that keeps the declaration honest.Why it was wrong
make-calib-corpus.pycaps records at 8,192 tokens and never rewrites them, so the inherited corpus kept 197-token records (8,193-11,970 tokens each) plus a 38,263-token one. The bf16 arm boots with a 7,436-token pool, so 197 of its 198 documents were unscoreable there and the earlier arms proved it the same way. The engine's own warning in the ledger names the true window while the header repeated the flag.Measured from the file the rig reads,
/home/dank/corpora/out/kld.jsonl: 197 records, 5,507,573 chars total, longest 36,855 chars, 3,757,189 scored chars at the recordedscore_fromoffsets. The rig documents that corpus as 197 prompts / 50,432 scored positions (197 x 256). With a 7,168-token ceiling the same text re-chunks into more, shorter documents; the ceiling is the fact that matters, not the row count.No vault page changes: this repo has no change diary (destroyed 2026-10-19), git holds the chronology, and the rig header above is the point-of-use record.
Verification
bash tests/run.sh- green, reported in the comment thread. The new seam also runs on its own (bats tests/qwen38_rechunk.bats, 4 tests).cwd: /home/blank/.omp/wt/wt-20261027-142335-6d1fc1f · session: 6d1fc1f
Tests:
bash tests/run.sh-> green, 76.6 s (502 tests, exit 0). The new seam, from that run:Two fixes fell out of the first red run:
refuse()now exits 2 (the argparse convention for unusable arguments) instead of 1, and the two small-ceiling cases pass--score-tail 16so the fixture's 100-300 token records can actually be re-chunked at a 128-token ceiling instead of tripping the ceiling-vs-tail refusal.