refactor(quant): one corpus plumbing module, and the builders out of main()

The three corpus tools now share scripts/qwen38_common.py: tokenizer load with the
venv guidance, one n_tokens (so a KLD count cannot drift from the corpus it scores),
JSONL read/write, an atomic writer, and refuse(). make-calib-corpus.py's 170-line
main() with three nested closures became classify / split_pools / fragment_text /
build_frags / pack_class / emit_calib / emit_kld / manifest; its magic numbers are
named constants. rechunk-kld-corpus.py's main became validate / cut / rechunk around
a Window record, so the round-trip proof sits in one function that can be tested
without a tokenizer. calib-push.py grew parse_args / load_prompts / push_round, and
a malformed corpus now refuses with a line number instead of raising KeyError.

Parity, measured on dankserver against the pre-refactor scripts at the same seed:
calib.jsonl and kld.jsonl carry identical records in identical order, the manifest
is identical, and the re-chunked corpus is byte-identical (md5 e824e67d...). The one
byte-level change is JSON escaping — write_jsonl emits literal UTF-8, which is what
the re-chunker already wrote; the parsed records do not move.
This commit is contained in:
Frank committed 2026-10-29 15:26:08 -07:00
1 parent 1e4fa14772
commit 3e5a37a21f
5 files changed
+405 -229

No files matched your search

+4 -1
View File
@@ -641,7 +641,10 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
- Convert campaign (dormant, kept for a future quant): `scripts/convert-swift15-w4dn8.sh`
`scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe`, `scripts/run-swift15-bf16.sh`
`scripts/kld-gate-swift15.sh`, `scripts/kld-then-calib.sh`, `scripts/calib-push.py`
`scripts/make-calib-corpus.py`, `scripts/derive-protected-experts.py`
`scripts/make-calib-corpus.py`, `scripts/rechunk-kld-corpus.py`,
`scripts/qwen38_common.py` (the tokenizer, JSONL and refusal plumbing those three
corpus tools share — one `n_tokens`, so every tool counts the same way),
`scripts/derive-protected-experts.py`
`scripts/apply-protected-cast.sh`, `scripts/verify-swift15-ple.py`.
- Probes: `scripts/probe-frank-archon-kv.py` (reads slots/window/dtype from the deployed env so
it cannot drift; leg D = decode), `scripts/frank-archon-expert-activity.py`