refactor(quant): one corpus plumbing module, and the builders out of main()
The three corpus tools now share scripts/qwen38_common.py: tokenizer load with the venv guidance, one n_tokens (so a KLD count cannot drift from the corpus it scores), JSONL read/write, an atomic writer, and refuse(). make-calib-corpus.py's 170-line main() with three nested closures became classify / split_pools / fragment_text / build_frags / pack_class / emit_calib / emit_kld / manifest; its magic numbers are named constants. rechunk-kld-corpus.py's main became validate / cut / rechunk around a Window record, so the round-trip proof sits in one function that can be tested without a tokenizer. calib-push.py grew parse_args / load_prompts / push_round, and a malformed corpus now refuses with a line number instead of raising KeyError. Parity, measured on dankserver against the pre-refactor scripts at the same seed: calib.jsonl and kld.jsonl carry identical records in identical order, the manifest is identical, and the re-chunked corpus is byte-identical (md5 e824e67d...). The one byte-level change is JSON escaping — write_jsonl emits literal UTF-8, which is what the re-chunker already wrote; the parsed records do not move.
This commit is contained in:
1 parent
1e4fa14772
commit
3e5a37a21f
5 files changed
+405
-229
No files matched your search
@@ -641,7 +641,10 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
|
||||
- Convert campaign (dormant, kept for a future quant): `scripts/convert-swift15-w4dn8.sh`
|
||||
`scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe`, `scripts/run-swift15-bf16.sh`
|
||||
`scripts/kld-gate-swift15.sh`, `scripts/kld-then-calib.sh`, `scripts/calib-push.py`
|
||||
`scripts/make-calib-corpus.py`, `scripts/derive-protected-experts.py`
|
||||
`scripts/make-calib-corpus.py`, `scripts/rechunk-kld-corpus.py`,
|
||||
`scripts/qwen38_common.py` (the tokenizer, JSONL and refusal plumbing those three
|
||||
corpus tools share — one `n_tokens`, so every tool counts the same way),
|
||||
`scripts/derive-protected-experts.py`
|
||||
`scripts/apply-protected-cast.sh`, `scripts/verify-swift15-ple.py`.
|
||||
- Probes: `scripts/probe-frank-archon-kv.py` (reads slots/window/dtype from the deployed env so
|
||||
it cannot drift; leg D = decode), `scripts/frank-archon-expert-activity.py`
|
||||
|
||||
Reference in new issue
Block a user