fix(quant): harden the campaign pipeline, then refactor its corpus tools onto one module #83

2026-10-29T15:14:44-07:00

dank commented

2026-10-29 15:14:44 -07:00

PR #83 — fix(quant): harden the campaign pipeline — wrong-name container races, an unproved token identity, and three silent-waste loops

Adversarial review of the whole qwen38-w5nl64 pipeline (convert → calib → kldref →
score, plus the corpus builder/re-chunker/pusher and the fp8 stacker). Every fix below is a
defect that either destroys hours silently or produces a number that is not the number
claimed; cosmetics and style were left alone.

Stacks on #82 (both touch scripts/frank-archon.env's neighbourhood); merge #82 first.

Defects found and fixed

correctness — a KLD reference that scores different tokens than the corpus holds

rechunk-kld-corpus.py wrote each window as text (tok.decode(window)) and asserted only
window lengths, while its own docstring promises "a record's windows concatenate back to that
record's own token sequence, unchanged". Decode→encode is not identity in general, so the
reference could be recorded against a token sequence the parent corpus never held — and the KLD
gate's whole meaning is that both sides score the same positions. Now every window is
re-encoded and the tool refuses on any mismatch.
make-calib-corpus.py crashed the entire build on any .json in the repo that is not a
content-keyed JSONL: the "has a content key" test was a substring match on the raw line, and a
list-shaped document raised TypeError. Non-conforming lines are now skipped.
make-calib-corpus.py advanced the KLD cursor by 2 per consumed fragment, so half the KLD
pool was never used. Now +1.

silent waste — rounds that cannot finish, taps that cannot fill

calib-push.py looped up to --max-rounds without ever checking that the tap grew. That is
exactly today's failure mode: a wrongly-pointed or read-only tap stops recording while the
engine keeps answering 200, and the driver re-pushes the whole 10 M-token corpus per round.
Now: bytes sealed per round are compared, a round with zero stored bytes aborts, and a round
with zero accepted prompts aborts.
calib-push.py divided by a possibly-zero elapsing time, hard-coded the served model id
(frank-archon) although the rig serves qwen4exp-qwen38-w5nl64, and recomputed the corpus'
char count every round. Fixed; --model and --tap-dir are now parameters.

races and false receipts — the runner and the converter

run-qwen38-bf16.sh's restore() started production without removing the rig container.
--rm only fires when podman itself exits, so a killed run leaves conmon alive holding the
GPUs and the service reports active while its engine starves — measured today, and it needed
a manual podman rm -f qwen38-bf16. restore() now removes the container first.
run-qwen38-bf16.sh: the calib tap directory is now proven writable before the boot (a typo'd
path used to auto-create an empty dir and burn the run), and RADIANCE_CALIB_TOKENS is
overridable (it was a magic 12 M inside the argv).
run-qwen38-bf16.sh: the score's read-only-reference guard compared strings, so
.../kld-ref/../kld-ref/x.json slipped past it and then mounted the reference dir writable.
Now also compared as resolved paths.
convert-qwen38-w5nl64.sh: --arm / --target-count with no value died on set -u's
"unbound variable"; an existing qwen38-convert-<arm> container (a killed run) was only
discovered after the plan phase; and a rad-convert that exited 0 without writing the
artifact produced an arithmetic error instead of a receipt failure. All three refuse early with
the fix named.
stack-qwen38-official-fp8.py: the verifier used bare assert, so python3 -O would have
skipped byte verification entirely; a PLE HD that the 128-lane pack cannot express was a
warning and the stack continued. Both are hard failures now.

speed — the corpus build

make-calib-corpus.py re-tokenized each fragment on every pass of a loop that cycles the pool
up to 40 times, and re-tokenized the entire KLD corpus a second time just to print
kld_scored_positions. Counts are computed once per fragment and the scored total is
accumulated during generation.

Verification

./tests/run.sh — 576 tests, 0 failures (see the job log in the PR comment).
python3 -m py_compile on all four tools; bash -n on both scripts; shellcheck -S warning
clean for both scripts.
calib-push.py --help and an import-level exercise of the new tap_bytes() (empty dir → 0,
one 100-byte file → 100, absent dir → 0).

Not changed, on purpose

convert-qwen38-w5nl64.sh's expert_opts single-quoted $CALIB is correct: the recipe reader
expands it from the container environment (-e CALIB=/data/calib), which shellcheck cannot see.
The arms stay sequential: three concurrent converts would each write ~141 GiB against one
volume — the peak-disk law, not a bug.

Refactor (second commit)

The three corpus tools now share scripts/qwen38_common.py: tokenizer load, one
n_tokens, JSONL read/write, atomic writer, refuse(). make-calib-corpus.py's
170-line main() with three nested closures is now classify / split_pools /
fragment_text / build_frags / pack_class / emit_calib / emit_kld /
manifest; rechunk-kld-corpus.py's main is validate / cut / rechunk around
a Window record, so the round-trip proof is one testable function; calib-push.py
grew parse_args / load_prompts / push_round.

Parity was measured on dankserver against the pre-refactor scripts at the same seed:
identical records in identical order for calib.jsonl and kld.jsonl, identical
manifest, byte-identical re-chunked corpus (md5 e824e67d3acbffe2b6cb2b1c50853457).
The only byte-level change is JSON escaping — write_jsonl emits literal UTF-8,
which the re-chunker already did; the parsed records do not move.

cwd: /home/blank/.omp/wt/wt-20261027-142335-6d1fc1f · session: 01a17f3f-3e2e-7509-aef8-be516c27cf98

# PR #83 — fix(quant): harden the campaign pipeline — wrong-name container races, an unproved token identity, and three silent-waste loops Adversarial review of the whole `qwen38-w5nl64` pipeline (`convert` → `calib` → `kldref` → `score`, plus the corpus builder/re-chunker/pusher and the fp8 stacker). Every fix below is a defect that either **destroys hours silently** or **produces a number that is not the number claimed**; cosmetics and style were left alone. **Stacks on #82** (both touch `scripts/frank-archon.env`'s neighbourhood); merge #82 first. ## Defects found and fixed ### correctness — a KLD reference that scores different tokens than the corpus holds - `rechunk-kld-corpus.py` wrote each window as **text** (`tok.decode(window)`) and asserted only window lengths, while its own docstring promises "a record's windows concatenate back to that record's own token sequence, unchanged". Decode→encode is not identity in general, so the reference could be recorded against a token sequence the parent corpus never held — and the KLD gate's whole meaning is that both sides score *the same positions*. Now every window is re-encoded and the tool refuses on any mismatch. - `make-calib-corpus.py` crashed the entire build on any `.json` in the repo that is not a content-keyed JSONL: the "has a content key" test was a substring match on the raw line, and a list-shaped document raised `TypeError`. Non-conforming lines are now skipped. - `make-calib-corpus.py` advanced the KLD cursor by **2** per consumed fragment, so half the KLD pool was never used. Now `+1`. ### silent waste — rounds that cannot finish, taps that cannot fill - `calib-push.py` looped up to `--max-rounds` **without ever checking that the tap grew**. That is exactly today's failure mode: a wrongly-pointed or read-only tap stops recording while the engine keeps answering 200, and the driver re-pushes the whole 10 M-token corpus per round. Now: bytes sealed per round are compared, a round with zero stored bytes aborts, and a round with zero accepted prompts aborts. - `calib-push.py` divided by a possibly-zero elapsing time, hard-coded the served model id (`frank-archon`) although the rig serves `qwen4exp-qwen38-w5nl64`, and recomputed the corpus' char count every round. Fixed; `--model` and `--tap-dir` are now parameters. ### races and false receipts — the runner and the converter - `run-qwen38-bf16.sh`'s `restore()` started production **without removing the rig container**. `--rm` only fires when podman itself exits, so a killed run leaves conmon alive holding the GPUs and the service reports `active` while its engine starves — measured today, and it needed a manual `podman rm -f qwen38-bf16`. `restore()` now removes the container first. - `run-qwen38-bf16.sh`: the calib tap directory is now proven writable before the boot (a typo'd path used to auto-create an empty dir and burn the run), and `RADIANCE_CALIB_TOKENS` is overridable (it was a magic 12 M inside the argv). - `run-qwen38-bf16.sh`: the score's read-only-reference guard compared strings, so `.../kld-ref/../kld-ref/x.json` slipped past it and then mounted the reference dir writable. Now also compared as resolved paths. - `convert-qwen38-w5nl64.sh`: `--arm` / `--target-count` with no value died on `set -u`'s "unbound variable"; an existing `qwen38-convert-<arm>` container (a killed run) was only discovered *after* the plan phase; and a `rad-convert` that exited 0 without writing the artifact produced an arithmetic error instead of a receipt failure. All three refuse early with the fix named. - `stack-qwen38-official-fp8.py`: the verifier used bare `assert`, so `python3 -O` would have skipped byte verification entirely; a PLE `HD` that the 128-lane pack cannot express was a warning and the stack continued. Both are hard failures now. ### speed — the corpus build - `make-calib-corpus.py` re-tokenized each fragment on every pass of a loop that cycles the pool up to 40 times, and re-tokenized the entire KLD corpus a second time just to print `kld_scored_positions`. Counts are computed once per fragment and the scored total is accumulated during generation. ## Verification - `./tests/run.sh` — 576 tests, 0 failures (see the job log in the PR comment). - `python3 -m py_compile` on all four tools; `bash -n` on both scripts; `shellcheck -S warning` clean for both scripts. - `calib-push.py --help` and an import-level exercise of the new `tap_bytes()` (empty dir → 0, one 100-byte file → 100, absent dir → 0). ## Not changed, on purpose - `convert-qwen38-w5nl64.sh`'s `expert_opts` single-quoted `$CALIB` is correct: the recipe reader expands it from the container environment (`-e CALIB=/data/calib`), which shellcheck cannot see. - The arms stay sequential: three concurrent converts would each write ~141 GiB against one volume — the peak-disk law, not a bug. ## Refactor (second commit) The three corpus tools now share `scripts/qwen38_common.py`: tokenizer load, one `n_tokens`, JSONL read/write, atomic writer, `refuse()`. `make-calib-corpus.py`'s 170-line `main()` with three nested closures is now `classify` / `split_pools` / `fragment_text` / `build_frags` / `pack_class` / `emit_calib` / `emit_kld` / `manifest`; `rechunk-kld-corpus.py`'s `main` is `validate` / `cut` / `rechunk` around a `Window` record, so the round-trip proof is one testable function; `calib-push.py` grew `parse_args` / `load_prompts` / `push_round`. Parity was measured on dankserver against the pre-refactor scripts at the same seed: identical records in identical order for `calib.jsonl` and `kld.jsonl`, identical manifest, byte-identical re-chunked corpus (md5 `e824e67d3acbffe2b6cb2b1c50853457`). The only byte-level change is JSON escaping — `write_jsonl` emits literal UTF-8, which the re-chunker already did; the parsed records do not move. --- cwd: /home/blank/.omp/wt/wt-20261027-142335-6d1fc1f · session: 01a17f3f-3e2e-7509-aef8-be516c27cf98

dank added 2 commits 2026-10-29 15:14:44 -07:00

fix(frank-archon): fold the pinned host pool back to 49,152; keep the 2,048 step e6fba5d97e

PR #81's pool half measured inert the day it deployed: the 66,560 boot printed the
same elastic 26.73 GiB / experts 20.72 / kv cache 5.64 / vram_weights 24.30 as the
49,152 boot and opened the pool with 0 B used, and the qwen38-w5nl64 rig moved
vram_kv 269.99 -> 271.25 MiB (window still 7,436 tokens: its 95.37 GiB n-gram table
claim decides, not the pool). The step half stays. Tests: 576 ok / 0 fail.

fix(quant): prove window token identity, gate the tap on progress, kill the runner's orphan race 1e4fa14772

Adversarial review of the campaign pipeline. Correctness: rechunk-kld-corpus.py wrote
each window as text and asserted only lengths, so a KLD reference could score a
different token sequence than the corpus holds — every window is now re-encoded and
the tool refuses on mismatch; make-calib-corpus.py no longer crashes on a .json that
is not a content-keyed JSONL, and its KLD cursor advanced by 2 per fragment (half the
pool unused). Silent waste: calib-push.py now aborts when the tap sealed nothing or
no prompt was accepted, instead of re-pushing 10M tokens per round; div-by-zero gone,
served model id parameterised. Races: run-qwen38-bf16.sh restore() removes the rig
container before starting production (today's orphan held the GPUs while the service
reported active), the calib tap dir is proven writable, the score ro-guard resolves
paths; convert-qwen38-w5nl64.sh refuses a missing --arm value, an existing
qwen38-convert-<arm> container, and a zero-exit convert that wrote no artifact.
stack-qwen38-official-fp8.py: bare asserts in verify() (skipped under -O) and a
non-128-lane PLE HD that only warned are hard failures. Speed: fragment token counts
computed once, KLD scored positions accumulated instead of a second full re-encode.
Tests: 576 ok / 0 fail.

dank changed title from ~~fix(quant): harden the campaign pipeline — token identity, tap progress, runner races~~ to fix(quant): harden the campaign pipeline, then refactor its corpus tools onto one module

2026-10-29 15:26:08 -07:00

dank added 1 commit 2026-10-29 15:26:08 -07:00

refactor(quant): one corpus plumbing module, and the builders out of main() 3e5a37a21f

The three corpus tools now share scripts/qwen38_common.py: tokenizer load with the
venv guidance, one n_tokens (so a KLD count cannot drift from the corpus it scores),
JSONL read/write, an atomic writer, and refuse(). make-calib-corpus.py's 170-line
main() with three nested closures became classify / split_pools / fragment_text /
build_frags / pack_class / emit_calib / emit_kld / manifest; its magic numbers are
named constants. rechunk-kld-corpus.py's main became validate / cut / rechunk around
a Window record, so the round-trip proof sits in one function that can be tested
without a tokenizer. calib-push.py grew parse_args / load_prompts / push_round, and
a malformed corpus now refuses with a line number instead of raising KeyError.

Parity, measured on dankserver against the pre-refactor scripts at the same seed:
calib.jsonl and kld.jsonl carry identical records in identical order, the manifest
is identical, and the re-chunked corpus is byte-identical (md5 e824e67d...). The one
byte-level change is JSON escaping — write_jsonl emits literal UTF-8, which is what
the re-chunker already wrote; the parsed records do not move.

dank merged commit da5eb968f9 into main

2026-10-29 15:29:34 -07:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dank/serverAppConfigs#83