fix(quant): prove window token identity, gate the tap on progress, kill the runner's orphan race
Adversarial review of the campaign pipeline. Correctness: rechunk-kld-corpus.py wrote each window as text and asserted only lengths, so a KLD reference could score a different token sequence than the corpus holds — every window is now re-encoded and the tool refuses on mismatch; make-calib-corpus.py no longer crashes on a .json that is not a content-keyed JSONL, and its KLD cursor advanced by 2 per fragment (half the pool unused). Silent waste: calib-push.py now aborts when the tap sealed nothing or no prompt was accepted, instead of re-pushing 10M tokens per round; div-by-zero gone, served model id parameterised. Races: run-qwen38-bf16.sh restore() removes the rig container before starting production (today's orphan held the GPUs while the service reported active), the calib tap dir is proven writable, the score ro-guard resolves paths; convert-qwen38-w5nl64.sh refuses a missing --arm value, an existing qwen38-convert-<arm> container, and a zero-exit convert that wrote no artifact. stack-qwen38-official-fp8.py: bare asserts in verify() (skipped under -O) and a non-128-lane PLE HD that only warned are hard failures. Speed: fragment token counts computed once, KLD scored positions accumulated instead of a second full re-encode. Tests: 576 ok / 0 fail.
This commit is contained in:
1 parent
e6fba5d97e
commit
1e4fa14772
6 files changed
+127
-32
No files matched your search
@@ -136,7 +136,12 @@ def main() -> None:
|
||||
if len(window) < a.score_tail:
|
||||
clamped += 1
|
||||
scored += len(window) - score_from
|
||||
child = {"prompt": tok.decode(window),
|
||||
text = tok.decode(window)
|
||||
if tok.encode(text, add_special_tokens=False).ids != window:
|
||||
refuse(f"{a.corpus}:{lineno} window at token {start} does not "
|
||||
"round-trip through text — the reference would score a "
|
||||
"different token sequence than the corpus holds")
|
||||
child = {"prompt": text,
|
||||
"score_from": score_from}
|
||||
if "source" in rec:
|
||||
child["source"] = rec["source"]
|
||||
|
||||
Reference in new issue
Block a user