fix(quant): prove window token identity, gate the tap on progress, kill the runner's orphan race
Adversarial review of the campaign pipeline. Correctness: rechunk-kld-corpus.py wrote each window as text and asserted only lengths, so a KLD reference could score a different token sequence than the corpus holds — every window is now re-encoded and the tool refuses on mismatch; make-calib-corpus.py no longer crashes on a .json that is not a content-keyed JSONL, and its KLD cursor advanced by 2 per fragment (half the pool unused). Silent waste: calib-push.py now aborts when the tap sealed nothing or no prompt was accepted, instead of re-pushing 10M tokens per round; div-by-zero gone, served model id parameterised. Races: run-qwen38-bf16.sh restore() removes the rig container before starting production (today's orphan held the GPUs while the service reported active), the calib tap dir is proven writable, the score ro-guard resolves paths; convert-qwen38-w5nl64.sh refuses a missing --arm value, an existing qwen38-convert-<arm> container, and a zero-exit convert that wrote no artifact. stack-qwen38-official-fp8.py: bare asserts in verify() (skipped under -O) and a non-128-lane PLE HD that only warned are hard failures. Speed: fragment token counts computed once, KLD scored positions accumulated instead of a second full re-encode. Tests: 576 ok / 0 fail.
This commit is contained in:
1 parent
e6fba5d97e
commit
1e4fa14772
6 files changed
+127
-32
No files matched your search
@@ -71,10 +71,12 @@ PLAN_ONLY=0
|
||||
KEEP_RECIPE=0
|
||||
while [ "$#" -gt 0 ]; do
|
||||
case "$1" in
|
||||
--arm) ARMS+=("$2"); shift 2 ;;
|
||||
--arm) [ "$#" -ge 2 ] || { echo "--arm needs a value (cd3|cd6|ao1)" >&2; exit 2; }
|
||||
ARMS+=("$2"); shift 2 ;;
|
||||
--dry-run) DRY=1; shift ;;
|
||||
--plan-only) PLAN_ONLY=1; shift ;;
|
||||
--target-count) TARGET_COUNT="$2"; shift 2 ;;
|
||||
--target-count) [ "$#" -ge 2 ] || { echo "--target-count needs a value" >&2; exit 2; }
|
||||
TARGET_COUNT="$2"; shift 2 ;;
|
||||
--keep-recipe) KEEP_RECIPE=1; shift ;;
|
||||
*) echo "unknown argument: $1" >&2; exit 2 ;;
|
||||
esac
|
||||
@@ -124,6 +126,13 @@ fi
|
||||
for a in "${ARMS[@]}"; do
|
||||
[ -e "$OUTDIR/qwen38-w5nl64-$a.rad" ] && {
|
||||
echo "$OUTDIR/qwen38-w5nl64-$a.rad already exists; move it to trash first" >&2; exit 2; }
|
||||
# A killed run leaves its container behind (--rm only fires when podman itself
|
||||
# exits), and the next arm then dies on "name already in use" AFTER the plan
|
||||
# phase; refuse here, where the fix is one `podman rm -f`.
|
||||
podman container exists "qwen38-convert-$a" && {
|
||||
echo "container qwen38-convert-$a still exists (a killed run?);" \
|
||||
"'podman rm -f qwen38-convert-$a' and re-run" >&2
|
||||
exit 2; }
|
||||
done
|
||||
|
||||
# --- the protected plane, spliced into a per-run copy of the template --------------------------
|
||||
@@ -203,6 +212,7 @@ for arm in "${ARMS[@]}"; do
|
||||
rc=0; run_convert "$arm" || rc=$?
|
||||
[ "$rc" -eq 0 ] || { echo "arm $arm: rad-convert rc=$rc; log: $STATE/convert-$arm-$TS.log" >&2; overall=1; continue; }
|
||||
out="$OUTDIR/qwen38-w5nl64-$arm.rad"
|
||||
[ -f "$out" ] || { echo "arm $arm: rad-convert exited 0 but wrote no $out; log: $STATE/convert-$arm-$TS.log" >&2; overall=1; continue; }
|
||||
got_mib=$(( $(stat -c %s "$out") / 1048576 ))
|
||||
# The receipt: the artifact is what the plan predicted, within the metadata the plan does not
|
||||
# count. An arm that came out materially smaller lost a plane somewhere.
|
||||
|
||||
Reference in new issue
Block a user