fix(quant): prove window token identity, gate the tap on progress, kill the runner's orphan race

Adversarial review of the campaign pipeline. Correctness: rechunk-kld-corpus.py wrote
each window as text and asserted only lengths, so a KLD reference could score a
different token sequence than the corpus holds — every window is now re-encoded and
the tool refuses on mismatch; make-calib-corpus.py no longer crashes on a .json that
is not a content-keyed JSONL, and its KLD cursor advanced by 2 per fragment (half the
pool unused). Silent waste: calib-push.py now aborts when the tap sealed nothing or
no prompt was accepted, instead of re-pushing 10M tokens per round; div-by-zero gone,
served model id parameterised. Races: run-qwen38-bf16.sh restore() removes the rig
container before starting production (today's orphan held the GPUs while the service
reported active), the calib tap dir is proven writable, the score ro-guard resolves
paths; convert-qwen38-w5nl64.sh refuses a missing --arm value, an existing
qwen38-convert-<arm> container, and a zero-exit convert that wrote no artifact.
stack-qwen38-official-fp8.py: bare asserts in verify() (skipped under -O) and a
non-128-lane PLE HD that only warned are hard failures. Speed: fragment token counts
computed once, KLD scored positions accumulated instead of a second full re-encode.
Tests: 576 ok / 0 fail.
This commit is contained in:
Frank committed 2026-10-29 15:14:43 -07:00
1 parent e6fba5d97e
commit 1e4fa14772
6 files changed
+127 -32

No files matched your search

+21 -4
View File
@@ -120,7 +120,8 @@ while [ $# -gt 0 ]; do
case "$1" in
kldref|calib|score) MODE="$1" ;;
--dry-run) DRY=1 ;;
--image) IMAGE="$2"; shift ;;
--image) [ "$#" -ge 2 ] || { echo "--image needs a reference" >&2; exit 2; }
IMAGE="$2"; shift ;;
*)
# score's two positional arguments, and nothing else may be bare.
if [ "$MODE" = score ] && [ "${#POS[@]}" -lt 2 ]; then
@@ -250,7 +251,8 @@ case "$MODE" in
# time the engine reads a recording and writes a report instead of the
# reverse. The reference dir mounts READ-ONLY — a scoring pass must not be
# able to touch the zero point — and the report gets its own writable bind.
if [ "$OUT_DIR" = "$KLD_REF_DIR" ] || same_dir "$OUT_DIR" "$KLD_REF_DIR"; then
if [ "$OUT_DIR" = "$KLD_REF_DIR" ] || same_dir "$OUT_DIR" "$KLD_REF_DIR" \
|| [ "$(realpath -m "$OUT_DIR")" = "$(realpath -m "$KLD_REF_DIR")" ]; then
echo "REFUSED: score output $SCORE_OUT is inside the reference dir $KLD_REF_DIR, which this mode mounts read-only" >&2
exit 2
fi
@@ -276,9 +278,17 @@ case "$MODE" in
# through MAX_SEQS, exactly as kldref and score do.
ARGS+=(--model "$MODEL_IN" --port 7770
--max-num-seqs "$MAX_SEQS" --max-num-batched-tokens "$MNBT")
# The tap is only real if this directory exists and the engine can write it:
# without the check a typo'd CALIB_DIR auto-creates a root-owned empty dir in
# whatever path was given, and the run burns hours serving prompts into a tap
# nobody reads. RADIANCE_CALIB_TOKENS is the whole corpus' floor, not a round's.
[ "$DRY" = 1 ] || [ -w "$CALIB_DIR" ] || {
echo "REFUSED: calib tap dir $CALIB_DIR is not writable (mkdir it, chown it, then re-run)" >&2
exit 2
}
MOUNTS+=(-v "$CALIB_DIR:/data/calib")
TAPENV=(-e RADIANCE_CALIB_DIR=/data/calib
-e RADIANCE_CALIB_TOKENS=12000000)
-e "RADIANCE_CALIB_TOKENS=${FRANK_QWEN38_CALIB_TOKENS:-12000000}")
;;
esac
@@ -352,7 +362,14 @@ cp -f "$TEMPLATE" "$TPL" || {
[ "$MODE" = kldref ] && mkdir -p "$KLD_REF_DIR"
[ "$MODE" = calib ] && mkdir -p "$CALIB_DIR"
restore() { sudo systemctl start "$SERVICE" >/dev/null 2>&1 || true; }
# Stop the campaign container BEFORE handing the cards back: a killed run leaves
# its conmon alive (--rm only fires when podman itself exits), and production
# then boots beside a container still holding the GPUs — measured 2026-10-29,
# where the service reported `active` while its engine starved on the rig's VRAM.
restore() {
podman rm -f "$C" >/dev/null 2>&1 || true
sudo systemctl start "$SERVICE" >/dev/null 2>&1 || true
}
trap restore EXIT
echo "run-qwen38-bf16 $MODE: stopping $SERVICE ($(date -Is)); log: $LOG"
sudo systemctl stop "$SERVICE" >/dev/null 2>&1 || true