fix(quant): a tap is only a tap when its rig is running

Four gates, all bought with the 2026-10-29 incident: the rig's calibration boot died of an OOM,
the chain's health loop expired without failing, and the pusher fed four rounds of 10M tokens
(86 minutes of prefill) into RESTORED PRODUCTION, recording nothing.

- run-qwen38-bf16.sh: --ngram-placement becomes a knob whose calib default is `disk` (the 95.37
  GiB table pin is what starved that boot: 230.22 GiB pinned of 246, 8.06 GiB available) while
  kldref and score keep `ram`. calib writes .rig-boot into the tap dir and retires it on a
  non-zero exit, so a dead boot cannot leave the tap looking rig-authorized.
- calib-push.py: refuses before the first prompt unless the marker exists, names this container,
  and that container is running. With the driver's existing tap-progress gate, a push can no
  longer spend hours feeding an endpoint that is not the rig.
- qwen38-calib-window.sh: the calibration window as a gated chain — stop on a failed phase
  instead of falling through the health wait, bank the 336 GiB reference container into trash
  only when both phases exit 0 and the tap holds grams, and print the whole plan on --dry-run.
- kld-gate-qwen38.sh: two-report mode (candidate vs the served payload) for the disk where the
  172.8 GiB FP8 anchor cannot live, plus the laws the report always carried and the gate never
  read: a p99.9 tail ceiling and every by_source slice passing mean, tail and top-1.
- frank-archon.md: the pipeline laws (marker, n-gram tier, three-metric acceptance) and a Files
  section that names the live campaign instead of the retired swift15 era.

Tests: 587/0 — 13 new behavioral cases (dry-run argv, marker refusals, two-report gate math).
This commit is contained in:
Frank committed 2026-10-29 16:14:16 -07:00
1 parent 3e5a37a21f
commit c238e32d6f
6 files changed
+523 -56

No files matched your search

+38 -1
View File
@@ -193,6 +193,25 @@ if ! [[ "$MAX_MODEL_LEN" =~ ^[0-9]+$ ]]; then
exit 2
fi
# Where the n-gram table lives. On the bf16 payload it is 95.37 GiB and `ram`
# PINS it: the calibration boot died at "startup stopped: out of memory" with
# 230.22 GiB pinned of 246.07 GiB and 8.06 GiB available (measured 2026-10-29),
# while the reference recording and every score boot had already succeeded with
# the same pool and slots. The difference that matters is the job: calib only
# needs the routed weights prefilled, so it reads the table from the container's
# own file tier and the pin never happens; kldref and score keep production's
# placement, because their numbers are only comparable when every other term of
# the shape is identical. FRANK_QWEN38_NGRAM_PLACEMENT overrides either default.
case "$MODE" in
calib) NGRAM_PLACEMENT="${FRANK_QWEN38_NGRAM_PLACEMENT:-disk}" ;;
*) NGRAM_PLACEMENT="${FRANK_QWEN38_NGRAM_PLACEMENT:-ram}" ;;
esac
case "$NGRAM_PLACEMENT" in
auto|vram|ram|disk) ;;
*) echo "REFUSED: FRANK_QWEN38_NGRAM_PLACEMENT must be auto|vram|ram|disk (got: $NGRAM_PLACEMENT)" >&2
exit 2 ;;
esac
# Every container this campaign boots is read through the ro /models mount, so
# the host path must live under it; the container-side spelling is what goes
# into the argv. Same rule for the container a score pass opens.
@@ -230,7 +249,7 @@ ARGS=(--tp 2 --placement expert_tiered --weights-disk-tier
--host-pool-mib "$HOST_POOL_MIB" --gpu-headroom-mib "$GPU_HEADROOM_MIB"
--deterministic --num-speculative-tokens 0
--kv-cache-dtype fp8 --max-model-len "$MAX_MODEL_LEN"
--ngram-placement ram
--ngram-placement "$NGRAM_PLACEMENT"
--override-chat-template /models/tpl.jinja)
MOUNTS=(-v "$MODELS:/models:ro")
TAPENV=()
@@ -289,6 +308,17 @@ case "$MODE" in
MOUNTS+=(-v "$CALIB_DIR:/data/calib")
TAPENV=(-e RADIANCE_CALIB_DIR=/data/calib
-e "RADIANCE_CALIB_TOKENS=${FRANK_QWEN38_CALIB_TOKENS:-12000000}")
# The tap's own authorization. A pusher must see this marker before it sends
# a prompt (calib-push.py --tap-marker), it names the container so a push
# aimed anywhere else refuses, and a boot that fails removes it again below.
# Measured 2026-10-29: with no marker, the chain pushed 4 rounds x 10M tokens
# into RESTORED PRODUCTION after the rig's boot died, and nothing recorded.
if [ "$DRY" = 1 ]; then
:
else
printf 'container=%s\nimage=%s\nmode=calib\nstarted=%s\npid=%s\n' \
"$C" "$IMAGE" "$(date -Is)" "$$" > "$CALIB_DIR/.rig-boot"
fi
;;
esac
@@ -377,5 +407,12 @@ podman rm -f "$C" >/dev/null 2>&1 || true
"${CMD[@]}" 2>&1 | tee "$LOG"
rc=${PIPESTATUS[0]}
# A boot that failed must not leave the tap marked as rig-authorized: the marker
# is what calib-push.py trusts, and the point of it is that a dead rig can never
# be pushed into again.
if [ "$MODE" = calib ] && [ "$rc" -ne 0 ] && [ -f "$CALIB_DIR/.rig-boot" ]; then
mv -f "$CALIB_DIR/.rig-boot" "$CALIB_DIR/.rig-boot.failed-$(date +%Y%m%d-%H%M%S)" 2>/dev/null || true
echo "the rig boot failed (rc=$rc): the tap marker is retired, so no pusher can take this tap for a live rig"
fi
echo "run-qwen38-bf16 $MODE exited rc=$rc; $SERVICE restored; log: $LOG"
exit "$rc"