fix(qwen38): one identity preflight owns every fact a calibration window needs

Each gate the window used was a proxy, and each lied at least once:
- /health answered 200 from the dying production process (window dead at second zero)
- podman inspect named a rig container that had bound nothing
- the space check asked a directory that merely looked related
- the identity check compared podman's CONTAINER-internal pids (1, 7) to an ss host pid

scripts/qwen38-preflight.sh now checks cwd, the digest a tag actually resolves to,
free space on the tap's own filesystem, stale grams, every /dev/kfd holder's host pid
(podman top <ctr> hpid) and the :7770 listener's host pid against the service's, the
corpus and the reference recording; it prints PASS/FAIL per fact and exits 2. The
window calls it before a card is touched, and its duplicated checks are gone.
This commit is contained in:
Frank committed 2026-10-30 16:26:07 -07:00
1 parent 167e6d6de3
commit 5c678b804c
4 files changed
+292 -27

No files matched your search

+12 -16
View File
@@ -40,7 +40,6 @@ MODELS="${FRANK_QWEN38_MODELS:-/home/dank/models}"
CAMPAIGN="${FRANK_QWEN38_CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}"
RAD="${FRANK_QWEN38_RAD:-$MODELS/rad/qwen38-bf16.rad}"
CORPUS="${Q38_WINDOW_CALIB_CORPUS:-/home/dank/corpora/out/calib-7168.jsonl}"
KLDREF="$CAMPAIGN/kld-ref/kld.json"
CALIB="${FRANK_QWEN38_CALIB_DIR:-$CAMPAIGN/calib}"
HEALTH_TIMEOUT="${Q38_WINDOW_HEALTH_TIMEOUT:-3600}"
ROUNDS="${Q38_WINDOW_ROUNDS:-6}"
@@ -48,22 +47,19 @@ RIG=qwen38-bf16
SERVICE=frank-archon
PY="${PY:-python3}"
# ---- preflight: everything this window needs, before a card is touched -------
[ -f "$KLDREF" ] || { echo "REFUSED: no reference recording at $KLDREF — record it first (scripts/run-qwen38-bf16.sh kldref); this window only fills the calibration tap" >&2; exit 2; }
# ---- preflight: every fact by identity, one owner (scripts/qwen38-preflight.sh) ----
# The checks that used to live here were proxies, and each lied at least once: the space check
# asked a directory that merely LOOKED related, `/health` answered for a dying production
# process, `podman inspect` named a container that existed but had bound nothing. The shared
# script binds each fact to its object — cwd, the digest the tag actually resolves to, free
# space on the tap's own filesystem, every /dev/kfd holder's HOST pid against the service's,
# the :7770 listener's host pid, the corpus and the reference recording. A resumed window
# keeps an existing tap, so the tap-empty fact is waived.
[ -f "$RAD" ] || { echo "REFUSED: no BF16 reference container at $RAD — nothing to serve; scripts/build-qwen38-bf16-rad.sh" >&2; exit 2; }
[ -f "$CORPUS" ] || { echo "REFUSED: no calibration corpus at $CORPUS" >&2; exit 2; }
[ -d "$CALIB" ] || { echo "REFUSED: no tap dir at $CALIB" >&2; exit 2; }
[ -w "$CALIB" ] || { echo "REFUSED: tap dir $CALIB is not writable by $(id -un)" >&2; exit 2; }
# One tap seal writes ~40 GiB (measured 2026-10-30). The runner refuses too; this
# names the same number before a card is touched, so a full volume never becomes
# a 17-hour dead run.
avail_mib=$(df --output=avail -B 1M "$CALIB" 2>/dev/null | tail -1 | tr -d ' ')
[[ "$avail_mib" =~ ^[0-9]+$ ]] || avail_mib=0
need_mib="${Q38_CALIB_MIN_FREE_MIB:-61440}"
[ "$avail_mib" -ge "$need_mib" ] || {
echo "REFUSED: tap dir $CALIB has ${avail_mib} MiB free; one seal writes ~40 GiB and this window needs ${need_mib} MiB (Q38_CALIB_MIN_FREE_MIB overrides)" >&2
exit 2
}
FRANK_QWEN38_CAMPAIGN="$CAMPAIGN" \
Q38_WINDOW_CALIB_CORPUS="$CORPUS" \
Q38_CALIB_MIN_FREE_MIB="${Q38_CALIB_MIN_FREE_MIB:-61440}" \
bash scripts/qwen38-preflight.sh --allow-existing-tap || exit 2
! [[ "$HEALTH_TIMEOUT" =~ ^[0-9]+$ ]] && { echo "REFUSED: Q38_WINDOW_HEALTH_TIMEOUT must be seconds (got: $HEALTH_TIMEOUT)" >&2; exit 2; }
! [[ "$ROUNDS" =~ ^[0-9]+$ ]] && { echo "REFUSED: Q38_WINDOW_ROUNDS must be a round count (got: $ROUNDS)" >&2; exit 2; }