fix(qwen38): pin calib to 1.2.0, gate the tap's disk, kill orphaned engines

A 17-hour calibration tap on 2026-10-29/30 wrote ZERO bytes: the runner booted
FRANK_ARCHON_IMAGE (1.8.1) for every mode, and RADIANCE_CALIB_DIR/TOKENS only
arms the expert tap on the 1.2.x line. 1.8.1 still carries the plugin strings and
the "calibration wrote layer Hessians" message, /health answered, /stats showed
1,700 documents and 118 tok/s, and nothing anywhere said the tap was dead.

Four gates, each with the measurement that motivated it:

- CALIB_IMAGE pins the 1.2.0 digest for calib mode (stilldeadcode/radiance
  @sha256:665775ed…), overridable only by naming --image deliberately. Proven by
  the engagement test: `calibration wrote layer Hessians after 10207 tokens`.
- The tap dir's filesystem must have >= 60 GiB free (Q38_CALIB_MIN_FREE_MIB).
  One seal wrote 39.4 GiB in seconds; pointed at a 37 GiB root it filled the
  filesystem to zero and every later write came back short.
- restore() kills orphaned `radiance` processes still holding /dev/kfd before it
  starts the service. `podman rm -f` does not kill the engine: an orphan made
  production's ar_oneshot_2rank_wht6 init fail `device error`, and the unit
  restarted 100 times with /health never answering.
- FRANK_QWEN38_CALIB_DIR / Q38_WINDOW_TRASH_ROOT make the tap dir and the bank
  target configurable, so the tap lives on /mnt/entropy instead of the root fs.
This commit is contained in:
Frank committed 2026-10-30 14:18:03 -07:00
1 parent f6298e3501
commit b4a1e6069e
2 files changed
+49 -4

No files matched your search

+12 -2
View File
@@ -36,7 +36,7 @@ CAMPAIGN="${FRANK_QWEN38_CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}"
RAD="${FRANK_QWEN38_RAD:-$MODELS/rad/qwen38-bf16.rad}"
CORPUS="${Q38_WINDOW_CALIB_CORPUS:-/home/dank/corpora/out/calib-7168.jsonl}"
KLDREF="$CAMPAIGN/kld-ref/kld.json"
CALIB="$CAMPAIGN/calib"
CALIB="${FRANK_QWEN38_CALIB_DIR:-$CAMPAIGN/calib}"
HEALTH_TIMEOUT="${Q38_WINDOW_HEALTH_TIMEOUT:-3600}"
ROUNDS="${Q38_WINDOW_ROUNDS:-6}"
RIG=qwen38-bf16
@@ -49,6 +49,16 @@ PY="${PY:-python3}"
[ -f "$CORPUS" ] || { echo "REFUSED: no calibration corpus at $CORPUS" >&2; exit 2; }
[ -d "$CALIB" ] || { echo "REFUSED: no tap dir at $CALIB" >&2; exit 2; }
[ -w "$CALIB" ] || { echo "REFUSED: tap dir $CALIB is not writable by $(id -un)" >&2; exit 2; }
# One tap seal writes ~40 GiB (measured 2026-10-30). The runner refuses too; this
# names the same number before a card is touched, so a full volume never becomes
# a 17-hour dead run.
avail_mib=$(df --output=avail -B 1M "$CALIB" 2>/dev/null | tail -1 | tr -d ' ')
[[ "$avail_mib" =~ ^[0-9]+$ ]] || avail_mib=0
need_mib="${Q38_CALIB_MIN_FREE_MIB:-61440}"
[ "$avail_mib" -ge "$need_mib" ] || {
echo "REFUSED: tap dir $CALIB has ${avail_mib} MiB free; one seal writes ~40 GiB and this window needs ${need_mib} MiB (Q38_CALIB_MIN_FREE_MIB overrides)" >&2
exit 2
}
! [[ "$HEALTH_TIMEOUT" =~ ^[0-9]+$ ]] && { echo "REFUSED: Q38_WINDOW_HEALTH_TIMEOUT must be seconds (got: $HEALTH_TIMEOUT)" >&2; exit 2; }
! [[ "$ROUNDS" =~ ^[0-9]+$ ]] && { echo "REFUSED: Q38_WINDOW_ROUNDS must be a round count (got: $ROUNDS)" >&2; exit 2; }
@@ -115,7 +125,7 @@ say "TAP BANKED: $(ls -1 "$CALIB" | wc -l) file(s) in $CALIB"
if [ "${Q38_WINDOW_KEEP_RAD:-0}" = "1" ]; then
say "reference .rad kept (Q38_WINDOW_KEEP_RAD=1)"
else
T="$HOME/.AGENTTRASH/$(date +%Y%m%d)-qwen38-bf16-rad"
T="${Q38_WINDOW_TRASH_ROOT:-$HOME/.AGENTTRASH}/$(date +%Y%m%d)-qwen38-bf16-rad"
mkdir -p "$T" || { echo "FAIL: cannot create trash dir $T" >&2; exit 1; }
mv "$RAD" "$T/" && say "reference .rad banked in $T (the checkpoint that rebuilds it stays)"
fi