fix(qwen38): pin calib to 1.2.0, gate the tap's disk, kill orphaned engines
A 17-hour calibration tap on 2026-10-29/30 wrote ZERO bytes: the runner booted FRANK_ARCHON_IMAGE (1.8.1) for every mode, and RADIANCE_CALIB_DIR/TOKENS only arms the expert tap on the 1.2.x line. 1.8.1 still carries the plugin strings and the "calibration wrote layer Hessians" message, /health answered, /stats showed 1,700 documents and 118 tok/s, and nothing anywhere said the tap was dead. Four gates, each with the measurement that motivated it: - CALIB_IMAGE pins the 1.2.0 digest for calib mode (stilldeadcode/radiance @sha256:665775ed…), overridable only by naming --image deliberately. Proven by the engagement test: `calibration wrote layer Hessians after 10207 tokens`. - The tap dir's filesystem must have >= 60 GiB free (Q38_CALIB_MIN_FREE_MIB). One seal wrote 39.4 GiB in seconds; pointed at a 37 GiB root it filled the filesystem to zero and every later write came back short. - restore() kills orphaned `radiance` processes still holding /dev/kfd before it starts the service. `podman rm -f` does not kill the engine: an orphan made production's ar_oneshot_2rank_wht6 init fail `device error`, and the unit restarted 100 times with /health never answering. - FRANK_QWEN38_CALIB_DIR / Q38_WINDOW_TRASH_ROOT make the tap dir and the bank target configurable, so the tap lives on /mnt/entropy instead of the root fs.
This commit is contained in:
1 parent
f6298e3501
commit
b4a1e6069e
2 files changed
+49
-4
No files matched your search
@@ -36,7 +36,7 @@ CAMPAIGN="${FRANK_QWEN38_CAMPAIGN:-/home/dank/campaign/qwen38-w5nl64}"
|
||||
RAD="${FRANK_QWEN38_RAD:-$MODELS/rad/qwen38-bf16.rad}"
|
||||
CORPUS="${Q38_WINDOW_CALIB_CORPUS:-/home/dank/corpora/out/calib-7168.jsonl}"
|
||||
KLDREF="$CAMPAIGN/kld-ref/kld.json"
|
||||
CALIB="$CAMPAIGN/calib"
|
||||
CALIB="${FRANK_QWEN38_CALIB_DIR:-$CAMPAIGN/calib}"
|
||||
HEALTH_TIMEOUT="${Q38_WINDOW_HEALTH_TIMEOUT:-3600}"
|
||||
ROUNDS="${Q38_WINDOW_ROUNDS:-6}"
|
||||
RIG=qwen38-bf16
|
||||
@@ -49,6 +49,16 @@ PY="${PY:-python3}"
|
||||
[ -f "$CORPUS" ] || { echo "REFUSED: no calibration corpus at $CORPUS" >&2; exit 2; }
|
||||
[ -d "$CALIB" ] || { echo "REFUSED: no tap dir at $CALIB" >&2; exit 2; }
|
||||
[ -w "$CALIB" ] || { echo "REFUSED: tap dir $CALIB is not writable by $(id -un)" >&2; exit 2; }
|
||||
# One tap seal writes ~40 GiB (measured 2026-10-30). The runner refuses too; this
|
||||
# names the same number before a card is touched, so a full volume never becomes
|
||||
# a 17-hour dead run.
|
||||
avail_mib=$(df --output=avail -B 1M "$CALIB" 2>/dev/null | tail -1 | tr -d ' ')
|
||||
[[ "$avail_mib" =~ ^[0-9]+$ ]] || avail_mib=0
|
||||
need_mib="${Q38_CALIB_MIN_FREE_MIB:-61440}"
|
||||
[ "$avail_mib" -ge "$need_mib" ] || {
|
||||
echo "REFUSED: tap dir $CALIB has ${avail_mib} MiB free; one seal writes ~40 GiB and this window needs ${need_mib} MiB (Q38_CALIB_MIN_FREE_MIB overrides)" >&2
|
||||
exit 2
|
||||
}
|
||||
! [[ "$HEALTH_TIMEOUT" =~ ^[0-9]+$ ]] && { echo "REFUSED: Q38_WINDOW_HEALTH_TIMEOUT must be seconds (got: $HEALTH_TIMEOUT)" >&2; exit 2; }
|
||||
! [[ "$ROUNDS" =~ ^[0-9]+$ ]] && { echo "REFUSED: Q38_WINDOW_ROUNDS must be a round count (got: $ROUNDS)" >&2; exit 2; }
|
||||
|
||||
@@ -115,7 +125,7 @@ say "TAP BANKED: $(ls -1 "$CALIB" | wc -l) file(s) in $CALIB"
|
||||
if [ "${Q38_WINDOW_KEEP_RAD:-0}" = "1" ]; then
|
||||
say "reference .rad kept (Q38_WINDOW_KEEP_RAD=1)"
|
||||
else
|
||||
T="$HOME/.AGENTTRASH/$(date +%Y%m%d)-qwen38-bf16-rad"
|
||||
T="${Q38_WINDOW_TRASH_ROOT:-$HOME/.AGENTTRASH}/$(date +%Y%m%d)-qwen38-bf16-rad"
|
||||
mkdir -p "$T" || { echo "FAIL: cannot create trash dir $T" >&2; exit 1; }
|
||||
mv "$RAD" "$T/" && say "reference .rad banked in $T (the checkpoint that rebuilds it stays)"
|
||||
fi
|
||||
|
||||
Reference in new issue
Block a user