Commit Graph
2 Commits
Author SHA1 Message Date
Frank b4a1e6069e fix(qwen38): pin calib to 1.2.0, gate the tap's disk, kill orphaned engines
A 17-hour calibration tap on 2026-10-29/30 wrote ZERO bytes: the runner booted
FRANK_ARCHON_IMAGE (1.8.1) for every mode, and RADIANCE_CALIB_DIR/TOKENS only
arms the expert tap on the 1.2.x line. 1.8.1 still carries the plugin strings and
the "calibration wrote layer Hessians" message, /health answered, /stats showed
1,700 documents and 118 tok/s, and nothing anywhere said the tap was dead.

Four gates, each with the measurement that motivated it:

- CALIB_IMAGE pins the 1.2.0 digest for calib mode (stilldeadcode/radiance
  @sha256:665775ed…), overridable only by naming --image deliberately. Proven by
  the engagement test: `calibration wrote layer Hessians after 10207 tokens`.
- The tap dir's filesystem must have >= 60 GiB free (Q38_CALIB_MIN_FREE_MIB).
  One seal wrote 39.4 GiB in seconds; pointed at a 37 GiB root it filled the
  filesystem to zero and every later write came back short.
- restore() kills orphaned `radiance` processes still holding /dev/kfd before it
  starts the service. `podman rm -f` does not kill the engine: an orphan made
  production's ar_oneshot_2rank_wht6 init fail `device error`, and the unit
  restarted 100 times with /health never answering.
- FRANK_QWEN38_CALIB_DIR / Q38_WINDOW_TRASH_ROOT make the tap dir and the bank
  target configurable, so the tap lives on /mnt/entropy instead of the root fs.
2026-10-30 14:18:03 -07:00
Frank c238e32d6f fix(quant): a tap is only a tap when its rig is running
Four gates, all bought with the 2026-10-29 incident: the rig's calibration boot died of an OOM,
the chain's health loop expired without failing, and the pusher fed four rounds of 10M tokens
(86 minutes of prefill) into RESTORED PRODUCTION, recording nothing.

- run-qwen38-bf16.sh: --ngram-placement becomes a knob whose calib default is `disk` (the 95.37
  GiB table pin is what starved that boot: 230.22 GiB pinned of 246, 8.06 GiB available) while
  kldref and score keep `ram`. calib writes .rig-boot into the tap dir and retires it on a
  non-zero exit, so a dead boot cannot leave the tap looking rig-authorized.
- calib-push.py: refuses before the first prompt unless the marker exists, names this container,
  and that container is running. With the driver's existing tap-progress gate, a push can no
  longer spend hours feeding an endpoint that is not the rig.
- qwen38-calib-window.sh: the calibration window as a gated chain — stop on a failed phase
  instead of falling through the health wait, bank the 336 GiB reference container into trash
  only when both phases exit 0 and the tap holds grams, and print the whole plan on --dry-run.
- kld-gate-qwen38.sh: two-report mode (candidate vs the served payload) for the disk where the
  172.8 GiB FP8 anchor cannot live, plus the laws the report always carried and the gate never
  read: a p99.9 tail ceiling and every by_source slice passing mean, tail and top-1.
- frank-archon.md: the pipeline laws (marker, n-gram tier, three-metric acceptance) and a Files
  section that names the live campaign instead of the retired swift15 era.

Tests: 587/0 — 13 new behavioral cases (dry-run argv, marker refusals, two-report gate math).
2026-10-29 16:14:16 -07:00