fix(qwen38): one identity preflight owns every fact a calibration window needs

Each gate the window used was a proxy, and each lied at least once:
- /health answered 200 from the dying production process (window dead at second zero)
- podman inspect named a rig container that had bound nothing
- the space check asked a directory that merely looked related
- the identity check compared podman's CONTAINER-internal pids (1, 7) to an ss host pid

scripts/qwen38-preflight.sh now checks cwd, the digest a tag actually resolves to,
free space on the tap's own filesystem, stale grams, every /dev/kfd holder's host pid
(podman top <ctr> hpid) and the :7770 listener's host pid against the service's, the
corpus and the reference recording; it prints PASS/FAIL per fact and exits 2. The
window calls it before a card is touched, and its duplicated checks are gone.
This commit is contained in:
Frank committed 2026-10-30 16:26:07 -07:00
1 parent 167e6d6de3
commit 5c678b804c
4 files changed
+292 -27

No files matched your search

+13
View File
@@ -571,6 +571,19 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
of falling through its health wait. All three exist because on 2026-10-29 the unguarded chain's
rig boot died of an OOM, the health loop expired without failing, and the pusher fed four rounds
of 10M tokens into **restored production** — 86 minutes of live prefill that recorded nothing.
- **A container that exists, a port that answers, and a `/health` 200 are three different claims —
bind identity before believing any of them.** Measured 2026-10-30: `podman inspect` named the rig
while the port still belonged to the *dying* production process (`health ok waited=0s`, window dead
at second zero), and the identity check written to catch that compared two different number spaces:
`podman top <ctr> pid` prints **container-internal** pids (`1`, `7`), never host pids, and a rootless
cgroup carries the container ID rather than the service name — so both "matches" can only ever fail.
The binding is `podman top <ctr> hpid` against the listener from
`ss -H -ltnp 'sport = :7770'`, and `fuser /dev/kfd` likewise resolved through podman.
`scripts/qwen38-preflight.sh` owns those facts now (cwd, the digest a tag actually resolves to,
free space on the tap's **own** filesystem, stale grams, every GPU holder and the listener by host
pid), prints `PASS`/`FAIL` per fact, and `qwen38-calib-window.sh` calls it before a card is touched.
Corollary for every ad-hoc driver: a detached launcher that loses its cwd fails with
`No such file or directory` in a log nobody reads — assert cwd, and never bypass the orchestrator.
- **The n-gram table's pin is what starves a rig boot, and the tier is a knob.** On the BF16 payload
the table is 95.37 GiB, `--ngram-placement ram` pins it, and the calibration boot died at
`startup stopped: out of memory` with 230.22 GiB pinned of 246.07 GiB and 8.06 GiB available —