fix(qwen38): one identity preflight owns every fact a calibration window needs
Each gate the window used was a proxy, and each lied at least once: - /health answered 200 from the dying production process (window dead at second zero) - podman inspect named a rig container that had bound nothing - the space check asked a directory that merely looked related - the identity check compared podman's CONTAINER-internal pids (1, 7) to an ss host pid scripts/qwen38-preflight.sh now checks cwd, the digest a tag actually resolves to, free space on the tap's own filesystem, stale grams, every /dev/kfd holder's host pid (podman top <ctr> hpid) and the :7770 listener's host pid against the service's, the corpus and the reference recording; it prints PASS/FAIL per fact and exits 2. The window calls it before a card is touched, and its duplicated checks are gone.
This commit is contained in:
1 parent
167e6d6de3
commit
5c678b804c
4 files changed
+292
-27
No files matched your search
@@ -571,6 +571,19 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
|
||||
of falling through its health wait. All three exist because on 2026-10-29 the unguarded chain's
|
||||
rig boot died of an OOM, the health loop expired without failing, and the pusher fed four rounds
|
||||
of 10M tokens into **restored production** — 86 minutes of live prefill that recorded nothing.
|
||||
- **A container that exists, a port that answers, and a `/health` 200 are three different claims —
|
||||
bind identity before believing any of them.** Measured 2026-10-30: `podman inspect` named the rig
|
||||
while the port still belonged to the *dying* production process (`health ok waited=0s`, window dead
|
||||
at second zero), and the identity check written to catch that compared two different number spaces:
|
||||
`podman top <ctr> pid` prints **container-internal** pids (`1`, `7`), never host pids, and a rootless
|
||||
cgroup carries the container ID rather than the service name — so both "matches" can only ever fail.
|
||||
The binding is `podman top <ctr> hpid` against the listener from
|
||||
`ss -H -ltnp 'sport = :7770'`, and `fuser /dev/kfd` likewise resolved through podman.
|
||||
`scripts/qwen38-preflight.sh` owns those facts now (cwd, the digest a tag actually resolves to,
|
||||
free space on the tap's **own** filesystem, stale grams, every GPU holder and the listener by host
|
||||
pid), prints `PASS`/`FAIL` per fact, and `qwen38-calib-window.sh` calls it before a card is touched.
|
||||
Corollary for every ad-hoc driver: a detached launcher that loses its cwd fails with
|
||||
`No such file or directory` in a log nobody reads — assert cwd, and never bypass the orchestrator.
|
||||
- **The n-gram table's pin is what starves a rig boot, and the tier is a knob.** On the BF16 payload
|
||||
the table is 95.37 GiB, `--ngram-placement ram` pins it, and the calibration boot died at
|
||||
`startup stopped: out of memory` with 230.22 GiB pinned of 246.07 GiB and 8.06 GiB available —
|
||||
|
||||
Reference in new issue
Block a user