fix(qwen38): the calibration window must prove the rig owns the socket
The runner stops frank-archon first, and the dying production process keeps its listener for a moment while the freshly created rig container is still loading. 'podman inspect says running' plus '/health answers' therefore BOTH hold for the wrong process: measured 2026-10-30, the window logged 'health ok waited=0s', the pusher refused on the container name, and a 12M window died at second zero. The gate now asks the kernel who owns :7770 (ss -ltnp) and requires that pid to be one of the rig container's processes (podman top). Two tests, written red first against the old rig_up: a foreign listener must fail the window without a push, and an in-container listener must proceed and bank. Also renames the new test fixture to tap_window_fixture: it had shadowed the existing window_fixture helper, which broke the window dry-run test when the whole file ran.
This commit is contained in:
1 parent
4043798053
commit
167e6d6de3
2 files changed
+98
-4
No files matched your search
@@ -12,8 +12,13 @@
|
||||
# (2026-10-29) the rig's boot died of an OOM, the health loop quietly fell through, and the
|
||||
# pusher fed four rounds of 10M tokens into RESTORED PRODUCTION with nothing recorded:
|
||||
#
|
||||
# 1. the runner must still be alive and its container must answer /health, or this script
|
||||
# kills it, restores the service and exits non-zero — no push, ever;
|
||||
# 1. the runner must still be alive and its container must answer /health AND own the
|
||||
# listening socket, or this script kills it, restores the service and exits non-zero —
|
||||
# no push, ever. The socket clause is not decoration: the runner stops frank-archon
|
||||
# first, the dying process keeps its listener for a moment, and a rig container that
|
||||
# is still loading has not bound the port yet, so `podman inspect` + `/health` both
|
||||
# report success for the WRONG process (measured 2026-10-30: `health ok waited=0s`,
|
||||
# then the pusher refused on the container name and the window died at second zero);
|
||||
# 2. the pusher refuses on its own unless the rig wrote its marker into the tap dir, the
|
||||
# marker names this container, and the container is running (scripts/calib-push.py
|
||||
# --tap-marker), so a second lock survives a mistake in this file;
|
||||
@@ -79,8 +84,18 @@ say "phase=calib serve begin"
|
||||
bash scripts/run-qwen38-bf16.sh calib & spid=$!
|
||||
|
||||
rig_up() {
|
||||
podman inspect -f '{{.State.Running}}' "$RIG" 2>/dev/null | grep -qx true \
|
||||
&& curl -sf --max-time 10 -o /dev/null http://127.0.0.1:7770/health
|
||||
podman inspect -f '{{.State.Running}}' "$RIG" 2>/dev/null | grep -qx true || return 1
|
||||
curl -sf --max-time 10 -o /dev/null http://127.0.0.1:7770/health || return 1
|
||||
# The answer must come from the CONTAINER. The runner stops frank-archon first, and the
|
||||
# dying production process keeps its listener for a moment while the freshly created
|
||||
# rig container is still loading — so `container running` + `/health` 200 both hold
|
||||
# before the rig has bound anything. Measured 2026-10-30: `health ok waited=0s`, then the
|
||||
# pusher refused on the container name and the window died at second zero. Ask the kernel
|
||||
# who owns the listening socket and require it to be a rig process.
|
||||
local lpid
|
||||
lpid=$(ss -H -ltnp 'sport = :7770' 2>/dev/null | grep -o 'pid=[0-9]*' | head -1 | cut -d= -f2)
|
||||
[ -n "${lpid:-}" ] || return 1
|
||||
podman top "$RIG" pid 2>/dev/null | grep -qw "$lpid"
|
||||
}
|
||||
|
||||
waited=0
|
||||
|
||||
Reference in new issue
Block a user