fix(qwen38): the calibration window must prove the rig owns the socket

The runner stops frank-archon first, and the dying production process keeps its
listener for a moment while the freshly created rig container is still loading.
'podman inspect says running' plus '/health answers' therefore BOTH hold for the
wrong process: measured 2026-10-30, the window logged 'health ok waited=0s',
the pusher refused on the container name, and a 12M window died at second zero.

The gate now asks the kernel who owns :7770 (ss -ltnp) and requires that pid to
be one of the rig container's processes (podman top). Two tests, written red
first against the old rig_up: a foreign listener must fail the window without a
push, and an in-container listener must proceed and bank.

Also renames the new test fixture to tap_window_fixture: it had shadowed the
existing window_fixture helper, which broke the window dry-run test when the
whole file ran.
This commit is contained in:
Frank committed 2026-10-30 15:54:57 -07:00
1 parent 4043798053
commit 167e6d6de3
2 files changed
+98 -4

No files matched your search

+19 -4
View File
@@ -12,8 +12,13 @@
# (2026-10-29) the rig's boot died of an OOM, the health loop quietly fell through, and the
# pusher fed four rounds of 10M tokens into RESTORED PRODUCTION with nothing recorded:
#
# 1. the runner must still be alive and its container must answer /health, or this script
# kills it, restores the service and exits non-zero — no push, ever;
# 1. the runner must still be alive and its container must answer /health AND own the
# listening socket, or this script kills it, restores the service and exits non-zero —
# no push, ever. The socket clause is not decoration: the runner stops frank-archon
# first, the dying process keeps its listener for a moment, and a rig container that
# is still loading has not bound the port yet, so `podman inspect` + `/health` both
# report success for the WRONG process (measured 2026-10-30: `health ok waited=0s`,
# then the pusher refused on the container name and the window died at second zero);
# 2. the pusher refuses on its own unless the rig wrote its marker into the tap dir, the
# marker names this container, and the container is running (scripts/calib-push.py
# --tap-marker), so a second lock survives a mistake in this file;
@@ -79,8 +84,18 @@ say "phase=calib serve begin"
bash scripts/run-qwen38-bf16.sh calib & spid=$!
rig_up() {
podman inspect -f '{{.State.Running}}' "$RIG" 2>/dev/null | grep -qx true \
&& curl -sf --max-time 10 -o /dev/null http://127.0.0.1:7770/health
podman inspect -f '{{.State.Running}}' "$RIG" 2>/dev/null | grep -qx true || return 1
curl -sf --max-time 10 -o /dev/null http://127.0.0.1:7770/health || return 1
# The answer must come from the CONTAINER. The runner stops frank-archon first, and the
# dying production process keeps its listener for a moment while the freshly created
# rig container is still loading — so `container running` + `/health` 200 both hold
# before the rig has bound anything. Measured 2026-10-30: `health ok waited=0s`, then the
# pusher refused on the container name and the window died at second zero. Ask the kernel
# who owns the listening socket and require it to be a rig process.
local lpid
lpid=$(ss -H -ltnp 'sport = :7770' 2>/dev/null | grep -o 'pid=[0-9]*' | head -1 | cut -d= -f2)
[ -n "${lpid:-}" ] || return 1
podman top "$RIG" pid 2>/dev/null | grep -qw "$lpid"
}
waited=0