fix(qwen38): the calibration window must prove the rig owns the socket #90

Merged
dank merged 1 commits from fix/quant-pipeline-hardening into main 2026-10-30 16:08:29 -07:00
Owner

What

The calibration window's serve gate now proves the listener belongs to the rig, not just that a
container is "running" and something answers /health.

Sequence that broke a 12 M window on 2026-10-30:

  1. the runner stops frank-archon (seconds before the rig can bind anything);
  2. the dying production process keeps its listener for a moment;
  3. podman inspect -f '{{.State.Running}}' qwen38-bf16 is already true — the container exists;
  4. /health answers 200 — from production.

The window logged phase=calib health ok waited=0s, the pusher (correctly gated on the container)
refused, and the window exited at second zero with FAIL: serve rc=1 push rc=2.

rig_up() now asks the kernel which pid owns :7770 and requires it to be one of the rig container's
processes:

lpid=$(ss -H -ltnp 'sport = :7770' | grep -o 'pid=[0-9]*' | head -1 | cut -d= -f2)
[ -n "$lpid" ] || return 1
podman top "$RIG" pid | grep -qw "$lpid"

Tests (written red first)

Both in tests/qwen38_campaign.bats, with stubbed curl/podman/ss and a stub rig:

  • a health answer from a listener outside the rig is not the rig — health 200, container running,
    listener pid owned by nobody in the container: the window must fail and must not push. Red
    before the fix (it pushed and succeeded), green after.
  • when the listener is inside the rig the window proceeds and banks the tap — positive path.

Also renames the new fixture to tap_window_fixture: it had shadowed the file's existing
window_fixture, so the window dry-run test got an empty path and failed only in whole-file runs.
That collision was mine, from adding tests earlier today.

Testing

./tests/run.sh -> ALL TESTS PASSED (591).


cwd: /home/blank/.omp/wt/wt-20261027-142335-6d1fc1f · session: 01a17f3f-3e2e-7509-aef8-be516c27cf98

## What The calibration window's serve gate now proves the **listener belongs to the rig**, not just that a container is "running" and something answers `/health`. Sequence that broke a 12 M window on 2026-10-30: 1. the runner stops `frank-archon` (seconds before the rig can bind anything); 2. the dying production process keeps its listener for a moment; 3. `podman inspect -f '{{.State.Running}}' qwen38-bf16` is already `true` — the container exists; 4. `/health` answers **200** — from production. The window logged `phase=calib health ok waited=0s`, the pusher (correctly gated on the container) refused, and the window exited at second zero with `FAIL: serve rc=1 push rc=2`. `rig_up()` now asks the kernel which pid owns `:7770` and requires it to be one of the rig container's processes: ```bash lpid=$(ss -H -ltnp 'sport = :7770' | grep -o 'pid=[0-9]*' | head -1 | cut -d= -f2) [ -n "$lpid" ] || return 1 podman top "$RIG" pid | grep -qw "$lpid" ``` ## Tests (written red first) Both in `tests/qwen38_campaign.bats`, with stubbed `curl`/`podman`/`ss` and a stub rig: - **a health answer from a listener outside the rig is not the rig** — health 200, container running, listener pid owned by nobody in the container: the window must fail and must **not** push. Red before the fix (it pushed and succeeded), green after. - **when the listener is inside the rig the window proceeds and banks the tap** — positive path. Also renames the new fixture to `tap_window_fixture`: it had shadowed the file's existing `window_fixture`, so the window dry-run test got an empty path and failed only in whole-file runs. That collision was mine, from adding tests earlier today. ## Testing `./tests/run.sh` -> **ALL TESTS PASSED** (591). --- cwd: /home/blank/.omp/wt/wt-20261027-142335-6d1fc1f · session: 01a17f3f-3e2e-7509-aef8-be516c27cf98
dank added 1 commit 2026-10-30 15:55:03 -07:00
The runner stops frank-archon first, and the dying production process keeps its
listener for a moment while the freshly created rig container is still loading.
'podman inspect says running' plus '/health answers' therefore BOTH hold for the
wrong process: measured 2026-10-30, the window logged 'health ok waited=0s',
the pusher refused on the container name, and a 12M window died at second zero.

The gate now asks the kernel who owns :7770 (ss -ltnp) and requires that pid to
be one of the rig container's processes (podman top). Two tests, written red
first against the old rig_up: a foreign listener must fail the window without a
push, and an in-container listener must proceed and bank.

Also renames the new test fixture to tap_window_fixture: it had shadowed the
existing window_fixture helper, which broke the window dry-run test when the
whole file ran.
dank merged commit 81e0995f7d into main 2026-10-30 16:08:29 -07:00
Sign in to join this conversation.
No Reviewers
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dank/serverAppConfigs#90