live: heartbeat sync 2026-10-30T17:23:49-07:00 (Refs #11)

agent/managed-skills/container-port-image-identity-binding/SKILL.md

@@ -0,0 +1,59 @@
+---
+name: container-port-image-identity-binding
+description: "Diagnose or write fleet gates that silently verify the WRONG process/image: /health answered by a dying container, podman inspect naming a merely-created container, cgroup lacking the service name, RepoDigests[0] vs the pin, detached launchers losing cwd. Bind every fact to its object by host pid."
+---
+
+# Container / port / image identity binding
+
+For fleet gates, restart checks, "is it really up?" triage, and any bench/window harness that stops
+a production service and starts a rig on the same port. Use when a check *passes* but the thing
+still misbehaves, or when the same class of silent failure keeps recurring.
+
+## The failure signature (all of these mean "you checked a proxy")
+
+| Check that lied | What it actually measured |
+|---|---|
+| `curl /health` = 200 | whichever process owned the socket, often the one being stopped |
+| `podman inspect -f '{{.State.Running}}'` | a container that was created, not one that bound a port |
+| `podman ps` / container name | existence, not service |
+| `pgrep -f "<pattern>"` | your own polling shell, or a sibling arm |
+| log mtime advancing | some request completed, not necessarily yours |
+| `df <dir>` | a filesystem that merely looks related to the one the bytes go to |
+| `podman image inspect -f '{{index .RepoDigests 0}}'` | ONE of several digests — a by-digest pull makes an image carry the tag digest first |
+| rootless cgroup path | the container **ID**, never the service/container name |
+
+## The bindings that work
+
+- **Listener → process:** `ss -H -ltnp 'sport = :<port>'` → `pid=`; needs privilege for foreign
+  processes (`sudo ss` when the listener is root's).
+- **Host pid → container:** `podman top <ctr> hpid` (header line first).
+  **`podman top <ctr> pid` prints container-internal pids (`1`, `7`) and compares against nothing** —
+  this is the single most expensive mistake in this class.
+- **Image pin:** `podman image inspect -f '{{range .RepoDigests}}{{println .}}{{end}}' <ref>` and test
+  *membership* of the pin, never position. Message the difference: "present but does not carry X".
+- **Device holder:** `fuser /dev/kfd` → pids → resolve each through `podman top <ctr> hpid`; anything
+  outside the allowed set is foreign (an orphaned engine wedges the next boot as `device error`).
+- **Detached launcher:** assert cwd (`cd "$(dirname "$0")/.."`; verify a known executable exists) and
+  write a `PASS/FAIL` line per fact. `setsid nohup … &` turns a lost cwd into silence, not an error.
+
+## Procedure
+
+1. Write ONE preflight that prints `PASS <fact> …` / `FAIL <fact> — reason`, exit 2 on any FAIL.
+   Facts: cwd, image digest membership, free space on the real target filesystem, stale artifacts,
+   GPU holders, listener, inputs, reference data.
+2. Call it as step 0 of the orchestrator, before any GPU/port is touched. Delete the duplicated
+   inline checks so the fact has one owner.
+3. Pin each lie with a bats test using PATH stubs (`podman`, `ss`, `fuser`): wrong digest, foreign
+   listener pid, foreign GPU pid, dirty dir, wrong cwd. Fixture files must be NON-EMPTY — a `-s`
+   fact on a `: > file` fixture fails for the wrong reason.
+4. Keep one driver. Ad-hoc copies beside the orchestrator drop guards, and two drivers racing on one
+   port produce phantom "strays" that look like a hijack.
+
+## Traps
+
+- `podman top … pid` vs `hpid` — see above.
+- Stub collision: a test stub that answers `top` for both the rig and production must branch on the
+  container name, or the test proves nothing.
+- A gate that reports a stale-but-present image must distinguish "absent" from "retagged/older" —
+  the operator's fix differs.
+- Verify a kill by re-reading the port; `podman rm -f` frees the cgroup, a pattern-kill may not.