---
name: container-port-image-identity-binding
description: "Diagnose or write fleet gates that silently verify the WRONG process/image: /health answered by a dying container, podman inspect naming a merely-created container, cgroup lacking the service name, RepoDigests[0] vs the pin, detached launchers losing cwd. Bind every fact to its object by host pid."
---

# Container / port / image identity binding

For fleet gates, restart checks, "is it really up?" triage, and any bench/window harness that stops
a production service and starts a rig on the same port. Use when a check *passes* but the thing
still misbehaves, or when the same class of silent failure keeps recurring.

## The failure signature (all of these mean "you checked a proxy")

| Check that lied | What it actually measured |
|---|---|
| `curl /health` = 200 | whichever process owned the socket, often the one being stopped |
| `podman inspect -f '{{.State.Running}}'` | a container that was created, not one that bound a port |
| `podman ps` / container name | existence, not service |
| `pgrep -f "<pattern>"` | your own polling shell, or a sibling arm |
| log mtime advancing | some request completed, not necessarily yours |
| `df <dir>` | a filesystem that merely looks related to the one the bytes go to |
| `podman image inspect -f '{{index .RepoDigests 0}}'` | ONE of several digests — a by-digest pull makes an image carry the tag digest first |
| rootless cgroup path | the container **ID**, never the service/container name |

## The bindings that work

- **Listener → process:** `ss -H -ltnp 'sport = :<port>'` → `pid=`; needs privilege for foreign
  processes (`sudo ss` when the listener is root's).
- **Host pid → container:** `podman top <ctr> hpid` (header line first).
  **`podman top <ctr> pid` prints container-internal pids (`1`, `7`) and compares against nothing** —
  this is the single most expensive mistake in this class.
- **Image pin:** `podman image inspect -f '{{range .RepoDigests}}{{println .}}{{end}}' <ref>` and test
  *membership* of the pin, never position. Message the difference: "present but does not carry X".
- **Device holder:** `fuser /dev/kfd` → pids → resolve each through `podman top <ctr> hpid`; anything
  outside the allowed set is foreign (an orphaned engine wedges the next boot as `device error`).
- **Detached launcher:** assert cwd (`cd "$(dirname "$0")/.."`; verify a known executable exists) and
  write a `PASS/FAIL` line per fact. `setsid nohup … &` turns a lost cwd into silence, not an error.

## Procedure

1. Write ONE preflight that prints `PASS <fact> …` / `FAIL <fact> — reason`, exit 2 on any FAIL.
   Facts: cwd, image digest membership, free space on the real target filesystem, stale artifacts,
   GPU holders, listener, inputs, reference data.
2. Call it as step 0 of the orchestrator, before any GPU/port is touched. Delete the duplicated
   inline checks so the fact has one owner.
3. Pin each lie with a bats test using PATH stubs (`podman`, `ss`, `fuser`): wrong digest, foreign
   listener pid, foreign GPU pid, dirty dir, wrong cwd. Fixture files must be NON-EMPTY — a `-s`
   fact on a `: > file` fixture fails for the wrong reason.
4. Keep one driver. Ad-hoc copies beside the orchestrator drop guards, and two drivers racing on one
   port produce phantom "strays" that look like a hijack.

## Traps

- `podman top … pid` vs `hpid` — see above.
- Stub collision: a test stub that answers `top` for both the rig and production must branch on the
  container name, or the test proves nothing.
- A gate that reports a stale-but-present image must distinguish "absent" from "retagged/older" —
  the operator's fix differs.
- Verify a kill by re-reading the port; `podman rm -f` frees the cgroup, a pattern-kill may not.