command-center/agent/managed-skills/podman-one-shot-hang-proofing/SKILL.md

name, description

name	description
podman-one-shot-hang-proofing	Run remote podman one-shot jobs hang-proof: container-as-unit timeout/reap protocol; diagnosing stalls where ssh never returns and containers outlive timeout

Use when running one-shot container jobs over SSH (grading sandboxes, CI executors) and a caller stalls, or timeout N podman run ... demonstrably fails to kill the workload.

Stall signatures (all three = this bug)

• SSH client process alive for hours, no established-socket data flowing
• podman ps shows the container Up far past the timeout; podman top <c> -o pid,stat,etime,args shows R/S workload
• endpoint/queue reports idle (in_flight: 0) while the caller awaits

The three laws

1. A timeout/SIGTERM on an attached podman run client kills only the client. The container is conmon-detached: it survives, inherits the ssh stdout pipe, ssh never sees EOF, the awaiting parent blocks forever.
2. podman wait has NO -t flag in podman 5.x (docker-ism) → every exec silently dies with rc 125. Bound the waiter with coreutils timeout -k 5 <T> podman wait <ctr> — killing a waiter is harmless.
3. podman rm -f is the only true container kill (escalates SIGTERM → SIGKILL on the container itself).

Hang-proof exec protocol (single ssh script)

cname=job-<unique>
podman run -d --name "$cname" --network none --memory 4g --cpus 2 <image> python /w/entry >/dev/null
timeout -k 5 "$T" podman wait "$cname" >/dev/null 2>&1; rc=$?
code=$(podman inspect -f '{{.State.ExitCode}}' "$cname" 2>/dev/null || echo -1)
podman logs "$cname" 2>/dev/null; echo "EXIT:${rc}:${code}"
podman rm -f "$cname" >/dev/null 2>&1

• rc=0, code=0 → success; rc=124 → timed out (container reaped); harvest output via podman logs.
• Wrap the local ssh spawn in a hard timeout of T + 180 s (kills transport half-hangs too).
• Unique --name per submission; the trailing rm -f is also the leak reaper.

Recovery of a live stall

1. ssh host podman rm -f <ctr> (warning "resorting to SIGKILL" confirms the theory).
2. Kill the hung local ssh + supervisor; queue/ledger resume (append-only ledger + takeover by heartbeat-pid liveness) resumes mid-eval; a hung step re-graded under the fixed protocol records a clean FAIL exit 124.

Test to keep (regression)

execPython("while True: time.sleep(1)", timeout=20) → expect exitCode 124, completion < 180 s, then podman ps --format {{.Names}} | grep -c <prefix> == 0.