---
name: podman-one-shot-hang-proofing
description: "Run remote podman one-shot jobs hang-proof: container-as-unit timeout/reap protocol; diagnosing stalls where ssh never returns and containers outlive timeout"
---

Use when running one-shot container jobs over SSH (grading sandboxes, CI executors) and a caller stalls, or `timeout N podman run ...` demonstrably fails to kill the workload.

## Stall signatures (all three = this bug)
- SSH client process alive for hours, no established-socket data flowing
- `podman ps` shows the container `Up` far past the timeout; `podman top <c> -o pid,stat,etime,args` shows `R`/`S` workload
- endpoint/queue reports idle (`in_flight: 0`) while the caller awaits

## The three laws
1. **A `timeout`/SIGTERM on an attached `podman run` client kills only the client.** The container is conmon-detached: it survives, inherits the ssh stdout pipe, ssh never sees EOF, the awaiting parent blocks forever.
2. **`podman wait` has NO `-t` flag in podman 5.x** (docker-ism) → every exec silently dies with rc 125. Bound the waiter with coreutils `timeout -k 5 <T> podman wait <ctr>` — killing a waiter is harmless.
3. **`podman rm -f` is the only true container kill** (escalates SIGTERM → SIGKILL on the container itself).

## Hang-proof exec protocol (single ssh script)
```bash
cname=job-<unique>
podman run -d --name "$cname" --network none --memory 4g --cpus 2 <image> python /w/entry >/dev/null
timeout -k 5 "$T" podman wait "$cname" >/dev/null 2>&1; rc=$?
code=$(podman inspect -f '{{.State.ExitCode}}' "$cname" 2>/dev/null || echo -1)
podman logs "$cname" 2>/dev/null; echo "EXIT:${rc}:${code}"
podman rm -f "$cname" >/dev/null 2>&1
```
- `rc=0, code=0` → success; `rc=124` → timed out (container reaped); harvest output via `podman logs`.
- Wrap the local ssh spawn in a **hard timeout of T + 180 s** (kills transport half-hangs too).
- Unique `--name` per submission; the trailing `rm -f` is also the leak reaper.

## Recovery of a live stall
1. `ssh host podman rm -f <ctr>` (warning "resorting to SIGKILL" confirms the theory).
2. Kill the hung local ssh + supervisor; queue/ledger resume (append-only ledger + takeover by heartbeat-pid liveness) resumes mid-eval; a hung step re-graded under the fixed protocol records a clean FAIL exit 124.

## Test to keep (regression)
`execPython("while True: time.sleep(1)", timeout=20)` → expect `exitCode 124`, completion < 180 s, then `podman ps --format {{.Names}} | grep -c <prefix>` == 0.