37 lines
2.4 KiB
Markdown
37 lines
2.4 KiB
Markdown
---
|
|
name: podman-one-shot-hang-proofing
|
|
description: "Run remote podman one-shot jobs hang-proof: container-as-unit timeout/reap protocol; diagnosing stalls where ssh never returns and containers outlive timeout"
|
|
---
|
|
|
|
Use when running one-shot container jobs over SSH (grading sandboxes, CI executors) and a caller stalls, or `timeout N podman run ...` demonstrably fails to kill the workload.
|
|
|
|
## Stall signatures (all three = this bug)
|
|
- SSH client process alive for hours, no established-socket data flowing
|
|
- `podman ps` shows the container `Up` far past the timeout; `podman top <c> -o pid,stat,etime,args` shows `R`/`S` workload
|
|
- endpoint/queue reports idle (`in_flight: 0`) while the caller awaits
|
|
|
|
## The three laws
|
|
1. **A `timeout`/SIGTERM on an attached `podman run` client kills only the client.** The container is conmon-detached: it survives, inherits the ssh stdout pipe, ssh never sees EOF, the awaiting parent blocks forever.
|
|
2. **`podman wait` has NO `-t` flag in podman 5.x** (docker-ism) → every exec silently dies with rc 125. Bound the waiter with coreutils `timeout -k 5 <T> podman wait <ctr>` — killing a waiter is harmless.
|
|
3. **`podman rm -f` is the only true container kill** (escalates SIGTERM → SIGKILL on the container itself).
|
|
|
|
## Hang-proof exec protocol (single ssh script)
|
|
```bash
|
|
cname=job-<unique>
|
|
podman run -d --name "$cname" --network none --memory 4g --cpus 2 <image> python /w/entry >/dev/null
|
|
timeout -k 5 "$T" podman wait "$cname" >/dev/null 2>&1; rc=$?
|
|
code=$(podman inspect -f '{{.State.ExitCode}}' "$cname" 2>/dev/null || echo -1)
|
|
podman logs "$cname" 2>/dev/null; echo "EXIT:${rc}:${code}"
|
|
podman rm -f "$cname" >/dev/null 2>&1
|
|
```
|
|
- `rc=0, code=0` → success; `rc=124` → timed out (container reaped); harvest output via `podman logs`.
|
|
- Wrap the local ssh spawn in a **hard timeout of T + 180 s** (kills transport half-hangs too).
|
|
- Unique `--name` per submission; the trailing `rm -f` is also the leak reaper.
|
|
|
|
## Recovery of a live stall
|
|
1. `ssh host podman rm -f <ctr>` (warning "resorting to SIGKILL" confirms the theory).
|
|
2. Kill the hung local ssh + supervisor; queue/ledger resume (append-only ledger + takeover by heartbeat-pid liveness) resumes mid-eval; a hung step re-graded under the fixed protocol records a clean FAIL exit 124.
|
|
|
|
## Test to keep (regression)
|
|
`execPython("while True: time.sleep(1)", timeout=20)` → expect `exitCode 124`, completion < 180 s, then `podman ps --format {{.Names}} | grep -c <prefix>` == 0.
|