live: heartbeat sync 2026-10-18T16:47:59-07:00 (Refs #11)

agent/managed-skills/podman-one-shot-hang-proofing/SKILL.md

@@ -0,0 +1,36 @@
+---
+name: podman-one-shot-hang-proofing
+description: "Run remote podman one-shot jobs hang-proof: container-as-unit timeout/reap protocol; diagnosing stalls where ssh never returns and containers outlive timeout"
+---
+
+Use when running one-shot container jobs over SSH (grading sandboxes, CI executors) and a caller stalls, or `timeout N podman run ...` demonstrably fails to kill the workload.
+
+## Stall signatures (all three = this bug)
+- SSH client process alive for hours, no established-socket data flowing
+- `podman ps` shows the container `Up` far past the timeout; `podman top <c> -o pid,stat,etime,args` shows `R`/`S` workload
+- endpoint/queue reports idle (`in_flight: 0`) while the caller awaits
+
+## The three laws
+1. **A `timeout`/SIGTERM on an attached `podman run` client kills only the client.** The container is conmon-detached: it survives, inherits the ssh stdout pipe, ssh never sees EOF, the awaiting parent blocks forever.
+2. **`podman wait` has NO `-t` flag in podman 5.x** (docker-ism) → every exec silently dies with rc 125. Bound the waiter with coreutils `timeout -k 5 <T> podman wait <ctr>` — killing a waiter is harmless.
+3. **`podman rm -f` is the only true container kill** (escalates SIGTERM → SIGKILL on the container itself).
+
+## Hang-proof exec protocol (single ssh script)
+```bash
+cname=job-<unique>
+podman run -d --name "$cname" --network none --memory 4g --cpus 2 <image> python /w/entry >/dev/null
+timeout -k 5 "$T" podman wait "$cname" >/dev/null 2>&1; rc=$?
+code=$(podman inspect -f '{{.State.ExitCode}}' "$cname" 2>/dev/null || echo -1)
+podman logs "$cname" 2>/dev/null; echo "EXIT:${rc}:${code}"
+podman rm -f "$cname" >/dev/null 2>&1
+```
+- `rc=0, code=0` → success; `rc=124` → timed out (container reaped); harvest output via `podman logs`.
+- Wrap the local ssh spawn in a **hard timeout of T + 180 s** (kills transport half-hangs too).
+- Unique `--name` per submission; the trailing `rm -f` is also the leak reaper.
+
+## Recovery of a live stall
+1. `ssh host podman rm -f <ctr>` (warning "resorting to SIGKILL" confirms the theory).
+2. Kill the hung local ssh + supervisor; queue/ledger resume (append-only ledger + takeover by heartbeat-pid liveness) resumes mid-eval; a hung step re-graded under the fixed protocol records a clean FAIL exit 124.
+
+## Test to keep (regression)
+`execPython("while True: time.sleep(1)", timeout=20)` → expect `exitCode 124`, completion < 180 s, then `podman ps --format {{.Names}} | grep -c <prefix>` == 0.