live: heartbeat sync 2026-10-30T16:20:49-07:00 (Refs #11)

agent/managed-skills/radiance-tap-run-observability/SKILL.md

@@ -0,0 +1,63 @@
+---
+name: radiance-tap-run-observability
+description: "Judge a live radiance calibration-tap run: which pin actually arms the tap, why a quiet engine log is not a hang, the watchdog that does not throw away accumulated grams, the seal's side effects, and the container-vs-port gate before pushing."
+---
+
+# Judging a live radiance calibration-tap run
+
+Use when a tap (layer Hessians for `rad-convert`'s `gptq:calib=`) is running for hours on a drained
+endpoint and you must tell progress from a stall without destroying the accumulation.
+
+## Which image actually arms the tap
+
+- **Verify `:latest` by digest, not by tag or by a local label.** `podman pull` then
+  `podman inspect … --format '{{.Labels}} {{.Created}}'`: a working box can hold a *stale* local
+  `:latest` whose label reads `version=dev/revision=unknown` while the registry's `latest` is a real
+  release (on 2026-10-30 that was v1.14.0, revision `4a61d7296bc5`). Compare the two before blaming
+  a version.
+- The arming line is the only ground truth that the tap is in the program:
+  `I[0] recorded passes: off, the program accumulates calibration Grams (gram_accum)`.
+  Its absence means no tap, however healthy `/health`, `/stats` and the pusher look — that is how a
+  17-hour run on a production pin wrote zero bytes while everything reported fine.
+- Reading source proves nothing about a build: the tap code was byte-identical in the version that
+  arms and the one that never writes. Judge the shipped image.
+
+## A quiet log is not a hang
+
+- The engine logs **one line per completed request** (`I req cmpl-… prompt=<n> tok`). At 60–70 tok/s
+  of prefill a 7.5 K-token document takes ~2 min alone, ~4 min at 2-way and ~8 min at 4-way
+  concurrency, and **nothing is logged in between**.
+- Concurrently, `W step: nothing has been logged for 21 s` fires on *step*-level silence. It is a
+  diagnostic, not a verdict: `/health` stays 200, `/stats` keeps reporting `running: N`, and
+  `kv_tokens_used` still steps in batch-sized chunks.
+- So a watchdog keyed on log mtime kills healthy cycles, and every kill discards the accumulation
+  (grams only materialise at the seal). **Key the watchdog on the completed-request count** and give
+  it a threshold far above the worst legitimate request latency (45 min for a 1.5 M-token floor).
+- Cheap liveness triple: `grep -c "req cmpl" log` climbing, `kv_tokens_used` moving, `podman ps`
+  showing the rig container. Throughput is bounded by expert streaming, so raising concurrency
+  lengthens requests without raising total tokens/s.
+
+## The seal and what it does
+
+- One seal is ~43 GB (24,768 `gram.r<rank>.blk.<L>.ffn_down_exps.<e>.bin` files) — size the tap
+  filesystem before starting, and rotate any previous tap out first.
+- The engine **exits by itself when the tap seals**. A client mid-flight then sees
+  `429 rate_limit_error: the server is shutting down`: that is success, not failure.
+- The seal line is `radiance: calibration wrote layer Hessians after <n> tokens to /data/calib`.
+  Since the tap materialises all at once, "grams > 1000 files" is a valid post-seal trigger for a
+  chained imatrix → convert pipeline.
+
+## Gate on the container, not the port
+
+After a script stops the production service, the dying process still answers `:7770` for seconds, so
+a health check can pass against the wrong thing. Require `podman ps` to name the rig container
+**and** health to pass. A pusher that trusts the port alone pushes into a dead rig.
+
+## Cards must be exclusive
+
+An orphaned engine keeps `/dev/kfd`, and the next container boot dies with
+`E startup stopped: device error` — which looks like a bad argument and is really a busy device.
+Kill by pid from `sudo fuser /dev/kfd`, never `pkill -f`; verify `fuser` is empty before the next
+boot. A genuinely wedged driver is a different failure: the *production* service then fails to serve
+too, and only a reboot clears it — so prove the card healthy by restarting production and taking one
+real chat turn before blaming your own shape.