live: heartbeat sync 2026-10-30T15:20:04-07:00 (Refs #11)

agent/managed-skills/radiance-tap-arm-and-seal-verify/SKILL.md

@@ -0,0 +1,81 @@
+---
+name: radiance-tap-arm-and-seal-verify
+description: "Prove quickly whether a radiance build writes calibration Grams (arming line, 8k-floor seal, engine exits on seal, 43 GB seal, GPU-exclusivity preflight, pgrep/health gate traps)."
+---
+
+# Verify a radiance calibration tap in 15 minutes
+
+Prove whether a radiance build writes calibration Grams (`RADIANCE_CALIB_DIR`) before
+committing hours of drained serving to it. The whole answer is two log lines and a
+directory size.
+
+## The two lines that decide it
+
+1. **Arming** — on the boot ledger, rank 0:
+   `recorded passes: off, the program accumulates calibration Grams (gram_accum)`
+   Absent = the tap is not in the program: nothing will ever be written, and
+   `/health`, `/stats` and a healthy push prove nothing. A 17-hour run on the
+   production pin (radiance 1.8.1) wrote **zero bytes** with no arming line.
+2. **Seal** — when the pooled token floor is reached:
+   `radiance: calibration wrote layer Hessians after N tokens to /data/calib`
+   printed once per rank. Then the directory holds `gram.r<rank>.<block>.<stem>.bin`
+   files (~24,768 files, ~43 GB). Check `du -sb`; a plausible-looking empty dir is
+   the failure mode.
+
+## The seal ends the server
+
+The engine **shuts itself down** once it seals. In-flight requests then receive
+`429 rate_limit_error: the server is shutting down`. That is SUCCESS, not an error:
+wait for the seal line, and treat a post-seal 429 as done. Do not retry into it.
+
+## Minimum proof run
+
+- Boot the candidate image with `RADIANCE_CALIB_DIR=/data/calib` and
+  `RADIANCE_CALIB_TOKENS=8192` (a small floor, not the campaign floor).
+- Push three or four ~7k-token documents concurrently; at ~64 tok/s aggregate the
+  seal lands in ~2 minutes.
+- One seal costs ~43 GB: point the tap at a filesystem with ≥60 GB free and check
+  before booting.
+- Consumer side is version-stable: `rad-convert` takes `gptq:calib=<dir>` and reads
+  `RADGRAM2` grams written by any pin, so the producer and the converter may be
+  different builds.
+
+## Shape: the vendor recipe does not fit two 32 GiB cards
+
+Vendor `docs/MOE-W4.md` prescribes `--max-num-batched-tokens 65536 --max-num-seqs 64`
+(~930 tok/s). On two 32 GiB R9700s the activation arena alone is 12.83 GiB of the
+27.95 GiB claimable and every variant stops in the VRAM resolver.
+The qwen4exp arch prices a **QSA score plane** of `rows × window / compress`:
+`the QSA score plane would be 3125 MiB -- 16384 rows by --max-model-len 200000 / 4.
+Lower either.` → 16,384 rows is refused outright; 8,192 rows costs ~1.56 GiB.
+The shape that fits and seals: production's own — 4 slots, `--max-num-batched-tokens
+2048`, fp8 KV, `expert_tiered` + weights disk tier, n-gram table on disk.
+
+## Exclusivity is not optional
+
+A leftover calibration container silently owns the cards and poisons every later
+boot with `E startup stopped: device error`. Before any boot:
+
+```
+sudo fuser /dev/kfd            # must print nothing
+podman ps --format '{{.Names}}'
+```
+
+After stopping a rig, `podman rm -f <name>` **and** kill any `radiance` pid still
+holding `/dev/kfd` (`kill -9` by the pid from `fuser`, never `pkill -f`). Then verify
+`fuser` is empty before booting the next shape — a 21-second `device error` at boot
+is a dirty device, not a shape refusal.
+
+Two gate traps, both measured:
+- `curl /health` right after stopping the service can be answered by the **dying**
+  production process, so "up at 10 s" is a lie. Gate on the container name in
+  `podman ps` **and** the marker file the rig writes, not on the port alone.
+- `pgrep -f "script.sh"` matches the polling shell's own command line; a
+  `while pgrep -f` wait loop then never exits. Gate on a file, a container name, or
+  the pid you captured at launch.
+
+## Report
+
+Write the outcome into the page/README that owns the campaign, not a diary: which
+build arms the tap, the fitting shape, the floor and its warrant, and the seal size.
+Numbers from a shape that is not running now are dead weight — keep the law.