live: heartbeat sync 2026-10-30T16:20:49-07:00 (Refs #11)

agent/managed-skills/radiance-calib-window-orchestrator/SKILL.md

@@ -0,0 +1,68 @@
+---
+name: radiance-calib-window-orchestrator
+description: "Run the fleet's radiance calibration window through the repo orchestrator (scripts/qwen38-calib-window.sh): its gates, the arming proof, the seal contract, and the traps that make a tap look healthy while writing nothing."
+---
+
+# Run the calibration window through the orchestrator, never a hand-rolled driver
+
+`serverAppConfigs/scripts/qwen38-calib-window.sh` is the only sanctioned driver for a radiance
+calibration tap. Run it from the deploy checkout on the inference host:
+
+```sh
+cd /home/dank/serverAppConfigs
+bash scripts/qwen38-calib-window.sh          # add --dry-run to print the plan only
+```
+
+## What it owns (do not re-implement any of it)
+
+| Gate | Detail |
+|---|---|
+| Inputs | reference recording `kld-ref/kld.json`, the BF16 `.rad`, the corpus (`Q38_WINDOW_CALIB_CORPUS`, default `calib-7168.jsonl` — the re-chunked file the rig's pool can hold) |
+| **Space** | tap dir must have `Q38_CALIB_MIN_FREE_MIB` (default **61440** MiB) free **before a card is touched** |
+| Rig | boots `scripts/run-qwen38-bf16.sh calib`, waits `/health` within `Q38_WINDOW_HEALTH_TIMEOUT` (3600 s), else kills the rig, restores the service, exits non-zero |
+| Marker | the runner writes `calib/.rig-boot`; `calib-push.py` refuses without it, refuses if it names another container, refuses if that container is not running |
+| Bank | the BF16 reference container moves to `~/.AGENTTRASH/<date>-qwen38-bf16-rad/` **only** if both phases exit 0 |
+
+A hand-rolled `podman run` + push loop skips the space gate and the corpus choice — that is how a
+43 GB seal lands on a volume with 35 GB free.
+
+## Prove the tap ARMS before spending an hour
+
+A build can carry every `RADIANCE_CALIB_*` string, seal message and plugin, and still never arm.
+The arming proof is the boot ledger line:
+
+```
+I[0] recorded passes: off, the program accumulates calibration Grams (gram_accum)
+```
+
+No such line ⇒ nothing will be written, however healthy `/health`, `/stats` and the push look
+(measured: 17 h, zero bytes, on a pin that lacked it). Diff the *sources* first if in doubt — the tap
+code can be identical across versions while one shipped build drops it.
+
+## Reading a live tap run
+
+- **A quiet log is not a hang.** The engine logs one line per *completed* request. At 2–4-way
+  concurrency a 7.5 K-token prefill takes 4–8 minutes; `kv_tokens_used` steps in 2,048-token
+  chunks and `W step: nothing has been logged for 21 s` is step-level silence. Count completed
+  requests (`req cmpl`), never log mtime — a log-mtime watchdog kills healthy cycles, and every
+  reboot discards the accumulated tap.
+- **Throughput is capped by expert streaming** (~60–70 tok/s on a disk-tier payload); concurrency
+  lengthens requests instead of raising throughput.
+- **A port that answers is not the rig.** A dying production process answers `:7770` for seconds
+  after the stop. Gate on `podman ps` naming the rig container.
+
+## The seal contract
+
+- One seal writes **~40 GiB** (24,768 `gram.r<rank>.blk.<L>.ffn_down_exps.<E>.bin` files).
+- `RADIANCE_CALIB_TOKENS` is a **pooled floor**: the engine sums Grams over prefill tokens and writes
+  at that count (engine default 65,536; the fleet runner defaults 1,500,000). The vendor's own
+  measurement: 10 M tokens beat 1 M by under 1 %.
+- After writing, **the engine exits by itself**; in-flight clients get
+  `429 rate_limit_error: the server is shutting down` — that is success, not failure.
+
+## Card exclusivity
+
+A leftover engine keeps `/dev/kfd`; the next boot dies `E startup stopped: device error` with the
+reason buried above the startup report. Kill by pid — `for p in $(sudo fuser /dev/kfd); do kill -9 $p; done` —
+and never `pkill -f <name>` (the pattern matches your own command line). Verify the device is empty
+before any boot and after any kill.