live: heartbeat sync 2026-10-30T16:20:49-07:00 (Refs #11)

agent/managed-skills/radiance-calib-window-serve-gate/SKILL.md

@@ -0,0 +1,85 @@
+---
+name: radiance-calib-window-serve-gate
+description: "Diagnose a radiance calibration window that dies at second zero or refuses its push: the health answer coming from the dying production socket, the listener-identity gate, the preflight trio (space, re-chunked corpus, KEEP_RAD), the 12M floor and the wide-step knob."
+---
+
+# radiance calibration window — the serve gate and its preflight
+
+Friction-earned 2026-10-30 on `dank/serverAppConfigs` (`scripts/qwen38-calib-window.sh`,
+`scripts/run-qwen38-bf16.sh`). A 12 M-token window died at second zero; the cause was the gate,
+not the engine.
+
+## The signature, and what it really means
+
+```
+phase=calib health ok waited=0s
+the rig container qwen38-bf16 is not running        <- pusher refuses
+phase=calib push end rc=2  ... FAIL: serve rc=1 push rc=2
+```
+
+`waited=0s` is the tell. The runner stops `frank-archon` **first**; the dying production process keeps
+its listener for a moment, while the freshly created rig container is already `State.Running=true` but
+has not bound the port yet. So `podman inspect` (container running) **and** `/health` 200 (production's
+socket) both hold for the wrong process.
+
+**The gate must prove the listener belongs to the rig:**
+
+```bash
+lpid=$(ss -H -ltnp 'sport = :7770' | grep -o 'pid=[0-9]*' | head -1 | cut -d= -f2)
+[ -n "$lpid" ] || return 1
+podman top "$RIG" pid | grep -qw "$lpid"
+```
+
+Landed as PR #90 with two red-first tests (foreign listener ⇒ no push, window fails; in-container
+listener ⇒ proceeds and banks). If you see `waited=0s` on a checkout older than that PR, the window
+there is pre-fix: do not run a long window on it.
+
+## Preflight trio — the window refuses correctly, so do not "fix" it
+
+1. **Tap dir ≥ 60 GiB free.** One seal writes **43 GB in 24,768 files**
+   (`gram.r<rank>.blk.<l>.ffn_down_exps.<e>.bin`). Point it at a 35 GiB root and the seal fills the
+   filesystem to zero. `Q38_CALIB_MIN_FREE_MIB` overrides — for *tests only* (fixtures on `/tmp` tmpfs).
+2. **The re-chunked corpus.** `Q38_WINDOW_CALIB_CORPUS` defaults to `calib-7168.jsonl`; pushing
+   `calib.jsonl` instead runs un-chunked documents through a window sized for chunks.
+3. **`Q38_WINDOW_KEEP_RAD=1` when two windows run back to back.** The window banks the BF16 reference
+   container into `~/.AGENTTRASH/` on success; without this, stage 2 has no reference to tap.
+
+## Floor and the "add more tokens later" trap
+
+`RADIANCE_CALIB_TOKENS` is the campaign's number, not the vendor's: the reference container this arm
+competes with was calibrated on 10 M tokens, so the fleet floor is **12 M**. Engine default is 65,536.
+
+**A sealed tap cannot be extended.** Hessians are VRAM accumulators; the seal writes and drops them.
+So "run 1 M to verify, then add 11 M" is two windows — 1 M then a fresh 12 M — not an append. Budget
+accordingly, and rotate the verification tap to `.AGENTTRASH/`.
+
+One round of the re-chunked corpus is **9.31 M tokens**, so a 12 M floor seals in round 1 plus part
+of round 2; a flat round is normal (the floor is pooled; `round_verdict` says "continue").
+
+## The wide-step knob (the tap's real cost)
+
+`Q38_CALIB_MNBT` / `Q38_CALIB_SEQS` set the calibration step and slots; everything else inherits
+production. The tap is **read-bound**: a 7,168-token document cut into four 2,048-token steps
+re-streams each layer's routed expert union four times, so a wider step is worth up to ~4× (the
+runner's own header says this).
+
+Wide steps are refused by the arch, not the budget, and the refusal scales with the window:
+
+```
+QSA score plane = rows x (--max-model-len / 4) x 4 bytes
+16,384 rows @ 200,000 window -> 3,125 MiB  -> REFUSED at boot
+16,384 rows @  32,768 window ->   512 MiB  -> legal
+```
+
+The calibration corpus is chunked to 7,168 tokens, so **32,768 is the window to calibrate at** —
+copying production's 200,000 window is what disqualifies the wide step.
+
+## Judging a run
+
+- **A quiet log is not a hang.** The engine logs one line per *completed* request; at 2-way a 7.5 K
+  prefill takes ~4 min. Watch `kv_tokens_used` (steps of the chunk size) and the `req cmpl` count.
+  A watchdog on log mtime kills healthy cycles and throws away accumulated tokens.
+- **Sealing is success.** The engine exits by itself; a mid-flight client gets
+  `429 rate_limit_error: the server is shutting down`.
+- **Kill by pid** (`sudo fuser /dev/kfd`), never `pkill -f`: an orphaned engine keeps the device and
+  the next boot dies in ~20 s with `E startup stopped: device error`, which reads like a bad shape.