live: heartbeat sync 2026-10-30T15:20:04-07:00 (Refs #11)

agent/managed-skills/radiance-w4-campaign/SKILL.md

@@ -1,27 +1,39 @@
 ---
 name: radiance-w4-campaign
-description: "Run a radiance GPTQ-w4 quantization campaign on dankserver: calibration → protected experts → convert → KLD gate → deploy, with the engine-pin, tap-engagement and MNBT laws."
+description: "Run a radiance GPTQ-w4 quantization campaign on dankserver: calibration tap pin and engagement proof, protected experts, convert, KLD gate, deploy — plus the tap-disk and orphan-engine laws."
 ---
 
-Pipeline for quantizing a BF16 Qwen3.8-Flash-Next checkpoint into a servable `qwen4exp` `.rad` on dankserver (frank-primary). Reference run: `serverAppConfigs-docs/LLM/frank-primary-radiance-swift15-w4dn8.md`; scripts in `scripts/` (`run-swift15-bf16.sh`, `calib-push.py`, `derive-protected-experts.py`, `apply-protected-cast.sh`, `convert-swift15-w4dn8.sh`, `kld-gate-swift15.sh`, `fleet-probes-swift15.sh`).
+Pipeline for quantizing a BF16 MoE checkpoint into a servable `.rad` on dankserver (frank-primary). Reference run: `serverAppConfigs-docs/LLM/frank-primary-radiance-swift15-w4dn8.md`; scripts in `scripts/` (`run-swift15-bf16.sh`, `run-<model>-bf16.sh`, `calib-push.py`, `derive-protected-experts.py`, `apply-protected-cast.sh`, `convert-<model>.sh`, `kld-gate-<model>.sh`, `fleet-probes-<model>.sh`).
 
 Sequence:
 1. **Fetch** checkpoint with per-shard LFS sha256 verification (`-f`, byte counts, no token on argv).
 2. **BF16 container**: `rad-convert` no-recipe (pure copy) — serves as the KLD reference engine.
-3. **KLD reference**: record logits on the SERVE pin (1.0.13) at MNBT 32768, `--max-model-len 16384`, corpus ≤16k prompts. Writes atomically at completion (`rows:` line = receipt). Never scp a running bash script (lazy-read law).
-4. **Calibration**: the tap arms on the **1.2.0** pin — see the section below before any hour is spent.
+3. **KLD reference**: record logits on the SERVE pin at the production geometry, `--max-model-len` under the pool, corpus ≤ that window. Writes atomically at completion (`rows:` line = receipt).
+4. **Calibration tap** — see the pin law and the engagement test below.
 5. **Protected experts**: derive from gram energy (share ≥3 % AND ≥3× routing share), splice between `# PROTECTED-BEGIN/END` markers of the recipe. Plugin law: ONE encoding per expert table; only whole bf16 experts legal beside it — no gate_up/down split.
 6. **Final convert** on the SERVE pin (CPU GPTQ, hours; disk gate ≥146 GiB free first).
-7. **KLD gate** on the serve pin: **MNBT ≤ 2048** (w4 streams in place only there; 32768 → `DID NOT FIT ... stored form is not the container's`), pool 40960, fp8 KV matching the production KV dtype. Gate: mean KL ≤ 0.03, top-1 ≥ 95 %.
-8. **Deploy** `./scripts/deploy.sh dankserver --service frank-archon`; verify `/v1/models` id = served-model-name (alias independent of file — harness models.yml untouched), boot ledger, streamed ≥128-token decode probe, fleet probes (vision needs ≥256-token budget + `reasoning_content` fallback).
+7. **KLD gate** on the serve pin at the production MNBT (a w4 does not stream in place outside the shape it was measured at), pool and KV dtype matching production. Gate: mean KL ≤ 0.03, top-1 ≥ 95 %.
+8. **Deploy** `./scripts/deploy.sh dankserver --service frank-archon`; verify `/v1/models` id = served-model-name, boot ledger, streamed ≥128-token decode probe, fleet probes (vision needs ≥512-token budget + `reasoning_content` fallback).
 
-## The calibration tap: pin, arming, engagement gate, output size
+## The tap pin law (measured 2026-10-30, cost: one 17-hour night)
 
-- **Pin: 1.2.0 arms the tap, the production pin does not.** `RADIANCE_CALIB_DIR` + `RADIANCE_CALIB_TOKENS` are the whole interface (there is no `--calib*` flag on any pin). Running a tap on the 1.8.1 serve image produced **zero bytes over 17 h / 7.3 M prompt tokens** at a healthy 118 tok/s — no error, no ledger line, no `/stats` counter. Swift15's runner pins `PIN_120` for the `calib` mode only; a runner that uses `FRANK_ARCHON_IMAGE` for every mode silently calibrates into a void.
-- **Strings prove code, not arming.** `CalibExperts`, `RADGRAM2` and the seal line `calibration wrote layer Hessians after %lld tokens` exist in `architectures/qwen4exp_fp8.so` from 1.2.4 through 1.8.1, and the boot ledger lists RADIANCE_* env **names** (a dump, not support). Neither is evidence the tap is engaged.
-- **Engagement gate, before any long drain.** Boot with a tiny floor (`RADIANCE_CALIB_TOKENS=8192`), push 2–3 corpus documents, and require the seal line plus bytes in the tap dir. The seal fires after ~10 k tokens regardless of prompt size, so the test costs minutes — the absence of this check cost 17 h.
-- **One seal writes ~40 GB** (per-expert gram + Hessian set, sized by expert width and layer count, essentially independent of how few tokens triggered it). Point `RADIANCE_CALIB_DIR` at a volume with ≥60 GiB free (`/mnt/entropy` on dankserver) and gate on free space before boot; landing it on a 37 GiB root filled the filesystem, and every subsequent write came back `was short` / `cannot write`.
-- The tap needs the bf16 container (`calibrate the bf16 container`), tp=2 with `--placement expert_tiered --weights-disk-tier` is fine (swift15 did it), and a sliced-expert layout is refused by name (`calibration runs at one rank; this rank holds a slice of every expert's width`). 1.2.0 lacks `--flash-attn`, `--metrics`, `--expert-nl`, `--busy-prefill-chunk`; it does have every measuring-shape flag (`--kv-cache-dtype`, `--max-model-len`, `--ngram-placement`, `--num-speculative-tokens`, `--override-chat-template`).
-- Engine rates: `prefill_tps` in `/stats` is a **rolling gauge**, never a cumulative counter — derive throughput from `total_requests × corpus mean tokens/doc`, or from `steps_prefill`. A wide calib step (MNBT 8192) turns ~2.5 steps/doc into 1.0 and buys ~1.2× on a disk-tiered bf16 rig, not 4.5×.
+`RADIANCE_CALIB_DIR` + `RADIANCE_CALIB_TOKENS` arms the per-expert tap on the **1.2.x** line only.
 
-Laws: stop frank-archon for every GPU phase (relinked unit: stop, mask fails; plain `systemctl` for a system unit is denied by polkit — use `sudo systemctl`); verify docs against `podman inspect <name> --format {{join .Config.Cmd " "}}` (rootless, as dank) before writing shapes; ff-pull fails on untracked files the merge tracks — trash first.
+- Every plugin from 1.2.4 through 1.8.1 still **contains** the machinery (`CalibExperts`, `RADGRAM2`, `calib_write_host`, the seal line `calibration wrote layer Hessians after %lld tokens`), so `strings` proves nothing about the pin.
+- A runner that boots `FRANK_ARCHON_IMAGE` (1.8.1) for `calib` serves perfectly — `/health` 200, 1,700 documents at 118 tok/s, `.rig-boot` written, no error anywhere — and writes **zero bytes**: the tap never arms.
+- The engine exposes **no tap counter** in `/stats` and prints **no arming line** in the boot ledger, so a dead tap is indistinguishable from a slow one for hours.
+- Default `calib` to the 1.2.0 digest (`stilldeadcode/radiance@sha256:665775ed1d90de6dbb25558e07fc60d20328f1346f96f3c6e00845eb763f01fa` on dankserver as of 2026-10-30); override only by naming `--image` on purpose.
+
+**Engagement test — 3 minutes, run it before any long tap:** boot with `RADIANCE_CALIB_TOKENS=8192`, push 2–3 real corpus documents, then require the seal line *and* bytes in the tap dir. On 1.2.0 that reads `calibration wrote layer Hessians after 10207 tokens to /data/calib`. No seal line in ten minutes = the pin or the arming is wrong; stop.
+
+## The tap's disk and process laws
+
+- **One seal writes ~40 GB** (39.4 GiB measured for an 8,192-token floor): the output is expert-shaped, not token-shaped. Put the tap dir on a volume with ≥60 GiB free and gate it before boot (`Q38_CALIB_MIN_FREE_MIB`). On a 37 GiB root fs it filled `/` to **0 bytes**; the engine's later writes came back `cannot write` / `write was short`, and *other* tooling on the host then failed too.
+- **Removing the container does not kill its engine.** `podman rm -f <rig>` leaves `podman-init -- radiance …` alive holding `/dev/kfd`; production then fails `ar_oneshot_2rank_wht6 … device error` at kernel init, exits **0**, and `Restart=always` loops it (counter reached 100) while `/health` never answers. Diagnose with `sudo fuser -v /dev/kfd`; kill **by pid** from `fuser /dev/kfd` + `/proc/<pid>/comm == radiance`, never `pkill -f` (the pattern matches the caller's own argv). Put that gate in the runner's restore trap.
+- The service stop/restore path needs `sudo systemctl` — polkit denies the unprivileged call intermittently (`Access denied … interactive authentication`), and a failed stop means the rig boots *beside* production.
+
+## Rate measurement
+
+`/stats` `prefill_tps` is a **rolling gauge**, never a cumulative counter: `gauge × uptime` invents rates (it once produced a bogus 4.5×). Derive tok/s from `total_requests` deltas × the corpus mean tokens/document, or from `steps_prefill`.
+
+Laws: stop production for every GPU phase; verify shapes against `podman inspect <name> --format {{join .Config.Cmd " "}}` (rootless, as dank) before writing them; ff-pull fails on untracked files the merge tracks — trash first.