live: heartbeat sync 2026-10-30T11:11:49-07:00 (Refs #11)

agent/managed-skills/radiance-w4-campaign/SKILL.md

@@ -3,16 +3,29 @@ name: radiance-w4-campaign
 description: "Run a radiance GPTQ-w4 quantization campaign on dankserver: calibration → protected experts → convert → KLD gate → deploy, with the engine-pin and MNBT laws."
 ---
 
-Pipeline for quantizing a BF16 Qwen3.8-Flash-Next checkpoint into a servable `qwen4exp` `.rad` on dankserver (frank-primary). Reference run: `serverAppConfigs-docs/LLM/frank-primary-radiance-swift15-w4dn8.md`; scripts in `scripts/` (`run-swift15-bf16.sh`, `calib-push.py`, `derive-protected-experts.py`, `apply-protected-cast.sh`, `convert-swift15-w4dn8.sh`, `kld-gate-swift15.sh`, `fleet-probes-swift15.sh`).
+Pipeline for quantizing a BF16 Qwen3.8-Flash-Next checkpoint into a servable `qwen4exp` `.rad` on dankserver (frank-primary). Reference run: `serverAppConfigs-docs/LLM/Quantization Production.md` + `scripts/` (`calib-push.py`, `qwen38-calib-window.sh`, `run-qwen38-bf16.sh`, `derive-protected-experts.py`, `apply-protected-cast.sh`, the convert script, the KLD gate, the fleet probes).
 
 Sequence:
 1. **Fetch** checkpoint with per-shard LFS sha256 verification (`-f`, byte counts, no token on argv).
-2. **BF16 container**: `rad-convert` no-recipe (pure copy) — serves as the KLD reference engine.
-3. **KLD reference**: record logits on the SERVE pin (1.0.13) at MNBT 32768, `--max-model-len 16384`, corpus ≤16k prompts. Writes atomically at completion (`rows:` line = receipt). Never scp a running bash script (lazy-read law).
-4. **Calibration**: engine **1.2.0** (only pin with `RADIANCE_CALIB_TOKENS`); push ~10M tokens of the model's own distribution; per-expert down-proj Grams land as RADGRAM2 files (`gram.r<rank>.blk.L.ffn_down_exps.<e>.bin`). Grams are engine-invariant across pins — the version boundary is by design.
-5. **Protected experts**: derive from gram energy (share ≥3 % AND ≥3× routing share), splice between `# PROTECTED-BEGIN/END` markers of the recipe. Plugin law: ONE encoding per expert table; only whole bf16 experts legal beside it — no gate_up/down split.
-6. **Final convert** on the SERVE pin (CPU GPTQ, hours; disk gate ≥146 GiB free first).
-7. **KLD gate** on the serve pin: **MNBT ≤ 2048** (w4 streams in place only there; 32768 → `DID NOT FIT ... stored form is not the container's`), pool 40960, fp8 KV matching the production KV dtype. Gate: mean KL ≤ 0.03, top-1 ≥ 95 %.
-8. **Deploy** `./scripts/deploy.sh dankserver --service frank-primary`; verify `/v1/models` id = served-model-name (alias independent of file — harness models.yml untouched), boot ledger, streamed ≥128-token decode probe, fleet probes (vision needs ≥256-token budget + `reasoning_content` fallback).
+2. **BF16 container**: `rad-convert` no-recipe (pure copy) — the KLD reference engine and the calibration rig.
+3. **KLD reference**: record logits on the serving shape, corpus ≤ the per-slot window. Writes atomically at completion (`rows:` line = receipt).
+4. **Calibration**: see the pin law below. Push the corpus through prefill; per-expert down-proj Grams land as `RADGRAM2` files (`gram.r<rank>.<tensor>.<e>.bin`).
+5. **Protected experts**: derive from gram energy (share ≥3 % AND ≥3× routing share), splice between the recipe's `# PROTECTED-BEGIN/END` markers. One encoding per expert table; only whole bf16 experts legal beside it — no gate_up/down split.
+6. **Final convert** on the serving pin (CPU GPTQ, hours; disk gate ≥146 GiB free first).
+7. **KLD gate**: MNBT ≤ 2048 when a w4 container streams in place at 32768 (`DID NOT FIT … stored form is not the container's`), pool 40960, KV dtype matching production. Gate: mean KL ≤ 0.03, top-1 ≥ 95 %.
+8. **Deploy** `./scripts/deploy.sh dankserver --service frank-archon`; verify `/v1/models` id, boot ledger, a streamed ≥128-token decode probe, and fleet probes (vision needs ≥512 budget on a thinking model + `reasoning_content` fallback).
 
-Laws: stop frank-primary for every GPU phase (relinked unit: stop, mask fails); verify docs against `podman inspect <name> --format {{join .Config.Cmd " "}}` (rootless, as dank) before writing shapes; ff-pull fails on untracked files the merge tracks — trash first.
+## The calibration pin law, and how a tap dies silently (measured 2026-10-30)
+
+- **`calib` MUST boot the 1.2.0 pin (digest `sha256:665775ed…`), never the production pin.** Setting `RADIANCE_CALIB_DIR` + `RADIANCE_CALIB_TOKENS` on 1.8.1 looks accepted and writes **nothing**: 17 h and 7.3 M prompt tokens produced zero bytes, with `/health` green, 118 tok/s and no error line. The tap code (`CalibExperts`, `RADGRAM2`, the seal line `calibration wrote layer Hessians after %lld tokens`) is present in every plugin from 1.2.4 through 1.8.1 — **string presence is not the arm**. Only the 1.2.0 plugin arms on the env pair, which is why swift15's runner pins `IMAGE` per mode (`calib` → `PIN_120`) and the qwen38 runner's single `FRANK_ARCHON_IMAGE` port was a silent regression.
+- **The boot ledger's `ENVIRONMENT -- the RADIANCE_* variables set, by name` block is a dump, not acceptance.** It lists names the engine may ignore.
+- **`/stats` has no calibration counter** and the tap dir stays empty until the floor is reached, so a dead tap is indistinguishable from a slow one for hours. Therefore every tap run needs an **engagement gate**: push 2–3 documents, then refuse to continue unless the tap dir holds bytes. Put it in the runner, not in an operator's memory.
+- Calibration at `--tp 2` is legal (`expert_tiered` + `--weights-disk-tier` streams whole experts); the plugin refuses only when a rank would hold a *slice* of every expert ("calibration runs at one rank … its Hessians would be of the slice").
+- The reported gain from a wide calib step is small: MNBT 2,048 → 8,192 cut steps/doc 2.5 → 1.0 and moved **118 tok/s vs ~94, ≈1.2×**, because the tap's cost is bytes read from the disk tier, not steps. Size the token floor from a counter measurement (`total_requests` deltas × corpus mean tokens/doc); **never** from `prefill_tps`, which is a rolling gauge (it manufactured a false 4.5× / 320 tok/s figure).
+
+## Ops laws that bite
+
+- **`systemctl` on the system units needs `sudo`** in scripts and over ssh. Bare `systemctl stop frank-archon` passes one day and returns `Access denied … requires interactive authentication` the next, and a runner that ignores the failure boots the rig *beside* live production. Check `systemctl is-active` after every stop/start and fail closed.
+- Stopping the unit leaves it `failed` (the engine exits non-zero on SIGTERM); `systemctl reset-failed` after the stop keeps a deliberate drain from reading as a fault.
+- A probe container started with `--rm` deletes its own log on exit; capture `podman logs` before cleanup, and never redirect `podman run -d` to a file expecting the engine's output — that file only receives the container id.
+- Stop the production service for every GPU phase (relinked unit: stop, mask fails); verify shapes against `podman inspect <name> --format {{join .Config.Cmd " "}}` (rootless, as dank) before writing them down.

agent/memories/sharpshooter/wt-20261027-142335-6d1fc1f-1hpiehxnq73s0/state.json

@@ -1,6 +1,6 @@
 {
 	"v": 1,
-	"lastConsolidatedAt": 1793380074621,
+	"lastConsolidatedAt": 1793383734650,
 	"lastResult": {
 		"at": 1793315963987,
 		"sessions": 1,