command-center/agent/managed-skills/radiance-w4-campaign/SKILL.md

name, description

name	description
radiance-w4-campaign	Run a radiance GPTQ-w4 quantization campaign on dankserver: calibration → protected experts → convert → KLD gate → deploy, with the engine-pin and MNBT laws.

Pipeline for quantizing a BF16 Qwen3.8-Flash-Next checkpoint into a servable qwen4exp .rad on dankserver (frank-primary). Reference run: serverAppConfigs-docs/LLM/Quantization Production.md + scripts/ (calib-push.py, qwen38-calib-window.sh, run-qwen38-bf16.sh, derive-protected-experts.py, apply-protected-cast.sh, the convert script, the KLD gate, the fleet probes).

Sequence:

1. Fetch checkpoint with per-shard LFS sha256 verification (-f, byte counts, no token on argv).
2. BF16 container: rad-convert no-recipe (pure copy) — the KLD reference engine and the calibration rig.
3. KLD reference: record logits on the serving shape, corpus ≤ the per-slot window. Writes atomically at completion (rows: line = receipt).
4. Calibration: see the pin law below. Push the corpus through prefill; per-expert down-proj Grams land as RADGRAM2 files (gram.r<rank>.<tensor>.<e>.bin).
5. Protected experts: derive from gram energy (share ≥3 % AND ≥3× routing share), splice between the recipe's # PROTECTED-BEGIN/END markers. One encoding per expert table; only whole bf16 experts legal beside it — no gate_up/down split.
6. Final convert on the serving pin (CPU GPTQ, hours; disk gate ≥146 GiB free first).
7. KLD gate: MNBT ≤ 2048 when a w4 container streams in place at 32768 (DID NOT FIT … stored form is not the container's), pool 40960, KV dtype matching production. Gate: mean KL ≤ 0.03, top-1 ≥ 95 %.
8. Deploy ./scripts/deploy.sh dankserver --service frank-archon; verify /v1/models id, boot ledger, a streamed ≥128-token decode probe, and fleet probes (vision needs ≥512 budget on a thinking model + reasoning_content fallback).

The calibration pin law, and how a tap dies silently (measured 2026-10-30)

• calib MUST boot the 1.2.0 pin (digest sha256:665775ed…), never the production pin. Setting RADIANCE_CALIB_DIR + RADIANCE_CALIB_TOKENS on 1.8.1 looks accepted and writes nothing: 17 h and 7.3 M prompt tokens produced zero bytes, with /health green, 118 tok/s and no error line. The tap code (CalibExperts, RADGRAM2, the seal line calibration wrote layer Hessians after %lld tokens) is present in every plugin from 1.2.4 through 1.8.1 — string presence is not the arm. Only the 1.2.0 plugin arms on the env pair, which is why swift15's runner pins IMAGE per mode (calib → PIN_120) and the qwen38 runner's single FRANK_ARCHON_IMAGE port was a silent regression.
• The boot ledger's ENVIRONMENT -- the RADIANCE_* variables set, by name block is a dump, not acceptance. It lists names the engine may ignore.
• /stats has no calibration counter and the tap dir stays empty until the floor is reached, so a dead tap is indistinguishable from a slow one for hours. Therefore every tap run needs an engagement gate: push 2–3 documents, then refuse to continue unless the tap dir holds bytes. Put it in the runner, not in an operator's memory.
• Calibration at --tp 2 is legal (expert_tiered + --weights-disk-tier streams whole experts); the plugin refuses only when a rank would hold a slice of every expert ("calibration runs at one rank … its Hessians would be of the slice").
• The reported gain from a wide calib step is small: MNBT 2,048 → 8,192 cut steps/doc 2.5 → 1.0 and moved 118 tok/s vs ~94, ≈1.2×, because the tap's cost is bytes read from the disk tier, not steps. Size the token floor from a counter measurement (total_requests deltas × corpus mean tokens/doc); never from prefill_tps, which is a rolling gauge (it manufactured a false 4.5× / 320 tok/s figure).

Ops laws that bite

• systemctl on the system units needs sudo in scripts and over ssh. Bare systemctl stop frank-archon passes one day and returns Access denied … requires interactive authentication the next, and a runner that ignores the failure boots the rig beside live production. Check systemctl is-active after every stop/start and fail closed.
• Stopping the unit leaves it failed (the engine exits non-zero on SIGTERM); systemctl reset-failed after the stop keeps a deliberate drain from reading as a fault.
• A probe container started with --rm deletes its own log on exit; capture podman logs before cleanup, and never redirect podman run -d to a file expecting the engine's output — that file only receives the container id.
• Stop the production service for every GPU phase (relinked unit: stop, mask fails); verify shapes against podman inspect <name> --format {{join .Config.Cmd " "}} (rootless, as dank) before writing them down.