32 lines
4.7 KiB
Markdown
32 lines
4.7 KiB
Markdown
---
|
||
name: radiance-w4-campaign
|
||
description: "Run a radiance GPTQ-w4 quantization campaign on dankserver: calibration → protected experts → convert → KLD gate → deploy, with the engine-pin and MNBT laws."
|
||
---
|
||
|
||
Pipeline for quantizing a BF16 Qwen3.8-Flash-Next checkpoint into a servable `qwen4exp` `.rad` on dankserver (frank-primary). Reference run: `serverAppConfigs-docs/LLM/Quantization Production.md` + `scripts/` (`calib-push.py`, `qwen38-calib-window.sh`, `run-qwen38-bf16.sh`, `derive-protected-experts.py`, `apply-protected-cast.sh`, the convert script, the KLD gate, the fleet probes).
|
||
|
||
Sequence:
|
||
1. **Fetch** checkpoint with per-shard LFS sha256 verification (`-f`, byte counts, no token on argv).
|
||
2. **BF16 container**: `rad-convert` no-recipe (pure copy) — the KLD reference engine and the calibration rig.
|
||
3. **KLD reference**: record logits on the serving shape, corpus ≤ the per-slot window. Writes atomically at completion (`rows:` line = receipt).
|
||
4. **Calibration**: see the pin law below. Push the corpus through prefill; per-expert down-proj Grams land as `RADGRAM2` files (`gram.r<rank>.<tensor>.<e>.bin`).
|
||
5. **Protected experts**: derive from gram energy (share ≥3 % AND ≥3× routing share), splice between the recipe's `# PROTECTED-BEGIN/END` markers. One encoding per expert table; only whole bf16 experts legal beside it — no gate_up/down split.
|
||
6. **Final convert** on the serving pin (CPU GPTQ, hours; disk gate ≥146 GiB free first).
|
||
7. **KLD gate**: MNBT ≤ 2048 when a w4 container streams in place at 32768 (`DID NOT FIT … stored form is not the container's`), pool 40960, KV dtype matching production. Gate: mean KL ≤ 0.03, top-1 ≥ 95 %.
|
||
8. **Deploy** `./scripts/deploy.sh dankserver --service frank-archon`; verify `/v1/models` id, boot ledger, a streamed ≥128-token decode probe, and fleet probes (vision needs ≥512 budget on a thinking model + `reasoning_content` fallback).
|
||
|
||
## The calibration pin law, and how a tap dies silently (measured 2026-10-30)
|
||
|
||
- **`calib` MUST boot the 1.2.0 pin (digest `sha256:665775ed…`), never the production pin.** Setting `RADIANCE_CALIB_DIR` + `RADIANCE_CALIB_TOKENS` on 1.8.1 looks accepted and writes **nothing**: 17 h and 7.3 M prompt tokens produced zero bytes, with `/health` green, 118 tok/s and no error line. The tap code (`CalibExperts`, `RADGRAM2`, the seal line `calibration wrote layer Hessians after %lld tokens`) is present in every plugin from 1.2.4 through 1.8.1 — **string presence is not the arm**. Only the 1.2.0 plugin arms on the env pair, which is why swift15's runner pins `IMAGE` per mode (`calib` → `PIN_120`) and the qwen38 runner's single `FRANK_ARCHON_IMAGE` port was a silent regression.
|
||
- **The boot ledger's `ENVIRONMENT -- the RADIANCE_* variables set, by name` block is a dump, not acceptance.** It lists names the engine may ignore.
|
||
- **`/stats` has no calibration counter** and the tap dir stays empty until the floor is reached, so a dead tap is indistinguishable from a slow one for hours. Therefore every tap run needs an **engagement gate**: push 2–3 documents, then refuse to continue unless the tap dir holds bytes. Put it in the runner, not in an operator's memory.
|
||
- Calibration at `--tp 2` is legal (`expert_tiered` + `--weights-disk-tier` streams whole experts); the plugin refuses only when a rank would hold a *slice* of every expert ("calibration runs at one rank … its Hessians would be of the slice").
|
||
- The reported gain from a wide calib step is small: MNBT 2,048 → 8,192 cut steps/doc 2.5 → 1.0 and moved **118 tok/s vs ~94, ≈1.2×**, because the tap's cost is bytes read from the disk tier, not steps. Size the token floor from a counter measurement (`total_requests` deltas × corpus mean tokens/doc); **never** from `prefill_tps`, which is a rolling gauge (it manufactured a false 4.5× / 320 tok/s figure).
|
||
|
||
## Ops laws that bite
|
||
|
||
- **`systemctl` on the system units needs `sudo`** in scripts and over ssh. Bare `systemctl stop frank-archon` passes one day and returns `Access denied … requires interactive authentication` the next, and a runner that ignores the failure boots the rig *beside* live production. Check `systemctl is-active` after every stop/start and fail closed.
|
||
- Stopping the unit leaves it `failed` (the engine exits non-zero on SIGTERM); `systemctl reset-failed` after the stop keeps a deliberate drain from reading as a fault.
|
||
- A probe container started with `--rm` deletes its own log on exit; capture `podman logs` before cleanup, and never redirect `podman run -d` to a file expecting the engine's output — that file only receives the container id.
|
||
- Stop the production service for every GPU phase (relinked unit: stop, mask fails); verify shapes against `podman inspect <name> --format {{join .Config.Cmd " "}}` (rootless, as dank) before writing them down.
|