From 750854131084e4f56be31b37727bd2475b0ea122 Mon Sep 17 00:00:00 2001 From: Frank Date: Fri, 30 Oct 2026 11:11:49 -0700 Subject: [PATCH] live: heartbeat sync 2026-10-30T11:11:49-07:00 (Refs #11) --- .../radiance-w4-campaign/SKILL.md | 31 +++++++++++++------ .../state.json | 2 +- 2 files changed, 23 insertions(+), 10 deletions(-) diff --git a/agent/managed-skills/radiance-w4-campaign/SKILL.md b/agent/managed-skills/radiance-w4-campaign/SKILL.md index 882c071..57583dd 100644 --- a/agent/managed-skills/radiance-w4-campaign/SKILL.md +++ b/agent/managed-skills/radiance-w4-campaign/SKILL.md @@ -3,16 +3,29 @@ name: radiance-w4-campaign description: "Run a radiance GPTQ-w4 quantization campaign on dankserver: calibration → protected experts → convert → KLD gate → deploy, with the engine-pin and MNBT laws." --- -Pipeline for quantizing a BF16 Qwen3.8-Flash-Next checkpoint into a servable `qwen4exp` `.rad` on dankserver (frank-primary). Reference run: `serverAppConfigs-docs/LLM/frank-primary-radiance-swift15-w4dn8.md`; scripts in `scripts/` (`run-swift15-bf16.sh`, `calib-push.py`, `derive-protected-experts.py`, `apply-protected-cast.sh`, `convert-swift15-w4dn8.sh`, `kld-gate-swift15.sh`, `fleet-probes-swift15.sh`). +Pipeline for quantizing a BF16 Qwen3.8-Flash-Next checkpoint into a servable `qwen4exp` `.rad` on dankserver (frank-primary). Reference run: `serverAppConfigs-docs/LLM/Quantization Production.md` + `scripts/` (`calib-push.py`, `qwen38-calib-window.sh`, `run-qwen38-bf16.sh`, `derive-protected-experts.py`, `apply-protected-cast.sh`, the convert script, the KLD gate, the fleet probes). Sequence: 1. **Fetch** checkpoint with per-shard LFS sha256 verification (`-f`, byte counts, no token on argv). -2. **BF16 container**: `rad-convert` no-recipe (pure copy) — serves as the KLD reference engine. -3. **KLD reference**: record logits on the SERVE pin (1.0.13) at MNBT 32768, `--max-model-len 16384`, corpus ≤16k prompts. Writes atomically at completion (`rows:` line = receipt). Never scp a running bash script (lazy-read law). -4. **Calibration**: engine **1.2.0** (only pin with `RADIANCE_CALIB_TOKENS`); push ~10M tokens of the model's own distribution; per-expert down-proj Grams land as RADGRAM2 files (`gram.r.blk.L.ffn_down_exps..bin`). Grams are engine-invariant across pins — the version boundary is by design. -5. **Protected experts**: derive from gram energy (share ≥3 % AND ≥3× routing share), splice between `# PROTECTED-BEGIN/END` markers of the recipe. Plugin law: ONE encoding per expert table; only whole bf16 experts legal beside it — no gate_up/down split. -6. **Final convert** on the SERVE pin (CPU GPTQ, hours; disk gate ≥146 GiB free first). -7. **KLD gate** on the serve pin: **MNBT ≤ 2048** (w4 streams in place only there; 32768 → `DID NOT FIT ... stored form is not the container's`), pool 40960, fp8 KV matching the production KV dtype. Gate: mean KL ≤ 0.03, top-1 ≥ 95 %. -8. **Deploy** `./scripts/deploy.sh dankserver --service frank-primary`; verify `/v1/models` id = served-model-name (alias independent of file — harness models.yml untouched), boot ledger, streamed ≥128-token decode probe, fleet probes (vision needs ≥256-token budget + `reasoning_content` fallback). +2. **BF16 container**: `rad-convert` no-recipe (pure copy) — the KLD reference engine and the calibration rig. +3. **KLD reference**: record logits on the serving shape, corpus ≤ the per-slot window. Writes atomically at completion (`rows:` line = receipt). +4. **Calibration**: see the pin law below. Push the corpus through prefill; per-expert down-proj Grams land as `RADGRAM2` files (`gram.r...bin`). +5. **Protected experts**: derive from gram energy (share ≥3 % AND ≥3× routing share), splice between the recipe's `# PROTECTED-BEGIN/END` markers. One encoding per expert table; only whole bf16 experts legal beside it — no gate_up/down split. +6. **Final convert** on the serving pin (CPU GPTQ, hours; disk gate ≥146 GiB free first). +7. **KLD gate**: MNBT ≤ 2048 when a w4 container streams in place at 32768 (`DID NOT FIT … stored form is not the container's`), pool 40960, KV dtype matching production. Gate: mean KL ≤ 0.03, top-1 ≥ 95 %. +8. **Deploy** `./scripts/deploy.sh dankserver --service frank-archon`; verify `/v1/models` id, boot ledger, a streamed ≥128-token decode probe, and fleet probes (vision needs ≥512 budget on a thinking model + `reasoning_content` fallback). -Laws: stop frank-primary for every GPU phase (relinked unit: stop, mask fails); verify docs against `podman inspect --format {{join .Config.Cmd " "}}` (rootless, as dank) before writing shapes; ff-pull fails on untracked files the merge tracks — trash first. +## The calibration pin law, and how a tap dies silently (measured 2026-10-30) + +- **`calib` MUST boot the 1.2.0 pin (digest `sha256:665775ed…`), never the production pin.** Setting `RADIANCE_CALIB_DIR` + `RADIANCE_CALIB_TOKENS` on 1.8.1 looks accepted and writes **nothing**: 17 h and 7.3 M prompt tokens produced zero bytes, with `/health` green, 118 tok/s and no error line. The tap code (`CalibExperts`, `RADGRAM2`, the seal line `calibration wrote layer Hessians after %lld tokens`) is present in every plugin from 1.2.4 through 1.8.1 — **string presence is not the arm**. Only the 1.2.0 plugin arms on the env pair, which is why swift15's runner pins `IMAGE` per mode (`calib` → `PIN_120`) and the qwen38 runner's single `FRANK_ARCHON_IMAGE` port was a silent regression. +- **The boot ledger's `ENVIRONMENT -- the RADIANCE_* variables set, by name` block is a dump, not acceptance.** It lists names the engine may ignore. +- **`/stats` has no calibration counter** and the tap dir stays empty until the floor is reached, so a dead tap is indistinguishable from a slow one for hours. Therefore every tap run needs an **engagement gate**: push 2–3 documents, then refuse to continue unless the tap dir holds bytes. Put it in the runner, not in an operator's memory. +- Calibration at `--tp 2` is legal (`expert_tiered` + `--weights-disk-tier` streams whole experts); the plugin refuses only when a rank would hold a *slice* of every expert ("calibration runs at one rank … its Hessians would be of the slice"). +- The reported gain from a wide calib step is small: MNBT 2,048 → 8,192 cut steps/doc 2.5 → 1.0 and moved **118 tok/s vs ~94, ≈1.2×**, because the tap's cost is bytes read from the disk tier, not steps. Size the token floor from a counter measurement (`total_requests` deltas × corpus mean tokens/doc); **never** from `prefill_tps`, which is a rolling gauge (it manufactured a false 4.5× / 320 tok/s figure). + +## Ops laws that bite + +- **`systemctl` on the system units needs `sudo`** in scripts and over ssh. Bare `systemctl stop frank-archon` passes one day and returns `Access denied … requires interactive authentication` the next, and a runner that ignores the failure boots the rig *beside* live production. Check `systemctl is-active` after every stop/start and fail closed. +- Stopping the unit leaves it `failed` (the engine exits non-zero on SIGTERM); `systemctl reset-failed` after the stop keeps a deliberate drain from reading as a fault. +- A probe container started with `--rm` deletes its own log on exit; capture `podman logs` before cleanup, and never redirect `podman run -d` to a file expecting the engine's output — that file only receives the container id. +- Stop the production service for every GPU phase (relinked unit: stop, mask fails); verify shapes against `podman inspect --format {{join .Config.Cmd " "}}` (rootless, as dank) before writing them down. diff --git a/agent/memories/sharpshooter/wt-20261027-142335-6d1fc1f-1hpiehxnq73s0/state.json b/agent/memories/sharpshooter/wt-20261027-142335-6d1fc1f-1hpiehxnq73s0/state.json index 0ae2166..2bc8c11 100644 --- a/agent/memories/sharpshooter/wt-20261027-142335-6d1fc1f-1hpiehxnq73s0/state.json +++ b/agent/memories/sharpshooter/wt-20261027-142335-6d1fc1f-1hpiehxnq73s0/state.json @@ -1,6 +1,6 @@ { "v": 1, - "lastConsolidatedAt": 1793380074621, + "lastConsolidatedAt": 1793383734650, "lastResult": { "at": 1793315963987, "sessions": 1,