59 lines
3.7 KiB
Markdown
59 lines
3.7 KiB
Markdown
---
|
||
name: radiance-calib-tap-sizing
|
||
description: "Decide whether a radiance calibration tap is needed, how large a token floor it wants, and which step/window shape makes it fast — before booking a multi-hour card window."
|
||
---
|
||
|
||
# Sizing a radiance calibration tap
|
||
|
||
Read this before promising a tap run (hours to days of exclusive cards). It answers three questions:
|
||
*what is the tap for*, *how many tokens*, and *which shape makes it fast*.
|
||
|
||
## 1. The tap is for the Hessian, not the imatrix
|
||
|
||
`tools/rad_convert.cpp` help: `--imatrix F` is *"the expert-popularity profile, the column
|
||
importance"* — a diagonal summary. GPTQ's input is separate: the recipe names `gptq calib=<dir>`,
|
||
and `<dir>` is the tap's gram set: **d × d fp32 second moments (`E[x xᵀ]`) per expert per layer**.
|
||
|
||
- An external/published imatrix **cannot** substitute. It carries no second moments, it is keyed by
|
||
another model's tensor names, and radiance warns that *"a legacy-format imatrix cannot carry
|
||
[per-expert counts] at all — the planner will start cold."*
|
||
- The imatrix you need is a free by-product: `scripts/grams-to-imatrix.py <tap dir> -o imatrix.gguf`.
|
||
- Without a tap the experts can only take the recipe's non-GPTQ rule (`rtn`) — a visibly worse arm.
|
||
So "we only want an imatrix" is not a reason to skip it, and "just fetch one" is not a path.
|
||
|
||
## 2. The shape decides the throughput — window and step are one decision
|
||
|
||
The arch prices a step against the window: **QSA score plane ≈ rows × (window ÷ 4) × 4 bytes**.
|
||
|
||
| window | step (rows) | plane | verdict |
|
||
|---|---|---|---|
|
||
| 200,000 | 16,384 | 3,125 MiB | refused at boot (`the QSA score plane would be … Lower either`) |
|
||
| 32,768 | 16,384 | 512 MiB | fits |
|
||
| 32,768 | 32,768 | 1 GiB | fits |
|
||
|
||
Tap prefill is bound by **expert re-reads per step**, not by tokens as such, so the small window is the
|
||
lever: calibration corpora are re-chunked to ≤7,168-token documents and need nowhere near a 200k
|
||
window, yet copying the production window is the reflex that disqualifies a wide step and pins the run
|
||
to 2,048-token steps (≈70 tok/s on 2× R9700 = ~48 h for 12 M tokens). A 8,192–16,384-token step at a
|
||
32,768 window re-reads experts 4–8× less often per token.
|
||
|
||
Always boot once and read the ledger (`activation arena`, `experts`, `kv cache`) before believing any
|
||
speed estimate: a wider step grows the arena and steals from the experts.
|
||
|
||
## 3. Before you spend the cards
|
||
|
||
- **Arming proof**: the boot must print `recorded passes: off, the program accumulates calibration
|
||
Grams (gram_accum)`. A build that prints nothing writes nothing, however healthy `/health` and
|
||
`/stats` look (a 17-hour run on a shipped pin produced zero files).
|
||
- **The rig must own the socket**: `ss -ltnp 'sport = :7770'` → pid, and that pid must appear in
|
||
`podman top <rig> pid`. After the runner stops production, the dying process answers `/health` for a
|
||
moment while the new container is still loading — "container running + health 200" is not the rig.
|
||
- **Liveness is completions, not log mtime**: the engine logs one line per *completed* request, so a
|
||
4–8 minute prefill looks like silence while `kv_tokens_used` steps in 2,048-token chunks.
|
||
- **Space**: one seal writes ~40 GiB in one burst; the preflight needs ≥60 GiB free in the tap dir.
|
||
- **The floor is a campaign decision, not a vendor default**: the engine default is 65,536 tokens; a
|
||
12 M floor over a ~9.31 M-token corpus seals in round 1 plus part of round 2, and a round that seals
|
||
nothing is normal (the floor is pooled — never treat a flat round as a failure).
|
||
- **Exit means success**: at the seal the engine exits by itself; a mid-flight client gets
|
||
`429 rate_limit_error: the server is shutting down`.
|