50 lines
4.7 KiB
Markdown
50 lines
4.7 KiB
Markdown
---
|
||
name: radiance-calib-tap-produce-latest
|
||
description: "Produce the radiance calibration tap (layer Hessians for GPTQ calib=) on released radiance: which image actually arms it, the only shape that fits 2x R9700, the floor, the seal contract, and the two silent traps. Use when a w4-MoE quant needs a tap, not when merely verifying a build writes."
|
||
---
|
||
|
||
Friction-earned 2026-10-30 on dankserver (2x R9700 32 GiB, qwen3.8-flash-next bf16 container, campaign `/home/dank/campaign/qwen38-w5nl64`).
|
||
|
||
## The image decides whether the tap exists
|
||
|
||
`RADIANCE_CALIB_DIR` + `RADIANCE_CALIB_TOKENS` arm the tap in the **source** (`mc.calib = calib_dir() != nullptr`, identical in v1.8.1 and v1.14.0 trees), but a boot either prints
|
||
|
||
```
|
||
I[0] recorded passes: off, the program accumulates calibration Grams (gram_accum)
|
||
```
|
||
|
||
or it does not — and if it does not, the tap writes **zero bytes for the whole run** while `/health`, `/stats` and the pusher all look healthy. Measured: production pin 1.8.1 = 17 hours, 0 bytes; `:latest` (v1.14.0, revision `4a61d7296bc5`, digest `69add58f7de1…`) on the same argv/env = sealed in minutes.
|
||
|
||
**Procedure:** boot the candidate image with the calib env and grep its log for `gram_accum` **before** pushing anything. Absence = wrong build; do not spend a window discovering it.
|
||
|
||
## The vendor calibration shape does not fit two 32 GiB cards
|
||
|
||
`docs/MOE-W4.md` prescribes `--max-num-batched-tokens 65536 --max-num-seqs 64` (~930 tok/s). On 2×32 GiB R9700s (27.95 GiB claimable) it dies in the VRAM resolver — arena alone 12.83 GiB; variants came up short by 3.19 GiB, 279 MiB, 59 MiB, 86 MiB, then a starved cache. Wider steps are refused by the arch before memory even matters:
|
||
|
||
```
|
||
radiance: qwen4exp: the QSA score plane would be 3125 MiB -- 16384 rows (--max-num-batched-tokens 16384)
|
||
by --max-model-len 200000 / compress ratio 4. Lower either.
|
||
```
|
||
|
||
What boots and seals: **4 slots, `--max-num-batched-tokens 2048`** (the quantised file tier's ceiling), `--kv-cache-dtype fp8`, `--weights-disk-tier`, `--placement expert_tiered`, `--ngram-placement disk`, `--max-model-len 200000` (pool 199,948 tokens: `kv cache 1.50 GiB … 1.0 full-length`). Prefill lands at 50–70 tok/s; the tap's cost is expert re-streaming per step, not the engine.
|
||
|
||
## Concurrency: 4 stalls, 2 survives
|
||
|
||
With `--parallel 4` the pusher sends four ~7.8 K-token prefills; **two complete (7,335 and 8,054 tokens), then a step never returns**: log frozen, `W step: nothing has been logged for 21 s` repeating, all threads in `futex`/`kfd_wait_on_events`, 0 MiB/s of reads, `/health` still 200 and `/stats` still `running: 4` with the pool 98 % idle. Reproduced on both a 32,768 and a 200,000 window, so it is not the pool. Production restarts fine afterwards (`chat 200` in 0.49 s) — the card is not wedged. Treat a frozen engine log with live requests as a hang, not a slow rig, and drive the pusher at `--parallel 2`.
|
||
|
||
## Floor and seal contract
|
||
|
||
- `RADIANCE_CALIB_TOKENS=1500000` is enough: the vendor's own measurement is that a pooled Gram from 10 M tokens beats one from 1 M by under 1 %; 6 M cost 29 h on this rig.
|
||
- One seal writes **~40 GiB** — 24,768 files `gram.r<rank>.blk.<l>.ffn_down_exps.<e>.bin`, 42.9 GB. Give the tap dir ≥60 GiB; a 37 GiB filesystem filled to zero and every later write came back short.
|
||
- The engine **exits by itself when the tap seals**, so a mid-flight client gets `429 rate_limit_error: the server is shutting down`. That is success.
|
||
- Grams are written only at the seal: a restart before it loses the accumulation but no artifact, so a self-healing driver (kill rig, reboot, re-push) is safe.
|
||
|
||
## Two traps that read as successes
|
||
|
||
- **A port that answers is not your rig.** After the runner stops production, the dying process still answers `:7770` for seconds; a health-only gate "passes" at 10 s on a container that is not up, and the pusher then refuses on container name (or worse, pushes into production). Gate on `podman ps` **naming the container**, then health.
|
||
- **A container-based GPU tool must own the cards.** An orphaned engine keeps `/dev/kfd` and the next boot dies with `E startup stopped: device error` (also produced by `kill -9`-ing an engine: let the container teardown run, or kill by exact pid from `sudo fuser /dev/kfd` — never `pkill -f`, which also matches your own command line).
|
||
|
||
## Downstream
|
||
|
||
The tap is consumed by `rad-convert` via `gptq:calib=<dir>`; the file contract (`RADGRAM2`, `gram.r<rank>.<stem>.bin`) is version-stable, so a latest-written tap converts on latest. `scripts/grams-to-imatrix.py <calib> -o imatrix.gguf` turns it into an imatrix. Arm output size is ~the served container's size (114 G here) — check free space before converting, not after.
|