86 lines
4.8 KiB
Markdown
86 lines
4.8 KiB
Markdown
---
|
||
name: radiance-calibration-tap-latest
|
||
description: "Capture radiance calibration Grams for GPTQ (RADIANCE_CALIB_DIR) on the R9700 rig: the tap only arms on latest, the fitting shape, the 12M floor rule, and how to read a run that looks hung."
|
||
---
|
||
|
||
Friction-earned 2026-10-30 on `dank/serverAppConfigs` (qwen38-w5nl64 campaign, 2×R9700 TP=2).
|
||
|
||
## The tap arms only on the latest radiance image
|
||
|
||
A 17-hour calib run on the *production* pin (1.8.1) wrote **zero bytes** into `RADIANCE_CALIB_DIR`
|
||
while `/health`, `/stats` and the pusher all reported healthy. The tap's arming code is byte-identical
|
||
in both trees — the shipped 1.8.1 build drops it. The tell is the boot line:
|
||
|
||
```
|
||
I[0] recorded passes: off, the program accumulates calibration Grams (gram_accum)
|
||
```
|
||
|
||
Present on latest (v1.14.0 = `:latest` 2026-10-30), absent on 1.8.1. Verify with
|
||
`podman inspect IMG --format '{{json .Labels}}'` → `org.opencontainers.image.version` /
|
||
`revision`, and pin the **index digest**, not the tag: `:latest` moves under you, and a local stale
|
||
`:latest` can be a 3-week-old `dev` build (check `.Created`).
|
||
|
||
Run the tap from a report container of the *same* model, but the tap needs `--ngram-placement disk`
|
||
for a calibration boot: the n-gram table pinned to RAM (`ram`) OOMs the rig's boot
|
||
(`startup stopped: out of memory`) while kldref/score boots succeed at the same pool.
|
||
|
||
## The fitting shape (vendor recipe does not boot 2×32 GiB)
|
||
|
||
The vendor's calibration line (`--max-num-batched-tokens 65536 --max-num-seqs 64`, ~930 tok/s)
|
||
assumes a bigger box: the activation arena alone is 12.83 GiB of 27.95 GiB claimable and every
|
||
variant stops in the VRAM resolver (short 3.19 GiB → 279 MiB → 59 MiB → 86 MiB → cache starved).
|
||
Wider steps are refused by the arch itself, pre-allocation:
|
||
|
||
```
|
||
qwen4exp: the QSA score plane would be 3125 MiB -- 16384 rows (--max-num-batched-tokens 16384)
|
||
by --max-model-len 200000 / compress ratio 4. Lower either.
|
||
```
|
||
|
||
What boots and seals: 4 slots, `--max-num-batched-tokens 2048` (the file-tier ceiling),
|
||
`--kv-cache-dtype fp8`, `--weights-disk-tier`, `--placement expert_tiered`,
|
||
`--ngram-placement disk`, `--max-model-len 200000`, `--num-speculative-tokens 0`, `--deterministic`.
|
||
Ledger at that shape: `elastic 22.8 GiB — experts first, kv cache 1.50 GiB … 1.0 full-length`
|
||
(pool ≈ 199,948 tokens). On 1.8.1 the same payload's pool was 7,436 tokens — the resolver, not the
|
||
window, is what changed.
|
||
|
||
## Floor
|
||
|
||
`RADIANCE_CALIB_TOKENS` is a **pooled** token floor. Engine default 65,536. Fleet rule (Frank,
|
||
2026-10-30): **12,000,000**, because the served quant this arm competes with was calibrated on a
|
||
10 M-token set. The vendor's "10 M beats 1 M by under 1 %" measures *their* plate, not ours.
|
||
Consequence: a ~9.3 M-token corpus round seals in round 1 plus part of round 2, and a round that
|
||
seals nothing is NORMAL (`round_verdict` → continue, never failure).
|
||
|
||
## Reading a run (the trap that cost two cycles)
|
||
|
||
- Throughput is **streaming-bound**, ~60–70 tok/s whatever the slot count; concurrency only lengthens
|
||
each request (2-way ≈ 4 min per 7.5 K-token prefill, 4-way ≈ 8).
|
||
- The engine logs **one line per completed request**. A log quiet for 8 minutes is therefore normal;
|
||
`W step: nothing has been logged for 21 s` is step-level silence. Liveness = **completed-request
|
||
count**, never log mtime. A watchdog on mtime kills healthy cycles and every false positive throws
|
||
away accumulated tap tokens.
|
||
- One seal writes **~43 GB** (24,768 `gram.r<rank>.blk.<l>.ffn_down_exps.<e>.bin`) and the engine
|
||
then **exits by itself**; a mid-flight client gets `429 rate_limit_error: the server is shutting
|
||
down` — success, not failure.
|
||
- A port that answers is not the rig: after the runner stops production the dying process answers
|
||
`:7770` briefly. Gate on `podman ps` naming the rig container, and on the `.rig-boot` marker.
|
||
|
||
## Orchestration and space
|
||
|
||
Use the repo's orchestrator, never a hand-rolled driver:
|
||
`bash scripts/qwen38-calib-window.sh` (gates: reference present, `.rad` present, tap dir ≥ 60 GiB,
|
||
healthy rig, marker, then push; banks the reference container only on success). Knobs:
|
||
`FRANK_QWEN38_CALIB_TOKENS`, `Q38_WINDOW_CALIB_CORPUS`, `Q38_WINDOW_ROUNDS`,
|
||
`Q38_CALIB_MIN_FREE_MIB`, `Q38_WINDOW_KEEP_RAD`.
|
||
|
||
A hand-rolled driver skips the space preflight; on this host the tap dir sat on `/` with 35 G free
|
||
while one seal writes 43 GB — the orchestrator refuses that, which is the point. Frank's rule: keep
|
||
artifacts on `/`, never `/mnt/entropy`.
|
||
|
||
## Then
|
||
|
||
`gram_files → scripts/grams-to-imatrix.py DIR -o imatrix.gguf` (stdlib-only, skips non-gram names
|
||
like `.rig-boot`) → `scripts/convert-qwen38-w5nl64.sh --plan-only` (space bill, `--plan-only` first)
|
||
→ convert (card-less, CPU BLAS) → `score` each arm + the served payload at the **production shape** →
|
||
`scripts/kld-gate-qwen38.sh` (mean KL ≤ 0.03, p99.9, top-1 ≥ 95 %, every `by_source` slice).
|