command-center/agent/managed-skills/radiance-calib-tap-produce-latest/SKILL.md

name, description

name	description
radiance-calib-tap-produce-latest	Produce the radiance calibration tap (layer Hessians for GPTQ calib=) on released radiance: which image actually arms it, the only shape that fits 2x R9700, the floor, the seal contract, and the two silent traps. Use when a w4-MoE quant needs a tap, not when merely verifying a build writes.

Friction-earned 2026-10-30 on dankserver (2x R9700 32 GiB, qwen3.8-flash-next bf16 container, campaign /home/dank/campaign/qwen38-w5nl64).

The image decides whether the tap exists

RADIANCE_CALIB_DIR + RADIANCE_CALIB_TOKENS arm the tap in the source (mc.calib = calib_dir() != nullptr, identical in v1.8.1 and v1.14.0 trees), but a boot either prints

I[0] recorded passes: off, the program accumulates calibration Grams (gram_accum)

or it does not — and if it does not, the tap writes zero bytes for the whole run while /health, /stats and the pusher all look healthy. Measured: production pin 1.8.1 = 17 hours, 0 bytes; :latest (v1.14.0, revision 4a61d7296bc5, digest 69add58f7de1…) on the same argv/env = sealed in minutes.

Procedure: boot the candidate image with the calib env and grep its log for gram_accum before pushing anything. Absence = wrong build; do not spend a window discovering it.

The vendor calibration shape does not fit two 32 GiB cards

docs/MOE-W4.md prescribes --max-num-batched-tokens 65536 --max-num-seqs 64 (~930 tok/s). On 2×32 GiB R9700s (27.95 GiB claimable) it dies in the VRAM resolver — arena alone 12.83 GiB; variants came up short by 3.19 GiB, 279 MiB, 59 MiB, 86 MiB, then a starved cache. Wider steps are refused by the arch before memory even matters:

radiance: qwen4exp: the QSA score plane would be 3125 MiB -- 16384 rows (--max-num-batched-tokens 16384)
          by --max-model-len 200000 / compress ratio 4. Lower either.

What boots and seals: 4 slots, --max-num-batched-tokens 2048 (the quantised file tier's ceiling), --kv-cache-dtype fp8, --weights-disk-tier, --placement expert_tiered, --ngram-placement disk, --max-model-len 200000 (pool 199,948 tokens: kv cache 1.50 GiB … 1.0 full-length). Prefill lands at 50–70 tok/s; the tap's cost is expert re-streaming per step, not the engine.

Concurrency: 4 stalls, 2 survives

With --parallel 4 the pusher sends four ~7.8 K-token prefills; two complete (7,335 and 8,054 tokens), then a step never returns: log frozen, W step: nothing has been logged for 21 s repeating, all threads in futex/kfd_wait_on_events, 0 MiB/s of reads, /health still 200 and /stats still running: 4 with the pool 98 % idle. Reproduced on both a 32,768 and a 200,000 window, so it is not the pool. Production restarts fine afterwards (chat 200 in 0.49 s) — the card is not wedged. Treat a frozen engine log with live requests as a hang, not a slow rig, and drive the pusher at --parallel 2.

Floor and seal contract

• RADIANCE_CALIB_TOKENS=1500000 is enough: the vendor's own measurement is that a pooled Gram from 10 M tokens beats one from 1 M by under 1 %; 6 M cost 29 h on this rig.
• One seal writes ~40 GiB — 24,768 files gram.r<rank>.blk.<l>.ffn_down_exps.<e>.bin, 42.9 GB. Give the tap dir ≥60 GiB; a 37 GiB filesystem filled to zero and every later write came back short.
• The engine exits by itself when the tap seals, so a mid-flight client gets 429 rate_limit_error: the server is shutting down. That is success.
• Grams are written only at the seal: a restart before it loses the accumulation but no artifact, so a self-healing driver (kill rig, reboot, re-push) is safe.

Two traps that read as successes

• A port that answers is not your rig. After the runner stops production, the dying process still answers :7770 for seconds; a health-only gate "passes" at 10 s on a container that is not up, and the pusher then refuses on container name (or worse, pushes into production). Gate on podman ps naming the container, then health.
• A container-based GPU tool must own the cards. An orphaned engine keeps /dev/kfd and the next boot dies with E startup stopped: device error (also produced by kill -9-ing an engine: let the container teardown run, or kill by exact pid from sudo fuser /dev/kfd — never pkill -f, which also matches your own command line).

Downstream

The tap is consumed by rad-convert via gptq:calib=<dir>; the file contract (RADGRAM2, gram.r<rank>.<stem>.bin) is version-stable, so a latest-written tap converts on latest. scripts/grams-to-imatrix.py <calib> -o imatrix.gguf turns it into an imatrix. Arm output size is ~the served container's size (114 G here) — check free space before converting, not after.