command-center/agent/managed-skills/radiance-calibration-tap-latest/SKILL.md

name, description

name	description
radiance-calibration-tap-latest	Capture radiance calibration Grams for GPTQ (RADIANCE_CALIB_DIR) on the R9700 rig: the tap only arms on latest, the fitting shape, the 12M floor rule, and how to read a run that looks hung.

Friction-earned 2026-10-30 on dank/serverAppConfigs (qwen38-w5nl64 campaign, 2×R9700 TP=2).

The tap arms only on the latest radiance image

A 17-hour calib run on the production pin (1.8.1) wrote zero bytes into RADIANCE_CALIB_DIR while /health, /stats and the pusher all reported healthy. The tap's arming code is byte-identical in both trees — the shipped 1.8.1 build drops it. The tell is the boot line:

I[0] recorded passes: off, the program accumulates calibration Grams (gram_accum)

Present on latest (v1.14.0 = :latest 2026-10-30), absent on 1.8.1. Verify with podman inspect IMG --format '{{json .Labels}}' → org.opencontainers.image.version / revision, and pin the index digest, not the tag: :latest moves under you, and a local stale :latest can be a 3-week-old dev build (check .Created).

Run the tap from a report container of the same model, but the tap needs --ngram-placement disk for a calibration boot: the n-gram table pinned to RAM (ram) OOMs the rig's boot (startup stopped: out of memory) while kldref/score boots succeed at the same pool.

The fitting shape (vendor recipe does not boot 2×32 GiB)

The vendor's calibration line (--max-num-batched-tokens 65536 --max-num-seqs 64, ~930 tok/s) assumes a bigger box: the activation arena alone is 12.83 GiB of 27.95 GiB claimable and every variant stops in the VRAM resolver (short 3.19 GiB → 279 MiB → 59 MiB → 86 MiB → cache starved). Wider steps are refused by the arch itself, pre-allocation:

qwen4exp: the QSA score plane would be 3125 MiB -- 16384 rows (--max-num-batched-tokens 16384)
          by --max-model-len 200000 / compress ratio 4. Lower either.

What boots and seals: 4 slots, --max-num-batched-tokens 2048 (the file-tier ceiling), --kv-cache-dtype fp8, --weights-disk-tier, --placement expert_tiered, --ngram-placement disk, --max-model-len 200000, --num-speculative-tokens 0, --deterministic. Ledger at that shape: elastic 22.8 GiB — experts first, kv cache 1.50 GiB … 1.0 full-length (pool ≈ 199,948 tokens). On 1.8.1 the same payload's pool was 7,436 tokens — the resolver, not the window, is what changed.

Floor

RADIANCE_CALIB_TOKENS is a pooled token floor. Engine default 65,536. Fleet rule (Frank, 2026-10-30): 12,000,000, because the served quant this arm competes with was calibrated on a 10 M-token set. The vendor's "10 M beats 1 M by under 1 %" measures their plate, not ours. Consequence: a ~9.3 M-token corpus round seals in round 1 plus part of round 2, and a round that seals nothing is NORMAL (round_verdict → continue, never failure).

Reading a run (the trap that cost two cycles)

• Throughput is streaming-bound, ~60–70 tok/s whatever the slot count; concurrency only lengthens each request (2-way ≈ 4 min per 7.5 K-token prefill, 4-way ≈ 8).
• The engine logs one line per completed request. A log quiet for 8 minutes is therefore normal; W step: nothing has been logged for 21 s is step-level silence. Liveness = completed-request count, never log mtime. A watchdog on mtime kills healthy cycles and every false positive throws away accumulated tap tokens.
• One seal writes ~43 GB (24,768 gram.r<rank>.blk.<l>.ffn_down_exps.<e>.bin) and the engine then exits by itself; a mid-flight client gets 429 rate_limit_error: the server is shutting down — success, not failure.
• A port that answers is not the rig: after the runner stops production the dying process answers :7770 briefly. Gate on podman ps naming the rig container, and on the .rig-boot marker.

Orchestration and space

Use the repo's orchestrator, never a hand-rolled driver: bash scripts/qwen38-calib-window.sh (gates: reference present, .rad present, tap dir ≥ 60 GiB, healthy rig, marker, then push; banks the reference container only on success). Knobs: FRANK_QWEN38_CALIB_TOKENS, Q38_WINDOW_CALIB_CORPUS, Q38_WINDOW_ROUNDS, Q38_CALIB_MIN_FREE_MIB, Q38_WINDOW_KEEP_RAD.

A hand-rolled driver skips the space preflight; on this host the tap dir sat on / with 35 G free while one seal writes 43 GB — the orchestrator refuses that, which is the point. Frank's rule: keep artifacts on /, never /mnt/entropy.

Then

gram_files → scripts/grams-to-imatrix.py DIR -o imatrix.gguf (stdlib-only, skips non-gram names like .rig-boot) → scripts/convert-qwen38-w5nl64.sh --plan-only (space bill, --plan-only first) → convert (card-less, CPU BLAS) → score each arm + the served payload at the production shape → scripts/kld-gate-qwen38.sh (mean KL ≤ 0.03, p99.9, top-1 ≥ 95 %, every by_source slice).