command-center/agent/managed-skills/radiance-calibration-tap-shape/SKILL.md

name, description

name	description
radiance-calibration-tap-shape	Run and measure a radiance GPTQ calibration tap on a disk-tiered BF16 rig — n-gram tier, step sized to the document, tap marker gates, and reading the real rate from /stats instead of the log.

Running and measuring a radiance calibration tap

A calibration tap (RADIANCE_CALIB_DIR + RADIANCE_CALIB_TOKENS) boots a BF16 reference container on the serving cards under --weights-disk-tier and prefills a corpus until the pooled token floor is reached, writing per-expert Grams for GPTQ. It is the one campaign phase whose cost is dominated by disk read volume, not by compute, and the shape that serves production is the wrong shape for it.

The four things that decide the tap's wall time

1. N-gram table tier. On the BF16 payload the table is tens of GiB and --ngram-placement ram PINS it. Beside a full host pool that leaves a few GiB of RAM and the boot dies at startup stopped: out of memory while the same pool and slot count had already booted the reference recording. Calib only needs the routed weights prefilled, so pass --ngram-placement disk for calib and keep ram for the KL modes — their numbers are only comparable at an identical shape.
2. Step ≥ document length. A document cut into N steps re-streams each layer's routed expert union N times from the container. With --max-num-batched-tokens below the document length the tap runs several times slower than with one wide step, and the KV pool does not change at all (kv_tokens_total is unaffected). This is legal for calib and NOT for a scoring boot: scoring numbers are only comparable at one shape, so the step is calib's own knob.
3. KV pool, not slots. When the pool holds barely one document, extra --max-num-seqs only queue (and preempt) requests; raising slots does not raise throughput.
4. The token floor is pooled, not per round. One corpus pass may be smaller than the floor, so a round that seals no bytes is NORMAL — a driver that treats a flat round as failure aborts one round early. Fail on a tap that loses bytes; let "never sealed" fail at max rounds.

Never push a tap that belongs to no rig

A tap dir is a directory; production is also an endpoint on the same port. Without proof, a pusher will happily feed millions of tokens into whatever answers and record nothing.

• The rig writes a marker into the tap dir at boot (container=, image=, pid=), retires it when its boot exits non-zero, and the pusher refuses unless the marker exists, names the container it targets, and that container is running.
• The window script stops on a failed phase instead of falling through its health wait, and banks a large reference artifact only when both phases exit 0 and the tap holds files.

Measuring the rate (do not read it off the log)

The campaign log is tee'd and block-buffered, so its request count lags by minutes.

curl -s http://127.0.0.1:7770/stats | python3 -c 'import json,sys; d=json.load(sys.stdin); \
  print({k: d[k] for k in ("uptime_s","prefill_tps","steps_prefill","total_requests","preemptions","kv_tokens_total")})'

prefill_tps is CUMULATIVE over uptime — tokens ≈ prefill_tps × uptime_s, so take two samples a couple of minutes apart and difference them. steps_prefill / total_requests proves whether a document is one step or many. Discard the first minutes after boot (warmup and the first requests are not the steady state).

Sizing the tap before paying for it

Probe one 5-minute sample at the candidate step, compute the ratio against the inherited shape, and choose the floor from the measured rate — not from an earlier campaign's runtime on a different corpus and shape. Then set RADIANCE_CALIB_TOKENS (via the runner's env knob) and launch ONE run; if the probe already beat the bar, that probe run IS the real run and needs no restart.

Deliverables that make it unattended

• a gated window script: boot → health with a hard timeout → push → stop → restore the service → bank the reference artifact only on success;
• a pusher that is marker-gated and progress-aware;
• a ledger line recording the shape used and the target chosen, so the next session does not re-derive it.