72 lines
4.2 KiB
Markdown
72 lines
4.2 KiB
Markdown
---
|
||
name: radiance-calibration-tap-shape
|
||
description: "Run and measure a radiance GPTQ calibration tap on a disk-tiered BF16 rig — n-gram tier, step sized to the document, tap marker gates, and reading the real rate from /stats instead of the log."
|
||
---
|
||
|
||
# Running and measuring a radiance calibration tap
|
||
|
||
A calibration tap (`RADIANCE_CALIB_DIR` + `RADIANCE_CALIB_TOKENS`) boots a BF16 reference container
|
||
on the serving cards under `--weights-disk-tier` and prefills a corpus until the pooled token floor
|
||
is reached, writing per-expert Grams for GPTQ. It is the one campaign phase whose cost is dominated
|
||
by **disk read volume**, not by compute, and the shape that serves production is the wrong shape for
|
||
it.
|
||
|
||
## The four things that decide the tap's wall time
|
||
|
||
1. **N-gram table tier.** On the BF16 payload the table is tens of GiB and `--ngram-placement ram`
|
||
PINS it. Beside a full host pool that leaves a few GiB of RAM and the boot dies at
|
||
`startup stopped: out of memory` while the same pool and slot count had already booted the
|
||
reference recording. Calib only needs the routed weights prefilled, so pass
|
||
`--ngram-placement disk` for calib and keep `ram` for the KL modes — their numbers are only
|
||
comparable at an identical shape.
|
||
2. **Step ≥ document length.** A document cut into N steps re-streams each layer's routed expert
|
||
union N times from the container. With `--max-num-batched-tokens` below the document length the
|
||
tap runs several times slower than with one wide step, and the KV pool does not change at all
|
||
(`kv_tokens_total` is unaffected). This is legal for calib and NOT for a scoring boot: scoring
|
||
numbers are only comparable at one shape, so the step is calib's own knob.
|
||
3. **KV pool, not slots.** When the pool holds barely one document, extra `--max-num-seqs` only
|
||
queue (and preempt) requests; raising slots does not raise throughput.
|
||
4. **The token floor is pooled, not per round.** One corpus pass may be smaller than the floor, so a
|
||
round that seals no bytes is NORMAL — a driver that treats a flat round as failure aborts one
|
||
round early. Fail on a tap that *loses* bytes; let "never sealed" fail at max rounds.
|
||
|
||
## Never push a tap that belongs to no rig
|
||
|
||
A tap dir is a directory; production is also an endpoint on the same port. Without proof, a pusher
|
||
will happily feed millions of tokens into whatever answers and record nothing.
|
||
|
||
- The rig writes a marker into the tap dir at boot (`container=`, `image=`, `pid=`), retires it
|
||
when its boot exits non-zero, and the pusher refuses unless the marker exists, names the
|
||
container it targets, and that container is running.
|
||
- The window script stops on a failed phase instead of falling through its health wait, and banks a
|
||
large reference artifact only when both phases exit 0 and the tap holds files.
|
||
|
||
## Measuring the rate (do not read it off the log)
|
||
|
||
The campaign log is `tee`'d and block-buffered, so its request count lags by minutes.
|
||
|
||
```bash
|
||
curl -s http://127.0.0.1:7770/stats | python3 -c 'import json,sys; d=json.load(sys.stdin); \
|
||
print({k: d[k] for k in ("uptime_s","prefill_tps","steps_prefill","total_requests","preemptions","kv_tokens_total")})'
|
||
```
|
||
|
||
`prefill_tps` is CUMULATIVE over uptime — `tokens ≈ prefill_tps × uptime_s`, so take two samples a
|
||
couple of minutes apart and difference them. `steps_prefill / total_requests` proves whether a
|
||
document is one step or many. Discard the first minutes after boot (warmup and the first requests
|
||
are not the steady state).
|
||
|
||
## Sizing the tap before paying for it
|
||
|
||
Probe one 5-minute sample at the candidate step, compute the ratio against the inherited shape, and
|
||
choose the floor from the measured rate — not from an earlier campaign's runtime on a different
|
||
corpus and shape. Then set `RADIANCE_CALIB_TOKENS` (via the runner's env knob) and launch ONE run;
|
||
if the probe already beat the bar, that probe run IS the real run and needs no restart.
|
||
|
||
## Deliverables that make it unattended
|
||
|
||
- a gated window script: boot → health with a hard timeout → push → stop → restore the service →
|
||
bank the reference artifact only on success;
|
||
- a pusher that is marker-gated and progress-aware;
|
||
- a ledger line recording the shape used and the target chosen, so the next session does not
|
||
re-derive it.
|