69 lines
3.7 KiB
Markdown
69 lines
3.7 KiB
Markdown
---
|
||
name: radiance-disk-tier-throughput-census
|
||
description: "Measure a radiance rig's real prefill rate and prove whether the bottleneck is disk, CPU or shape — for calibration taps, KLD recording or score boots; use when a rate looks wrong, an ETA must be quoted, or someone asks about CPU allocation."
|
||
---
|
||
|
||
# Census a radiance rig's throughput without lying to yourself
|
||
|
||
Ask three questions in order: what is the rate, what is the binding resource, and does the shape
|
||
explain it. Every number below comes from a kernel or engine counter — never from a gauge.
|
||
|
||
## 1. The rate: counters, never `prefill_tps`
|
||
|
||
`/stats` on :7770 exposes `prefill_tps`, and it is a **rolling gauge**. Reading 69.5 → 149.1 across
|
||
120 s and multiplying by `uptime_s` produced a bogus "320 tok/s / 10.4 h" verdict that had to be
|
||
retracted (measured 2026-10-29). Derive the rate from counters over a ≥240 s window:
|
||
|
||
```bash
|
||
python3 - <<'PY'
|
||
import json, time, urllib.request
|
||
def stats(): return json.load(urllib.request.urlopen("http://127.0.0.1:7770/stats", timeout=10))
|
||
a = stats(); t0 = time.time(); time.sleep(240); b = stats(); t1 = time.time(); dt = t1 - t0
|
||
docs = b["total_requests"] - a["total_requests"]
|
||
steps = b["steps_prefill"] - a["steps_prefill"]
|
||
MEAN = 4238.0 # tokens/doc, from the corpus: total_tokens / documents
|
||
print("docs %d in %.0fs -> %.0f tok/s | steps/doc %.2f" % (docs, dt, docs * MEAN / dt, steps / docs))
|
||
PY
|
||
```
|
||
|
||
`steps/doc` is the read-amplification signal: a document cut into N prefill steps re-streams each
|
||
layer's routed expert union N times when the weights are on the disk tier, so N > 1 is leaked time.
|
||
|
||
## 2. The binding resource: diskstats and the idle/iowait split
|
||
|
||
```bash
|
||
# per-device read bandwidth (parent device, not the partition)
|
||
awk '{print $3, $6*512}' /proc/diskstats | grep -E "^nvme"
|
||
# host busy = total - idle - iowait (summing /proc/stat's first 8 fields WITHOUT subtracting is
|
||
# the classic error: the sum already covers idle, so it reads ~800% on an 8-core box)
|
||
```
|
||
|
||
Then split CPU by process: sample `/proc/<pid>/stat` fields 14+15 (utimes + stimes) twice.
|
||
|
||
Two hard readings that decide the argument:
|
||
|
||
- `radiance` compute at ~0.26 cores while the disk delivers ~3.2 GB/s and `iowait` is ~29 % of the
|
||
box ⇒ **disk-bound**. More cores or a `--cpuset-cpus` pin cannot add bytes; pinning only caps it.
|
||
- A container with `cpu_quota=0` and an empty `CpusetCpus` (check `podman inspect -f
|
||
'{{.HostConfig.CpuQuota}} {{.HostConfig.CpusetCpus}}'`) is already unrestricted — "allocate more
|
||
CPU" is a no-op, and "allocate 6 of 8" is a ceiling.
|
||
|
||
Sanity relation: tok/s × (GiB streamed per document ÷ tokens per document) should match the measured
|
||
device bandwidth. If it does not, the shape, not the hardware, is the story.
|
||
|
||
## 3. The shape: what may legally be widened
|
||
|
||
- `kldref` and `score` must keep production's `--max-num-batched-tokens` and `--max-num-seqs`: their
|
||
KL numbers are only comparable across boots at an identical shape (the audit law).
|
||
- A **calibration tap** is exempt — its output is layer Hessians, nothing compares it. Give it a step
|
||
that covers a whole document (`Q38_CALIB_MNBT` ≥ the corpus window) and read `steps/doc`: on a
|
||
7,168-token corpus, 2,048 → 8,192 moved 2.5 → 1.00 steps/doc and 94 → 113 tok/s (**1.2×**, not the
|
||
4× the step-count ratio suggests — total bytes are unchanged, so do not promise more).
|
||
- The KV pool, not the step, bounds concurrency: when `kv_tokens_total` ≈ one document, requests queue
|
||
and preempt; extra slots buy nothing.
|
||
|
||
## Reporting
|
||
|
||
Quote tok/s, steps/doc, device GB/s and the busy/iowait split together, plus the shape each number
|
||
was taken at. A rate without its shape is not a measurement; a gauge without its window is not a rate.
|