radiance runbook: MNBT A/B verdict + rad-tune tune-dir bind-mount

Snapshot of a parallel session's completed docs edit (mtime 03:07,
was blocking the deploy tree-clean gate): --max-num-batched-tokens
2048 A/B-verified (231/460/448 t/s cold prefill ladder), rad-tune
cache survives via FRANK_PRIMARY_STATE_DIR/tune bind-mount
(36 GEMM rows, generated 2026-10-06).
This commit is contained in:
Frank
2026-10-06 03:08:30 -07:00
parent 31b1ec397b
commit 0f9faf05ea
@@ -112,7 +112,7 @@ same served alias (`frank-primary`), same container identity
| `--gpu-headroom-mib 96 --expert-vs-cache-ratio 0.82` | publisher values | 0.82 = measured 99 % expert-cache hit; 0.87 → 0 % (the cliff) |
| `--kv-cache-dtype fp8` | **7.32 KiB/token** | 12 QSA full-attention layers, 2 KV heads × 256 dim, K+V; a 200K-token sequence is 1.43 GiB. KV is sized from the VRAM the ratio leaves |
| `--num-speculative-tokens 3` | MTP head | the checkpoint carries `mtp.*` (3,101 tensors, `mtp_num_hidden_layers: 1`); the plugin's probe returns draft depth 3 — the publisher's operating point |
| `--max-num-batched-tokens 2048` | prefill chunk | matches the `--checkpoint-interval` default |
| `--max-num-batched-tokens 2048` | prefill chunk | A/B-verified optimum (1024/2048/4096 ladder, see "Tuning levers") |
| `--prefix-cache-host-mib 4096 --prefix-cache-dir /data/kvcache/frank-primary --prefix-cache-disk-mib 65536` | 3 tiers | host tier pinned (see the ledger), disk tier in `/home/dank/radiance-state` (mounted rw at `/data`; `/` is ext4, mapped-tier-eligible); a session snapshot moves with its blocks |
| `--ngram-placement ram` | **PLE pinned to host RAM** | Frank's pin law. The PLE n-gram table is 51.2 GB (320,001,536 rows × 160 dims = 29 % of all parameters) served via io_uring reads; `ram` **refuses the boot** if memlock is too small |
| `--override-chat-template /templates/template.jinja` | froggeric v22.5 | vendored `scripts/qwen38-froggeric-v22.5.jinja` (28,234 B, `qwen3.8-froggeric-v22.5`); rad-convert bakes the checkpoint's own template, this overrides it, and the engine derives the reasoning/tool-call reply format from it |
@@ -245,11 +245,29 @@ Post-cutover checklist (each is observable, not assumed):
declared over the seqs × (1+spec) row bound.
- **`FRANK_PRIMARY_EXTRA_ARGS`**: the boot-ladder escape hatch, expanded
unquoted.
- **`rad-tune -m <rad> --tp 2`**: benchmarks the tunable space at this
model's shapes on this machine, writes `$RADIANCE_HOME/tune/<machine>.tune`
which the engine's selector reads at boot. Declare at the shapes you
serve (`--tp 2`), or you measure a program the serve never runs.
- **`rad-tune -m <rad> --tp 2 --num-speculative-tokens 3`**: benchmarks
the tunable space at this model's shapes on this machine, writes
`$RADIANCE_HOME/tune/<machine>.tune` which the engine's selector reads at
boot. Declare at the shapes you serve (`--tp 2
--num-speculative-tokens 3`), or you measure a program the serve never
runs. `$RADIANCE_HOME` is `/opt/radiance/share/radiance` — in the IMAGE — so a
`--rm` scratch container's cache dies with it: the serve script
bind-mounts `$FRANK_PRIMARY_STATE_DIR/tune` (`/home/dank/radiance-state/tune`)
at `/opt/radiance/share/radiance/tune`, so the cache (36 GEMM rows,
`gfx1201-x3-AMD_Radeon_Graphics.tune`, generated 2026-10-06 in a
stopped-service window) survives and is read at every production boot.
The two attention requests have no shape recipe (`attn_paged`,
`attn_paged_gate_quant` — "see tools/opshapes.h") and are not tuned.
`RADIANCE_TUNE` pins a kernel for an A/B — an instrument, not a setting.
- **`FRANK_PRIMARY_MAX_BATCHED_TOKENS` (MNBT)**: A/B'd 2026-10-06 in a
stopped-service window (real-text probes: 20,417-token cold prefill,
1,985-token warm prefill, 256-token decode). The prefill ladder is
NON-MONOTONIC with an interior optimum at 2048: 1024 → 2048 → 4096 =
231 → 460 → 448 t/s cold, 221 → 412 → 427 t/s warm. Decode is flat within
noise (24.0 / 25.1 / 24.3 t/s); draft acceptance 60.1 / 67.1 / 63.3 %.
**2048 wins** (peak prefill + peak acceptance) — it stays the production
value. 1024 halves prefill (the batch too small to saturate the card);
4096 is a hair off 2048 (the chunk past the kernel's sweet spot).
- **Diagnostics**: `rad-info` (container: encodings, recipe, profile),
`--debug-graph` (what declared vs what issued), `rad-kbench` (kernel
correctness/speed vs libref), `/stats` + `/sessions` (live), and the