radiance runbook: MNBT A/B verdict + rad-tune tune-dir bind-mount
Snapshot of a parallel session's completed docs edit (mtime 03:07, was blocking the deploy tree-clean gate): --max-num-batched-tokens 2048 A/B-verified (231/460/448 t/s cold prefill ladder), rad-tune cache survives via FRANK_PRIMARY_STATE_DIR/tune bind-mount (36 GEMM rows, generated 2026-10-06).
This commit is contained in:
@@ -112,7 +112,7 @@ same served alias (`frank-primary`), same container identity
|
||||
| `--gpu-headroom-mib 96 --expert-vs-cache-ratio 0.82` | publisher values | 0.82 = measured 99 % expert-cache hit; 0.87 → 0 % (the cliff) |
|
||||
| `--kv-cache-dtype fp8` | **7.32 KiB/token** | 12 QSA full-attention layers, 2 KV heads × 256 dim, K+V; a 200K-token sequence is 1.43 GiB. KV is sized from the VRAM the ratio leaves |
|
||||
| `--num-speculative-tokens 3` | MTP head | the checkpoint carries `mtp.*` (3,101 tensors, `mtp_num_hidden_layers: 1`); the plugin's probe returns draft depth 3 — the publisher's operating point |
|
||||
| `--max-num-batched-tokens 2048` | prefill chunk | matches the `--checkpoint-interval` default |
|
||||
| `--max-num-batched-tokens 2048` | prefill chunk | A/B-verified optimum (1024/2048/4096 ladder, see "Tuning levers") |
|
||||
| `--prefix-cache-host-mib 4096 --prefix-cache-dir /data/kvcache/frank-primary --prefix-cache-disk-mib 65536` | 3 tiers | host tier pinned (see the ledger), disk tier in `/home/dank/radiance-state` (mounted rw at `/data`; `/` is ext4, mapped-tier-eligible); a session snapshot moves with its blocks |
|
||||
| `--ngram-placement ram` | **PLE pinned to host RAM** | Frank's pin law. The PLE n-gram table is 51.2 GB (320,001,536 rows × 160 dims = 29 % of all parameters) served via io_uring reads; `ram` **refuses the boot** if memlock is too small |
|
||||
| `--override-chat-template /templates/template.jinja` | froggeric v22.5 | vendored `scripts/qwen38-froggeric-v22.5.jinja` (28,234 B, `qwen3.8-froggeric-v22.5`); rad-convert bakes the checkpoint's own template, this overrides it, and the engine derives the reasoning/tool-call reply format from it |
|
||||
@@ -245,11 +245,29 @@ Post-cutover checklist (each is observable, not assumed):
|
||||
declared over the seqs × (1+spec) row bound.
|
||||
- **`FRANK_PRIMARY_EXTRA_ARGS`**: the boot-ladder escape hatch, expanded
|
||||
unquoted.
|
||||
- **`rad-tune -m <rad> --tp 2`**: benchmarks the tunable space at this
|
||||
model's shapes on this machine, writes `$RADIANCE_HOME/tune/<machine>.tune`
|
||||
which the engine's selector reads at boot. Declare at the shapes you
|
||||
serve (`--tp 2`), or you measure a program the serve never runs.
|
||||
- **`rad-tune -m <rad> --tp 2 --num-speculative-tokens 3`**: benchmarks
|
||||
the tunable space at this model's shapes on this machine, writes
|
||||
`$RADIANCE_HOME/tune/<machine>.tune` which the engine's selector reads at
|
||||
boot. Declare at the shapes you serve (`--tp 2
|
||||
--num-speculative-tokens 3`), or you measure a program the serve never
|
||||
runs. `$RADIANCE_HOME` is `/opt/radiance/share/radiance` — in the IMAGE — so a
|
||||
`--rm` scratch container's cache dies with it: the serve script
|
||||
bind-mounts `$FRANK_PRIMARY_STATE_DIR/tune` (`/home/dank/radiance-state/tune`)
|
||||
at `/opt/radiance/share/radiance/tune`, so the cache (36 GEMM rows,
|
||||
`gfx1201-x3-AMD_Radeon_Graphics.tune`, generated 2026-10-06 in a
|
||||
stopped-service window) survives and is read at every production boot.
|
||||
The two attention requests have no shape recipe (`attn_paged`,
|
||||
`attn_paged_gate_quant` — "see tools/opshapes.h") and are not tuned.
|
||||
`RADIANCE_TUNE` pins a kernel for an A/B — an instrument, not a setting.
|
||||
- **`FRANK_PRIMARY_MAX_BATCHED_TOKENS` (MNBT)**: A/B'd 2026-10-06 in a
|
||||
stopped-service window (real-text probes: 20,417-token cold prefill,
|
||||
1,985-token warm prefill, 256-token decode). The prefill ladder is
|
||||
NON-MONOTONIC with an interior optimum at 2048: 1024 → 2048 → 4096 =
|
||||
231 → 460 → 448 t/s cold, 221 → 412 → 427 t/s warm. Decode is flat within
|
||||
noise (24.0 / 25.1 / 24.3 t/s); draft acceptance 60.1 / 67.1 / 63.3 %.
|
||||
**2048 wins** (peak prefill + peak acceptance) — it stays the production
|
||||
value. 1024 halves prefill (the batch too small to saturate the card);
|
||||
4096 is a hair off 2048 (the chunk past the kernel's sweet spot).
|
||||
- **Diagnostics**: `rad-info` (container: encodings, recipe, profile),
|
||||
`--debug-graph` (what declared vs what issued), `rad-kbench` (kernel
|
||||
correctness/speed vs libref), `/stats` + `/sessions` (live), and the
|
||||
|
||||
Reference in New Issue
Block a user