@@ -96,7 +96,7 @@ on stop. Boot budget 1800 s (the publisher's compose allows a 15-minute `start_p
| Spec / batch | `SPEC_TOKENS=2` — MTP depth 2, the `.rad` carries the model's own `mtp.*` head (`mtp.fc_hidden` /`fc_embedding` /`draft_head` ); `MAX_BATCHED_TOKENS=2048` (MNBT — 2560 collapses decode, Laws) |
| Prefix cache | `PREFIX_CACHE_MIB=8192` , pinned **host tier only ** — the 64 GiB disk tier is gone, so an evicted prefix is re-prefilled; `STATE_DIR=/home/dank/radiance-state` now exists only to hold the rad-tune cache |
| Template | `TEMPLATE=scripts/qwen38-froggeric-v22.5.jinja` (28,234 B) via `--override-chat-template` — froggeric's fixed Qwen3.8 template (the repo root's `chat_template.jinja` , vendored); the checkpoint's own template is baked into the `.rad` by rad-convert and this overrides it, and radiance derives the reasoning and tool-call reply format from it |
| Step sharing | `BUSY_PREFILL_CHUNK=auto` (a shared chunk reserves a row plus the draft window per waiting decoder) · `PREFILL_INTERLEAVE=0 ` — held at zero until the decode-vs-prefill buckets ask for it |
| Step sharing | `BUSY_PREFILL_CHUNK=auto` — prefill-speed first: it reserves 3 tokens per waiting decoder and ⅛ of the budget per waiting request, and does **not ** shrink the chunk for decoders alone · `PREFILL_INTERLEAVE=1 ` — t he decoder dial, **armed 2026-10-20 ** : one decode-only step after each chunk step taken beside decoders; meter `engine_steps_held_total` , price is the prefilling prompt's TTFT |
| TTM gate | `TTM_MIN_MIB=150000` , `TTM_TARGET_MIB=204800` |
| Boot | `RAD_MIN_MIB=110000` deployer floor; the listener binds only after the full load (health ~100 s, budget 1800 s) |
@@ -280,30 +280,92 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
- **MNBT 2048.** 2560 collapses decode to 2.6-7.7 t/s (MTP resonance); the w4 file tier streams
in place only at MNBT ≤ 2048 and the activation arena scales with the chunk (32768 → 33.3
GiB, refused).
- **Decode pays for the chunk it shares a step with.** Census of the live engine taken at the
3-slot shape on 1.0 .13 (` /stats ` + ` /metrics `, cumulative over the 2026-10-15 boot):
1,435,585 decode-only steps at **18.7 ms** each against 127,991 prefill+mixed steps
at **317 ms** — 60 % of engine wall-clock spent inside 8 % of the steps, because the
chunk is ` min(remaining, checkpoint interval 2048, MNBT 2048) ` = 2048 rows and every
decode row in that step waits the who le chunk. Day of Prometheus (120 s step, 721
samples): ` decode_tokens_per_second` median **13.3 t/s** in the 56 % of samples where
` prefill_tokens_per_second` ≥ 1000, against 106-240 t/s in the light-prefill buckets;
bucketed by mean step time it is 128.7 t/s at 0.02-0.05 s steps and 12.0 t/s at
0.15-0.4 s. The stall is **during**, not after — decode in windows 2- 14 min after a
busy-prefill window sits at the idle baseline (58-64 vs 77-85 t/s, the residual
confounded by co-scheduling), so no lingering expert-residency penalty at 2-minute
resolution, and ` preemptions: 0` rules out preempt-and-re-prefill. Method:
[[LLM Performance Diagnosis]] §9. The law is shape-independent; the fix lives at 1.8.1 (next law),
and nobody has re-run the census since the migration — so the fix is ordered, **not verified**.
- **The knobs for it arrived with 1.8.1, and only one of them is on.** ` --busy-prefill-chunk` caps the
shared chunk so a step with waiting decoders takes a smaller piece (live: ` auto`, the engine picks;
the explicit value is 512), and ` --prefill-interleave N` inserts N decode-only steps after a chunk
step taken beside decoders. ` interleave` is deliberately **0** — the env comment holds it there until
the decode-vs-prefill buckets ask for it, and those buckets have not been read since the migration.
The vendor prices a 1024 chunk at about a fifth off prefill throughput to halve the stall; that is a
knob for choosing tomorrow's chunk, not a measurement of this shape. MNBT remains an arm and not an
edit: 2560 collapses decode through MTP resonance. Any chunk change re-scores draft acceptance before
it is trusted.
- **Decode pays for the chunk it shares a step with, and ` --busy-prefill-chunk auto ` does not
protect it.** Re-measured on the live 1.8 .1 shape (34.8 h boot, Prometheus at 60 s,
` engine_step_seconds{,_decode}_total` / ` engine_steps{,_decode,_mixed,_prefill}_total` /
` prefill_tokens_total`): decode-only steps run at **15.5 ms**, chunk steps (a chunk alone, or a
chunk with decode rows beside it) at **166 ms** — 4.7 % of the steps hold **34 % of the engine's
step time**, so a session decoding whi le another prompts advances one token per 166 ms. Bucket the
harness's ` radiance: decode_tokens_per_second` by ` radiance:prefill_tokens_per_second ` (15 s
scrapes; per-stream rate = the gauge ÷ ` vllm:num_requests_running`): median **166 t/s** under light
prefill, 81 t/s at 1-5k, **13 t/s above 5k** — a **12.7×
sessions running each stream sits at 6.7 t/s = 149 ms a token, one token per shared step. Nothing
else moves: decode-*only* steps stay 14-18 ms in every bucket (14.1 ms in a prefill-free minute,
17.9 ms in the deep one) and the relationship is lag-0 only (r = −
30 s on, decode back at baseline by 5 min) — residency is not the mechanism either. The second
victim is the **arriving** request: through a high-prefill window TTFT p90 goes 2.4 → **18.0 s**
and 15.5 % of requests wait over 10 s for their first token, with ` http_queue_depth ` at 2-3, while
median TTFT (~1.9 s) and per-token latency barely move. Preemption is real but burst-shaped: 264
` vllm:num_preemptions_total` in the boot, 108 of them inside 105 s of a 6.5k-tok/s prefill. The
chunk is ` min(remaining, checkpoint interval, MNBT, busy_cap)`; the first three read 2048 here,
and ` busy_cap` is the fourth law.
- **` auto` is a prefill-speed dial, not a decoder-protection dial.** ` Scheduler::busy_cap` reserves
` 1 + spec` rows per decoding sequence behind the chunk (3 tokens at MTP 2) plus one eighth of the
budget per *waiting request or waiting prefill*, and the code refuses to do more for decoders:
"not for decode rows alone: their reserve is a few tokens, so the chunk ends a quantum short and
the next step finishes the interval… Halving would hold a long prompt to half-interval chunks while
anything decodes behind it, which costs more" (` core/sched/scheduler.cpp:874-878`, 1.8.1). Live
proof: ` prefill_chunks_capped` 5,018 against 62,386 chunk steps — 8 %, and those are the minutes
with a waiter, not the minutes with a decoder. An explicit ` N` is the only value that applies
"whenever the chunk shares the step".
- **At MNBT 2048 no chunk value keeps both the arena loan and the staged copies.** The arena is a
ladder of levels (256 and 1024 rows here) and a step's rows are its **whole** token count —
` max(n_tok, n_seq × `, ` core/engine.cpp:1142` — so a 1024-token chunk with two decode
rows is a 1030-row step: it falls back to the full plan, which "lends nothing", and it arms the
prefill stager. The stager's floor is the largest lending level plus one, 1025 rows here — named on
the boot line ` prefill staging: … a step of more than 1024 tokens copies the pooled experts of 48
routed layers onto the card up to 3 layers ahead` — and below it "every routed layer reads its
pooled experts synchronously and the prefill slows far more than sharing saves" (GUIDE §4.5). The
vendor's own number for naive shrinking, on this architecture: four decoders with a long prompt
beside them finished in **29.5 s at a 2048 chunk and 31.9 s at 512**.
- **A big step takes the arena from the expert slab and keeps it for the whole prefill.** The arena
lends 987.77 MiB (steps ≤ 256 rows) or 670.26 MiB (≤ 1024) to the slab; a full-plan step reclaims
it **synchronously before the step** — units revert to their host copies (an address change, not a
copy) and both read streams drain — and ` arena_tick` regrows the loan only **2 s after the last big
step** (` core/engine.cpp:1129-1202`), so every step of the prefill run, decode-only included, runs
with a slab ~780 expert units smaller over a link also carrying the staged copies. ` /diagnostics`
says the binding constraint in one line: "a prefill step streams every routed layer's pooled
experts onto the card, so its speed is bounded by the link; decode reads the ones it routes to".
Measured size of the whole effect: the ~10 % above. Real, small, not worth chasing.
- **` --prefill-interleave` is the dial that buys decode its cadence; armed at 1 on 2026-10-20.**
N decode-only steps follow any chunk step taken beside decoders; held steps admit nothing, so it
costs the prefilling prompt its TTFT and an arriving request its first step. Rate arithmetic at the
two measured step times and 0.657 acceptance (≈2.3 tokens a step): N=0 → ~14 t/s, **N=1 (armed) →
~23**, N=2 → ~33, N=4 → ~48, N=8 → ~69, with the prefill slowed by (166 + 15.5N)/166. **Arithmetic, not
measurement** — the arm is chosen by the §9 bucket pair plus ` steps_held`, and ` auto` is not a
substitute for it.
- **` --prefill-span` is the shape that keeps the loan, and it is priced.** It needs MNBT = N
intervals, so ` --prefill-span 2 --max-num-batched-tokens 4096`: a *lone* prefill carries 2
intervals a step (and on an expert-tiered box only from 8 intervals / 16k tokens, where the span
starts to win), while **every other step is planned at the shared budget of 2048** — which is a
lending level *and* keeps the stager's ring, the one combination ` --busy-prefill-chunk` cannot
express. Costs per GUIDE §4.5: 320 MiB a card of expert residency, ~3 % off the KV pool (§4.3's
cliff is sharp), a decode step uploads twice the per-step arrays at 4096, and checkpoints inside a
span are not written, so a partially-shared prompt resumes one interval earlier. On the vendor's
second box (2×
75k — hence their own rule: measure your own prompts at two ranks before turning it on. MNBT 4096
does not boot at 3 slots; this shape is 2.
- **Most of this endpoint's prefill work is replay of tokens the prefix cache already had.** Over
the same window ` prefill_tokens_total` 65.2 M against ` prefix_hit_cut_tokens_total` 24.9 M = **38 %
replayed**, and it is not an eviction: a hit on a hybrid model reuses only as far as the last
**linear-state** checkpoint (` reuse = min(h.n_attn_tokens, h.n_linear_tokens)`,
` core/sched/scheduler.cpp:953-978`), and the retention default keeps a **geometric ladder** - about
` log2(depth/interval) + 2` snapshots - so a turn that diverges ` d` behind the tip can resume as
deep as ` tip − `. The live ladder for a 200k session at interval 2048 is 2048 · 131072 · 163840
· … · 198656: diverge at 120k and 21.7k tokens replay, diverge at 50k and 48k. Every
` prefix_hit_cuts_total` carries ` reason="no_snapshot"` (the other two reasons stay dead), so the
cause object agrees and ` hit_cut_tokens` is the cost. ` --checkpoint-policy` admits exactly
**` geometric-backoff` | ` tip-only` | ` keep-all`** and refuses anything else at boot. It governs
retention only - the writes happen at every interval whatever the policy says (only ` --prefill-span`
skips them) - so ` keep-all` buys a replay bounded by one interval at any depth for **pinned host RAM
alone**: 97 snapshots a 200k session, ` --checkpoint-slots auto` asking 196 ×
against a store budget of an eighth of host RAM (31.5 GiB here, so ` share` admits it) with ` room` -
the pinning room left beside the expert pool at boot - as the hard bound the ledger line names.
Retention also spares a snapshot a later turn resumed from (` core/mem/prefix.cpp:709-723`), so a
chain that keeps resuming at one deep point stops paying for it. Shortening
` --checkpoint-interval` does **not** buy the deep case: the coarsest ladder scale is the largest
` interval × ` under the depth, so the deepest gap stays ≈ half the prompt (129,024 tokens at
interval 2048 against 130,816 at 256) while every cut multiplies the snapshot writes.
- **Slots cost KV, rows, and window — never speed.** ` --max-num-seqs` carries no radiance constraint: it
adds ≈110 MiB of recurrent state per slot (the live ` kv_full_backing` arithmetic prices them exactly),
declares the fused TP all-reduce over ` seqs × ` rows, and ` wht6` is bit-exact up to 25 rows
@@ -396,12 +458,35 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
4600 MT/s ended it, with zero ` gdn_hip` faults either side; the prefix-caching / GDN re-seed
theory was confounded on both arms. Consumer non-ECC raises no MCE for a silent flip and a
60 s / 8 GiB pattern check covers 0.5 % of 246 GiB — silent RAM corruption outranks every
software theory.
software theory. **That mitigation is a BIOS setting and it is not in effect now:** ` sudo
dmidecode -t memory | grep -i configured` answers 4800 MT/s on 2026-10-20, and the only event
between the fix and today is the 2026-10-12 power failure that also reset TTM. Check the speed
before crediting any software revert with a fix — and before believing a clean run of the arm
you just tested, because the confounder is still in the slot.
- **TP=2 here** is legal since the 2026-09-20 ProArt B850 swap (x8/x8 on CPU root ports,
` Routing+`/` ReqEn+` end to end) only because radiance's wire is ` libr4d`'s own one-shot
all-reduce; RCCL needs atomics on the whole chain and fails ` hipErrorIllegalState` without
them (). ` wht6` buys +5.6-6.3 % prefill for ~1.3 % of 8-stream
decode, ` exact` is the bit-exact alternative.
- **The link is already at the board's ceiling — do not chase the ` link-below-max` warning.**
Both root ports (` 00:01.1`, ` 00:01.3`) are **x8 max** at 32 GT/s, so PCIe 5.0 x8 ≈ 31.5 GB/s is
the whole budget per card and the cards' own ` pcie_dpm` tables expose only x8 states.
` /diagnostics` still warns that the on-card switch hop (` 01:00.0`, ` 05:00.0`) runs "x8 of its
x16 while the card is 100 % busy" and suggests a reboot or a reseat — pointless when the host
side of the chain is x8 max, which the same endpoint reports two findings later as
` link-capped`. The width that matters is the minimum of the chain, and it is x8 either way, so
the only lever on a link-bound prefill is **less traffic over the link** (more expert
residency, fewer pooled experts), never a wider link. P2P resolves **on** (` auto` asks the
driver, the two cards reach each other's memory), so the collectives do not hairpin through host
RAM, and the engine's copies ride one DMA queue that "give[s] way to the collectives" — the
all-reduce wins the link, the staged experts wait.
- **Eight cores is the whole host, media stack included.** Dankserver is a Ryzen 7 7700X (8
cores / 8 threads, one NUMA node, ` nodes: -1` for both cards) running Jellyfin, Plex, Calibre
and RomM beside the engine, and ` --http-threads` defaults to one thread per core. Measured
during a heavy prefill: load average ≈ 1.0, ` jellyfin` 11 % CPU, ` radiance` 7 % — **CPU
contention is not the current decode culprit**, so do not spend an experiment window on it;
re-read ` uptime` and ` ps -eo pcpu,comm --sort=-pcpu | head` first if a future collapse looks
different from the step-sharing one.
- ` --prefix-cache-dir` creation is **not recursive** — the serve script ` mkdir -p`s the path or
the disk tier silently never opens. The tier does survive a payload swap when arch and
tokenizer match (it recovered 900,176 blocks on the 2026-10-15 swap; ` cache_n` proves a hit).
@@ -445,11 +530,19 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
## Contract (the wire)
- **Live surface:** ` /health` (== ` /ping`), ` /v1/models`, ` /v1/chat/completions`
` /v1/completions`, ` /metrics`, ` /stats`, ` /sessions`, ` /dashboard`. ` /health` answers 200
JSON (model id, ` in_flight`, ` queue_depth`) while serving and **503 only while stopping**
(` core/server/admin.cpp:385`), and the listener binds *after* the full load (` main.cpp `:
` engine.start()` then ` engine.serve()`) — during the load the port refuses connections, so a
bare ` curl -fsS …/health` gate is honest and needs no body grep.
` /v1/completions`, ` /metrics`, ` /stats`, ` /sessions`, ` /dashboard`, **` /diagnostics `** . ``
answers 200 JSON (model id, ` in_flight`, ` queue_depth`) while serving and **503 only while
stopping** (` core/server/admin.cpp:385`), and the listener binds *after* the full load
(` main.cpp `: `` then ` engine.serve()`) — during the load the port refuses
connections, so a bare ` curl -fsS …/health` gate is honest and needs no body grep.
- **` /diagnostics` is the instrument for "is the box, not the config, the limit"** — one call
returns ` configuration` (every flag, so drift from the env file is checkable without the host),
` environment`, ` derived` (the VRAM split as the ledger prints it), ` hardware` (host, per-card
` link_chain` + ` pcie_dpm` + clocks + power cap + DMA engine choice + P2P answer) and ` findings`
(level / what / why / fix). On this box it reports ` experts-from-host` (5.99 GiB a rank crosses
PCIe when used), ` link-capped` (root ports are x8 max — the board), ` swap-in-use`,
` tp-wire-lossy`, and **` link-below-max` as a ` warn` at 100 % card busy on both cards** — which is
structural on this board; see the x8-ceiling law under *This host, and the deploy path*.
- **Served id.** ` FRANK_ARCHON_SERVED_NAME=frank-archon` (the wire rename of 2026-10-09 moved
only the alias; container, unit, scripts and env keep the service name). The env's contract
line is that **radiance validates the requested model name, so every client — harness, bench
@@ -476,12 +569,28 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
and 16,384 patches per pass. Probe with a ≥ 256 px solid-colour PNG and a ≥ 256 token budget
— at 64 tokens you get empty ` content` and the colour only inside ` reasoning_content`.
- **Capacity behaviour:** a prompt over ` --max-model-len` is refused, never truncated; a request
longer than the KV pool's token count is accepted then refused as too large to fit; under the
3-slot contract a **fourth** full-height conversation is refused, not paged or squeezed.
longer than the KV pool's token count is accepted then refused as too large to fit; a request
beyond ` --max-num-seqs` is held by the HTTP port (` queue_cap`, default 8 ×
answered 503 only past that, never paged or squeezed. Because the cache funds **1.8** full-height
sessions, admitting a second near-full-height conversation can **preempt a running sequence from
the back** to free its blocks (` core/sched/scheduler.cpp:1291-1322`); the victim is recomputed, so
its session halts and then re-prefills. Meter: ` vllm:num_preemptions_total` (264 in the current
boot, all in prefill-coincident bursts — 108 inside 105 s); ` /stats preemptions` is the cumulative
count and ` radiance:num_requests_preempted` only the instant depth.
- **Observability:** ` /metrics` carries vLLM-compatible names plus ` radiance:*`, scraped by the
Dankclaw Prometheus job at ` 192.168.0.235:7770` over plain LAN routing (the netavark
published-port quirk is same-host only). The boot ledger — the referee for every shape numb er
above — is in the container log (` podman logs frank-archon`)
Dankclaw Prometheus job at ` 192.168.0.235:7770` (15 s interval) over plain LAN routing (the
netavark published-port quirk is same-host only). **Latency lives under ` vllm: `, cadence und er
` radiance:`** — TTFT / inter-token / per-output-token / e2e / decode-time histograms and the
running/waiting gauges are ` vllm:*` with no radiance-native equivalent, so a ` radiance:`-only
query misses every latency answer. The step-phase counters are all scraped:
` engine_steps{,_decode,_mixed,_prefill,_held}_total`, ` engine_step_seconds{,_decode}_total`,
` engine_prefill_chunks_capped_total`, ` prefill_tokens_total`, ` decode_tokens_per_second`,
` prefill_tokens_per_second`, ` prefix_hit_cut_tokens_total`, ` http_queue_depth`. Two holes: the
engine's own step-detail histograms (` prefill_steps_duration`, ` prefill_step_tokens`,
` token_queue_wait`) are exported but reach no target, and the **arena / expert slab / heat engine /
prefill stager have no metric at all** — that half of every law above is read from the boot ledger,
` /stats` and ` /diagnostics`, never from a graph. The boot ledger — the referee for every shape
number here — is in the container log (` podman logs frank-archon`);
` /home/dank/logs/frank_archon.log` is where the serve script redirects ` podman run`.
## Files