dank/serverAppConfigs
1
0
Fork 0
Code Issues 4 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

docs(vault): answer "what are we doing wrong" on decode-vs-prefill

Browse Source 
frank-archon Laws: add the measured interference census (decode-only step
15.5 ms vs 166 ms chunk steps; 4.7% of steps hold 34% of step time; median
decode 166 -> 13 t/s above 5k prefill tok/s, per-stream 6.7 t/s = one token
per shared step; TTFT p90 2.4 -> 18.0 s for arriving requests), the lag test
that kills the residency theory, the busy_cap arithmetic, and the correction
that --prefill-interleave is the decoder dial and auto is not.

Facts that change how an operator reads this box: /diagnostics exists and
reports link-below-max at 100% busy; latency metrics live under vllm:, cadence
under radiance:; preemption is vllm:num_preemptions_total (264 this boot, all
bursts) while the radiance gauge is instant depth; both root ports are x8 Gen5
max by board (31.5 GB/s ceiling, not 64); DDR is back at 4800 MT/s while the
4600 corruption fix is recorded -- verify before blaming garbage tokens.

§9 (LLM Performance Diagnosis): method corrected to the phase-ratio counters,
dead 1.0.13 numbers destroyed, four PromQL traps added (metric prefix, 1 s EMA
gauges vs 15 s scrape, counter resets across builds, 2-minute mean diluting the
chunk), instruments that cannot be reached from Prometheus, and the knob list
priced from the pinned 1.8.1 source instead of the stale guide.
This commit is contained in:
Frank
2026-10-20 22:08:00 -07:00
parent 687dbfbabe
commit 9f7457e39e
2 changed files with 216 additions and 71 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md
+80 -35
View File
@@ -239,45 +239,90 @@ prefill is over. `preemptions` separates the third shape: a preempted-and-recomp
sequence stops producing entirely and then re-prefills from token 0, which is a
stall, not sludge.
Knobs, in the order worth trying them (radiance GUIDE §4.5):
**Split the step ledger before choosing a knob.** Three counters decide which mechanism you are
looking at, and none is a rate: `engine_step_seconds_decode_total / engine_steps_decode_total` is
the mean **decode-only** step, and `(step_seconds_total − step_seconds_decode_total) /
(steps_total − steps_decode_total)` is the mean **chunk step** (a chunk alone, or a chunk with
decode rows beside it). On frank-archon's live 1.8.1 shape that pair is **15.5 ms against 166 ms**,
with 4.7 % of the steps holding 34 % of the step time. Decode-only steps themselves move only
14.1 → 16.4 ms between a prefill-free minute and a heavy one, and the lagged buckets stay flat: so
a decode-rate collapse on that box is **step time sharing**, and the eviction/residency story is
worth ~10 % of one step — not a lever.
- `--prefill-interleave N` — after a chunk step taken beside decoders, the next N
steps carry decode rows only and admit nothing. Decoders keep their cadence; the
prompt pays that in TTFT. Counts as `engine_steps_held_total`.
- `--busy-prefill-chunk MODE` — the chunk's share of a step that has other work:
`auto` (default) reserves a row plus the draft window per decode row behind it and
an eighth of the budget per waiting prefill/request, `off` lets the chunk take the
whole step (admission never runs while it does), `N` pins the share. Never under
the prefill stager's smallest pass — the startup line names it, and below it every
routed layer reads its pooled experts synchronously, which costs the prefill far
more than sharing saves. Counts as `engine_prefill_chunks_capped_total`.
- `--max-num-batched-tokens` — the blunt rung, and the only one older pins have. The vendor's own
curve prices **halving the shared chunk at roughly a fifth of prefill throughput** (a 48-layer MoE
at 200k on two cards — knob pricing, not this endpoint's number, so treat it as the order of
magnitude a chunk decision costs). Raising it past the checkpoint interval buys nothing and loses
expert residency to the arena.
- `--prefill-span N` — the inverse trade: a *lone* prefill carries N intervals a step while every
shared step is planned at a smaller shared budget, which on an expert-tiered box lends that arena
back to the expert slab. The vendor's own measurement says the win is **long-prompt-only**: nothing
at short prompts, a large TTFT gain at the far end of the window.
**Four traps in the PromQL above.** Learn them before trusting a bucket:
- **Latency is `vllm:`, cadence is `radiance:`.** The TTFT / inter-token / per-output-token / e2e
histograms and the running/waiting gauges exist only under the vLLM compatibility prefix; preemption
is `vllm:num_preemptions_total` (the counter) while `radiance:num_requests_preempted` is the
*instant* depth. A `radiance:`-only query finds no latency and no preemption, and reads as "the
endpoint never stalls anyone".
- **The tps gauges are 1 s decayed windows, batch-pooled.** Per-stream rate is
`decode_tokens_per_second / vllm:num_requests_running`. With a 15 s scrape, a 120 s step samples one
scrape in eight — use step ≤ 30 s for a bucket table.
- **Every counter resets at a restart and the metric set changes with the build.** Prove the window is
one shape (`up`, or the first sample of a counter that only the current build exports) before
ranking buckets; frank-archon's 48 h window straddled two containers, and one bucket's "collapse"
turned out to belong entirely to the older one.
- **A 2-minute mean step time dilutes the chunk.** The sharp variable is the phase ratio in the query
above, never `step_seconds/steps` — 4.7 % of steps cannot move a mean that is 95 % decode.
Two traps. **The arena/slab coupling makes a shared step doubly expensive on an
expert-tiered shape**: the full-plan arena takes VRAM the resident expert slab would have had, so the
decode rows inside that step also read experts over the link — a smaller shared budget helps twice.
And **re-check
`prefix_hit_rate` / `prefix_evictions` after any chunk change**: the arena and the KV
pool are the same memory, and the failure looks like turn two taking as long as turn
one.
Three instruments the PromQL above cannot reach:
Availability is a version question, not a config question: `--busy-prefill-chunk` and
`--prefill-interleave` are absent from older pins — prove it on the running container
(`podman exec <c> <binary> --help | grep -iE "prefill|chunk"`), never from the vendored
tree, and check the source clone's depth before dating a flag from `git log -S`
(a `--depth 1` clone reports its own tip for every search).
- **The engine's own step-detail histograms (`prefill_steps_duration`, `prefill_step_tokens`,
`token_queue_wait`) are exported but reach no target**, and the arena / expert slab / heat engine /
prefill stager have **no metric at all**. Per-request forensics mean `curl localhost:7770/stats` on
the endpoint's own host — same phase counters plus `preemptions` (the count), the chunk geometry,
and the expert/link byte totals. `/diagnostics` answers "is the *box* the limit" in one call:
configuration drift, the VRAM split, each card's link chain and clocks, findings with fixes.
- **`prefix_hit_cut_tokens_total` measures prefill load you created for yourself.** On a hybrid
(linear + paged) model a prefix hit reuses only as far as the last **linear-state** checkpoint, so
cached tokens are replayed as prefill and stall decoders for it; frank-archon ran 38 % of all its
prefill tokens that way, every `prefix_hit_cuts_total` labelled `reason="no_snapshot"`. A high
prefill rate is not always a client that sends new tokens.
- **Bucket the arriving request too** — `histogram_quantile` over
`vllm:time_to_first_token_seconds_bucket` by the same prefill series. The decode gauge only sees
sequences that are already running; the one waiting for its first token pays the same steps and
shows in no decode number.
Run on: [[frank-archon]] — the census was taken on its 3-slot radiance 1.0.13 era (budget 2048 =
interval 2048); the live 1.8.1 shape carries the two chunk knobs. The measured census and the verdict
for this fleet's shape are that page's.
Knobs, in the order worth trying them (radiance GUIDE §4.5; the pinned source is the referee):
- `--prefill-interleave N` — after a chunk step taken beside decoders, the next N steps carry decode
rows only and admit nothing. **The knob that acts on the measured mechanism.** Decoders keep their
cadence; the prefilling prompt pays in TTFT and an arriving request waits out the held steps.
Counts as `engine_steps_held_total` — if that counter is zero, nothing is protecting anyone.
- `--busy-prefill-chunk N` — pins a shared chunk's share of the step. **`auto` is a prefill-speed
setting, not decoder protection:** it reserves `1 + spec` rows per decoding sequence plus an eighth
of the budget per *waiting* request or prefill, and the source refuses to halve for "decode rows
alone", so a shape can run `auto` and still lose every chunk step. Prove engagement from
`engine_prefill_chunks_capped_total` against the chunk-step count. The floor is the prefill
stager's smallest pass — the largest arena level that lends, plus one, named on the startup line
`prefill staging: … a step of more than N tokens copies the pooled experts of …` — and under it
every routed layer reads its pooled experts synchronously: the vendor prices the whole job (four
decoders plus a long prompt) at **29.5 s with a 2048 chunk and 31.9 s at 512**, so shrinking the
chunk blindly buys latency in both directions.
- `--prefill-span N` — needs `--max-num-batched-tokens` of N intervals. A *lone* prefill then
carries N intervals a step (on an expert-tiered box only from 8 intervals, where the vendor's own
measurement says a span starts to win), while **every other step is planned at the shared
budget** — a lending arena level that also keeps the stager's ring, the one combination the chunk
knob cannot express. Priced by the vendor at 320 MiB a card of expert residency, ~3 % off the KV
pool, and TTFT-neutral on one of his two 2×R9700 boxes: measure your own prompt lengths first.
- `--max-num-batched-tokens` alone — the blunt rung, and the only one older pins have. It raises the
arena the expert slab loses, and on this payload its resonance is violent (MNBT 2560 collapses
decode — [[frank-archon]]).
Two traps. **The arena/slab coupling makes a shared step expensive twice:** the full-plan arena is
reclaimed from the expert slab *synchronously before the step*, and the loan regrows only after a
quiet interval, so every step of a prefill run — the decode-only ones included — runs on the smaller
slab. And **re-check `prefix_hit_rate` / `prefix_evictions` after any chunk or span change**: the
arena, the staging ring and the KV pool are the same memory, and the failure looks like turn two
taking exactly as long as turn one.
Availability is a version question, not a config question: `--busy-prefill-chunk`,
`--prefill-interleave` and `--prefill-span` are absent from older pins — prove it on the running
container (`podman exec <c> <binary> --help | grep -iE "prefill|chunk"`) or from `/diagnostics`'s
`configuration` block, never from a vendored tree, and check a source clone's depth before dating a
flag from `git log -S` (a `--depth 1` clone reports its own tip for every search).
Run on: [[frank-archon]] — owner of the live census, the measured step pair, and the arms.
## Related
serverAppConfigs-docs/LLM/frank-archon.md
+136 -36
View File
@@ -96,7 +96,7 @@ on stop. Boot budget 1800 s (the publisher's compose allows a 15-minute `start_p
| Spec / batch | `SPEC_TOKENS=2` — MTP depth 2, the `.rad` carries the model's own `mtp.*` head (`mtp.fc_hidden`/`fc_embedding`/`draft_head`); `MAX_BATCHED_TOKENS=2048` (MNBT — 2560 collapses decode, Laws) |
| Prefix cache | `PREFIX_CACHE_MIB=8192`, pinned **host tier only** — the 64 GiB disk tier is gone, so an evicted prefix is re-prefilled; `STATE_DIR=/home/dank/radiance-state` now exists only to hold the rad-tune cache |
| Template | `TEMPLATE=scripts/qwen38-froggeric-v22.5.jinja` (28,234 B) via `--override-chat-template` — froggeric's fixed Qwen3.8 template (the repo root's `chat_template.jinja`, vendored); the checkpoint's own template is baked into the `.rad` by rad-convert and this overrides it, and radiance derives the reasoning and tool-call reply format from it |
| Step sharing | `BUSY_PREFILL_CHUNK=auto` (a shared chunk reserves a row plus the draft window per waiting decoder) · `PREFILL_INTERLEAVE=0` — held at zero until the decode-vs-prefill buckets ask for it |
| Step sharing | `BUSY_PREFILL_CHUNK=auto` — prefill-speed first: it reserves 3 tokens per waiting decoder and ⅛ of the budget per waiting request, and does **not** shrink the chunk for decoders alone · `PREFILL_INTERLEAVE=0` — the decoder dial, held at zero until the §9 buckets ask for it |
| TTM gate | `TTM_MIN_MIB=150000`, `TTM_TARGET_MIB=204800` |
| Boot | `RAD_MIN_MIB=110000` deployer floor; the listener binds only after the full load (health ~100 s, budget 1800 s) |
@@ -280,30 +280,83 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
- **MNBT 2048.** 2560 collapses decode to 2.6-7.7 t/s (MTP resonance); the w4 file tier streams
in place only at MNBT ≤ 2048 and the activation arena scales with the chunk (32768 → 33.3
GiB, refused).
- **Decode pays for the chunk it shares a step with.** Census of the live engine taken at the
3-slot shape on 1.0.13 (`/stats` + `/metrics`, cumulative over the 2026-10-15 boot):
1,435,585 decode-only steps at **18.7 ms** each against 127,991 prefill+mixed steps
at **317 ms** — 60 % of engine wall-clock spent inside 8 % of the steps, because the
chunk is `min(remaining, checkpoint interval 2048, MNBT 2048)` = 2048 rows and every
decode row in that step waits the whole chunk. Day of Prometheus (120 s step, 721
samples): `decode_tokens_per_second` median **13.3 t/s** in the 56 % of samples where
`prefill_tokens_per_second` ≥ 1000, against 106-240 t/s in the light-prefill buckets;
bucketed by mean step time it is 128.7 t/s at 0.02-0.05 s steps and 12.0 t/s at
0.15-0.4 s. The stall is **during**, not after — decode in windows 2-14 min after a
busy-prefill window sits at the idle baseline (58-64 vs 77-85 t/s, the residual
confounded by co-scheduling), so no lingering expert-residency penalty at 2-minute
resolution, and `preemptions: 0` rules out preempt-and-re-prefill. Method:
[[LLM Performance Diagnosis]] §9. The law is shape-independent; the fix lives at 1.8.1 (next law),
and nobody has re-run the census since the migration — so the fix is ordered, **not verified**.
- **The knobs for it arrived with 1.8.1, and only one of them is on.** `--busy-prefill-chunk` caps the
shared chunk so a step with waiting decoders takes a smaller piece (live: `auto`, the engine picks;
the explicit value is 512), and `--prefill-interleave N` inserts N decode-only steps after a chunk
step taken beside decoders. `interleave` is deliberately **0** — the env comment holds it there until
the decode-vs-prefill buckets ask for it, and those buckets have not been read since the migration.
The vendor prices a 1024 chunk at about a fifth off prefill throughput to halve the stall; that is a
knob for choosing tomorrow's chunk, not a measurement of this shape. MNBT remains an arm and not an
edit: 2560 collapses decode through MTP resonance. Any chunk change re-scores draft acceptance before
it is trusted.
- **Decode pays for the chunk it shares a step with, and `--busy-prefill-chunk auto` does not
protect it.** Re-measured on the live 1.8.1 shape (34.8 h boot, Prometheus at 60 s,
`engine_step_seconds{,_decode}_total` / `engine_steps{,_decode,_mixed,_prefill}_total` /
`prefill_tokens_total`): decode-only steps run at **15.5 ms**, chunk steps (a chunk alone, or a
chunk with decode rows beside it) at **166 ms** — 4.7 % of the steps hold **34 % of the engine's
step time**, so a session decoding while another prompts advances one token per 166 ms. Bucket the
harness's `radiance:decode_tokens_per_second` by `radiance:prefill_tokens_per_second` (15 s
scrapes; per-stream rate = the gauge ÷ `vllm:num_requests_running`): median **166 t/s** under light
prefill, 81 t/s at 1-5k, **13 t/s above 5k** — a **12.7×** collapse, and at 13 t/s with two
sessions running each stream sits at 6.7 t/s = 149 ms a token, one token per shared step. Nothing
else moves: decode-*only* steps stay 14-18 ms in every bucket (14.1 ms in a prefill-free minute,
17.9 ms in the deep one) and the relationship is lag-0 only (r = −0.74 at lag 0, |r| ≤ 0.19 from
30 s on, decode back at baseline by 5 min) — residency is not the mechanism either. The second
victim is the **arriving** request: through a high-prefill window TTFT p90 goes 2.4 → **18.0 s**
and 15.5 % of requests wait over 10 s for their first token, with `http_queue_depth` at 2-3, while
median TTFT (~1.9 s) and per-token latency barely move. Preemption is real but burst-shaped: 264
`vllm:num_preemptions_total` in the boot, 108 of them inside 105 s of a 6.5k-tok/s prefill. The
chunk is `min(remaining, checkpoint interval, MNBT, busy_cap)`; the first three read 2048 here,
and `busy_cap` is the fourth law.
- **`auto` is a prefill-speed dial, not a decoder-protection dial.** `Scheduler::busy_cap` reserves
`1 + spec` rows per decoding sequence behind the chunk (3 tokens at MTP 2) plus one eighth of the
budget per *waiting request or waiting prefill*, and the code refuses to do more for decoders:
"not for decode rows alone: their reserve is a few tokens, so the chunk ends a quantum short and
the next step finishes the interval… Halving would hold a long prompt to half-interval chunks while
anything decodes behind it, which costs more" (`core/sched/scheduler.cpp:874-878`, 1.8.1). Live
proof: `prefill_chunks_capped` 5,018 against 62,386 chunk steps — 8 %, and those are the minutes
with a waiter, not the minutes with a decoder. An explicit `N` is the only value that applies
"whenever the chunk shares the step".
- **At MNBT 2048 no chunk value keeps both the arena loan and the staged copies.** The arena is a
ladder of levels (256 and 1024 rows here) and a step's rows are its **whole** token count —
`max(n_tok, n_seq × (1 + spec))`, `core/engine.cpp:1142` — so a 1024-token chunk with two decode
rows is a 1030-row step: it falls back to the full plan, which "lends nothing", and it arms the
prefill stager. The stager's floor is the largest lending level plus one, 1025 rows here — named on
the boot line `prefill staging: … a step of more than 1024 tokens copies the pooled experts of 48
routed layers onto the card up to 3 layers ahead` — and below it "every routed layer reads its
pooled experts synchronously and the prefill slows far more than sharing saves" (GUIDE §4.5). The
vendor's own number for naive shrinking, on this architecture: four decoders with a long prompt
beside them finished in **29.5 s at a 2048 chunk and 31.9 s at 512**.
- **A big step takes the arena from the expert slab and keeps it for the whole prefill.** The arena
lends 987.77 MiB (steps ≤ 256 rows) or 670.26 MiB (≤ 1024) to the slab; a full-plan step reclaims
it **synchronously before the step** — units revert to their host copies (an address change, not a
copy) and both read streams drain — and `arena_tick` regrows the loan only **2 s after the last big
step** (`core/engine.cpp:1129-1202`), so every step of the prefill run, decode-only included, runs
with a slab ~780 expert units smaller over a link also carrying the staged copies. `/diagnostics`
says the binding constraint in one line: "a prefill step streams every routed layer's pooled
experts onto the card, so its speed is bounded by the link; decode reads the ones it routes to".
Measured size of the whole effect: the ~10 % above. Real, small, not worth chasing.
- **`--prefill-interleave` is the dial that buys decode its cadence; it is off (`steps_held` 0).**
N decode-only steps follow any chunk step taken beside decoders; held steps admit nothing, so it
costs the prefilling prompt its TTFT and an arriving request its first step. Rate arithmetic at the
two measured step times and 0.657 acceptance (≈2.3 tokens a step): N=0 → ~14 t/s, N=2 → ~33,
N=4 → ~48, N=8 → ~69, with the prefill slowed by (166 + 15.5N)/166. **Arithmetic, not
measurement** — the arm is chosen by the §9 bucket pair plus `steps_held`, and `auto` is not a
substitute for it.
- **`--prefill-span` is the shape that keeps the loan, and it is priced.** It needs MNBT = N
intervals, so `--prefill-span 2 --max-num-batched-tokens 4096`: a *lone* prefill carries 2
intervals a step (and on an expert-tiered box only from 8 intervals / 16k tokens, where the span
starts to win), while **every other step is planned at the shared budget of 2048** — which is a
lending level *and* keeps the stager's ring, the one combination `--busy-prefill-chunk` cannot
express. Costs per GUIDE §4.5: 320 MiB a card of expert residency, ~3 % off the KV pool (§4.3's
cliff is sharp), a decode step uploads twice the per-step arrays at 4096, and checkpoints inside a
span are not written, so a partially-shared prompt resumes one interval earlier. On the vendor's
second box (2×R9700, w4nl64, TP2) it was TTFT-neutral — 2.61 vs 2.56 s at 18k, 10.72 vs 10.55 at
75k — hence their own rule: measure your own prompts at two ranks before turning it on. MNBT 4096
does not boot at 3 slots; this shape is 2.
- **Most of this endpoint's prefill work is replay of tokens the prefix cache already had.** Over
the same window `prefill_tokens_total` 65.2 M against `prefix_hit_cut_tokens_total` 24.9 M = **38 %
replayed**, and it is not an eviction: a hit on a hybrid model reuses only as far as the last
**linear-state** checkpoint (`reuse = min(h.n_attn_tokens, h.n_linear_tokens)`,
`core/sched/scheduler.cpp:953-978`), and the default `checkpoint policy geometric-backoff, base
2048` thins the snapshots deep in a prompt, so a hit at 150k can resume at 128k. The cause object
agrees — every `prefix_hit_cuts_total` carries `reason="no_snapshot"` (`windowed_group` and
`adopt_failed` are dead) — but the event count is small and the tokens are the cost, so read
`hit_cut_tokens`. An arm, not an edit:
`--checkpoint-policy floor` (or a shorter `--checkpoint-interval`) buys the replay back with
pinned-RAM checkpoint slots, and those slots come out of the same pinned host memory the expert
pool claims — a resolver fight, and `prefix_hit_cut_tokens_total` is the meter.
- **Slots cost KV, rows, and window — never speed.** `--max-num-seqs` carries no radiance constraint: it
adds ≈110 MiB of recurrent state per slot (the live `kv_full_backing` arithmetic prices them exactly),
declares the fused TP all-reduce over `seqs × (1 + spec)` rows, and `wht6` is bit-exact up to 25 rows
@@ -396,12 +449,35 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
4600 MT/s ended it, with zero `gdn_hip` faults either side; the prefix-caching / GDN re-seed
theory was confounded on both arms. Consumer non-ECC raises no MCE for a silent flip and a
60 s / 8 GiB pattern check covers 0.5 % of 246 GiB — silent RAM corruption outranks every
software theory.
software theory. **That mitigation is a BIOS setting and it is not in effect now:** `sudo
dmidecode -t memory | grep -i configured` answers 4800 MT/s on 2026-10-20, and the only event
between the fix and today is the 2026-10-12 power failure that also reset TTM. Check the speed
before crediting any software revert with a fix — and before believing a clean run of the arm
you just tested, because the confounder is still in the slot.
- **TP=2 here** is legal since the 2026-09-20 ProArt B850 swap (x8/x8 on CPU root ports,
`Routing+`/`ReqEn+` end to end) only because radiance's wire is `libr4d`'s own one-shot
all-reduce; RCCL needs atomics on the whole chain and fails `hipErrorIllegalState` without
them (). `wht6` buys +5.6-6.3 % prefill for ~1.3 % of 8-stream
decode, `exact` is the bit-exact alternative.
- **The link is already at the board's ceiling — do not chase the `link-below-max` warning.**
Both root ports (`00:01.1`, `00:01.3`) are **x8 max** at 32 GT/s, so PCIe 5.0 x8 ≈ 31.5 GB/s is
the whole budget per card and the cards' own `pcie_dpm` tables expose only x8 states.
`/diagnostics` still warns that the on-card switch hop (`01:00.0`, `05:00.0`) runs "x8 of its
x16 while the card is 100 % busy" and suggests a reboot or a reseat — pointless when the host
side of the chain is x8 max, which the same endpoint reports two findings later as
`link-capped`. The width that matters is the minimum of the chain, and it is x8 either way, so
the only lever on a link-bound prefill is **less traffic over the link** (more expert
residency, fewer pooled experts), never a wider link. P2P resolves **on** (`auto` asks the
driver, the two cards reach each other's memory), so the collectives do not hairpin through host
RAM, and the engine's copies ride one DMA queue that "give[s] way to the collectives" — the
all-reduce wins the link, the staged experts wait.
- **Eight cores is the whole host, media stack included.** Dankserver is a Ryzen 7 7700X (8
cores / 8 threads, one NUMA node, `nodes: -1` for both cards) running Jellyfin, Plex, Calibre
and RomM beside the engine, and `--http-threads` defaults to one thread per core. Measured
during a heavy prefill: load average ≈ 1.0, `jellyfin` 11 % CPU, `radiance` 7 % — **CPU
contention is not the current decode culprit**, so do not spend an experiment window on it;
re-read `uptime` and `ps -eo pcpu,comm --sort=-pcpu | head` first if a future collapse looks
different from the step-sharing one.
- `--prefix-cache-dir` creation is **not recursive** — the serve script `mkdir -p`s the path or
the disk tier silently never opens. The tier does survive a payload swap when arch and
tokenizer match (it recovered 900,176 blocks on the 2026-10-15 swap; `cache_n` proves a hit).
@@ -445,11 +521,19 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
## Contract (the wire)
- **Live surface:** `/health` (== `/ping`), `/v1/models`, `/v1/chat/completions`
`/v1/completions`, `/metrics`, `/stats`, `/sessions`, `/dashboard`. `/health` answers 200
JSON (model id, `in_flight`, `queue_depth`) while serving and **503 only while stopping**
(`core/server/admin.cpp:385`), and the listener binds *after* the full load (`main.cpp`:
`engine.start()` then `engine.serve()`) — during the load the port refuses connections, so a
bare `curl -fsS …/health` gate is honest and needs no body grep.
`/v1/completions`, `/metrics`, `/stats`, `/sessions`, `/dashboard`, **`/diagnostics`**. `/health`
answers 200 JSON (model id, `in_flight`, `queue_depth`) while serving and **503 only while
stopping** (`core/server/admin.cpp:385`), and the listener binds *after* the full load
(`main.cpp`: `engine.start()` then `engine.serve()`) — during the load the port refuses
connections, so a bare `curl -fsS …/health` gate is honest and needs no body grep.
- **`/diagnostics` is the instrument for "is the box, not the config, the limit"** — one call
returns `configuration` (every flag, so drift from the env file is checkable without the host),
`environment`, `derived` (the VRAM split as the ledger prints it), `hardware` (host, per-card
`link_chain` + `pcie_dpm` + clocks + power cap + DMA engine choice + P2P answer) and `findings`
(level / what / why / fix). On this box it reports `experts-from-host` (5.99 GiB a rank crosses
PCIe when used), `link-capped` (root ports are x8 max — the board), `swap-in-use`,
`tp-wire-lossy`, and **`link-below-max` as a `warn` at 100 % card busy on both cards** — which is
structural on this board; see the x8-ceiling law under *This host, and the deploy path*.
- **Served id.** `FRANK_ARCHON_SERVED_NAME=frank-archon` (the wire rename of 2026-10-09 moved
only the alias; container, unit, scripts and env keep the service name). The env's contract
line is that **radiance validates the requested model name, so every client — harness, bench
@@ -476,12 +560,28 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
and 16,384 patches per pass. Probe with a ≥ 256 px solid-colour PNG and a ≥ 256 token budget
— at 64 tokens you get empty `content` and the colour only inside `reasoning_content`.
- **Capacity behaviour:** a prompt over `--max-model-len` is refused, never truncated; a request
longer than the KV pool's token count is accepted then refused as too large to fit; under the
3-slot contract a **fourth** full-height conversation is refused, not paged or squeezed.
longer than the KV pool's token count is accepted then refused as too large to fit; a request
beyond `--max-num-seqs` is held by the HTTP port (`queue_cap`, default 8 × slots = 16 here) and
answered 503 only past that, never paged or squeezed. Because the cache funds **1.8** full-height
sessions, admitting a second near-full-height conversation can **preempt a running sequence from
the back** to free its blocks (`core/sched/scheduler.cpp:1291-1322`); the victim is recomputed, so
its session halts and then re-prefills. Meter: `vllm:num_preemptions_total` (264 in the current
boot, all in prefill-coincident bursts — 108 inside 105 s); `/stats preemptions` is the cumulative
count and `radiance:num_requests_preempted` only the instant depth.
- **Observability:** `/metrics` carries vLLM-compatible names plus `radiance:*`, scraped by the
Dankclaw Prometheus job at `192.168.0.235:7770` over plain LAN routing (the netavark
published-port quirk is same-host only). The boot ledger — the referee for every shape number
above — is in the container log (`podman logs frank-archon`)
Dankclaw Prometheus job at `192.168.0.235:7770` (15 s interval) over plain LAN routing (the
netavark published-port quirk is same-host only). **Latency lives under `vllm:`, cadence under
`radiance:`** — TTFT / inter-token / per-output-token / e2e / decode-time histograms and the
running/waiting gauges are `vllm:*` with no radiance-native equivalent, so a `radiance:`-only
query misses every latency answer. The step-phase counters are all scraped:
`engine_steps{,_decode,_mixed,_prefill,_held}_total`, `engine_step_seconds{,_decode}_total`,
`engine_prefill_chunks_capped_total`, `prefill_tokens_total`, `decode_tokens_per_second`,
`prefill_tokens_per_second`, `prefix_hit_cut_tokens_total`, `http_queue_depth`. Two holes: the
engine's own step-detail histograms (`prefill_steps_duration`, `prefill_step_tokens`,
`token_queue_wait`) are exported but reach no target, and the **arena / expert slab / heat engine /
prefill stager have no metric at all** — that half of every law above is read from the boot ledger,
`/stats` and `/diagnostics`, never from a graph. The boot ledger — the referee for every shape
number here — is in the container log (`podman logs frank-archon`);
`/home/dank/logs/frank_archon.log` is where the serve script redirects `podman run`.
## Files