diff --git a/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md b/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md index 3ba2114..4457cc3 100644 --- a/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md +++ b/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md @@ -239,45 +239,90 @@ prefill is over. `preemptions` separates the third shape: a preempted-and-recomp sequence stops producing entirely and then re-prefills from token 0, which is a stall, not sludge. -Knobs, in the order worth trying them (radiance GUIDE §4.5): +**Split the step ledger before choosing a knob.** Three counters decide which mechanism you are +looking at, and none is a rate: `engine_step_seconds_decode_total / engine_steps_decode_total` is +the mean **decode-only** step, and `(step_seconds_total − step_seconds_decode_total) / +(steps_total − steps_decode_total)` is the mean **chunk step** (a chunk alone, or a chunk with +decode rows beside it). On frank-archon's live 1.8.1 shape that pair is **15.5 ms against 166 ms**, +with 4.7 % of the steps holding 34 % of the step time. Decode-only steps themselves move only +14.1 → 16.4 ms between a prefill-free minute and a heavy one, and the lagged buckets stay flat: so +a decode-rate collapse on that box is **step time sharing**, and the eviction/residency story is +worth ~10 % of one step — not a lever. -- `--prefill-interleave N` — after a chunk step taken beside decoders, the next N - steps carry decode rows only and admit nothing. Decoders keep their cadence; the - prompt pays that in TTFT. Counts as `engine_steps_held_total`. -- `--busy-prefill-chunk MODE` — the chunk's share of a step that has other work: - `auto` (default) reserves a row plus the draft window per decode row behind it and - an eighth of the budget per waiting prefill/request, `off` lets the chunk take the - whole step (admission never runs while it does), `N` pins the share. Never under - the prefill stager's smallest pass — the startup line names it, and below it every - routed layer reads its pooled experts synchronously, which costs the prefill far - more than sharing saves. Counts as `engine_prefill_chunks_capped_total`. -- `--max-num-batched-tokens` — the blunt rung, and the only one older pins have. The vendor's own - curve prices **halving the shared chunk at roughly a fifth of prefill throughput** (a 48-layer MoE - at 200k on two cards — knob pricing, not this endpoint's number, so treat it as the order of - magnitude a chunk decision costs). Raising it past the checkpoint interval buys nothing and loses - expert residency to the arena. -- `--prefill-span N` — the inverse trade: a *lone* prefill carries N intervals a step while every - shared step is planned at a smaller shared budget, which on an expert-tiered box lends that arena - back to the expert slab. The vendor's own measurement says the win is **long-prompt-only**: nothing - at short prompts, a large TTFT gain at the far end of the window. +**Four traps in the PromQL above.** Learn them before trusting a bucket: +- **Latency is `vllm:`, cadence is `radiance:`.** The TTFT / inter-token / per-output-token / e2e + histograms and the running/waiting gauges exist only under the vLLM compatibility prefix; preemption + is `vllm:num_preemptions_total` (the counter) while `radiance:num_requests_preempted` is the + *instant* depth. A `radiance:`-only query finds no latency and no preemption, and reads as "the + endpoint never stalls anyone". +- **The tps gauges are 1 s decayed windows, batch-pooled.** Per-stream rate is + `decode_tokens_per_second / vllm:num_requests_running`. With a 15 s scrape, a 120 s step samples one + scrape in eight — use step ≤ 30 s for a bucket table. +- **Every counter resets at a restart and the metric set changes with the build.** Prove the window is + one shape (`up`, or the first sample of a counter that only the current build exports) before + ranking buckets; frank-archon's 48 h window straddled two containers, and one bucket's "collapse" + turned out to belong entirely to the older one. +- **A 2-minute mean step time dilutes the chunk.** The sharp variable is the phase ratio in the query + above, never `step_seconds/steps` — 4.7 % of steps cannot move a mean that is 95 % decode. -Two traps. **The arena/slab coupling makes a shared step doubly expensive on an -expert-tiered shape**: the full-plan arena takes VRAM the resident expert slab would have had, so the -decode rows inside that step also read experts over the link — a smaller shared budget helps twice. -And **re-check -`prefix_hit_rate` / `prefix_evictions` after any chunk change**: the arena and the KV -pool are the same memory, and the failure looks like turn two taking as long as turn -one. +Three instruments the PromQL above cannot reach: -Availability is a version question, not a config question: `--busy-prefill-chunk` and -`--prefill-interleave` are absent from older pins — prove it on the running container -(`podman exec --help | grep -iE "prefill|chunk"`), never from the vendored -tree, and check the source clone's depth before dating a flag from `git log -S` -(a `--depth 1` clone reports its own tip for every search). +- **The engine's own step-detail histograms (`prefill_steps_duration`, `prefill_step_tokens`, + `token_queue_wait`) are exported but reach no target**, and the arena / expert slab / heat engine / + prefill stager have **no metric at all**. Per-request forensics mean `curl localhost:7770/stats` on + the endpoint's own host — same phase counters plus `preemptions` (the count), the chunk geometry, + and the expert/link byte totals. `/diagnostics` answers "is the *box* the limit" in one call: + configuration drift, the VRAM split, each card's link chain and clocks, findings with fixes. +- **`prefix_hit_cut_tokens_total` measures prefill load you created for yourself.** On a hybrid + (linear + paged) model a prefix hit reuses only as far as the last **linear-state** checkpoint, so + cached tokens are replayed as prefill and stall decoders for it; frank-archon ran 38 % of all its + prefill tokens that way, every `prefix_hit_cuts_total` labelled `reason="no_snapshot"`. A high + prefill rate is not always a client that sends new tokens. +- **Bucket the arriving request too** — `histogram_quantile` over + `vllm:time_to_first_token_seconds_bucket` by the same prefill series. The decode gauge only sees + sequences that are already running; the one waiting for its first token pays the same steps and + shows in no decode number. -Run on: [[frank-archon]] — the census was taken on its 3-slot radiance 1.0.13 era (budget 2048 = -interval 2048); the live 1.8.1 shape carries the two chunk knobs. The measured census and the verdict -for this fleet's shape are that page's. +Knobs, in the order worth trying them (radiance GUIDE §4.5; the pinned source is the referee): + +- `--prefill-interleave N` — after a chunk step taken beside decoders, the next N steps carry decode + rows only and admit nothing. **The knob that acts on the measured mechanism.** Decoders keep their + cadence; the prefilling prompt pays in TTFT and an arriving request waits out the held steps. + Counts as `engine_steps_held_total` — if that counter is zero, nothing is protecting anyone. +- `--busy-prefill-chunk N` — pins a shared chunk's share of the step. **`auto` is a prefill-speed + setting, not decoder protection:** it reserves `1 + spec` rows per decoding sequence plus an eighth + of the budget per *waiting* request or prefill, and the source refuses to halve for "decode rows + alone", so a shape can run `auto` and still lose every chunk step. Prove engagement from + `engine_prefill_chunks_capped_total` against the chunk-step count. The floor is the prefill + stager's smallest pass — the largest arena level that lends, plus one, named on the startup line + `prefill staging: … a step of more than N tokens copies the pooled experts of …` — and under it + every routed layer reads its pooled experts synchronously: the vendor prices the whole job (four + decoders plus a long prompt) at **29.5 s with a 2048 chunk and 31.9 s at 512**, so shrinking the + chunk blindly buys latency in both directions. +- `--prefill-span N` — needs `--max-num-batched-tokens` of N intervals. A *lone* prefill then + carries N intervals a step (on an expert-tiered box only from 8 intervals, where the vendor's own + measurement says a span starts to win), while **every other step is planned at the shared + budget** — a lending arena level that also keeps the stager's ring, the one combination the chunk + knob cannot express. Priced by the vendor at 320 MiB a card of expert residency, ~3 % off the KV + pool, and TTFT-neutral on one of his two 2×R9700 boxes: measure your own prompt lengths first. +- `--max-num-batched-tokens` alone — the blunt rung, and the only one older pins have. It raises the + arena the expert slab loses, and on this payload its resonance is violent (MNBT 2560 collapses + decode — [[frank-archon]]). + +Two traps. **The arena/slab coupling makes a shared step expensive twice:** the full-plan arena is +reclaimed from the expert slab *synchronously before the step*, and the loan regrows only after a +quiet interval, so every step of a prefill run — the decode-only ones included — runs on the smaller +slab. And **re-check `prefix_hit_rate` / `prefix_evictions` after any chunk or span change**: the +arena, the staging ring and the KV pool are the same memory, and the failure looks like turn two +taking exactly as long as turn one. + +Availability is a version question, not a config question: `--busy-prefill-chunk`, +`--prefill-interleave` and `--prefill-span` are absent from older pins — prove it on the running +container (`podman exec --help | grep -iE "prefill|chunk"`) or from `/diagnostics`'s +`configuration` block, never from a vendored tree, and check a source clone's depth before dating a +flag from `git log -S` (a `--depth 1` clone reports its own tip for every search). + +Run on: [[frank-archon]] — owner of the live census, the measured step pair, and the arms. ## Related diff --git a/serverAppConfigs-docs/LLM/frank-archon.md b/serverAppConfigs-docs/LLM/frank-archon.md index 6c5a3ab..a6d7f2f 100644 --- a/serverAppConfigs-docs/LLM/frank-archon.md +++ b/serverAppConfigs-docs/LLM/frank-archon.md @@ -96,7 +96,7 @@ on stop. Boot budget 1800 s (the publisher's compose allows a 15-minute `start_p | Spec / batch | `SPEC_TOKENS=2` — MTP depth 2, the `.rad` carries the model's own `mtp.*` head (`mtp.fc_hidden`/`fc_embedding`/`draft_head`); `MAX_BATCHED_TOKENS=2048` (MNBT — 2560 collapses decode, Laws) | | Prefix cache | `PREFIX_CACHE_MIB=8192`, pinned **host tier only** — the 64 GiB disk tier is gone, so an evicted prefix is re-prefilled; `STATE_DIR=/home/dank/radiance-state` now exists only to hold the rad-tune cache | | Template | `TEMPLATE=scripts/qwen38-froggeric-v22.5.jinja` (28,234 B) via `--override-chat-template` — froggeric's fixed Qwen3.8 template (the repo root's `chat_template.jinja`, vendored); the checkpoint's own template is baked into the `.rad` by rad-convert and this overrides it, and radiance derives the reasoning and tool-call reply format from it | -| Step sharing | `BUSY_PREFILL_CHUNK=auto` (a shared chunk reserves a row plus the draft window per waiting decoder) · `PREFILL_INTERLEAVE=0` — held at zero until the decode-vs-prefill buckets ask for it | +| Step sharing | `BUSY_PREFILL_CHUNK=auto` — prefill-speed first: it reserves 3 tokens per waiting decoder and ⅛ of the budget per waiting request, and does **not** shrink the chunk for decoders alone · `PREFILL_INTERLEAVE=0` — the decoder dial, held at zero until the §9 buckets ask for it | | TTM gate | `TTM_MIN_MIB=150000`, `TTM_TARGET_MIB=204800` | | Boot | `RAD_MIN_MIB=110000` deployer floor; the listener binds only after the full load (health ~100 s, budget 1800 s) | @@ -280,30 +280,83 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10). - **MNBT 2048.** 2560 collapses decode to 2.6-7.7 t/s (MTP resonance); the w4 file tier streams in place only at MNBT ≤ 2048 and the activation arena scales with the chunk (32768 → 33.3 GiB, refused). -- **Decode pays for the chunk it shares a step with.** Census of the live engine taken at the - 3-slot shape on 1.0.13 (`/stats` + `/metrics`, cumulative over the 2026-10-15 boot): - 1,435,585 decode-only steps at **18.7 ms** each against 127,991 prefill+mixed steps - at **317 ms** — 60 % of engine wall-clock spent inside 8 % of the steps, because the - chunk is `min(remaining, checkpoint interval 2048, MNBT 2048)` = 2048 rows and every - decode row in that step waits the whole chunk. Day of Prometheus (120 s step, 721 - samples): `decode_tokens_per_second` median **13.3 t/s** in the 56 % of samples where - `prefill_tokens_per_second` ≥ 1000, against 106-240 t/s in the light-prefill buckets; - bucketed by mean step time it is 128.7 t/s at 0.02-0.05 s steps and 12.0 t/s at - 0.15-0.4 s. The stall is **during**, not after — decode in windows 2-14 min after a - busy-prefill window sits at the idle baseline (58-64 vs 77-85 t/s, the residual - confounded by co-scheduling), so no lingering expert-residency penalty at 2-minute - resolution, and `preemptions: 0` rules out preempt-and-re-prefill. Method: - [[LLM Performance Diagnosis]] §9. The law is shape-independent; the fix lives at 1.8.1 (next law), - and nobody has re-run the census since the migration — so the fix is ordered, **not verified**. -- **The knobs for it arrived with 1.8.1, and only one of them is on.** `--busy-prefill-chunk` caps the - shared chunk so a step with waiting decoders takes a smaller piece (live: `auto`, the engine picks; - the explicit value is 512), and `--prefill-interleave N` inserts N decode-only steps after a chunk - step taken beside decoders. `interleave` is deliberately **0** — the env comment holds it there until - the decode-vs-prefill buckets ask for it, and those buckets have not been read since the migration. - The vendor prices a 1024 chunk at about a fifth off prefill throughput to halve the stall; that is a - knob for choosing tomorrow's chunk, not a measurement of this shape. MNBT remains an arm and not an - edit: 2560 collapses decode through MTP resonance. Any chunk change re-scores draft acceptance before - it is trusted. +- **Decode pays for the chunk it shares a step with, and `--busy-prefill-chunk auto` does not + protect it.** Re-measured on the live 1.8.1 shape (34.8 h boot, Prometheus at 60 s, + `engine_step_seconds{,_decode}_total` / `engine_steps{,_decode,_mixed,_prefill}_total` / + `prefill_tokens_total`): decode-only steps run at **15.5 ms**, chunk steps (a chunk alone, or a + chunk with decode rows beside it) at **166 ms** — 4.7 % of the steps hold **34 % of the engine's + step time**, so a session decoding while another prompts advances one token per 166 ms. Bucket the + harness's `radiance:decode_tokens_per_second` by `radiance:prefill_tokens_per_second` (15 s + scrapes; per-stream rate = the gauge ÷ `vllm:num_requests_running`): median **166 t/s** under light + prefill, 81 t/s at 1-5k, **13 t/s above 5k** — a **12.7×** collapse, and at 13 t/s with two + sessions running each stream sits at 6.7 t/s = 149 ms a token, one token per shared step. Nothing + else moves: decode-*only* steps stay 14-18 ms in every bucket (14.1 ms in a prefill-free minute, + 17.9 ms in the deep one) and the relationship is lag-0 only (r = −0.74 at lag 0, |r| ≤ 0.19 from + 30 s on, decode back at baseline by 5 min) — residency is not the mechanism either. The second + victim is the **arriving** request: through a high-prefill window TTFT p90 goes 2.4 → **18.0 s** + and 15.5 % of requests wait over 10 s for their first token, with `http_queue_depth` at 2-3, while + median TTFT (~1.9 s) and per-token latency barely move. Preemption is real but burst-shaped: 264 + `vllm:num_preemptions_total` in the boot, 108 of them inside 105 s of a 6.5k-tok/s prefill. The + chunk is `min(remaining, checkpoint interval, MNBT, busy_cap)`; the first three read 2048 here, + and `busy_cap` is the fourth law. +- **`auto` is a prefill-speed dial, not a decoder-protection dial.** `Scheduler::busy_cap` reserves + `1 + spec` rows per decoding sequence behind the chunk (3 tokens at MTP 2) plus one eighth of the + budget per *waiting request or waiting prefill*, and the code refuses to do more for decoders: + "not for decode rows alone: their reserve is a few tokens, so the chunk ends a quantum short and + the next step finishes the interval… Halving would hold a long prompt to half-interval chunks while + anything decodes behind it, which costs more" (`core/sched/scheduler.cpp:874-878`, 1.8.1). Live + proof: `prefill_chunks_capped` 5,018 against 62,386 chunk steps — 8 %, and those are the minutes + with a waiter, not the minutes with a decoder. An explicit `N` is the only value that applies + "whenever the chunk shares the step". +- **At MNBT 2048 no chunk value keeps both the arena loan and the staged copies.** The arena is a + ladder of levels (256 and 1024 rows here) and a step's rows are its **whole** token count — + `max(n_tok, n_seq × (1 + spec))`, `core/engine.cpp:1142` — so a 1024-token chunk with two decode + rows is a 1030-row step: it falls back to the full plan, which "lends nothing", and it arms the + prefill stager. The stager's floor is the largest lending level plus one, 1025 rows here — named on + the boot line `prefill staging: … a step of more than 1024 tokens copies the pooled experts of 48 + routed layers onto the card up to 3 layers ahead` — and below it "every routed layer reads its + pooled experts synchronously and the prefill slows far more than sharing saves" (GUIDE §4.5). The + vendor's own number for naive shrinking, on this architecture: four decoders with a long prompt + beside them finished in **29.5 s at a 2048 chunk and 31.9 s at 512**. +- **A big step takes the arena from the expert slab and keeps it for the whole prefill.** The arena + lends 987.77 MiB (steps ≤ 256 rows) or 670.26 MiB (≤ 1024) to the slab; a full-plan step reclaims + it **synchronously before the step** — units revert to their host copies (an address change, not a + copy) and both read streams drain — and `arena_tick` regrows the loan only **2 s after the last big + step** (`core/engine.cpp:1129-1202`), so every step of the prefill run, decode-only included, runs + with a slab ~780 expert units smaller over a link also carrying the staged copies. `/diagnostics` + says the binding constraint in one line: "a prefill step streams every routed layer's pooled + experts onto the card, so its speed is bounded by the link; decode reads the ones it routes to". + Measured size of the whole effect: the ~10 % above. Real, small, not worth chasing. +- **`--prefill-interleave` is the dial that buys decode its cadence; it is off (`steps_held` 0).** + N decode-only steps follow any chunk step taken beside decoders; held steps admit nothing, so it + costs the prefilling prompt its TTFT and an arriving request its first step. Rate arithmetic at the + two measured step times and 0.657 acceptance (≈2.3 tokens a step): N=0 → ~14 t/s, N=2 → ~33, + N=4 → ~48, N=8 → ~69, with the prefill slowed by (166 + 15.5N)/166. **Arithmetic, not + measurement** — the arm is chosen by the §9 bucket pair plus `steps_held`, and `auto` is not a + substitute for it. +- **`--prefill-span` is the shape that keeps the loan, and it is priced.** It needs MNBT = N + intervals, so `--prefill-span 2 --max-num-batched-tokens 4096`: a *lone* prefill carries 2 + intervals a step (and on an expert-tiered box only from 8 intervals / 16k tokens, where the span + starts to win), while **every other step is planned at the shared budget of 2048** — which is a + lending level *and* keeps the stager's ring, the one combination `--busy-prefill-chunk` cannot + express. Costs per GUIDE §4.5: 320 MiB a card of expert residency, ~3 % off the KV pool (§4.3's + cliff is sharp), a decode step uploads twice the per-step arrays at 4096, and checkpoints inside a + span are not written, so a partially-shared prompt resumes one interval earlier. On the vendor's + second box (2×R9700, w4nl64, TP2) it was TTFT-neutral — 2.61 vs 2.56 s at 18k, 10.72 vs 10.55 at + 75k — hence their own rule: measure your own prompts at two ranks before turning it on. MNBT 4096 + does not boot at 3 slots; this shape is 2. +- **Most of this endpoint's prefill work is replay of tokens the prefix cache already had.** Over + the same window `prefill_tokens_total` 65.2 M against `prefix_hit_cut_tokens_total` 24.9 M = **38 % + replayed**, and it is not an eviction: a hit on a hybrid model reuses only as far as the last + **linear-state** checkpoint (`reuse = min(h.n_attn_tokens, h.n_linear_tokens)`, + `core/sched/scheduler.cpp:953-978`), and the default `checkpoint policy geometric-backoff, base + 2048` thins the snapshots deep in a prompt, so a hit at 150k can resume at 128k. The cause object + agrees — every `prefix_hit_cuts_total` carries `reason="no_snapshot"` (`windowed_group` and + `adopt_failed` are dead) — but the event count is small and the tokens are the cost, so read + `hit_cut_tokens`. An arm, not an edit: + `--checkpoint-policy floor` (or a shorter `--checkpoint-interval`) buys the replay back with + pinned-RAM checkpoint slots, and those slots come out of the same pinned host memory the expert + pool claims — a resolver fight, and `prefix_hit_cut_tokens_total` is the meter. - **Slots cost KV, rows, and window — never speed.** `--max-num-seqs` carries no radiance constraint: it adds ≈110 MiB of recurrent state per slot (the live `kv_full_backing` arithmetic prices them exactly), declares the fused TP all-reduce over `seqs × (1 + spec)` rows, and `wht6` is bit-exact up to 25 rows @@ -396,12 +449,35 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10). 4600 MT/s ended it, with zero `gdn_hip` faults either side; the prefix-caching / GDN re-seed theory was confounded on both arms. Consumer non-ECC raises no MCE for a silent flip and a 60 s / 8 GiB pattern check covers 0.5 % of 246 GiB — silent RAM corruption outranks every - software theory. + software theory. **That mitigation is a BIOS setting and it is not in effect now:** `sudo + dmidecode -t memory | grep -i configured` answers 4800 MT/s on 2026-10-20, and the only event + between the fix and today is the 2026-10-12 power failure that also reset TTM. Check the speed + before crediting any software revert with a fix — and before believing a clean run of the arm + you just tested, because the confounder is still in the slot. - **TP=2 here** is legal since the 2026-09-20 ProArt B850 swap (x8/x8 on CPU root ports, `Routing+`/`ReqEn+` end to end) only because radiance's wire is `libr4d`'s own one-shot all-reduce; RCCL needs atomics on the whole chain and fails `hipErrorIllegalState` without them (). `wht6` buys +5.6-6.3 % prefill for ~1.3 % of 8-stream decode, `exact` is the bit-exact alternative. +- **The link is already at the board's ceiling — do not chase the `link-below-max` warning.** + Both root ports (`00:01.1`, `00:01.3`) are **x8 max** at 32 GT/s, so PCIe 5.0 x8 ≈ 31.5 GB/s is + the whole budget per card and the cards' own `pcie_dpm` tables expose only x8 states. + `/diagnostics` still warns that the on-card switch hop (`01:00.0`, `05:00.0`) runs "x8 of its + x16 while the card is 100 % busy" and suggests a reboot or a reseat — pointless when the host + side of the chain is x8 max, which the same endpoint reports two findings later as + `link-capped`. The width that matters is the minimum of the chain, and it is x8 either way, so + the only lever on a link-bound prefill is **less traffic over the link** (more expert + residency, fewer pooled experts), never a wider link. P2P resolves **on** (`auto` asks the + driver, the two cards reach each other's memory), so the collectives do not hairpin through host + RAM, and the engine's copies ride one DMA queue that "give[s] way to the collectives" — the + all-reduce wins the link, the staged experts wait. +- **Eight cores is the whole host, media stack included.** Dankserver is a Ryzen 7 7700X (8 + cores / 8 threads, one NUMA node, `nodes: -1` for both cards) running Jellyfin, Plex, Calibre + and RomM beside the engine, and `--http-threads` defaults to one thread per core. Measured + during a heavy prefill: load average ≈ 1.0, `jellyfin` 11 % CPU, `radiance` 7 % — **CPU + contention is not the current decode culprit**, so do not spend an experiment window on it; + re-read `uptime` and `ps -eo pcpu,comm --sort=-pcpu | head` first if a future collapse looks + different from the step-sharing one. - `--prefix-cache-dir` creation is **not recursive** — the serve script `mkdir -p`s the path or the disk tier silently never opens. The tier does survive a payload swap when arch and tokenizer match (it recovered 900,176 blocks on the 2026-10-15 swap; `cache_n` proves a hit). @@ -445,11 +521,19 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10). ## Contract (the wire) - **Live surface:** `/health` (== `/ping`), `/v1/models`, `/v1/chat/completions` - `/v1/completions`, `/metrics`, `/stats`, `/sessions`, `/dashboard`. `/health` answers 200 - JSON (model id, `in_flight`, `queue_depth`) while serving and **503 only while stopping** - (`core/server/admin.cpp:385`), and the listener binds *after* the full load (`main.cpp`: - `engine.start()` then `engine.serve()`) — during the load the port refuses connections, so a - bare `curl -fsS …/health` gate is honest and needs no body grep. + `/v1/completions`, `/metrics`, `/stats`, `/sessions`, `/dashboard`, **`/diagnostics`**. `/health` + answers 200 JSON (model id, `in_flight`, `queue_depth`) while serving and **503 only while + stopping** (`core/server/admin.cpp:385`), and the listener binds *after* the full load + (`main.cpp`: `engine.start()` then `engine.serve()`) — during the load the port refuses + connections, so a bare `curl -fsS …/health` gate is honest and needs no body grep. +- **`/diagnostics` is the instrument for "is the box, not the config, the limit"** — one call + returns `configuration` (every flag, so drift from the env file is checkable without the host), + `environment`, `derived` (the VRAM split as the ledger prints it), `hardware` (host, per-card + `link_chain` + `pcie_dpm` + clocks + power cap + DMA engine choice + P2P answer) and `findings` + (level / what / why / fix). On this box it reports `experts-from-host` (5.99 GiB a rank crosses + PCIe when used), `link-capped` (root ports are x8 max — the board), `swap-in-use`, + `tp-wire-lossy`, and **`link-below-max` as a `warn` at 100 % card busy on both cards** — which is + structural on this board; see the x8-ceiling law under *This host, and the deploy path*. - **Served id.** `FRANK_ARCHON_SERVED_NAME=frank-archon` (the wire rename of 2026-10-09 moved only the alias; container, unit, scripts and env keep the service name). The env's contract line is that **radiance validates the requested model name, so every client — harness, bench @@ -476,12 +560,28 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10). and 16,384 patches per pass. Probe with a ≥ 256 px solid-colour PNG and a ≥ 256 token budget — at 64 tokens you get empty `content` and the colour only inside `reasoning_content`. - **Capacity behaviour:** a prompt over `--max-model-len` is refused, never truncated; a request - longer than the KV pool's token count is accepted then refused as too large to fit; under the - 3-slot contract a **fourth** full-height conversation is refused, not paged or squeezed. + longer than the KV pool's token count is accepted then refused as too large to fit; a request + beyond `--max-num-seqs` is held by the HTTP port (`queue_cap`, default 8 × slots = 16 here) and + answered 503 only past that, never paged or squeezed. Because the cache funds **1.8** full-height + sessions, admitting a second near-full-height conversation can **preempt a running sequence from + the back** to free its blocks (`core/sched/scheduler.cpp:1291-1322`); the victim is recomputed, so + its session halts and then re-prefills. Meter: `vllm:num_preemptions_total` (264 in the current + boot, all in prefill-coincident bursts — 108 inside 105 s); `/stats preemptions` is the cumulative + count and `radiance:num_requests_preempted` only the instant depth. - **Observability:** `/metrics` carries vLLM-compatible names plus `radiance:*`, scraped by the - Dankclaw Prometheus job at `192.168.0.235:7770` over plain LAN routing (the netavark - published-port quirk is same-host only). The boot ledger — the referee for every shape number - above — is in the container log (`podman logs frank-archon`) + Dankclaw Prometheus job at `192.168.0.235:7770` (15 s interval) over plain LAN routing (the + netavark published-port quirk is same-host only). **Latency lives under `vllm:`, cadence under + `radiance:`** — TTFT / inter-token / per-output-token / e2e / decode-time histograms and the + running/waiting gauges are `vllm:*` with no radiance-native equivalent, so a `radiance:`-only + query misses every latency answer. The step-phase counters are all scraped: + `engine_steps{,_decode,_mixed,_prefill,_held}_total`, `engine_step_seconds{,_decode}_total`, + `engine_prefill_chunks_capped_total`, `prefill_tokens_total`, `decode_tokens_per_second`, + `prefill_tokens_per_second`, `prefix_hit_cut_tokens_total`, `http_queue_depth`. Two holes: the + engine's own step-detail histograms (`prefill_steps_duration`, `prefill_step_tokens`, + `token_queue_wait`) are exported but reach no target, and the **arena / expert slab / heat engine / + prefill stager have no metric at all** — that half of every law above is read from the boot ledger, + `/stats` and `/diagnostics`, never from a graph. The boot ledger — the referee for every shape + number here — is in the container log (`podman logs frank-archon`); `/home/dank/logs/frank_archon.log` is where the serve script redirects `podman run`. ## Files