From 687dbfbabe34b2fb87bfebd342fb01574c63a835 Mon Sep 17 00:00:00 2001 From: Frank Date: Tue, 20 Oct 2026 22:07:55 -0700 Subject: [PATCH 1/4] docs(scripts): correct frank-archon's step-sharing and slot-count comments The env comment claimed BUSY_PREFILL_CHUNK=auto protects decode rows; the 1.8.1 source (core/sched/scheduler.cpp busy_cap) reserves 1+spec rows per decoding sequence and 1/8 of the budget per waiting request, and refuses to shrink the chunk for decoders alone -- auto is a prefill-speed dial. The interleave comment carried a 512 that is not this build's auto value (0 until the held-step counter asks; 64 in help mode). The serve header still described three decode slots; MAX_SEQS has been 2 since the 2026-10-19 reslice. No behavior change. --- scripts/frank-archon.env | 23 +++++++++++++++-------- scripts/serve-frank-archon.sh | 17 ++++++++++------- 2 files changed, 25 insertions(+), 15 deletions(-) diff --git a/scripts/frank-archon.env b/scripts/frank-archon.env index 287372c..65ac22c 100644 --- a/scripts/frank-archon.env +++ b/scripts/frank-archon.env @@ -93,14 +93,21 @@ FRANK_ARCHON_SPEC_TOKENS=2 FRANK_ARCHON_MAX_BATCHED_TOKENS=2048 # Step sharing. A prefill chunk that shares a step with decoding sequences holds # the device for the chunk's whole time, and every decode row in that step waits -# that long for its next token (the census: [[LLM Performance Diagnosis]] §9). -# `auto` leaves each decode row behind the chunk a row plus its draft window and -# keeps a share for whatever is waiting to be admitted; a number pins the chunk's -# cap; `off` is the pre-1.4 behaviour this endpoint ran for two eras, where the -# chunk took the step and admission never ran. Interleave inserts N decode-only -# steps after a chunk step taken beside decoders: it protects their cadence and -# holds a waiting request out of those steps, so it stays 0 until the post-migration -# decode-vs-prefill buckets say it is wanted. +# that long for its next token: measured at this shape, decode-only steps 15.5 ms +# against chunk steps 166 ms, which is the harness's ~146 t/s falling to ~11 t/s +# while another session prefills (law and census: [[frank-archon]]). +# `auto` is a PREFILL-SPEED dial, not a decoder dial: it reserves one row plus the +# draft window per decoding sequence (3 tokens) and an eighth of the budget per +# waiting request or waiting prefill, and the source states it will not halve for +# decode rows alone. Live proof: prefill_chunks_capped 5,018 across 62,386 chunk +# steps. A number is the only value that applies whenever the chunk shares a step, +# and it must not go under the prefill stager's smallest pass -- 1025 rows here, +# named on the boot line "prefill staging: ... a step of more than 1024 tokens" -- +# because below it every routed layer reads its pooled experts synchronously, so +# 512 is a trap and the vendor measures the whole job slower there. Interleave +# inserts N decode-only steps after a chunk step taken beside decoders: it protects +# their cadence and holds a waiting request out of those steps. That is the decoder +# dial, and it is 0 until the decode-vs-prefill buckets ask for it. FRANK_ARCHON_BUSY_PREFILL_CHUNK=auto FRANK_ARCHON_PREFILL_INTERLEAVE=0 # Prefix cache: pinned host memory only. The disk tier was removed outright diff --git a/scripts/serve-frank-archon.sh b/scripts/serve-frank-archon.sh index 2cf39f3..d56fbdc 100755 --- a/scripts/serve-frank-archon.sh +++ b/scripts/serve-frank-archon.sh @@ -15,13 +15,16 @@ # log states every budget before anything is allocated (weights vs KV # split, the expert host pool's coverage of the worst case, the PLE pin, # the P2P answer per pair) — read it after every resize. -# Deployed shape (2026-10-19, radiance 1.8.1): THREE decode slots x 200,000 -# tokens, fp8 KV, 44 GiB/rank pinned expert pool, MTP depth 2, prefix cache in -# pinned host RAM, PLE pinned in RAM. The per-card split between the expert slab -# and the KV pool is NO LONGER an operator number — 1.3.0 removed the VRAM pair -# and the budget resolver derives it from headroom, the cache bound and the host -# pool, so the slab size and the pool's token count are LEDGER lines: after every -# boot read them out of the log and check the pool still backs 3 x 200,000. +# Deployed shape (2026-10-19, radiance 1.8.1): TWO decode slots x 200,000 tokens, +# fp8 KV, 44 GiB/rank pinned expert pool, MTP depth 2, prefix cache in pinned host +# RAM, PLE pinned in RAM. The per-card split between the expert slab and the KV +# pool is NO LONGER an operator number — 1.3.0 removed the VRAM pair and the budget +# resolver derives it from headroom, the cache bound and the host pool, so the slab +# size and the pool's token count are LEDGER lines. After every boot read them, plus +# the two lines that size a prefill step — "activation arena: ... lends ... while +# steps are at most N tokens" (the arena levels the expert slab borrows) and +# "prefill staging: ... a step of more than N tokens ..." (the chunk size below which +# pooled experts are read synchronously) — and the cache's session backing. # # The root unit (scripts/frank-archon.service, User=dank, # LimitMEMLOCK=infinity) grants the pin budget: 2 x 44 GiB expert pools + From 9f7457e39e0aefd303ea37b829dd228f47293f71 Mon Sep 17 00:00:00 2001 From: Frank Date: Tue, 20 Oct 2026 22:08:00 -0700 Subject: [PATCH 2/4] docs(vault): answer "what are we doing wrong" on decode-vs-prefill MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit frank-archon Laws: add the measured interference census (decode-only step 15.5 ms vs 166 ms chunk steps; 4.7% of steps hold 34% of step time; median decode 166 -> 13 t/s above 5k prefill tok/s, per-stream 6.7 t/s = one token per shared step; TTFT p90 2.4 -> 18.0 s for arriving requests), the lag test that kills the residency theory, the busy_cap arithmetic, and the correction that --prefill-interleave is the decoder dial and auto is not. Facts that change how an operator reads this box: /diagnostics exists and reports link-below-max at 100% busy; latency metrics live under vllm:, cadence under radiance:; preemption is vllm:num_preemptions_total (264 this boot, all bursts) while the radiance gauge is instant depth; both root ports are x8 Gen5 max by board (31.5 GB/s ceiling, not 64); DDR is back at 4800 MT/s while the 4600 corruption fix is recorded -- verify before blaming garbage tokens. §9 (LLM Performance Diagnosis): method corrected to the phase-ratio counters, dead 1.0.13 numbers destroyed, four PromQL traps added (metric prefix, 1 s EMA gauges vs 15 s scrape, counter resets across builds, 2-minute mean diluting the chunk), instruments that cannot be reached from Prometheus, and the knob list priced from the pinned 1.8.1 source instead of the stale guide. --- .../LLM/LLM Performance Diagnosis.md | 115 ++++++++---- serverAppConfigs-docs/LLM/frank-archon.md | 172 ++++++++++++++---- 2 files changed, 216 insertions(+), 71 deletions(-) diff --git a/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md b/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md index 3ba2114..4457cc3 100644 --- a/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md +++ b/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md @@ -239,45 +239,90 @@ prefill is over. `preemptions` separates the third shape: a preempted-and-recomp sequence stops producing entirely and then re-prefills from token 0, which is a stall, not sludge. -Knobs, in the order worth trying them (radiance GUIDE §4.5): +**Split the step ledger before choosing a knob.** Three counters decide which mechanism you are +looking at, and none is a rate: `engine_step_seconds_decode_total / engine_steps_decode_total` is +the mean **decode-only** step, and `(step_seconds_total − step_seconds_decode_total) / +(steps_total − steps_decode_total)` is the mean **chunk step** (a chunk alone, or a chunk with +decode rows beside it). On frank-archon's live 1.8.1 shape that pair is **15.5 ms against 166 ms**, +with 4.7 % of the steps holding 34 % of the step time. Decode-only steps themselves move only +14.1 → 16.4 ms between a prefill-free minute and a heavy one, and the lagged buckets stay flat: so +a decode-rate collapse on that box is **step time sharing**, and the eviction/residency story is +worth ~10 % of one step — not a lever. -- `--prefill-interleave N` — after a chunk step taken beside decoders, the next N - steps carry decode rows only and admit nothing. Decoders keep their cadence; the - prompt pays that in TTFT. Counts as `engine_steps_held_total`. -- `--busy-prefill-chunk MODE` — the chunk's share of a step that has other work: - `auto` (default) reserves a row plus the draft window per decode row behind it and - an eighth of the budget per waiting prefill/request, `off` lets the chunk take the - whole step (admission never runs while it does), `N` pins the share. Never under - the prefill stager's smallest pass — the startup line names it, and below it every - routed layer reads its pooled experts synchronously, which costs the prefill far - more than sharing saves. Counts as `engine_prefill_chunks_capped_total`. -- `--max-num-batched-tokens` — the blunt rung, and the only one older pins have. The vendor's own - curve prices **halving the shared chunk at roughly a fifth of prefill throughput** (a 48-layer MoE - at 200k on two cards — knob pricing, not this endpoint's number, so treat it as the order of - magnitude a chunk decision costs). Raising it past the checkpoint interval buys nothing and loses - expert residency to the arena. -- `--prefill-span N` — the inverse trade: a *lone* prefill carries N intervals a step while every - shared step is planned at a smaller shared budget, which on an expert-tiered box lends that arena - back to the expert slab. The vendor's own measurement says the win is **long-prompt-only**: nothing - at short prompts, a large TTFT gain at the far end of the window. +**Four traps in the PromQL above.** Learn them before trusting a bucket: +- **Latency is `vllm:`, cadence is `radiance:`.** The TTFT / inter-token / per-output-token / e2e + histograms and the running/waiting gauges exist only under the vLLM compatibility prefix; preemption + is `vllm:num_preemptions_total` (the counter) while `radiance:num_requests_preempted` is the + *instant* depth. A `radiance:`-only query finds no latency and no preemption, and reads as "the + endpoint never stalls anyone". +- **The tps gauges are 1 s decayed windows, batch-pooled.** Per-stream rate is + `decode_tokens_per_second / vllm:num_requests_running`. With a 15 s scrape, a 120 s step samples one + scrape in eight — use step ≤ 30 s for a bucket table. +- **Every counter resets at a restart and the metric set changes with the build.** Prove the window is + one shape (`up`, or the first sample of a counter that only the current build exports) before + ranking buckets; frank-archon's 48 h window straddled two containers, and one bucket's "collapse" + turned out to belong entirely to the older one. +- **A 2-minute mean step time dilutes the chunk.** The sharp variable is the phase ratio in the query + above, never `step_seconds/steps` — 4.7 % of steps cannot move a mean that is 95 % decode. -Two traps. **The arena/slab coupling makes a shared step doubly expensive on an -expert-tiered shape**: the full-plan arena takes VRAM the resident expert slab would have had, so the -decode rows inside that step also read experts over the link — a smaller shared budget helps twice. -And **re-check -`prefix_hit_rate` / `prefix_evictions` after any chunk change**: the arena and the KV -pool are the same memory, and the failure looks like turn two taking as long as turn -one. +Three instruments the PromQL above cannot reach: -Availability is a version question, not a config question: `--busy-prefill-chunk` and -`--prefill-interleave` are absent from older pins — prove it on the running container -(`podman exec --help | grep -iE "prefill|chunk"`), never from the vendored -tree, and check the source clone's depth before dating a flag from `git log -S` -(a `--depth 1` clone reports its own tip for every search). +- **The engine's own step-detail histograms (`prefill_steps_duration`, `prefill_step_tokens`, + `token_queue_wait`) are exported but reach no target**, and the arena / expert slab / heat engine / + prefill stager have **no metric at all**. Per-request forensics mean `curl localhost:7770/stats` on + the endpoint's own host — same phase counters plus `preemptions` (the count), the chunk geometry, + and the expert/link byte totals. `/diagnostics` answers "is the *box* the limit" in one call: + configuration drift, the VRAM split, each card's link chain and clocks, findings with fixes. +- **`prefix_hit_cut_tokens_total` measures prefill load you created for yourself.** On a hybrid + (linear + paged) model a prefix hit reuses only as far as the last **linear-state** checkpoint, so + cached tokens are replayed as prefill and stall decoders for it; frank-archon ran 38 % of all its + prefill tokens that way, every `prefix_hit_cuts_total` labelled `reason="no_snapshot"`. A high + prefill rate is not always a client that sends new tokens. +- **Bucket the arriving request too** — `histogram_quantile` over + `vllm:time_to_first_token_seconds_bucket` by the same prefill series. The decode gauge only sees + sequences that are already running; the one waiting for its first token pays the same steps and + shows in no decode number. -Run on: [[frank-archon]] — the census was taken on its 3-slot radiance 1.0.13 era (budget 2048 = -interval 2048); the live 1.8.1 shape carries the two chunk knobs. The measured census and the verdict -for this fleet's shape are that page's. +Knobs, in the order worth trying them (radiance GUIDE §4.5; the pinned source is the referee): + +- `--prefill-interleave N` — after a chunk step taken beside decoders, the next N steps carry decode + rows only and admit nothing. **The knob that acts on the measured mechanism.** Decoders keep their + cadence; the prefilling prompt pays in TTFT and an arriving request waits out the held steps. + Counts as `engine_steps_held_total` — if that counter is zero, nothing is protecting anyone. +- `--busy-prefill-chunk N` — pins a shared chunk's share of the step. **`auto` is a prefill-speed + setting, not decoder protection:** it reserves `1 + spec` rows per decoding sequence plus an eighth + of the budget per *waiting* request or prefill, and the source refuses to halve for "decode rows + alone", so a shape can run `auto` and still lose every chunk step. Prove engagement from + `engine_prefill_chunks_capped_total` against the chunk-step count. The floor is the prefill + stager's smallest pass — the largest arena level that lends, plus one, named on the startup line + `prefill staging: … a step of more than N tokens copies the pooled experts of …` — and under it + every routed layer reads its pooled experts synchronously: the vendor prices the whole job (four + decoders plus a long prompt) at **29.5 s with a 2048 chunk and 31.9 s at 512**, so shrinking the + chunk blindly buys latency in both directions. +- `--prefill-span N` — needs `--max-num-batched-tokens` of N intervals. A *lone* prefill then + carries N intervals a step (on an expert-tiered box only from 8 intervals, where the vendor's own + measurement says a span starts to win), while **every other step is planned at the shared + budget** — a lending arena level that also keeps the stager's ring, the one combination the chunk + knob cannot express. Priced by the vendor at 320 MiB a card of expert residency, ~3 % off the KV + pool, and TTFT-neutral on one of his two 2×R9700 boxes: measure your own prompt lengths first. +- `--max-num-batched-tokens` alone — the blunt rung, and the only one older pins have. It raises the + arena the expert slab loses, and on this payload its resonance is violent (MNBT 2560 collapses + decode — [[frank-archon]]). + +Two traps. **The arena/slab coupling makes a shared step expensive twice:** the full-plan arena is +reclaimed from the expert slab *synchronously before the step*, and the loan regrows only after a +quiet interval, so every step of a prefill run — the decode-only ones included — runs on the smaller +slab. And **re-check `prefix_hit_rate` / `prefix_evictions` after any chunk or span change**: the +arena, the staging ring and the KV pool are the same memory, and the failure looks like turn two +taking exactly as long as turn one. + +Availability is a version question, not a config question: `--busy-prefill-chunk`, +`--prefill-interleave` and `--prefill-span` are absent from older pins — prove it on the running +container (`podman exec --help | grep -iE "prefill|chunk"`) or from `/diagnostics`'s +`configuration` block, never from a vendored tree, and check a source clone's depth before dating a +flag from `git log -S` (a `--depth 1` clone reports its own tip for every search). + +Run on: [[frank-archon]] — owner of the live census, the measured step pair, and the arms. ## Related diff --git a/serverAppConfigs-docs/LLM/frank-archon.md b/serverAppConfigs-docs/LLM/frank-archon.md index 6c5a3ab..a6d7f2f 100644 --- a/serverAppConfigs-docs/LLM/frank-archon.md +++ b/serverAppConfigs-docs/LLM/frank-archon.md @@ -96,7 +96,7 @@ on stop. Boot budget 1800 s (the publisher's compose allows a 15-minute `start_p | Spec / batch | `SPEC_TOKENS=2` — MTP depth 2, the `.rad` carries the model's own `mtp.*` head (`mtp.fc_hidden`/`fc_embedding`/`draft_head`); `MAX_BATCHED_TOKENS=2048` (MNBT — 2560 collapses decode, Laws) | | Prefix cache | `PREFIX_CACHE_MIB=8192`, pinned **host tier only** — the 64 GiB disk tier is gone, so an evicted prefix is re-prefilled; `STATE_DIR=/home/dank/radiance-state` now exists only to hold the rad-tune cache | | Template | `TEMPLATE=scripts/qwen38-froggeric-v22.5.jinja` (28,234 B) via `--override-chat-template` — froggeric's fixed Qwen3.8 template (the repo root's `chat_template.jinja`, vendored); the checkpoint's own template is baked into the `.rad` by rad-convert and this overrides it, and radiance derives the reasoning and tool-call reply format from it | -| Step sharing | `BUSY_PREFILL_CHUNK=auto` (a shared chunk reserves a row plus the draft window per waiting decoder) · `PREFILL_INTERLEAVE=0` — held at zero until the decode-vs-prefill buckets ask for it | +| Step sharing | `BUSY_PREFILL_CHUNK=auto` — prefill-speed first: it reserves 3 tokens per waiting decoder and ⅛ of the budget per waiting request, and does **not** shrink the chunk for decoders alone · `PREFILL_INTERLEAVE=0` — the decoder dial, held at zero until the §9 buckets ask for it | | TTM gate | `TTM_MIN_MIB=150000`, `TTM_TARGET_MIB=204800` | | Boot | `RAD_MIN_MIB=110000` deployer floor; the listener binds only after the full load (health ~100 s, budget 1800 s) | @@ -280,30 +280,83 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10). - **MNBT 2048.** 2560 collapses decode to 2.6-7.7 t/s (MTP resonance); the w4 file tier streams in place only at MNBT ≤ 2048 and the activation arena scales with the chunk (32768 → 33.3 GiB, refused). -- **Decode pays for the chunk it shares a step with.** Census of the live engine taken at the - 3-slot shape on 1.0.13 (`/stats` + `/metrics`, cumulative over the 2026-10-15 boot): - 1,435,585 decode-only steps at **18.7 ms** each against 127,991 prefill+mixed steps - at **317 ms** — 60 % of engine wall-clock spent inside 8 % of the steps, because the - chunk is `min(remaining, checkpoint interval 2048, MNBT 2048)` = 2048 rows and every - decode row in that step waits the whole chunk. Day of Prometheus (120 s step, 721 - samples): `decode_tokens_per_second` median **13.3 t/s** in the 56 % of samples where - `prefill_tokens_per_second` ≥ 1000, against 106-240 t/s in the light-prefill buckets; - bucketed by mean step time it is 128.7 t/s at 0.02-0.05 s steps and 12.0 t/s at - 0.15-0.4 s. The stall is **during**, not after — decode in windows 2-14 min after a - busy-prefill window sits at the idle baseline (58-64 vs 77-85 t/s, the residual - confounded by co-scheduling), so no lingering expert-residency penalty at 2-minute - resolution, and `preemptions: 0` rules out preempt-and-re-prefill. Method: - [[LLM Performance Diagnosis]] §9. The law is shape-independent; the fix lives at 1.8.1 (next law), - and nobody has re-run the census since the migration — so the fix is ordered, **not verified**. -- **The knobs for it arrived with 1.8.1, and only one of them is on.** `--busy-prefill-chunk` caps the - shared chunk so a step with waiting decoders takes a smaller piece (live: `auto`, the engine picks; - the explicit value is 512), and `--prefill-interleave N` inserts N decode-only steps after a chunk - step taken beside decoders. `interleave` is deliberately **0** — the env comment holds it there until - the decode-vs-prefill buckets ask for it, and those buckets have not been read since the migration. - The vendor prices a 1024 chunk at about a fifth off prefill throughput to halve the stall; that is a - knob for choosing tomorrow's chunk, not a measurement of this shape. MNBT remains an arm and not an - edit: 2560 collapses decode through MTP resonance. Any chunk change re-scores draft acceptance before - it is trusted. +- **Decode pays for the chunk it shares a step with, and `--busy-prefill-chunk auto` does not + protect it.** Re-measured on the live 1.8.1 shape (34.8 h boot, Prometheus at 60 s, + `engine_step_seconds{,_decode}_total` / `engine_steps{,_decode,_mixed,_prefill}_total` / + `prefill_tokens_total`): decode-only steps run at **15.5 ms**, chunk steps (a chunk alone, or a + chunk with decode rows beside it) at **166 ms** — 4.7 % of the steps hold **34 % of the engine's + step time**, so a session decoding while another prompts advances one token per 166 ms. Bucket the + harness's `radiance:decode_tokens_per_second` by `radiance:prefill_tokens_per_second` (15 s + scrapes; per-stream rate = the gauge ÷ `vllm:num_requests_running`): median **166 t/s** under light + prefill, 81 t/s at 1-5k, **13 t/s above 5k** — a **12.7×** collapse, and at 13 t/s with two + sessions running each stream sits at 6.7 t/s = 149 ms a token, one token per shared step. Nothing + else moves: decode-*only* steps stay 14-18 ms in every bucket (14.1 ms in a prefill-free minute, + 17.9 ms in the deep one) and the relationship is lag-0 only (r = −0.74 at lag 0, |r| ≤ 0.19 from + 30 s on, decode back at baseline by 5 min) — residency is not the mechanism either. The second + victim is the **arriving** request: through a high-prefill window TTFT p90 goes 2.4 → **18.0 s** + and 15.5 % of requests wait over 10 s for their first token, with `http_queue_depth` at 2-3, while + median TTFT (~1.9 s) and per-token latency barely move. Preemption is real but burst-shaped: 264 + `vllm:num_preemptions_total` in the boot, 108 of them inside 105 s of a 6.5k-tok/s prefill. The + chunk is `min(remaining, checkpoint interval, MNBT, busy_cap)`; the first three read 2048 here, + and `busy_cap` is the fourth law. +- **`auto` is a prefill-speed dial, not a decoder-protection dial.** `Scheduler::busy_cap` reserves + `1 + spec` rows per decoding sequence behind the chunk (3 tokens at MTP 2) plus one eighth of the + budget per *waiting request or waiting prefill*, and the code refuses to do more for decoders: + "not for decode rows alone: their reserve is a few tokens, so the chunk ends a quantum short and + the next step finishes the interval… Halving would hold a long prompt to half-interval chunks while + anything decodes behind it, which costs more" (`core/sched/scheduler.cpp:874-878`, 1.8.1). Live + proof: `prefill_chunks_capped` 5,018 against 62,386 chunk steps — 8 %, and those are the minutes + with a waiter, not the minutes with a decoder. An explicit `N` is the only value that applies + "whenever the chunk shares the step". +- **At MNBT 2048 no chunk value keeps both the arena loan and the staged copies.** The arena is a + ladder of levels (256 and 1024 rows here) and a step's rows are its **whole** token count — + `max(n_tok, n_seq × (1 + spec))`, `core/engine.cpp:1142` — so a 1024-token chunk with two decode + rows is a 1030-row step: it falls back to the full plan, which "lends nothing", and it arms the + prefill stager. The stager's floor is the largest lending level plus one, 1025 rows here — named on + the boot line `prefill staging: … a step of more than 1024 tokens copies the pooled experts of 48 + routed layers onto the card up to 3 layers ahead` — and below it "every routed layer reads its + pooled experts synchronously and the prefill slows far more than sharing saves" (GUIDE §4.5). The + vendor's own number for naive shrinking, on this architecture: four decoders with a long prompt + beside them finished in **29.5 s at a 2048 chunk and 31.9 s at 512**. +- **A big step takes the arena from the expert slab and keeps it for the whole prefill.** The arena + lends 987.77 MiB (steps ≤ 256 rows) or 670.26 MiB (≤ 1024) to the slab; a full-plan step reclaims + it **synchronously before the step** — units revert to their host copies (an address change, not a + copy) and both read streams drain — and `arena_tick` regrows the loan only **2 s after the last big + step** (`core/engine.cpp:1129-1202`), so every step of the prefill run, decode-only included, runs + with a slab ~780 expert units smaller over a link also carrying the staged copies. `/diagnostics` + says the binding constraint in one line: "a prefill step streams every routed layer's pooled + experts onto the card, so its speed is bounded by the link; decode reads the ones it routes to". + Measured size of the whole effect: the ~10 % above. Real, small, not worth chasing. +- **`--prefill-interleave` is the dial that buys decode its cadence; it is off (`steps_held` 0).** + N decode-only steps follow any chunk step taken beside decoders; held steps admit nothing, so it + costs the prefilling prompt its TTFT and an arriving request its first step. Rate arithmetic at the + two measured step times and 0.657 acceptance (≈2.3 tokens a step): N=0 → ~14 t/s, N=2 → ~33, + N=4 → ~48, N=8 → ~69, with the prefill slowed by (166 + 15.5N)/166. **Arithmetic, not + measurement** — the arm is chosen by the §9 bucket pair plus `steps_held`, and `auto` is not a + substitute for it. +- **`--prefill-span` is the shape that keeps the loan, and it is priced.** It needs MNBT = N + intervals, so `--prefill-span 2 --max-num-batched-tokens 4096`: a *lone* prefill carries 2 + intervals a step (and on an expert-tiered box only from 8 intervals / 16k tokens, where the span + starts to win), while **every other step is planned at the shared budget of 2048** — which is a + lending level *and* keeps the stager's ring, the one combination `--busy-prefill-chunk` cannot + express. Costs per GUIDE §4.5: 320 MiB a card of expert residency, ~3 % off the KV pool (§4.3's + cliff is sharp), a decode step uploads twice the per-step arrays at 4096, and checkpoints inside a + span are not written, so a partially-shared prompt resumes one interval earlier. On the vendor's + second box (2×R9700, w4nl64, TP2) it was TTFT-neutral — 2.61 vs 2.56 s at 18k, 10.72 vs 10.55 at + 75k — hence their own rule: measure your own prompts at two ranks before turning it on. MNBT 4096 + does not boot at 3 slots; this shape is 2. +- **Most of this endpoint's prefill work is replay of tokens the prefix cache already had.** Over + the same window `prefill_tokens_total` 65.2 M against `prefix_hit_cut_tokens_total` 24.9 M = **38 % + replayed**, and it is not an eviction: a hit on a hybrid model reuses only as far as the last + **linear-state** checkpoint (`reuse = min(h.n_attn_tokens, h.n_linear_tokens)`, + `core/sched/scheduler.cpp:953-978`), and the default `checkpoint policy geometric-backoff, base + 2048` thins the snapshots deep in a prompt, so a hit at 150k can resume at 128k. The cause object + agrees — every `prefix_hit_cuts_total` carries `reason="no_snapshot"` (`windowed_group` and + `adopt_failed` are dead) — but the event count is small and the tokens are the cost, so read + `hit_cut_tokens`. An arm, not an edit: + `--checkpoint-policy floor` (or a shorter `--checkpoint-interval`) buys the replay back with + pinned-RAM checkpoint slots, and those slots come out of the same pinned host memory the expert + pool claims — a resolver fight, and `prefix_hit_cut_tokens_total` is the meter. - **Slots cost KV, rows, and window — never speed.** `--max-num-seqs` carries no radiance constraint: it adds ≈110 MiB of recurrent state per slot (the live `kv_full_backing` arithmetic prices them exactly), declares the fused TP all-reduce over `seqs × (1 + spec)` rows, and `wht6` is bit-exact up to 25 rows @@ -396,12 +449,35 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10). 4600 MT/s ended it, with zero `gdn_hip` faults either side; the prefix-caching / GDN re-seed theory was confounded on both arms. Consumer non-ECC raises no MCE for a silent flip and a 60 s / 8 GiB pattern check covers 0.5 % of 246 GiB — silent RAM corruption outranks every - software theory. + software theory. **That mitigation is a BIOS setting and it is not in effect now:** `sudo + dmidecode -t memory | grep -i configured` answers 4800 MT/s on 2026-10-20, and the only event + between the fix and today is the 2026-10-12 power failure that also reset TTM. Check the speed + before crediting any software revert with a fix — and before believing a clean run of the arm + you just tested, because the confounder is still in the slot. - **TP=2 here** is legal since the 2026-09-20 ProArt B850 swap (x8/x8 on CPU root ports, `Routing+`/`ReqEn+` end to end) only because radiance's wire is `libr4d`'s own one-shot all-reduce; RCCL needs atomics on the whole chain and fails `hipErrorIllegalState` without them (). `wht6` buys +5.6-6.3 % prefill for ~1.3 % of 8-stream decode, `exact` is the bit-exact alternative. +- **The link is already at the board's ceiling — do not chase the `link-below-max` warning.** + Both root ports (`00:01.1`, `00:01.3`) are **x8 max** at 32 GT/s, so PCIe 5.0 x8 ≈ 31.5 GB/s is + the whole budget per card and the cards' own `pcie_dpm` tables expose only x8 states. + `/diagnostics` still warns that the on-card switch hop (`01:00.0`, `05:00.0`) runs "x8 of its + x16 while the card is 100 % busy" and suggests a reboot or a reseat — pointless when the host + side of the chain is x8 max, which the same endpoint reports two findings later as + `link-capped`. The width that matters is the minimum of the chain, and it is x8 either way, so + the only lever on a link-bound prefill is **less traffic over the link** (more expert + residency, fewer pooled experts), never a wider link. P2P resolves **on** (`auto` asks the + driver, the two cards reach each other's memory), so the collectives do not hairpin through host + RAM, and the engine's copies ride one DMA queue that "give[s] way to the collectives" — the + all-reduce wins the link, the staged experts wait. +- **Eight cores is the whole host, media stack included.** Dankserver is a Ryzen 7 7700X (8 + cores / 8 threads, one NUMA node, `nodes: -1` for both cards) running Jellyfin, Plex, Calibre + and RomM beside the engine, and `--http-threads` defaults to one thread per core. Measured + during a heavy prefill: load average ≈ 1.0, `jellyfin` 11 % CPU, `radiance` 7 % — **CPU + contention is not the current decode culprit**, so do not spend an experiment window on it; + re-read `uptime` and `ps -eo pcpu,comm --sort=-pcpu | head` first if a future collapse looks + different from the step-sharing one. - `--prefix-cache-dir` creation is **not recursive** — the serve script `mkdir -p`s the path or the disk tier silently never opens. The tier does survive a payload swap when arch and tokenizer match (it recovered 900,176 blocks on the 2026-10-15 swap; `cache_n` proves a hit). @@ -445,11 +521,19 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10). ## Contract (the wire) - **Live surface:** `/health` (== `/ping`), `/v1/models`, `/v1/chat/completions` - `/v1/completions`, `/metrics`, `/stats`, `/sessions`, `/dashboard`. `/health` answers 200 - JSON (model id, `in_flight`, `queue_depth`) while serving and **503 only while stopping** - (`core/server/admin.cpp:385`), and the listener binds *after* the full load (`main.cpp`: - `engine.start()` then `engine.serve()`) — during the load the port refuses connections, so a - bare `curl -fsS …/health` gate is honest and needs no body grep. + `/v1/completions`, `/metrics`, `/stats`, `/sessions`, `/dashboard`, **`/diagnostics`**. `/health` + answers 200 JSON (model id, `in_flight`, `queue_depth`) while serving and **503 only while + stopping** (`core/server/admin.cpp:385`), and the listener binds *after* the full load + (`main.cpp`: `engine.start()` then `engine.serve()`) — during the load the port refuses + connections, so a bare `curl -fsS …/health` gate is honest and needs no body grep. +- **`/diagnostics` is the instrument for "is the box, not the config, the limit"** — one call + returns `configuration` (every flag, so drift from the env file is checkable without the host), + `environment`, `derived` (the VRAM split as the ledger prints it), `hardware` (host, per-card + `link_chain` + `pcie_dpm` + clocks + power cap + DMA engine choice + P2P answer) and `findings` + (level / what / why / fix). On this box it reports `experts-from-host` (5.99 GiB a rank crosses + PCIe when used), `link-capped` (root ports are x8 max — the board), `swap-in-use`, + `tp-wire-lossy`, and **`link-below-max` as a `warn` at 100 % card busy on both cards** — which is + structural on this board; see the x8-ceiling law under *This host, and the deploy path*. - **Served id.** `FRANK_ARCHON_SERVED_NAME=frank-archon` (the wire rename of 2026-10-09 moved only the alias; container, unit, scripts and env keep the service name). The env's contract line is that **radiance validates the requested model name, so every client — harness, bench @@ -476,12 +560,28 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10). and 16,384 patches per pass. Probe with a ≥ 256 px solid-colour PNG and a ≥ 256 token budget — at 64 tokens you get empty `content` and the colour only inside `reasoning_content`. - **Capacity behaviour:** a prompt over `--max-model-len` is refused, never truncated; a request - longer than the KV pool's token count is accepted then refused as too large to fit; under the - 3-slot contract a **fourth** full-height conversation is refused, not paged or squeezed. + longer than the KV pool's token count is accepted then refused as too large to fit; a request + beyond `--max-num-seqs` is held by the HTTP port (`queue_cap`, default 8 × slots = 16 here) and + answered 503 only past that, never paged or squeezed. Because the cache funds **1.8** full-height + sessions, admitting a second near-full-height conversation can **preempt a running sequence from + the back** to free its blocks (`core/sched/scheduler.cpp:1291-1322`); the victim is recomputed, so + its session halts and then re-prefills. Meter: `vllm:num_preemptions_total` (264 in the current + boot, all in prefill-coincident bursts — 108 inside 105 s); `/stats preemptions` is the cumulative + count and `radiance:num_requests_preempted` only the instant depth. - **Observability:** `/metrics` carries vLLM-compatible names plus `radiance:*`, scraped by the - Dankclaw Prometheus job at `192.168.0.235:7770` over plain LAN routing (the netavark - published-port quirk is same-host only). The boot ledger — the referee for every shape number - above — is in the container log (`podman logs frank-archon`) + Dankclaw Prometheus job at `192.168.0.235:7770` (15 s interval) over plain LAN routing (the + netavark published-port quirk is same-host only). **Latency lives under `vllm:`, cadence under + `radiance:`** — TTFT / inter-token / per-output-token / e2e / decode-time histograms and the + running/waiting gauges are `vllm:*` with no radiance-native equivalent, so a `radiance:`-only + query misses every latency answer. The step-phase counters are all scraped: + `engine_steps{,_decode,_mixed,_prefill,_held}_total`, `engine_step_seconds{,_decode}_total`, + `engine_prefill_chunks_capped_total`, `prefill_tokens_total`, `decode_tokens_per_second`, + `prefill_tokens_per_second`, `prefix_hit_cut_tokens_total`, `http_queue_depth`. Two holes: the + engine's own step-detail histograms (`prefill_steps_duration`, `prefill_step_tokens`, + `token_queue_wait`) are exported but reach no target, and the **arena / expert slab / heat engine / + prefill stager have no metric at all** — that half of every law above is read from the boot ledger, + `/stats` and `/diagnostics`, never from a graph. The boot ledger — the referee for every shape + number here — is in the container log (`podman logs frank-archon`); `/home/dank/logs/frank_archon.log` is where the serve script redirects `podman run`. ## Files From b1fc1113c60172bc603539b41934244905da74d0 Mon Sep 17 00:00:00 2001 From: Frank Date: Tue, 20 Oct 2026 22:34:13 -0700 Subject: [PATCH 3/4] docs(vault): replace an invented --checkpoint-policy name with the three that exist MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The replay law named `--checkpoint-policy floor`; config.cpp admits exactly geometric-backoff (default), tip-only, keep-all and refuses anything else at boot, so the recorded arm could never be deployed. Replaced with the real semantics, priced from core/mem/prefix.{h,cpp}: the default keeps a geometric ladder (9 snapshots for a 200k session at interval 2048 -- exactly the live ledger line -- so a turn diverging d behind the tip can resume at tip-3d: diverge at 120k, replay 21.7k); retention is not the write path, so keep-all buys a bounded replay for pinned host RAM alone (196 slots x 111.98 MiB = 21.4 GiB against the store's eighth-of-host budget of 31.5 GiB, hard bound = room); and shortening --checkpoint-interval cannot fix the deep case -- the coarsest ladder scale is interval x 2^k under the depth, so the deepest gap stays about half the prompt at every interval. §9 keeps only the three-name locator. --- .../LLM/LLM Performance Diagnosis.md | 4 ++- serverAppConfigs-docs/LLM/frank-archon.md | 25 +++++++++++++------ 2 files changed, 20 insertions(+), 9 deletions(-) diff --git a/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md b/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md index 4457cc3..ed44f86 100644 --- a/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md +++ b/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md @@ -277,7 +277,9 @@ Three instruments the PromQL above cannot reach: (linear + paged) model a prefix hit reuses only as far as the last **linear-state** checkpoint, so cached tokens are replayed as prefill and stall decoders for it; frank-archon ran 38 % of all its prefill tokens that way, every `prefix_hit_cuts_total` labelled `reason="no_snapshot"`. A high - prefill rate is not always a client that sends new tokens. + prefill rate is not always a client that sends new tokens. Retention is `--checkpoint-policy` - + exactly three names, `geometric-backoff` (default), `tip-only`, `keep-all`, one owner: + [[frank-archon]]. - **Bucket the arriving request too** — `histogram_quantile` over `vllm:time_to_first_token_seconds_bucket` by the same prefill series. The decode gauge only sees sequences that are already running; the one waiting for its first token pays the same steps and diff --git a/serverAppConfigs-docs/LLM/frank-archon.md b/serverAppConfigs-docs/LLM/frank-archon.md index a6d7f2f..855c2e7 100644 --- a/serverAppConfigs-docs/LLM/frank-archon.md +++ b/serverAppConfigs-docs/LLM/frank-archon.md @@ -349,14 +349,23 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10). the same window `prefill_tokens_total` 65.2 M against `prefix_hit_cut_tokens_total` 24.9 M = **38 % replayed**, and it is not an eviction: a hit on a hybrid model reuses only as far as the last **linear-state** checkpoint (`reuse = min(h.n_attn_tokens, h.n_linear_tokens)`, - `core/sched/scheduler.cpp:953-978`), and the default `checkpoint policy geometric-backoff, base - 2048` thins the snapshots deep in a prompt, so a hit at 150k can resume at 128k. The cause object - agrees — every `prefix_hit_cuts_total` carries `reason="no_snapshot"` (`windowed_group` and - `adopt_failed` are dead) — but the event count is small and the tokens are the cost, so read - `hit_cut_tokens`. An arm, not an edit: - `--checkpoint-policy floor` (or a shorter `--checkpoint-interval`) buys the replay back with - pinned-RAM checkpoint slots, and those slots come out of the same pinned host memory the expert - pool claims — a resolver fight, and `prefix_hit_cut_tokens_total` is the meter. + `core/sched/scheduler.cpp:953-978`), and the retention default keeps a **geometric ladder** - about + `log2(depth/interval) + 2` snapshots - so a turn that diverges `d` behind the tip can resume as + deep as `tip − 3d`. The live ladder for a 200k session at interval 2048 is 2048 · 131072 · 163840 + · … · 198656: diverge at 120k and 21.7k tokens replay, diverge at 50k and 48k. Every + `prefix_hit_cuts_total` carries `reason="no_snapshot"` (the other two reasons stay dead), so the + cause object agrees and `hit_cut_tokens` is the cost. `--checkpoint-policy` admits exactly + **`geometric-backoff` | `tip-only` | `keep-all`** and refuses anything else at boot. It governs + retention only - the writes happen at every interval whatever the policy says (only `--prefill-span` + skips them) - so `keep-all` buys a replay bounded by one interval at any depth for **pinned host RAM + alone**: 97 snapshots a 200k session, `--checkpoint-slots auto` asking 196 × 111.98 MiB = 21.4 GiB + against a store budget of an eighth of host RAM (31.5 GiB here, so `share` admits it) with `room` - + the pinning room left beside the expert pool at boot - as the hard bound the ledger line names. + Retention also spares a snapshot a later turn resumed from (`core/mem/prefix.cpp:709-723`), so a + chain that keeps resuming at one deep point stops paying for it. Shortening + `--checkpoint-interval` does **not** buy the deep case: the coarsest ladder scale is the largest + `interval × 2^k` under the depth, so the deepest gap stays ≈ half the prompt (129,024 tokens at + interval 2048 against 130,816 at 256) while every cut multiplies the snapshot writes. - **Slots cost KV, rows, and window — never speed.** `--max-num-seqs` carries no radiance constraint: it adds ≈110 MiB of recurrent state per slot (the live `kv_full_backing` arithmetic prices them exactly), declares the fused TP all-reduce over `seqs × (1 + spec)` rows, and `wht6` is bit-exact up to 25 rows From aa75b455438e217f786c78a924c0a3fd2d974ea0 Mon Sep 17 00:00:00 2001 From: Frank Date: Tue, 20 Oct 2026 22:52:31 -0700 Subject: [PATCH 4/4] feat(frank-archon): arm --prefill-interleave at 1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A prefill chunk that shares a step with decoding sequences holds the device for the chunk's whole time: measured on this shape, decode-only steps 15.5 ms against chunk steps 166 ms, and 4.7 % of the steps carry 34 % of the engine's step time. Harness-visible decode falls 166 -> 13 tok/s (per stream 6.7 = one token per shared step) once concurrent prefill passes 5k tok/s, and TTFT p90 for an arriving request goes 2.4 -> 18.0 s. BUSY_PREFILL_CHUNK stays `auto`: it is a prefill-speed dial, and the source refuses to halve the chunk for decode rows alone -- which is why engine_steps_held_total read zero for the whole 34.7 h boot. PREFILL_INTERLEAVE is the knob that acts on the measured mechanism: N decode-only steps follow any chunk step taken beside decoders. N=1 is the conservative first arm -- a stream takes its token from the chunk step and one from the held step (~2 tokens per 182 ms instead of one per 166 ms); the cost lands on the prefilling prompt's TTFT and on requests still queued, not on anything decoding. Rollback: PREFILL_INTERLEAVE=0. Verification: the flag must reach argv (the serve script only passes it when > 0), engine_steps_held_total must leave zero, then the §9 bucket pair must move before N rises. --- scripts/frank-archon.env | 10 ++++++++-- serverAppConfigs-docs/LLM/frank-archon.md | 8 ++++---- 2 files changed, 12 insertions(+), 6 deletions(-) diff --git a/scripts/frank-archon.env b/scripts/frank-archon.env index 65ac22c..3adb083 100644 --- a/scripts/frank-archon.env +++ b/scripts/frank-archon.env @@ -107,9 +107,15 @@ FRANK_ARCHON_MAX_BATCHED_TOKENS=2048 # 512 is a trap and the vendor measures the whole job slower there. Interleave # inserts N decode-only steps after a chunk step taken beside decoders: it protects # their cadence and holds a waiting request out of those steps. That is the decoder -# dial, and it is 0 until the decode-vs-prefill buckets ask for it. +# dial. ARMED at 1 on 2026-10-20: at N=1 a stream takes its token from the chunk step +# AND one from the held step, ~2 tokens per 182 ms instead of one per 166 ms; the +# price falls on the prefilling prompt (its chunks lose every held step) and on any +# request still waiting to enter. `--prefill-span` re-funds a prefill only when it is +# ALONE, so it cannot buy this back while decoders run. Meter: +# radiance:engine_steps_held_total must leave zero, and the §9 buckets must move +# before N rises. FRANK_ARCHON_BUSY_PREFILL_CHUNK=auto -FRANK_ARCHON_PREFILL_INTERLEAVE=0 +FRANK_ARCHON_PREFILL_INTERLEAVE=1 # Prefix cache: pinned host memory only. The disk tier was removed outright # (--prefix-cache-dir and --prefix-cache-disk-mib are refusals), so what used to # be restored from NVMe is now re-prefilled. The tier is sized in ENTRIES and one diff --git a/serverAppConfigs-docs/LLM/frank-archon.md b/serverAppConfigs-docs/LLM/frank-archon.md index 855c2e7..9027e47 100644 --- a/serverAppConfigs-docs/LLM/frank-archon.md +++ b/serverAppConfigs-docs/LLM/frank-archon.md @@ -96,7 +96,7 @@ on stop. Boot budget 1800 s (the publisher's compose allows a 15-minute `start_p | Spec / batch | `SPEC_TOKENS=2` — MTP depth 2, the `.rad` carries the model's own `mtp.*` head (`mtp.fc_hidden`/`fc_embedding`/`draft_head`); `MAX_BATCHED_TOKENS=2048` (MNBT — 2560 collapses decode, Laws) | | Prefix cache | `PREFIX_CACHE_MIB=8192`, pinned **host tier only** — the 64 GiB disk tier is gone, so an evicted prefix is re-prefilled; `STATE_DIR=/home/dank/radiance-state` now exists only to hold the rad-tune cache | | Template | `TEMPLATE=scripts/qwen38-froggeric-v22.5.jinja` (28,234 B) via `--override-chat-template` — froggeric's fixed Qwen3.8 template (the repo root's `chat_template.jinja`, vendored); the checkpoint's own template is baked into the `.rad` by rad-convert and this overrides it, and radiance derives the reasoning and tool-call reply format from it | -| Step sharing | `BUSY_PREFILL_CHUNK=auto` — prefill-speed first: it reserves 3 tokens per waiting decoder and ⅛ of the budget per waiting request, and does **not** shrink the chunk for decoders alone · `PREFILL_INTERLEAVE=0` — the decoder dial, held at zero until the §9 buckets ask for it | +| Step sharing | `BUSY_PREFILL_CHUNK=auto` — prefill-speed first: it reserves 3 tokens per waiting decoder and ⅛ of the budget per waiting request, and does **not** shrink the chunk for decoders alone · `PREFILL_INTERLEAVE=1` — the decoder dial, **armed 2026-10-20**: one decode-only step after each chunk step taken beside decoders; meter `engine_steps_held_total`, price is the prefilling prompt's TTFT | | TTM gate | `TTM_MIN_MIB=150000`, `TTM_TARGET_MIB=204800` | | Boot | `RAD_MIN_MIB=110000` deployer floor; the listener binds only after the full load (health ~100 s, budget 1800 s) | @@ -327,11 +327,11 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10). says the binding constraint in one line: "a prefill step streams every routed layer's pooled experts onto the card, so its speed is bounded by the link; decode reads the ones it routes to". Measured size of the whole effect: the ~10 % above. Real, small, not worth chasing. -- **`--prefill-interleave` is the dial that buys decode its cadence; it is off (`steps_held` 0).** +- **`--prefill-interleave` is the dial that buys decode its cadence; armed at 1 on 2026-10-20.** N decode-only steps follow any chunk step taken beside decoders; held steps admit nothing, so it costs the prefilling prompt its TTFT and an arriving request its first step. Rate arithmetic at the - two measured step times and 0.657 acceptance (≈2.3 tokens a step): N=0 → ~14 t/s, N=2 → ~33, - N=4 → ~48, N=8 → ~69, with the prefill slowed by (166 + 15.5N)/166. **Arithmetic, not + two measured step times and 0.657 acceptance (≈2.3 tokens a step): N=0 → ~14 t/s, **N=1 (armed) → + ~23**, N=2 → ~33, N=4 → ~48, N=8 → ~69, with the prefill slowed by (166 + 15.5N)/166. **Arithmetic, not measurement** — the arm is chosen by the §9 bucket pair plus `steps_held`, and `auto` is not a substitute for it. - **`--prefill-span` is the shape that keeps the loan, and it is priced.** It needs MNBT = N