Merge pull request 'feat(frank-archon): arm --prefill-interleave at 1' (#57) from wt/20261020-213526 into main

A prefill chunk that shares a step with decoding sequences holds the device for
the chunk's whole time: measured on this shape, decode-only steps 15.5 ms
against chunk steps 166 ms, and 4.7 % of the steps carry 34 % of the engine's
step time. Harness-visible decode falls 166 -> 13 tok/s (per stream 6.7 = one
token per shared step) once concurrent prefill passes 5k tok/s, and TTFT p90 for
an arriving request goes 2.4 -> 18.0 s.

BUSY_PREFILL_CHUNK stays `auto`: it is a prefill-speed dial, and the source
refuses to halve the chunk for decode rows alone -- which is why
engine_steps_held_total read zero for the whole 34.7 h boot. PREFILL_INTERLEAVE
is the knob that acts on the measured mechanism: N decode-only steps follow any
chunk step taken beside decoders. N=1 is the conservative first arm -- a stream
takes its token from the chunk step and one from the held step (~2 tokens per
182 ms instead of one per 166 ms); the cost lands on the prefilling prompt's
TTFT and on requests still queued, not on anything decoding.

Rollback: PREFILL_INTERLEAVE=0. Verification: the flag must reach argv (the
serve script only passes it when > 0), engine_steps_held_total must leave zero,
then the §9 bucket pair must move before N rises.

scripts/frank-archon.env

@@ -93,16 +93,29 @@ FRANK_ARCHON_SPEC_TOKENS=2
 FRANK_ARCHON_MAX_BATCHED_TOKENS=2048
 # Step sharing. A prefill chunk that shares a step with decoding sequences holds
 # the device for the chunk's whole time, and every decode row in that step waits
-# that long for its next token (the census: [[LLM Performance Diagnosis]] §9).
-# `auto` leaves each decode row behind the chunk a row plus its draft window and
-# keeps a share for whatever is waiting to be admitted; a number pins the chunk's
-# cap; `off` is the pre-1.4 behaviour this endpoint ran for two eras, where the
-# chunk took the step and admission never ran. Interleave inserts N decode-only
-# steps after a chunk step taken beside decoders: it protects their cadence and
-# holds a waiting request out of those steps, so it stays 0 until the post-migration
-# decode-vs-prefill buckets say it is wanted.
+# that long for its next token: measured at this shape, decode-only steps 15.5 ms
+# against chunk steps 166 ms, which is the harness's ~146 t/s falling to ~11 t/s
+# while another session prefills (law and census: [[frank-archon]]).
+# `auto` is a PREFILL-SPEED dial, not a decoder dial: it reserves one row plus the
+# draft window per decoding sequence (3 tokens) and an eighth of the budget per
+# waiting request or waiting prefill, and the source states it will not halve for
+# decode rows alone. Live proof: prefill_chunks_capped 5,018 across 62,386 chunk
+# steps. A number is the only value that applies whenever the chunk shares a step,
+# and it must not go under the prefill stager's smallest pass -- 1025 rows here,
+# named on the boot line "prefill staging: ... a step of more than 1024 tokens" --
+# because below it every routed layer reads its pooled experts synchronously, so
+# 512 is a trap and the vendor measures the whole job slower there. Interleave
+# inserts N decode-only steps after a chunk step taken beside decoders: it protects
+# their cadence and holds a waiting request out of those steps. That is the decoder
+# dial. ARMED at 1 on 2026-10-20: at N=1 a stream takes its token from the chunk step
+# AND one from the held step, ~2 tokens per 182 ms instead of one per 166 ms; the
+# price falls on the prefilling prompt (its chunks lose every held step) and on any
+# request still waiting to enter. `--prefill-span` re-funds a prefill only when it is
+# ALONE, so it cannot buy this back while decoders run. Meter:
+# radiance:engine_steps_held_total must leave zero, and the §9 buckets must move
+# before N rises.
 FRANK_ARCHON_BUSY_PREFILL_CHUNK=auto
-FRANK_ARCHON_PREFILL_INTERLEAVE=0
+FRANK_ARCHON_PREFILL_INTERLEAVE=1
 # Prefix cache: pinned host memory only. The disk tier was removed outright
 # (--prefix-cache-dir and --prefix-cache-disk-mib are refusals), so what used to
 # be restored from NVMe is now re-prefilled. The tier is sized in ENTRIES and one

scripts/serve-frank-archon.sh

@@ -15,13 +15,16 @@
 # log states every budget before anything is allocated (weights vs KV
 # split, the expert host pool's coverage of the worst case, the PLE pin,
 # the P2P answer per pair) — read it after every resize.
-# Deployed shape (2026-10-19, radiance 1.8.1): THREE decode slots x 200,000
-# tokens, fp8 KV, 44 GiB/rank pinned expert pool, MTP depth 2, prefix cache in
-# pinned host RAM, PLE pinned in RAM. The per-card split between the expert slab
-# and the KV pool is NO LONGER an operator number — 1.3.0 removed the VRAM pair
-# and the budget resolver derives it from headroom, the cache bound and the host
-# pool, so the slab size and the pool's token count are LEDGER lines: after every
-# boot read them out of the log and check the pool still backs 3 x 200,000.
+# Deployed shape (2026-10-19, radiance 1.8.1): TWO decode slots x 200,000 tokens,
+# fp8 KV, 44 GiB/rank pinned expert pool, MTP depth 2, prefix cache in pinned host
+# RAM, PLE pinned in RAM. The per-card split between the expert slab and the KV
+# pool is NO LONGER an operator number — 1.3.0 removed the VRAM pair and the budget
+# resolver derives it from headroom, the cache bound and the host pool, so the slab
+# size and the pool's token count are LEDGER lines. After every boot read them, plus
+# the two lines that size a prefill step — "activation arena: ... lends ... while
+# steps are at most N tokens" (the arena levels the expert slab borrows) and
+# "prefill staging: ... a step of more than N tokens ..." (the chunk size below which
+# pooled experts are read synchronously) — and the cache's session backing.
 #
 # The root unit (scripts/frank-archon.service, User=dank,
 # LimitMEMLOCK=infinity) grants the pin budget: 2 x 44 GiB expert pools +

serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md

@@ -239,45 +239,92 @@ prefill is over. `preemptions` separates the third shape: a preempted-and-recomp
 sequence stops producing entirely and then re-prefills from token 0, which is a
 stall, not sludge.
 
-Knobs, in the order worth trying them (radiance GUIDE §4.5):
+**Split the step ledger before choosing a knob.** Three counters decide which mechanism you are
+looking at, and none is a rate: `engine_step_seconds_decode_total / engine_steps_decode_total` is
+the mean **decode-only** step, and `(step_seconds_total − step_seconds_decode_total) /
+(steps_total − steps_decode_total)` is the mean **chunk step** (a chunk alone, or a chunk with
+decode rows beside it). On frank-archon's live 1.8.1 shape that pair is **15.5 ms against 166 ms**,
+with 4.7 % of the steps holding 34 % of the step time. Decode-only steps themselves move only
+14.1 → 16.4 ms between a prefill-free minute and a heavy one, and the lagged buckets stay flat: so
+a decode-rate collapse on that box is **step time sharing**, and the eviction/residency story is
+worth ~10 % of one step — not a lever.
 
-- `--prefill-interleave N` — after a chunk step taken beside decoders, the next N
-  steps carry decode rows only and admit nothing. Decoders keep their cadence; the
-  prompt pays that in TTFT. Counts as `engine_steps_held_total`.
-- `--busy-prefill-chunk MODE` — the chunk's share of a step that has other work:
-  `auto` (default) reserves a row plus the draft window per decode row behind it and
-  an eighth of the budget per waiting prefill/request, `off` lets the chunk take the
-  whole step (admission never runs while it does), `N` pins the share. Never under
-  the prefill stager's smallest pass — the startup line names it, and below it every
-  routed layer reads its pooled experts synchronously, which costs the prefill far
-  more than sharing saves. Counts as `engine_prefill_chunks_capped_total`.
-- `--max-num-batched-tokens` — the blunt rung, and the only one older pins have. The vendor's own
-  curve prices **halving the shared chunk at roughly a fifth of prefill throughput** (a 48-layer MoE
-  at 200k on two cards — knob pricing, not this endpoint's number, so treat it as the order of
-  magnitude a chunk decision costs). Raising it past the checkpoint interval buys nothing and loses
-  expert residency to the arena.
-- `--prefill-span N` — the inverse trade: a *lone* prefill carries N intervals a step while every
-  shared step is planned at a smaller shared budget, which on an expert-tiered box lends that arena
-  back to the expert slab. The vendor's own measurement says the win is **long-prompt-only**: nothing
-  at short prompts, a large TTFT gain at the far end of the window.
+**Four traps in the PromQL above.** Learn them before trusting a bucket:
+- **Latency is `vllm:`, cadence is `radiance:`.** The TTFT / inter-token / per-output-token / e2e
+  histograms and the running/waiting gauges exist only under the vLLM compatibility prefix; preemption
+  is `vllm:num_preemptions_total` (the counter) while `radiance:num_requests_preempted` is the
+  *instant* depth. A `radiance:`-only query finds no latency and no preemption, and reads as "the
+  endpoint never stalls anyone".
+- **The tps gauges are 1 s decayed windows, batch-pooled.** Per-stream rate is
+  `decode_tokens_per_second / vllm:num_requests_running`. With a 15 s scrape, a 120 s step samples one
+  scrape in eight — use step ≤ 30 s for a bucket table.
+- **Every counter resets at a restart and the metric set changes with the build.** Prove the window is
+  one shape (`up`, or the first sample of a counter that only the current build exports) before
+  ranking buckets; frank-archon's 48 h window straddled two containers, and one bucket's "collapse"
+  turned out to belong entirely to the older one.
+- **A 2-minute mean step time dilutes the chunk.** The sharp variable is the phase ratio in the query
+  above, never `step_seconds/steps` — 4.7 % of steps cannot move a mean that is 95 % decode.
 
-Two traps. **The arena/slab coupling makes a shared step doubly expensive on an
-expert-tiered shape**: the full-plan arena takes VRAM the resident expert slab would have had, so the
-decode rows inside that step also read experts over the link — a smaller shared budget helps twice.
-And **re-check
-`prefix_hit_rate` / `prefix_evictions` after any chunk change**: the arena and the KV
-pool are the same memory, and the failure looks like turn two taking as long as turn
-one.
+Three instruments the PromQL above cannot reach:
 
-Availability is a version question, not a config question: `--busy-prefill-chunk` and
-`--prefill-interleave` are absent from older pins — prove it on the running container
-(`podman exec <c> <binary> --help | grep -iE "prefill|chunk"`), never from the vendored
-tree, and check the source clone's depth before dating a flag from `git log -S`
-(a `--depth 1` clone reports its own tip for every search).
+- **The engine's own step-detail histograms (`prefill_steps_duration`, `prefill_step_tokens`,
+  `token_queue_wait`) are exported but reach no target**, and the arena / expert slab / heat engine /
+  prefill stager have **no metric at all**. Per-request forensics mean `curl localhost:7770/stats` on
+  the endpoint's own host — same phase counters plus `preemptions` (the count), the chunk geometry,
+  and the expert/link byte totals. `/diagnostics` answers "is the *box* the limit" in one call:
+  configuration drift, the VRAM split, each card's link chain and clocks, findings with fixes.
+- **`prefix_hit_cut_tokens_total` measures prefill load you created for yourself.** On a hybrid
+  (linear + paged) model a prefix hit reuses only as far as the last **linear-state** checkpoint, so
+  cached tokens are replayed as prefill and stall decoders for it; frank-archon ran 38 % of all its
+  prefill tokens that way, every `prefix_hit_cuts_total` labelled `reason="no_snapshot"`. A high
+  prefill rate is not always a client that sends new tokens. Retention is `--checkpoint-policy` -
+  exactly three names, `geometric-backoff` (default), `tip-only`, `keep-all`, one owner:
+  [[frank-archon]].
+- **Bucket the arriving request too** — `histogram_quantile` over
+  `vllm:time_to_first_token_seconds_bucket` by the same prefill series. The decode gauge only sees
+  sequences that are already running; the one waiting for its first token pays the same steps and
+  shows in no decode number.
 
-Run on: [[frank-archon]] — the census was taken on its 3-slot radiance 1.0.13 era (budget 2048 =
-interval 2048); the live 1.8.1 shape carries the two chunk knobs. The measured census and the verdict
-for this fleet's shape are that page's.
+Knobs, in the order worth trying them (radiance GUIDE §4.5; the pinned source is the referee):
+
+- `--prefill-interleave N` — after a chunk step taken beside decoders, the next N steps carry decode
+  rows only and admit nothing. **The knob that acts on the measured mechanism.** Decoders keep their
+  cadence; the prefilling prompt pays in TTFT and an arriving request waits out the held steps.
+  Counts as `engine_steps_held_total` — if that counter is zero, nothing is protecting anyone.
+- `--busy-prefill-chunk N` — pins a shared chunk's share of the step. **`auto` is a prefill-speed
+  setting, not decoder protection:** it reserves `1 + spec` rows per decoding sequence plus an eighth
+  of the budget per *waiting* request or prefill, and the source refuses to halve for "decode rows
+  alone", so a shape can run `auto` and still lose every chunk step. Prove engagement from
+  `engine_prefill_chunks_capped_total` against the chunk-step count. The floor is the prefill
+  stager's smallest pass — the largest arena level that lends, plus one, named on the startup line
+  `prefill staging: … a step of more than N tokens copies the pooled experts of …` — and under it
+  every routed layer reads its pooled experts synchronously: the vendor prices the whole job (four
+  decoders plus a long prompt) at **29.5 s with a 2048 chunk and 31.9 s at 512**, so shrinking the
+  chunk blindly buys latency in both directions.
+- `--prefill-span N` — needs `--max-num-batched-tokens` of N intervals. A *lone* prefill then
+  carries N intervals a step (on an expert-tiered box only from 8 intervals, where the vendor's own
+  measurement says a span starts to win), while **every other step is planned at the shared
+  budget** — a lending arena level that also keeps the stager's ring, the one combination the chunk
+  knob cannot express. Priced by the vendor at 320 MiB a card of expert residency, ~3 % off the KV
+  pool, and TTFT-neutral on one of his two 2×R9700 boxes: measure your own prompt lengths first.
+- `--max-num-batched-tokens` alone — the blunt rung, and the only one older pins have. It raises the
+  arena the expert slab loses, and on this payload its resonance is violent (MNBT 2560 collapses
+  decode — [[frank-archon]]).
+
+Two traps. **The arena/slab coupling makes a shared step expensive twice:** the full-plan arena is
+reclaimed from the expert slab *synchronously before the step*, and the loan regrows only after a
+quiet interval, so every step of a prefill run — the decode-only ones included — runs on the smaller
+slab. And **re-check `prefix_hit_rate` / `prefix_evictions` after any chunk or span change**: the
+arena, the staging ring and the KV pool are the same memory, and the failure looks like turn two
+taking exactly as long as turn one.
+
+Availability is a version question, not a config question: `--busy-prefill-chunk`,
+`--prefill-interleave` and `--prefill-span` are absent from older pins — prove it on the running
+container (`podman exec <c> <binary> --help | grep -iE "prefill|chunk"`) or from `/diagnostics`'s
+`configuration` block, never from a vendored tree, and check a source clone's depth before dating a
+flag from `git log -S` (a `--depth 1` clone reports its own tip for every search).
+
+Run on: [[frank-archon]] — owner of the live census, the measured step pair, and the arms.
 
 ## Related
 

serverAppConfigs-docs/LLM/frank-archon.md

@@ -96,7 +96,7 @@ on stop. Boot budget 1800 s (the publisher's compose allows a 15-minute `start_p
 | Spec / batch | `SPEC_TOKENS=2` — MTP depth 2, the `.rad` carries the model's own `mtp.*` head (`mtp.fc_hidden`/`fc_embedding`/`draft_head`); `MAX_BATCHED_TOKENS=2048` (MNBT — 2560 collapses decode, Laws) |
 | Prefix cache | `PREFIX_CACHE_MIB=8192`, pinned **host tier only** — the 64 GiB disk tier is gone, so an evicted prefix is re-prefilled; `STATE_DIR=/home/dank/radiance-state` now exists only to hold the rad-tune cache |
 | Template | `TEMPLATE=scripts/qwen38-froggeric-v22.5.jinja` (28,234 B) via `--override-chat-template` — froggeric's fixed Qwen3.8 template (the repo root's `chat_template.jinja`, vendored); the checkpoint's own template is baked into the `.rad` by rad-convert and this overrides it, and radiance derives the reasoning and tool-call reply format from it |
-| Step sharing | `BUSY_PREFILL_CHUNK=auto` (a shared chunk reserves a row plus the draft window per waiting decoder) · `PREFILL_INTERLEAVE=0` — held at zero until the decode-vs-prefill buckets ask for it |
+| Step sharing | `BUSY_PREFILL_CHUNK=auto` — prefill-speed first: it reserves 3 tokens per waiting decoder and ⅛ of the budget per waiting request, and does **not** shrink the chunk for decoders alone · `PREFILL_INTERLEAVE=1` — the decoder dial, **armed 2026-10-20**: one decode-only step after each chunk step taken beside decoders; meter `engine_steps_held_total`, price is the prefilling prompt's TTFT |
 | TTM gate | `TTM_MIN_MIB=150000`, `TTM_TARGET_MIB=204800` |
 | Boot | `RAD_MIN_MIB=110000` deployer floor; the listener binds only after the full load (health ~100 s, budget 1800 s) |
 
@@ -280,30 +280,92 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
 - **MNBT 2048.** 2560 collapses decode to 2.6-7.7 t/s (MTP resonance); the w4 file tier streams
   in place only at MNBT ≤ 2048 and the activation arena scales with the chunk (32768 → 33.3
   GiB, refused).
-- **Decode pays for the chunk it shares a step with.** Census of the live engine taken at the
-  3-slot shape on 1.0.13 (`/stats` + `/metrics`, cumulative over the 2026-10-15 boot):
-  1,435,585 decode-only steps at **18.7 ms** each against 127,991 prefill+mixed steps
-  at **317 ms** — 60 % of engine wall-clock spent inside 8 % of the steps, because the
-  chunk is `min(remaining, checkpoint interval 2048, MNBT 2048)` = 2048 rows and every
-  decode row in that step waits the whole chunk. Day of Prometheus (120 s step, 721
-  samples): `decode_tokens_per_second` median **13.3 t/s** in the 56 % of samples where
-  `prefill_tokens_per_second` ≥ 1000, against 106-240 t/s in the light-prefill buckets;
-  bucketed by mean step time it is 128.7 t/s at 0.02-0.05 s steps and 12.0 t/s at
-  0.15-0.4 s. The stall is **during**, not after — decode in windows 2-14 min after a
-  busy-prefill window sits at the idle baseline (58-64 vs 77-85 t/s, the residual
-  confounded by co-scheduling), so no lingering expert-residency penalty at 2-minute
-  resolution, and `preemptions: 0` rules out preempt-and-re-prefill. Method:
-  [[LLM Performance Diagnosis]] §9. The law is shape-independent; the fix lives at 1.8.1 (next law),
-  and nobody has re-run the census since the migration — so the fix is ordered, **not verified**.
-- **The knobs for it arrived with 1.8.1, and only one of them is on.** `--busy-prefill-chunk` caps the
-  shared chunk so a step with waiting decoders takes a smaller piece (live: `auto`, the engine picks;
-  the explicit value is 512), and `--prefill-interleave N` inserts N decode-only steps after a chunk
-  step taken beside decoders. `interleave` is deliberately **0** — the env comment holds it there until
-  the decode-vs-prefill buckets ask for it, and those buckets have not been read since the migration.
-  The vendor prices a 1024 chunk at about a fifth off prefill throughput to halve the stall; that is a
-  knob for choosing tomorrow's chunk, not a measurement of this shape. MNBT remains an arm and not an
-  edit: 2560 collapses decode through MTP resonance. Any chunk change re-scores draft acceptance before
-  it is trusted.
+- **Decode pays for the chunk it shares a step with, and `--busy-prefill-chunk auto` does not
+  protect it.** Re-measured on the live 1.8.1 shape (34.8 h boot, Prometheus at 60 s,
+  `engine_step_seconds{,_decode}_total` / `engine_steps{,_decode,_mixed,_prefill}_total` /
+  `prefill_tokens_total`): decode-only steps run at **15.5 ms**, chunk steps (a chunk alone, or a
+  chunk with decode rows beside it) at **166 ms** — 4.7 % of the steps hold **34 % of the engine's
+  step time**, so a session decoding while another prompts advances one token per 166 ms. Bucket the
+  harness's `radiance:decode_tokens_per_second` by `radiance:prefill_tokens_per_second` (15 s
+  scrapes; per-stream rate = the gauge ÷ `vllm:num_requests_running`): median **166 t/s** under light
+  prefill, 81 t/s at 1-5k, **13 t/s above 5k** — a **12.7×** collapse, and at 13 t/s with two
+  sessions running each stream sits at 6.7 t/s = 149 ms a token, one token per shared step. Nothing
+  else moves: decode-*only* steps stay 14-18 ms in every bucket (14.1 ms in a prefill-free minute,
+  17.9 ms in the deep one) and the relationship is lag-0 only (r = −0.74 at lag 0, |r| ≤ 0.19 from
+  30 s on, decode back at baseline by 5 min) — residency is not the mechanism either. The second
+  victim is the **arriving** request: through a high-prefill window TTFT p90 goes 2.4 → **18.0 s**
+  and 15.5 % of requests wait over 10 s for their first token, with `http_queue_depth` at 2-3, while
+  median TTFT (~1.9 s) and per-token latency barely move. Preemption is real but burst-shaped: 264
+  `vllm:num_preemptions_total` in the boot, 108 of them inside 105 s of a 6.5k-tok/s prefill. The
+  chunk is `min(remaining, checkpoint interval, MNBT, busy_cap)`; the first three read 2048 here,
+  and `busy_cap` is the fourth law.
+- **`auto` is a prefill-speed dial, not a decoder-protection dial.** `Scheduler::busy_cap` reserves
+  `1 + spec` rows per decoding sequence behind the chunk (3 tokens at MTP 2) plus one eighth of the
+  budget per *waiting request or waiting prefill*, and the code refuses to do more for decoders:
+  "not for decode rows alone: their reserve is a few tokens, so the chunk ends a quantum short and
+  the next step finishes the interval… Halving would hold a long prompt to half-interval chunks while
+  anything decodes behind it, which costs more" (`core/sched/scheduler.cpp:874-878`, 1.8.1). Live
+  proof: `prefill_chunks_capped` 5,018 against 62,386 chunk steps — 8 %, and those are the minutes
+  with a waiter, not the minutes with a decoder. An explicit `N` is the only value that applies
+  "whenever the chunk shares the step".
+- **At MNBT 2048 no chunk value keeps both the arena loan and the staged copies.** The arena is a
+  ladder of levels (256 and 1024 rows here) and a step's rows are its **whole** token count —
+  `max(n_tok, n_seq × (1 + spec))`, `core/engine.cpp:1142` — so a 1024-token chunk with two decode
+  rows is a 1030-row step: it falls back to the full plan, which "lends nothing", and it arms the
+  prefill stager. The stager's floor is the largest lending level plus one, 1025 rows here — named on
+  the boot line `prefill staging: … a step of more than 1024 tokens copies the pooled experts of 48
+  routed layers onto the card up to 3 layers ahead` — and below it "every routed layer reads its
+  pooled experts synchronously and the prefill slows far more than sharing saves" (GUIDE §4.5). The
+  vendor's own number for naive shrinking, on this architecture: four decoders with a long prompt
+  beside them finished in **29.5 s at a 2048 chunk and 31.9 s at 512**.
+- **A big step takes the arena from the expert slab and keeps it for the whole prefill.** The arena
+  lends 987.77 MiB (steps ≤ 256 rows) or 670.26 MiB (≤ 1024) to the slab; a full-plan step reclaims
+  it **synchronously before the step** — units revert to their host copies (an address change, not a
+  copy) and both read streams drain — and `arena_tick` regrows the loan only **2 s after the last big
+  step** (`core/engine.cpp:1129-1202`), so every step of the prefill run, decode-only included, runs
+  with a slab ~780 expert units smaller over a link also carrying the staged copies. `/diagnostics`
+  says the binding constraint in one line: "a prefill step streams every routed layer's pooled
+  experts onto the card, so its speed is bounded by the link; decode reads the ones it routes to".
+  Measured size of the whole effect: the ~10 % above. Real, small, not worth chasing.
+- **`--prefill-interleave` is the dial that buys decode its cadence; armed at 1 on 2026-10-20.**
+  N decode-only steps follow any chunk step taken beside decoders; held steps admit nothing, so it
+  costs the prefilling prompt its TTFT and an arriving request its first step. Rate arithmetic at the
+  two measured step times and 0.657 acceptance (≈2.3 tokens a step): N=0 → ~14 t/s, **N=1 (armed) →
+  ~23**, N=2 → ~33, N=4 → ~48, N=8 → ~69, with the prefill slowed by (166 + 15.5N)/166. **Arithmetic, not
+  measurement** — the arm is chosen by the §9 bucket pair plus `steps_held`, and `auto` is not a
+  substitute for it.
+- **`--prefill-span` is the shape that keeps the loan, and it is priced.** It needs MNBT = N
+  intervals, so `--prefill-span 2 --max-num-batched-tokens 4096`: a *lone* prefill carries 2
+  intervals a step (and on an expert-tiered box only from 8 intervals / 16k tokens, where the span
+  starts to win), while **every other step is planned at the shared budget of 2048** — which is a
+  lending level *and* keeps the stager's ring, the one combination `--busy-prefill-chunk` cannot
+  express. Costs per GUIDE §4.5: 320 MiB a card of expert residency, ~3 % off the KV pool (§4.3's
+  cliff is sharp), a decode step uploads twice the per-step arrays at 4096, and checkpoints inside a
+  span are not written, so a partially-shared prompt resumes one interval earlier. On the vendor's
+  second box (2×R9700, w4nl64, TP2) it was TTFT-neutral — 2.61 vs 2.56 s at 18k, 10.72 vs 10.55 at
+  75k — hence their own rule: measure your own prompts at two ranks before turning it on. MNBT 4096
+  does not boot at 3 slots; this shape is 2.
+- **Most of this endpoint's prefill work is replay of tokens the prefix cache already had.** Over
+  the same window `prefill_tokens_total` 65.2 M against `prefix_hit_cut_tokens_total` 24.9 M = **38 %
+  replayed**, and it is not an eviction: a hit on a hybrid model reuses only as far as the last
+  **linear-state** checkpoint (`reuse = min(h.n_attn_tokens, h.n_linear_tokens)`,
+  `core/sched/scheduler.cpp:953-978`), and the retention default keeps a **geometric ladder** - about
+  `log2(depth/interval) + 2` snapshots - so a turn that diverges `d` behind the tip can resume as
+  deep as `tip − 3d`. The live ladder for a 200k session at interval 2048 is 2048 · 131072 · 163840
+  · … · 198656: diverge at 120k and 21.7k tokens replay, diverge at 50k and 48k. Every
+  `prefix_hit_cuts_total` carries `reason="no_snapshot"` (the other two reasons stay dead), so the
+  cause object agrees and `hit_cut_tokens` is the cost. `--checkpoint-policy` admits exactly
+  **`geometric-backoff` | `tip-only` | `keep-all`** and refuses anything else at boot. It governs
+  retention only - the writes happen at every interval whatever the policy says (only `--prefill-span`
+  skips them) - so `keep-all` buys a replay bounded by one interval at any depth for **pinned host RAM
+  alone**: 97 snapshots a 200k session, `--checkpoint-slots auto` asking 196 × 111.98 MiB = 21.4 GiB
+  against a store budget of an eighth of host RAM (31.5 GiB here, so `share` admits it) with `room` -
+  the pinning room left beside the expert pool at boot - as the hard bound the ledger line names.
+  Retention also spares a snapshot a later turn resumed from (`core/mem/prefix.cpp:709-723`), so a
+  chain that keeps resuming at one deep point stops paying for it. Shortening
+  `--checkpoint-interval` does **not** buy the deep case: the coarsest ladder scale is the largest
+  `interval × 2^k` under the depth, so the deepest gap stays ≈ half the prompt (129,024 tokens at
+  interval 2048 against 130,816 at 256) while every cut multiplies the snapshot writes.
 - **Slots cost KV, rows, and window — never speed.** `--max-num-seqs` carries no radiance constraint: it
   adds ≈110 MiB of recurrent state per slot (the live `kv_full_backing` arithmetic prices them exactly),
   declares the fused TP all-reduce over `seqs × (1 + spec)` rows, and `wht6` is bit-exact up to 25 rows
@@ -396,12 +458,35 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
   4600 MT/s ended it, with zero `gdn_hip` faults either side; the prefix-caching / GDN re-seed
   theory was confounded on both arms. Consumer non-ECC raises no MCE for a silent flip and a
   60 s / 8 GiB pattern check covers 0.5 % of 246 GiB — silent RAM corruption outranks every
-  software theory.
+  software theory. **That mitigation is a BIOS setting and it is not in effect now:** `sudo
+  dmidecode -t memory | grep -i configured` answers 4800 MT/s on 2026-10-20, and the only event
+  between the fix and today is the 2026-10-12 power failure that also reset TTM. Check the speed
+  before crediting any software revert with a fix — and before believing a clean run of the arm
+  you just tested, because the confounder is still in the slot.
 - **TP=2 here** is legal since the 2026-09-20 ProArt B850 swap (x8/x8 on CPU root ports,
   `Routing+`/`ReqEn+` end to end) only because radiance's wire is `libr4d`'s own one-shot
   all-reduce; RCCL needs atomics on the whole chain and fails `hipErrorIllegalState` without
   them (). `wht6` buys +5.6-6.3 % prefill for ~1.3 % of 8-stream
   decode, `exact` is the bit-exact alternative.
+- **The link is already at the board's ceiling — do not chase the `link-below-max` warning.**
+  Both root ports (`00:01.1`, `00:01.3`) are **x8 max** at 32 GT/s, so PCIe 5.0 x8 ≈ 31.5 GB/s is
+  the whole budget per card and the cards' own `pcie_dpm` tables expose only x8 states.
+  `/diagnostics` still warns that the on-card switch hop (`01:00.0`, `05:00.0`) runs "x8 of its
+  x16 while the card is 100 % busy" and suggests a reboot or a reseat — pointless when the host
+  side of the chain is x8 max, which the same endpoint reports two findings later as
+  `link-capped`. The width that matters is the minimum of the chain, and it is x8 either way, so
+  the only lever on a link-bound prefill is **less traffic over the link** (more expert
+  residency, fewer pooled experts), never a wider link. P2P resolves **on** (`auto` asks the
+  driver, the two cards reach each other's memory), so the collectives do not hairpin through host
+  RAM, and the engine's copies ride one DMA queue that "give[s] way to the collectives" — the
+  all-reduce wins the link, the staged experts wait.
+- **Eight cores is the whole host, media stack included.** Dankserver is a Ryzen 7 7700X (8
+  cores / 8 threads, one NUMA node, `nodes: -1` for both cards) running Jellyfin, Plex, Calibre
+  and RomM beside the engine, and `--http-threads` defaults to one thread per core. Measured
+  during a heavy prefill: load average ≈ 1.0, `jellyfin` 11 % CPU, `radiance` 7 % — **CPU
+  contention is not the current decode culprit**, so do not spend an experiment window on it;
+  re-read `uptime` and `ps -eo pcpu,comm --sort=-pcpu | head` first if a future collapse looks
+  different from the step-sharing one.
 - `--prefix-cache-dir` creation is **not recursive** — the serve script `mkdir -p`s the path or
   the disk tier silently never opens. The tier does survive a payload swap when arch and
   tokenizer match (it recovered 900,176 blocks on the 2026-10-15 swap; `cache_n` proves a hit).
@@ -445,11 +530,19 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
 ## Contract (the wire)
 
 - **Live surface:** `/health` (== `/ping`), `/v1/models`, `/v1/chat/completions`
-  `/v1/completions`, `/metrics`, `/stats`, `/sessions`, `/dashboard`. `/health` answers 200
-  JSON (model id, `in_flight`, `queue_depth`) while serving and **503 only while stopping**
-  (`core/server/admin.cpp:385`), and the listener binds *after* the full load (`main.cpp`:
-  `engine.start()` then `engine.serve()`) — during the load the port refuses connections, so a
-  bare `curl -fsS …/health` gate is honest and needs no body grep.
+  `/v1/completions`, `/metrics`, `/stats`, `/sessions`, `/dashboard`, **`/diagnostics`**. `/health`
+  answers 200 JSON (model id, `in_flight`, `queue_depth`) while serving and **503 only while
+  stopping** (`core/server/admin.cpp:385`), and the listener binds *after* the full load
+  (`main.cpp`: `engine.start()` then `engine.serve()`) — during the load the port refuses
+  connections, so a bare `curl -fsS …/health` gate is honest and needs no body grep.
+- **`/diagnostics` is the instrument for "is the box, not the config, the limit"** — one call
+  returns `configuration` (every flag, so drift from the env file is checkable without the host),
+  `environment`, `derived` (the VRAM split as the ledger prints it), `hardware` (host, per-card
+  `link_chain` + `pcie_dpm` + clocks + power cap + DMA engine choice + P2P answer) and `findings`
+  (level / what / why / fix). On this box it reports `experts-from-host` (5.99 GiB a rank crosses
+  PCIe when used), `link-capped` (root ports are x8 max — the board), `swap-in-use`,
+  `tp-wire-lossy`, and **`link-below-max` as a `warn` at 100 % card busy on both cards** — which is
+  structural on this board; see the x8-ceiling law under *This host, and the deploy path*.
 - **Served id.** `FRANK_ARCHON_SERVED_NAME=frank-archon` (the wire rename of 2026-10-09 moved
   only the alias; container, unit, scripts and env keep the service name). The env's contract
   line is that **radiance validates the requested model name, so every client — harness, bench
@@ -476,12 +569,28 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
   and 16,384 patches per pass. Probe with a ≥ 256 px solid-colour PNG and a ≥ 256 token budget
   — at 64 tokens you get empty `content` and the colour only inside `reasoning_content`.
 - **Capacity behaviour:** a prompt over `--max-model-len` is refused, never truncated; a request
-  longer than the KV pool's token count is accepted then refused as too large to fit; under the
-  3-slot contract a **fourth** full-height conversation is refused, not paged or squeezed.
+  longer than the KV pool's token count is accepted then refused as too large to fit; a request
+  beyond `--max-num-seqs` is held by the HTTP port (`queue_cap`, default 8 × slots = 16 here) and
+  answered 503 only past that, never paged or squeezed. Because the cache funds **1.8** full-height
+  sessions, admitting a second near-full-height conversation can **preempt a running sequence from
+  the back** to free its blocks (`core/sched/scheduler.cpp:1291-1322`); the victim is recomputed, so
+  its session halts and then re-prefills. Meter: `vllm:num_preemptions_total` (264 in the current
+  boot, all in prefill-coincident bursts — 108 inside 105 s); `/stats preemptions` is the cumulative
+  count and `radiance:num_requests_preempted` only the instant depth.
 - **Observability:** `/metrics` carries vLLM-compatible names plus `radiance:*`, scraped by the
-  Dankclaw Prometheus job at `192.168.0.235:7770` over plain LAN routing (the netavark
-  published-port quirk is same-host only). The boot ledger — the referee for every shape number
-  above — is in the container log (`podman logs frank-archon`)
+  Dankclaw Prometheus job at `192.168.0.235:7770` (15 s interval) over plain LAN routing (the
+  netavark published-port quirk is same-host only). **Latency lives under `vllm:`, cadence under
+  `radiance:`** — TTFT / inter-token / per-output-token / e2e / decode-time histograms and the
+  running/waiting gauges are `vllm:*` with no radiance-native equivalent, so a `radiance:`-only
+  query misses every latency answer. The step-phase counters are all scraped:
+  `engine_steps{,_decode,_mixed,_prefill,_held}_total`, `engine_step_seconds{,_decode}_total`,
+  `engine_prefill_chunks_capped_total`, `prefill_tokens_total`, `decode_tokens_per_second`,
+  `prefill_tokens_per_second`, `prefix_hit_cut_tokens_total`, `http_queue_depth`. Two holes: the
+  engine's own step-detail histograms (`prefill_steps_duration`, `prefill_step_tokens`,
+  `token_queue_wait`) are exported but reach no target, and the **arena / expert slab / heat engine /
+  prefill stager have no metric at all** — that half of every law above is read from the boot ledger,
+  `/stats` and `/diagnostics`, never from a graph. The boot ledger — the referee for every shape
+  number here — is in the container log (`podman logs frank-archon`);
   `/home/dank/logs/frank_archon.log` is where the serve script redirects `podman run`.
 
 ## Files

shared/grafana/provisioning/dashboards/host-system.json

@@ -0,0 +1,144 @@
+{
+  "uid": "host-system",
+  "title": "Hosts \u2014 Dankclaw & Dankserver",
+  "description": "node_exporter for both servers. Starter set \u2014 throw-at-the-wall breadth; refine the windows and drop the noise. The picker selects the host; CPU/load/memory/filesystem/network are per-node.",
+  "tags": ["host", "system"],
+  "timezone": "browser",
+  "schemaVersion": 39,
+  "refresh": "30s",
+  "time": { "from": "now-6h", "to": "now" },
+  "templating": {
+    "list": [
+      {
+        "name": "node",
+        "label": "host",
+        "type": "query",
+        "datasource": { "type": "prometheus", "uid": "prometheus" },
+        "query": { "query": "label_values(up{job=~\"node-.*\"}, job)", "refId": "A" },
+        "includeAll": true,
+        "multi": true,
+        "current": { "text": "All", "value": "$__all" }
+      }
+    ]
+  },
+  "panels": [
+    {
+      "id": 1, "type": "stat", "title": "Uptime",
+      "description": "Time since boot. A drop means a reboot.",
+      "gridPos": { "h": 4, "w": 6, "x": 0, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "s" }, "overrides": [] },
+      "targets": [{ "expr": "time() - node_boot_time_seconds{job=~\"$node\"}", "legendFormat": "{{job}}", "refId": "A" }]
+    },
+    {
+      "id": 2, "type": "stat", "title": "Memory used",
+      "description": "1 - MemAvailable/MemTotal. The honest \"in use\" figure (excludes reclaimable cache).",
+      "gridPos": { "h": 4, "w": 6, "x": 6, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1, "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 0.8 }, { "color": "red", "value": 0.92 }] } }, "overrides": [] },
+      "targets": [{ "expr": "1 - node_memory_MemAvailable_bytes{job=~\"$node\"} / node_memory_MemTotal_bytes{job=~\"$node\"}", "legendFormat": "{{job}}", "refId": "A" }]
+    },
+    {
+      "id": 3, "type": "stat", "title": "Load 5m / core",
+      "description": "Per-core load5. >1 sustained means runnable tasks are queuing behind cores.",
+      "gridPos": { "h": 4, "w": 6, "x": 12, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 0.7 }, { "color": "red", "value": 1 }] } }, "overrides": [] },
+      "targets": [{ "expr": "node_load5{job=~\"$node\"} / count by (job) (node_cpu_seconds_total{job=~\"$node\",mode=\"idle\"})", "legendFormat": "{{job}}", "refId": "A" }]
+    },
+    {
+      "id": 4, "type": "stat", "title": "Open file handles",
+      "description": "FDs allocated vs the kernel maximum. Rising toward the max precedes EMFILE across the host.",
+      "gridPos": { "h": 4, "w": 6, "x": 18, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "percentunit", "max": 1, "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 0.7 }, { "color": "red", "value": 0.9 }] } }, "overrides": [] },
+      "targets": [{ "expr": "node_filefd_allocated{job=~\"$node\"} / node_filefd_maximum{job=~\"$node\"}", "legendFormat": "{{job}}", "refId": "A" }]
+    },
+    {
+      "id": 5, "type": "timeseries", "title": "CPU busy %",
+      "description": "CPU seconds/s in each mode = cores busy per mode; the stack sums to total cores in use (16/host here). A user or system spike points at what is loading the box; steal > 0 means the hypervisor is throttling.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 4 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short", "min": 0, "custom": { "fillOpacity": 10, "lineWidth": 1, "stacking": { "mode": "normal" } } }, "overrides": [] },
+      "targets": [
+        { "expr": "sum by (job,mode) (rate(node_cpu_seconds_total{job=~\"$node\",mode!=\"idle\",mode!=\"iowait\"}[5m]))", "legendFormat": "{{job}} {{mode}}", "refId": "A" }
+      ]
+    },
+    {
+      "id": 6, "type": "timeseries", "title": "Load average",
+      "description": "load1/5/15 per host.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 4 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] },
+      "targets": [
+        { "expr": "node_load1{job=~\"$node\"}", "legendFormat": "{{job}} load1", "refId": "A" },
+        { "expr": "node_load5{job=~\"$node\"}", "legendFormat": "{{job}} load5", "refId": "B" },
+        { "expr": "node_load15{job=~\"$node\"}", "legendFormat": "{{job}} load15", "refId": "C" }
+      ]
+    },
+    {
+      "id": 7, "type": "timeseries", "title": "Memory breakdown",
+      "description": "Total, free, buffers+cache. Reclaimable cache is not pressure; MemAvailable is the real headroom.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 12 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "bytes" }, "overrides": [] },
+      "targets": [
+        { "expr": "node_memory_MemTotal_bytes{job=~\"$node\"} - node_memory_MemAvailable_bytes{job=~\"$node\"}", "legendFormat": "{{job}} used", "refId": "A" },
+        { "expr": "node_memory_MemAvailable_bytes{job=~\"$node\"}", "legendFormat": "{{job}} available", "refId": "B" }
+      ]
+    },
+    {
+      "id": 8, "type": "timeseries", "title": "Filesystem free %",
+      "description": "Per mountpoint. The media/library stacks fill disks.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 12 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1, "thresholds": { "mode": "absolute", "steps": [{ "color": "red", "value": null }, { "color": "green", "value": 0.15 }] } }, "overrides": [] },
+      "targets": [
+        { "expr": "node_filesystem_avail_bytes{job=~\"$node\",fstype!~\"tmpfs|overlay|squashfs\"} / node_filesystem_size_bytes{job=~\"$node\",fstype!~\"tmpfs|overlay|squashfs\"}", "legendFormat": "{{job}} {{mountpoint}}", "refId": "A" }
+      ]
+    },
+    {
+      "id": 9, "type": "timeseries", "title": "Disk throughput",
+      "description": "Read/write bytes/s per device.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 20 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "Bps" }, "overrides": [] },
+      "targets": [
+        { "expr": "sum by (job,device) (rate(node_disk_read_bytes_total{job=~\"$node\"}[5m]))", "legendFormat": "{{job}} {{device}} R", "refId": "A" },
+        { "expr": "-sum by (job,device) (rate(node_disk_written_bytes_total{job=~\"$node\"}[5m]))", "legendFormat": "{{job}} {{device}} W", "refId": "B" }
+      ]
+    },
+    {
+      "id": 10, "type": "timeseries", "title": "Disk busy %",
+      "description": "Fraction of time the device had I/O in flight. Near 1 = the device is the bottleneck.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 20 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1 }, "overrides": [] },
+      "targets": [
+        { "expr": "rate(node_disk_io_time_seconds_total{job=~\"$node\"}[5m])", "legendFormat": "{{job}} {{device}}", "refId": "A" }
+      ]
+    },
+    {
+      "id": 11, "type": "timeseries", "title": "Network throughput",
+      "description": "Receive (up) / transmit (down) bytes/s per interface.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 28 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "Bps" }, "overrides": [] },
+      "targets": [
+        { "expr": "sum by (job,device) (rate(node_network_receive_bytes_total{job=~\"$node\",device!~\"lo|veth.*|br.*|docker.*\"}[5m]))", "legendFormat": "{{job}} {{device}} RX", "refId": "A" },
+        { "expr": "-sum by (job,device) (rate(node_network_transmit_bytes_total{job=~\"$node\",device!~\"lo|veth.*|br.*|docker.*\"}[5m]))", "legendFormat": "{{job}} {{device}} TX", "refId": "B" }
+      ]
+    },
+    {
+      "id": 12, "type": "timeseries", "title": "Network errors / drops",
+      "description": "Interface errors and drops \u2014 the signature of a saturated link or a bad NIC/cable.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 28 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "ops" }, "overrides": [] },
+      "targets": [
+        { "expr": "sum by (job,device) (rate(node_network_receive_errs_total{job=~\"$node\"}[5m]))", "legendFormat": "{{job}} {{device}} RX err", "refId": "A" },
+        { "expr": "sum by (job,device) (rate(node_network_transmit_errs_total{job=~\"$node\"}[5m]))", "legendFormat": "{{job}} {{device}} TX err", "refId": "B" }
+      ]
+    }
+  ]
+}

shared/grafana/provisioning/dashboards/logs-overview.json

@@ -0,0 +1,75 @@
+{
+  "uid": "logs-overview",
+  "title": "Logs \u2014 Loki (systemd journal)",
+  "description": "Everything in Loki arrives as the journal (job=systemd-journal); the container is a JSON field parsed at query time, not a stream label. Starter breadth \u2014 the volume/error panels group by the parsed container_name. Loki keeps ~7 days; wide ranges are cheap counts, not the raw log panel.",
+  "tags": ["logs", "loki"],
+  "timezone": "browser",
+  "schemaVersion": 39,
+  "refresh": "30s",
+  "time": { "from": "now-1h", "to": "now" },
+  "templating": {
+    "list": [
+      {
+        "name": "container",
+        "label": "container",
+        "type": "query",
+        "datasource": { "type": "loki", "uid": "loki" },
+        "query": { "query": "{job=\"systemd-journal\"} | json", "label": "container_name", "refId": "A" },
+        "includeAll": true,
+        "multi": true,
+        "refresh": 2,
+        "current": { "text": "All", "value": "$__all" }
+      }
+    ]
+  },
+  "panels": [
+    {
+      "id": 1, "type": "timeseries", "title": "Log volume by container (lines/s)",
+      "description": "Parse the journal JSON, count lines per container. Spikes = a chatty or crashing unit.",
+      "gridPos": { "h": 9, "w": 24, "x": 0, "y": 0 },
+      "datasource": { "type": "loki", "uid": "loki" },
+      "fieldConfig": { "defaults": { "unit": "reqps", "custom": { "fillOpacity": 20, "stacking": { "mode": "normal" } } }, "overrides": [] },
+      "targets": [{ "expr": "sum by (container_name) (count_over_time({job=\"systemd-journal\"} | json [$__auto]))", "legendFormat": "{{container_name}}", "refId": "A" }]
+    },
+    {
+      "id": 2, "type": "timeseries", "title": "Total log rate",
+      "description": "All journal lines per second.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 9 },
+      "datasource": { "type": "loki", "uid": "loki" },
+      "fieldConfig": { "defaults": { "unit": "reqps" }, "overrides": [] },
+      "targets": [{ "expr": "sum(count_over_time({job=\"systemd-journal\"} [$__auto]))", "legendFormat": "all lines/s", "refId": "A" }]
+    },
+    {
+      "id": 3, "type": "timeseries", "title": "Error-level line rate",
+      "description": "Lines matching error/panic/fatal/fail. A blunt text match \u2014 the journal carries no structured level for all units.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 9 },
+      "datasource": { "type": "loki", "uid": "loki" },
+      "fieldConfig": { "defaults": { "unit": "reqps", "color": { "mode": "fixed", "fixedColor": "red" } }, "overrides": [] },
+      "targets": [{ "expr": "sum(count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal|fail)\" [$__auto]))", "legendFormat": "error lines/s", "refId": "A" }]
+    },
+    {
+      "id": 4, "type": "timeseries", "title": "Errors by container",
+      "description": "Which unit is throwing. Rank by the top offenders.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 17 },
+      "datasource": { "type": "loki", "uid": "loki" },
+      "fieldConfig": { "defaults": { "unit": "reqps" }, "overrides": [] },
+      "targets": [{ "expr": "sum by (container_name) (count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal)\" | json [$__auto]))", "legendFormat": "{{container_name}}", "refId": "A" }]
+    },
+    {
+      "id": 5, "type": "stat", "title": "Error lines (last 15m)",
+      "description": "Absolute count over the window, not a rate.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 17 },
+      "datasource": { "type": "loki", "uid": "loki" },
+      "fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 50 }, { "color": "red", "value": 500 }] } }, "overrides": [] },
+      "targets": [{ "expr": "sum(count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal)\" [15m]))", "instant": true, "legendFormat": "errors/15m", "refId": "A" }]
+    },
+    {
+      "id": 6, "type": "logs", "title": "Journal (filtered)",
+      "description": "Raw lines for the selected containers. Set the container picker; keep the window tight.",
+      "gridPos": { "h": 12, "w": 24, "x": 0, "y": 25 },
+      "datasource": { "type": "loki", "uid": "loki" },
+      "options": { "showTime": true, "wrapLogMessage": true, "sortOrder": "Descending", "enableLogDetails": true },
+      "targets": [{ "expr": "{job=\"systemd-journal\"} | json | container_name=~\"$container\"", "refId": "A" }]
+    }
+  ]
+}

shared/grafana/provisioning/dashboards/observability.json

@@ -0,0 +1,96 @@
+{
+  "uid": "observability",
+  "title": "Observability \u2014 self-monitoring",
+  "description": "The health of the stack that watches the fleet: Prometheus scrape/TSDB, Loki ingest, Grafana. If this is red the dashboards lie. Note: Alertmanager is running but NOT scraped by Prometheus (no job) \u2014 its metrics are absent by design gap, not fault.",
+  "tags": ["observability", "meta"],
+  "timezone": "browser",
+  "schemaVersion": 39,
+  "refresh": "30s",
+  "time": { "from": "now-6h", "to": "now" },
+  "panels": [
+    {
+      "id": 1, "type": "stat", "title": "Targets up",
+      "gridPos": { "h": 4, "w": 4, "x": 0, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }] } }, "overrides": [] },
+      "targets": [{ "expr": "count(up == 1)", "instant": true, "refId": "A" }]
+    },
+    {
+      "id": 2, "type": "stat", "title": "Targets down",
+      "description": "Known structural downs: comfyui (no metrics endpoint), gitea (metrics off), and the homepage 'services' probes. Real regressions ride up from here too \u2014 check the table before dismissing.",
+      "gridPos": { "h": 4, "w": 4, "x": 4, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "red", "value": 1 }] } }, "overrides": [] },
+      "targets": [{ "expr": "count(up == 0)", "instant": true, "refId": "A" }]
+    },
+    {
+      "id": 3, "type": "stat", "title": "TSDB head series",
+      "description": "Active series in memory. A jump = a new exporter or a label-cardinality leak.",
+      "gridPos": { "h": 4, "w": 5, "x": 8, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] },
+      "targets": [{ "expr": "prometheus_tsdb_head_series", "instant": true, "refId": "A" }]
+    },
+    {
+      "id": 4, "type": "stat", "title": "Discovered targets",
+      "gridPos": { "h": 4, "w": 5, "x": 13, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] },
+      "targets": [{ "expr": "sum(prometheus_sd_discovered_targets)", "instant": true, "refId": "A" }]
+    },
+    {
+      "id": 5, "type": "stat", "title": "Prometheus mem",
+      "gridPos": { "h": 4, "w": 3, "x": 18, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "bytes" }, "overrides": [] },
+      "targets": [{ "expr": "process_resident_memory_bytes{job=\"prometheus\"}", "instant": true, "refId": "A" }]
+    },
+    {
+      "id": 6, "type": "stat", "title": "Grafana up",
+      "gridPos": { "h": 4, "w": 3, "x": 21, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "mappings": [{ "type": "value", "options": { "1": { "text": "UP", "color": "green" }, "0": { "text": "DOWN", "color": "red" } } }], "thresholds": { "mode": "absolute", "steps": [{ "color": "red", "value": null }, { "color": "green", "value": 1 }] } }, "overrides": [] },
+      "targets": [{ "expr": "up{job=\"grafana\"}", "instant": true, "refId": "A" }]
+    },
+    {
+      "id": 7, "type": "table", "title": "Target health",
+      "description": "Every scrape target and its current up. Sort by Value to surface regressions.",
+      "gridPos": { "h": 9, "w": 12, "x": 0, "y": 4 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "options": { "sortBy": [{ "displayName": "Value", "sortOrder": "Asc" }] },
+      "targets": [{ "expr": "up", "format": "table", "instant": true, "refId": "A" }]
+    },
+    {
+      "id": 8, "type": "timeseries", "title": "Scrape interval drift (p99 actual)",
+      "description": "Actual seconds between scrapes (summary quantile). Above the 15s scrape_interval = Prometheus is falling behind its own cadence (slow endpoints or overload).",
+      "gridPos": { "h": 9, "w": 12, "x": 12, "y": 4 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "s" }, "overrides": [] },
+      "targets": [{ "expr": "prometheus_target_interval_length_seconds{quantile=\"0.99\"}", "legendFormat": "{{instance}}", "refId": "A" }]
+    },
+    {
+      "id": 9, "type": "timeseries", "title": "Scrape samples/s",
+      "description": "Total samples ingested across all jobs.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 13 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] },
+      "targets": [{ "expr": "sum(rate(scrape_samples_scraped[5m]))", "legendFormat": "samples/s", "refId": "A" }]
+    },
+    {
+      "id": 10, "type": "timeseries", "title": "Loki request rate",
+      "description": "Loki served requests/s (push, query). A drop to zero means ingest or query has stopped.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 13 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "reqps" }, "overrides": [] },
+      "targets": [{ "expr": "sum by (route) (rate(loki_request_duration_seconds_count[5m]))", "legendFormat": "{{route}}", "refId": "A" }]
+    },
+    {
+      "id": 11, "type": "timeseries", "title": "Exporter process CPU",
+      "description": "CPU by monitored process \u2014 spot a runaway exporter or the Prometheus process itself.",
+      "gridPos": { "h": 8, "w": 24, "x": 0, "y": 21 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "percentunit" }, "overrides": [] },
+      "targets": [{ "expr": "sum by (job) (rate(process_cpu_seconds_total[5m]))", "legendFormat": "{{job}}", "refId": "A" }]
+    }
+  ]
+}

tests/e2e/playwright.config.ts

@@ -31,8 +31,12 @@ export default defineConfig({
       // container name, so it needs its own pass condition). 18- joined on
       // 2026-09-24 (OpenWebUI SSO through authentik). 20- joined on 2026-10-21
       // (Grafana's OAuth2 relying-party login, which the edge-only 13- never
-      // follows).
-      testMatch: /(10|12|13|16|17|18|20)-.*\.spec\.ts/,
+      // follows). 21- joined 2026-10-21 (the open-notebook vhost, deployed
+      // 2026-10-19 - the newest guarded app, in neither GUARDED list nor any
+      // spec until now: the 17-hermes "uncovered vhost broke silently" class).
+      // 22- joins Prometheus's app-is-really-there proof (its own API), closing
+      // the "not a login wall but maybe a 404" hole 20- closed for Grafana.
+      testMatch: /(10|12|13|16|17|18|20|21|22)-.*\.spec\.ts/,
       use: { storageState: ".auth/qa-e2e.json" },
       dependencies: ["setup"],
     },

tests/e2e/specs/14-guard.spec.ts

@@ -15,7 +15,7 @@ const BASE = "https://www.franksdank.cloud";
 // Every vhost that Caddy routes through anubis:8923 (see dankclaw/Caddyfile).
 // Keep this list in lockstep with the Caddyfile: a new guarded vhost must be
 // added here or the guard-chain audit is incomplete.
-const GUARDED = ["www", "search", "romm", "files", "seerr", "grafana", "prometheus", "loki"];
+const GUARDED = ["www", "search", "romm", "files", "seerr", "grafana", "prometheus", "loki", "notebook"];
 
 // Markers that prove app content leaked past the guard to an anonymous user.
 const LEAK_MARKERS: Record<string, string[]> = {
@@ -25,6 +25,7 @@ const LEAK_MARKERS: Record<string, string[]> = {
   "seerr.franksdank.cloud": ["Request Movie"],
   "grafana.franksdank.cloud": ["Dashboards"],
   "prometheus.franksdank.cloud": ["Expression Browser", "Graph"],
+  "notebook.franksdank.cloud": ["Open Notebook"],
 };
 
 test("homepage root (/) is guarded for unauthenticated users", async ({ page }) => {

tests/e2e/specs/21-notebook.spec.ts

@@ -0,0 +1,52 @@
+import { expect, test } from "@playwright/test";
+
+// open-notebook (notebook.franksdank.cloud), deployed 2026-10-19, is a guarded
+// vhost whose outpost (notebook-proxy) fronts an app with its OWN password
+// login - the same shape as hermes (17-) and OpenWebUI (18-): the guard chain
+// (Anubis PoW -> authentik outpost) must hand off to the app, and the app's own
+// login page is the expected landing. Until now this host appeared in neither
+// the 13-/14- GUARDED lists nor any dedicated spec, so a broken upstream would
+// have stayed invisible - the exact 17-hermes "uncovered vhost" failure class.
+// 14- now covers the unauthenticated gate; this spec owns authenticated
+// reachability with a POSITIVE app marker, not just "not a login wall".
+
+const VHOST = "https://notebook.franksdank.cloud/";
+const CHALLENGE = "Making sure you're not a bot";
+
+test("notebook vhost reaches the open-notebook app through the guard chain", async ({ page }) => {
+  test.setTimeout(180_000);
+  await page.goto(VHOST);
+
+  // The Anubis proof-of-work clears in a real browser and the outpost hands off
+  // to the app; the user must never be parked on an authentik flow.
+  await expect
+    .poll(async () => !(await page.locator("body").innerText()).includes(CHALLENGE), { timeout: 90_000 })
+    .toBe(true);
+  await expect
+    .poll(() => new URL(page.url()).hostname, { timeout: 60_000 })
+    .toBe("notebook.franksdank.cloud");
+  expect(
+    new URL(page.url()).pathname.startsWith("/if/"),
+    "must not be parked on an authentik flow",
+  ).toBe(false);
+  expect(
+    await page.locator('input[name="uidField"]').count(),
+    "must not show the authentik login form",
+  ).toBe(0);
+
+  // Positive identity: the page really is open-notebook's own login, not a
+  // Caddy error or the IdP shell. The app answering its own login is the pass.
+  await expect(page, "destination title must be Open Notebook").toHaveTitle(/open notebook/i, {
+    timeout: 30_000,
+  });
+  await expect(page.locator("body"), "open-notebook login marker").toContainText(
+    /enter your password/i,
+    { timeout: 30_000 },
+  );
+  const body = await page.locator("body").innerText();
+  expect(body, "must not leak the IdP flow shell").not.toContain("Powered by authentik");
+  expect(body, "must not be a Caddy upstream error").not.toContain("Bad Gateway");
+
+  const resp = await page.request.get(VHOST);
+  expect(resp.status(), "open-notebook must answer, not error").toBeLessThan(500);
+});

tests/e2e/specs/22-prometheus-sso.spec.ts

@@ -0,0 +1,33 @@
+import { expect, test } from "@playwright/test";
+
+// Prometheus (prometheus.franksdank.cloud) sits behind the same embedded outpost
+// as Grafana. 13-sso and 11-admin only ever assert it is "not a login wall and
+// under 500" - which a Prometheus serving a 404 or a blank page would ALSO pass.
+// This is the app-is-really-there proof 20- provides for Grafana: with the
+// standard session the outpost hands off silently to the Prometheus UI, and the
+// app's own API answers success. A guard-route or upstream regression that lets
+// any non-Prometheus answer through fails here.
+
+test("SSO: Prometheus app answers its own API through authentik", async ({ page, request }) => {
+  test.setTimeout(180_000);
+  await page.goto("https://prometheus.franksdank.cloud/");
+
+  // Challenge clears, outpost hands off silently to the app (no credential prompt).
+  await expect
+    .poll(
+      async () => !(await page.locator("body").innerText()).includes("Making sure you're not a bot"),
+      { timeout: 90_000 },
+    )
+    .toBe(true);
+  await expect
+    .poll(() => new URL(page.url()).hostname, { timeout: 60_000 })
+    .toBe("prometheus.franksdank.cloud");
+  expect(await page.locator('input[name="uidField"]').count(), "no authentik form").toBe(0);
+  await expect(page, "must land on the Prometheus UI").toHaveTitle(/prometheus/i, { timeout: 30_000 });
+
+  // Load-bearing assertion: the Prometheus API itself answers, authenticated,
+  // through the outpost. An IdP bounce, a 404, or a raw upstream error fails here.
+  const api = await request.get("https://prometheus.franksdank.cloud/api/v1/status/buildinfo");
+  expect(api.status(), "Prometheus API must answer 200 through the guard").toBe(200);
+  expect((await api.json()).status, "Prometheus API must report success").toBe("success");
+});