docs(probe): price the corpus chars/token ratio at run time

The stored 3.28 chars/token (measured 2026-10-11) had drifted to 3.20 as the repo's docs grew, so leg B sent ~201,900-token prompts and took a legitimate 400 from the engine's own window check — which reads as a capacity-contract failure on a healthy endpoint. One ~4k-token request now prices the ratio, and the leg sizes to 97 % of the deployed window.
docs(vault): the four-slot ledger, and the resolver's funding law from the source
2026-10-20 23:26:53 -07:00 · 2026-10-20 23:26:53 -07:00 · 2026-10-20 23:10:05 -07:00 · 2026-10-20 22:52:31 -07:00 · 2026-10-20 22:34:13 -07:00 · 2026-10-20 22:08:00 -07:00
18 changed files with 989 additions and 197 deletions
@@ -47,9 +47,24 @@ Environment=GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=${GRAFANA_OAUTH_CLIENT_SECRET}
 # client_id query parameter — /application/o/<slug>/authorize/ does not exist.
 Environment=GF_AUTH_GENERIC_OAUTH_AUTH_URL=https://auth.franksdank.cloud/application/o/authorize/
 Environment=GF_AUTH_GENERIC_OAUTH_TOKEN_URL=https://auth.franksdank.cloud/application/o/token/
-# ?api=1 = introspection-style userinfo: returns custom claims (groups) too.
+# Standard OIDC userinfo. Grafana v13 fetches <api_url>/emails for the address
-Environment=GF_AUTH_GENERIC_OAUTH_API_URL=https://auth.franksdank.cloud/application/o/userinfo/?api=1
+# ONLY when the token carries no `email` claim (authentik has no /emails path and
-Environment=GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP=false
+# 404s it, failing the login) — so the claim must actually be there: the fix is
 # the grafana-email-scope mapping (see scripts/authentik-grafana-sso-fix.sh),
 # not a Grafana flag. Do NOT append ?api=1 to this URL: Grafana then GETs
 # <...>/userinfo/?api=1/emails, a 403.
 Environment=GF_AUTH_GENERIC_OAUTH_API_URL=https://auth.franksdank.cloud/application/o/userinfo/
 # authentik's scopes are openid/email/profile. `user:email` (a GitLab-ism) is
 # not a known scope here; requesting it makes the authorize view crash.
 # Quoted: quadlet word-splits Environment=, so an unquoted spaced list
 # truncates SCOPES to its first token.
 Environment="GF_AUTH_GENERIC_OAUTH_SCOPES=openid email profile"
 Environment=GF_AUTH_GENERIC_OAUTH_EMAIL_ATTRIBUTE_PATH=email
 Environment=GF_AUTH_GENERIC_OAUTH_LOGIN_ATTRIBUTE_PATH=sub
 Environment=GF_AUTH_GENERIC_OAUTH_NAME_ATTRIBUTE_PATH=preferred_username
 # First SSO login auto-provisions the user. Safe: unauthenticated traffic never
 # reaches this callback — the Anubis -> Authentik guard fronts every request.
 Environment=GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP=true
 Environment=GF_AUTH_GENERIC_OAUTH_AUTO_UPSERT=true
 Environment=GF_AUTH_GENERIC_OAUTH_ALLOW_ASSIGN_ORG=true
 Environment=GF_AUTH_GENERIC_OAUTH_ORG_ID=1
@@ -0,0 +1,81 @@
 #!/usr/bin/env bash
 # Repair the Grafana OAuth2 relying-party SSO on Dankclaw's authentik.
 #
 # grafana.franksdank.cloud logs in through authentik as an OAuth2 relying party.
 # Two DB-side facts (the provider was created by hand via `ak shell`, never
 # repo-managed) silently broke that login, and BOTH had previously been
 # mis-recorded here as a single "the property-mapping evaluator is broken, so
 # userinfo returns only `sub`" myth. They are neither, and the evaluator is fine:
 #
 #   1. provider.authorization_flow was NULL. The authorize view builds a flow
 #      planner from it and crashes with
 #      `AttributeError: 'NoneType' object has no attribute 'slug'` — an HTTP 500
 #      "Server Error — Powered by authentik", before any claim is touched.
 #      Fix: point it at default-provider-authorization-implicit-consent, the
 #      flow every other provider in this instance already uses.
 #   2. The `grafana-email-scope` ScopeMapping's expression was the bare string
 #      `user.email`. authentik merges a scope mapping's result into the token as
 #      a MAPPING: a returned dict adds its keys as claims; a returned string is
 #      keyed under the mapping's NAME ("grafana-email-scope"), so no `email`
 #      claim is ever emitted. Grafana, finding no email, falls back to
 #      GET <api_url>/emails, authentik 404s that path, and the login dies with
 #      "Login failed: InternalError". Fix: return
 #      {"email": user.email, "email_verified": True} — the shape the stock
 #      OpenID 'email' mapping uses. The dict-valued `grafana-profile-scope`
 #      mapping has always worked, which is the proof the evaluator is fine.
 #
 # This is authentik DB state. It is NOT in any deployable file: a blueprint
 # re-import, or an ak-shell re-creation of the grafana provider, reverts BOTH
 # facts and the symptom returns verbatim. Re-run this after any such rebuild.
 # Every step is idempotent.
 #
 # Grafana's own half — SCOPES="openid email profile", a standard userinfo API_URL
 # with no `?api=1`, and ALLOW_SIGN_UP=true — lives in
 # dankclaw/observability/quadlet/grafana.container and deploys normally.
 #
 # Usage: authentik-grafana-sso-fix.sh          (override host: AUTHENTIK_HOST=…)
 set -euo pipefail
 HOST="${AUTHENTIK_HOST:-dankclaw}"
 ssh "$HOST" 'podman exec -i authentik-server ak shell' <<'PYEOF'
 from authentik.flows.models import Flow
 from authentik.providers.oauth2.models import OAuth2Provider
 p = OAuth2Provider.objects.filter(name="grafana").first()
 if not p:
    raise SystemExit("no OAuth2Provider named 'grafana'")
 # 1. The authorization flow every other provider uses.
 want = Flow.objects.filter(slug="default-provider-authorization-implicit-consent").first()
 if not want:
    raise SystemExit("default-provider-authorization-implicit-consent flow is missing")
 if p.authorization_flow_id != want.pk:
    p.authorization_flow = want
    p.save()
    print(f"set grafana.authorization_flow -> {want.slug}")
 else:
    print(f"grafana.authorization_flow already {want.slug}")
 # 2. The email scope mapping must RETURN A DICT, not a bare string.
 m = p.property_mappings.filter(name="grafana-email-scope").first()
 if not m:
    raise SystemExit("grafana-email-scope mapping is missing")
 fixed = 'return {"email": user.email, "email_verified": True}'
 if m.expression.strip() != fixed:
    m.expression = fixed
    m.save()
    print("repaired grafana-email-scope -> returns {email, email_verified}")
 else:
    print("grafana-email-scope expression already correct")
 # 3. The id_token must carry the scope claims; Grafana reads identity from it.
 if not p.include_claims_in_id_token:
    p.include_claims_in_id_token = True
    p.save()
    print("enabled include_claims_in_id_token")
 else:
    print("include_claims_in_id_token already on")
 PYEOF
 echo "Done. Verify: sign in at https://grafana.franksdank.cloud via Authentik;"
 echo "the round trip must land on / and /api/user must answer 200 with an email."
@@ -46,31 +46,35 @@ FRANK_ARCHON_TP=2
 # stays EXACT for single-sequence decode and any step of <= 25 rows — see
 # MAX_SEQS x (1 + SPEC_TOKENS) below.
 FRANK_ARCHON_TP_WIRE=wht6
-# Decode slots. A slot costs KV, never decode; the pool is sized to this
+# Decode slots. A slot costs KV and pinned RAM, never decode: the cache target is
-# contract and nothing more. At 1.8.1 the budget resolver caps the cache at
+# exactly MAX_SEQS x MAX_MODEL_LEN and the checkpoint store asks (1 + what the
-# 556,348 fp8 tokens — 2.8 sessions at 200,000 — so a THIRD full-height
+# retention policy keeps) per slot. Ordered 2026-10-20: FOUR slots over a pool of at
-# conversation answers 400 while /health and every shorter chat work fine
+# least 700,000 fp8 tokens, rounding UP if the resolver cannot land it exactly. The
-# (3 x 50k streams run at 77.6 t/s aggregate). Frank's trade: declare the two
+# target is therefore 4 x 200,000 = 800,000 tokens and HOST_POOL_MIB decides whether
-# slots the engine can hold and let the freed ~1.1 GiB go to the expert plane,
+# the cache gets it (expert_vram_floor, core/place/planner.cpp:100-104: the card must
-# instead of advertising a third slot that only exists at shorter prompts.
+# keep expert-plane-minus-host-room). The 2-slot shape was the answer to a
-FRANK_ARCHON_MAX_SEQS=2
+# 365,204-token pool that backed 1.8 full-height sessions; at this pool the third and
 # fourth conversations are backed by the cache instead of over-committing it.
 FRANK_ARCHON_MAX_SEQS=4
 # Served window. The engine treats `kv group tokens >= max-model-len` as a HARD
 # limit, and models.yml contextWindow MUST quote this number (over-length is a
 # refusal, not a truncation). The pool is sized for seqs x maxlen, not one
 # sequence.
 FRANK_ARCHON_MAX_MODEL_LEN=200000
 FRANK_ARCHON_PLACEMENT=expert_tiered
-# Pinned host pool for expert rows that do not fit VRAM. Under the budget
+# Pinned host pool for expert rows that do not fit VRAM, PINNED ONCE PER RANK: the
-# resolver this is ALSO the lever that decides which experts stay on the card:
+# host must pin tp x this number, and the refusal that names it is
-# the card's leftover goes to the expert plane first and the KV cache takes the
+# core/engine_bringup.cpp:250-265. Under the budget resolver this is ALSO the KV
-# rest, so growing the pool moves experts off the card and shrinking it spills
+# lever -- the card keeps expert-plane-minus-this-room, so every MiB added moves one
-# them onto VRAM the cache wanted. Under-sizing it is a boot failure (36,864
+# MiB of experts off the card and into the cache. Derived from the boot ledger (a
-# refused a boot). The ledger's coverage line is the referee, not this file.
+# 23.73 GiB floor at 44,928 room puts plane + 2 units at 67.60 GiB a rank), 49,152
-# Measured, not guessed: 45,568 (PR #30) moved host_pool to 45.09 GiB and left
+# drops the floor to 19.6 GiB and leaves the cache ~6.9 GiB, which is enough to back
-# vram_kv byte-identical at 4.20 GiB / 556,348 tokens on both ranks — extra
+# the whole 4 x 200,000 target; the leftover returns to the experts. Cost is paid
-# host RAM above what the expert plane's footprint needs buys no cache. Back at
+# twice over: pinned RAM (tp x 48 GiB, against the TTM target below) and expert
-# 44,928, the size that boots clean.
+# residency -- ~3.2 GiB a card of experts now lives in host RAM and is read over the
-FRANK_ARCHON_HOST_POOL_MIB=44928
+# x8 link when the router picks it. The ledger's coverage line is the referee, not
 # this file: if vram_kv comes back under 700,000 tokens, raise this 1024 and redeploy.
 FRANK_ARCHON_HOST_POOL_MIB=49152
 # VRAM left unclaimed per card (publisher default 96 MiB); the engine's settle
 # floor is built from it.
 FRANK_ARCHON_GPU_HEADROOM_MIB=96
@@ -93,16 +97,29 @@ FRANK_ARCHON_SPEC_TOKENS=2
 FRANK_ARCHON_MAX_BATCHED_TOKENS=2048
 # Step sharing. A prefill chunk that shares a step with decoding sequences holds
 # the device for the chunk's whole time, and every decode row in that step waits
-# that long for its next token (the census: [[LLM Performance Diagnosis]] §9).
+# that long for its next token: measured at this shape, decode-only steps 15.5 ms
-# `auto` leaves each decode row behind the chunk a row plus its draft window and
+# against chunk steps 166 ms, which is the harness's ~146 t/s falling to ~11 t/s
-# keeps a share for whatever is waiting to be admitted; a number pins the chunk's
+# while another session prefills (law and census: [[frank-archon]]).
-# cap; `off` is the pre-1.4 behaviour this endpoint ran for two eras, where the
+# `auto` is a PREFILL-SPEED dial, not a decoder dial: it reserves one row plus the
-# chunk took the step and admission never ran. Interleave inserts N decode-only
+# draft window per decoding sequence (3 tokens) and an eighth of the budget per
-# steps after a chunk step taken beside decoders: it protects their cadence and
+# waiting request or waiting prefill, and the source states it will not halve for
-# holds a waiting request out of those steps, so it stays 0 until the post-migration
+# decode rows alone. Live proof: prefill_chunks_capped 5,018 across 62,386 chunk
-# decode-vs-prefill buckets say it is wanted.
+# steps. A number is the only value that applies whenever the chunk shares a step,
 # and it must not go under the prefill stager's smallest pass -- 1025 rows here,
 # named on the boot line "prefill staging: ... a step of more than 1024 tokens" --
 # because below it every routed layer reads its pooled experts synchronously, so
 # 512 is a trap and the vendor measures the whole job slower there. Interleave
 # inserts N decode-only steps after a chunk step taken beside decoders: it protects
 # their cadence and holds a waiting request out of those steps. That is the decoder
 # dial. ARMED at 1 on 2026-10-20: at N=1 a stream takes its token from the chunk step
 # AND one from the held step, ~2 tokens per 182 ms instead of one per 166 ms; the
 # price falls on the prefilling prompt (its chunks lose every held step) and on any
 # request still waiting to enter. `--prefill-span` re-funds a prefill only when it is
 # ALONE, so it cannot buy this back while decoders run. Meter:
 # radiance:engine_steps_held_total must leave zero, and the §9 buckets must move
 # before N rises.
 FRANK_ARCHON_BUSY_PREFILL_CHUNK=auto
-FRANK_ARCHON_PREFILL_INTERLEAVE=0
+FRANK_ARCHON_PREFILL_INTERLEAVE=1
 # Prefix cache: pinned host memory only. The disk tier was removed outright
 # (--prefix-cache-dir and --prefix-cache-disk-mib are refusals), so what used to
 # be restored from NVMe is now re-prefilled. The tier is sized in ENTRIES and one
@@ -118,9 +135,10 @@ FRANK_ARCHON_NGRAM_PLACEMENT=ram
 # RESETS pages_limit to half of RAM on every boot; below the plan the load dies
 # mid-pin with an anonymous allocator error. ttm-preflight.sh raises the knob
 # itself before refusing by name, so recovery is a boot, not a visit.
-# Floor = 2 x HOST_POOL + 48,824 (static weights + arena) + PREFIX_CACHE + MNBT
+# Floor = 2 x HOST_POOL + 48,824 (static weights + arena) + PREFIX_CACHE + MNBT + the
-# per the bats ledger test; at the 44,928 pool that computes to 148,920 MiB.
+# auto checkpoint store (MAX_SEQS x 10 slots of 111.98 MiB), per the bats ledger test;
-FRANK_ARCHON_TTM_MIN_MIB=150000
+# at the 49,152 pool with 4 slots that computes to 161,848 MiB.
 FRANK_ARCHON_TTM_MIN_MIB=162000
 FRANK_ARCHON_TTM_TARGET_MIB=204800
 # Escape hatch for experiments. Unset -> no extra argv; the serve script expands
 # this unquoted on purpose, so keep multi-flag values as one string.
@@ -147,10 +147,21 @@ def main():
    deep = corpus(160_000)
    wide = corpus(1_100_000)
-    # 3.28 chars/token is the measured ratio of this corpus through the
+    # The corpus's chars/token ratio is MEASURED here, not stored. A stored 3.28
-    # froggeric template (646,000 chars -> 196,919 tokens at the 2026-10-11
+    # (646,000 chars -> 196,919 tokens, 2026-10-11) had drifted to 3.20 by
-    # deploy); 98.5 % of the window leaves the response room under the ceiling.
+    # 2026-10-20 (646,299 chars -> 201,876 tokens) as the repo's docs grew, and a
-    chars = int(0.985 * maxlen * 3.28)
+    # stale constant sends leg B past --max-model-len, where the engine answers a
    # perfectly legitimate 400 ("the request (201900 tokens) exceeds the available
    # context size (200000)") and the leg reads as a capacity failure. One small
    # request prices the ratio; 97 % of the window leaves the response room.
    sample = prompt(wide, 4000, "CAL", "cal-marker")
    cal, _, _ = post({"model": MODEL,
                      "messages": [{"role": "user", "content": sample}],
                      "max_tokens": 1, "temperature": 0.0,
                      "chat_template_kwargs": {"enable_thinking": False}})
    ratio = len(sample) / cal["usage"]["prompt_tokens"]
    print(f"[cal] corpus ratio {ratio:.2f} chars/token")
    chars = int(0.97 * maxlen * ratio)
    # Each marker's FIRST hyphen-separated word is unique, because show()
    # scores the hit on that word — a shared prefix would score every slot HIT
    # on any answer that says the word.
@@ -176,7 +187,7 @@ def main():
                print(f"[B-{s}] FAILED after {time.time() - t0:.1f}s: {e!r}")
    print(f"[B] total {time.time() - t0:.1f}s")
    # D — concurrency decode: seqs streams, 512 tokens each, ~25 % depth.
-    dchars = int(0.25 * maxlen * 3.28)
+    dchars = int(0.25 * maxlen * ratio)
    dbodies = [chat(deep, dchars, f"D{i + 1}-9137", f"flint-marker-{i}", 512)
               for i in range(seqs)]
    print(f"[D] {seqs} concurrent streams x 512 tokens, "
@@ -15,18 +15,21 @@
 # log states every budget before anything is allocated (weights vs KV
 # split, the expert host pool's coverage of the worst case, the PLE pin,
 # the P2P answer per pair) — read it after every resize.
-# Deployed shape (2026-10-19, radiance 1.8.1): THREE decode slots x 200,000
+# Deployed shape (2026-10-20, radiance 1.8.1): FOUR decode slots x 200,000 tokens over a
-# tokens, fp8 KV, 44 GiB/rank pinned expert pool, MTP depth 2, prefix cache in
+# >= 700,000-token fp8 KV pool, 48 GiB/rank pinned expert pool, MTP depth 2, prefix cache
-# pinned host RAM, PLE pinned in RAM. The per-card split between the expert slab
+# in pinned host RAM, PLE pinned in RAM. The per-card split between the expert slab and the KV
-# and the KV pool is NO LONGER an operator number — 1.3.0 removed the VRAM pair
+# pool is NO LONGER an operator number — 1.3.0 removed the VRAM pair and the budget
-# and the budget resolver derives it from headroom, the cache bound and the host
+# resolver derives it from headroom, the cache bound and the host pool, so the slab
-# pool, so the slab size and the pool's token count are LEDGER lines: after every
+# size and the pool's token count are LEDGER lines. After every boot read them, plus
-# boot read them out of the log and check the pool still backs 3 x 200,000.
+# the two lines that size a prefill step — "activation arena: ... lends ... while
 # steps are at most N tokens" (the arena levels the expert slab borrows) and
 # "prefill staging: ... a step of more than N tokens ..." (the chunk size below which
 # pooled experts are read synchronously) — and the cache's session backing.
 #
 # The root unit (scripts/frank-archon.service, User=dank,
-# LimitMEMLOCK=infinity) grants the pin budget: 2 x 44 GiB expert pools +
+# LimitMEMLOCK=infinity) grants the pin budget: 2 x 48 GiB expert pools + the 47.7 GiB PLE
-# the 47.7 GiB PLE table + 8 GiB prefix host tier — ~147 GiB of pinned
+# table + 8 GiB prefix host tier + the checkpoint store (slots x 112 MiB) — ~159 GiB of
-# pages, which is ALSO why this script refuses to start when ttm's
+# pinned pages, which is ALSO why this script refuses to start when ttm's
 # pages_limit/pages_max cap is under FRANK_ARCHON_TTM_MIN_MIB (see below).
 set -euo pipefail
@@ -81,7 +81,7 @@ Two provisioning rules, each verified live by an outage:
   appears. Create the Application with `provider_id` set (the UI does this
   automatically; raw API/ORM drives must do it explicitly).
-## OAuth2 relying-party quirks (this fork, verified live 2026-08-31)
+## OAuth2 relying-party quirks (this fork; corrected live 2026-10-20)
 The Grafana RP provider (`grafana`, pk 9, confidential client) was created and
 verified end-to-end via `ak shell` ORM drives. This fork's OAuth2 provider
@@ -100,23 +100,32 @@ implementation diverges from upstream in ways that bit us; all verified live:
 - **Provider resolution is by `client_id` query param** on the global
  `/application/o/authorize/` route; per-slug authorize paths do not exist in
  this build. Token/userinfo live at `/application/o/token/` and
-  `/application/o/userinfo/?api=1`.
+  `/application/o/userinfo/` — do **not** append `?api=1` to a Grafana API URL.
- **Property-mapping evaluator is broken → userinfo returns `sub` only.**
+- **NULL `authorization_flow` → authorize 500, `AttributeError: 'NoneType'
-  `PropertyMappingEvaluator.evaluate(*args, **kwargs)` forwards kwargs to
+  object has no attribute 'slug'`.** A provider created via raw `ak shell`
-  `BaseEvaluator.evaluate(expression_source)`, so the first context kwarg
+  `create()` leaves `authorization_flow` unset; the authorize view then crashes
-  becomes the *expression source* instead of the context; every call raises a
+  building the flow planner (`flows/planner.py`) and returns "Server Error —
-  swallowed `PropertyMappingExpressionException` and the claim is skipped.
+  Powered by authentik" before any claim is built. This was previously
-  Scope-grant filtering has the same class of bug: `ScopeMapping` has no
+  mis-attributed to a broken evaluator; the crash is the missing flow. Fix: set
-  provider FK (linkage lives in the `ProviderPropertyMapping` through model)
+  `authorization_flow = default-provider-authorization-implicit-consent` (the
-  so `filter(provider=...)` always yields an empty queryset and tokens asked
+  flow every other provider in this instance already uses).
-  to carry scopes get zero. Net effect: userinfo/introspection answers with
+- **A scope mapping that returns a non-dict emits no claim.** authentik merges a
-  `{sub}` only — email/groups/entitlements claims cannot be delivered by this
+  granted scope mapping's evaluated result into the token **as a mapping**: a
-  build until the evaluator is patched or upgraded. Clients consuming this
+  dict contributes its keys as claims; a bare string is keyed under the
-  provider must key identity off `sub`.
+  mapping's NAME, so nothing usable appears. `grafana-email-scope` returned the
- Scope mappings for this provider exist — `grafana-email-scope` and
+  bare string `user.email`, so no `email` claim landed — the evaluator was
-  `grafana-profile-scope` are both rows in `authentik_core_propertymapping`
+  never broken (the dict-valued `grafana-profile-scope` mapping has always
-  (read-only `SELECT`, 2026-10-18) — and will start emitting claims the moment
+  delivered, and the stock OpenID mappings return dicts). Fix:
-  the evaluator bug is fixed; no config change is needed then.
+  `return {"email": user.email, "email_verified": True}`. With no email, Grafana
  v13 falls back to GET `<api_url>/emails`; authentik 404s it → "Login failed:
  InternalError".
 - **Grafana reads identity from the standard userinfo endpoint** (`API_URL` with
  no `?api=1`); no `USE_ID_TOKEN` is needed once the `email` claim exists.
 - **Net (2026-10-20):** the evaluator is fine; email/profile claims reach RP
  clients. Both IdP fixes are idempotent in
  `scripts/authentik-grafana-sso-fix.sh` (re-run after an ak-shell re-creation of
  the provider or a blueprint import, which revert them). Grafana's half plus the
  verified outcome (Viewer, `login = <sub hex>`, email present) are in [[Grafana]].
 ## LDAP provider objects (provisioned via `ak shell` ORM)
@@ -4,7 +4,7 @@ description: "Grafana visualizes the metrics that Prometheus collects from every
 tags:
  - infrastructure
  - grafana
-timestamp: 2026-09-11
+timestamp: 2026-10-20
 ---
 # Grafana
@@ -64,19 +64,28 @@ Grafana responds on `/api/health` with 200 and `"database":"ok"`; measured 2026-
 - Change the default admin credentials at first boot (env or UI).
 - Dashboards provisioned as code (dashboard JSON plus a YAML provider, `shared/grafana/provisioning/dashboards/`) survive container recreation; UI-created ones live in the data volume — back it up.
 - **Auth = OAuth2 relying-party, not auth_proxy.** The embedded outpost carries
-  no provider token (`token=None`), so no identity header ever reaches Grafana;
+  no provider token (`token=None`), so no identity header reaches Grafana; the
-  RP mode is topology-independent. The full verified contract (provider pk 9,
+  login is Grafana's own "Sign in with Authentik" button against the `grafana`
-  client_id `grafana-21ca190cde75d60f`, global authorize/token/userinfo paths,
+  confidential-client provider. **Verified end-to-end 2026-10-20** after four
-  Basic client auth, form-encoded token body, and the fork's userinfo-claims
+  independent breakages had made it fail. The two Authentik-side causes (a NULL
-  bug that leaves identity at the `sub` hash only) is in
+  `authorization_flow` → authorize 500; an email scope mapping that returned a
-  [[Authentik Outpost Reference]] → "OAuth2 relying-party quirks". Consequences for Grafana:
+  bare string so no `email` claim was emitted) and the full wire contract live in
-  The unit sets `ALLOW_SIGN_UP=false`, `AUTO_UPSERT=true`, `ALLOW_ASSIGN_ORG=true`
+  [[Authentik Outpost Reference]] → "OAuth2 relying-party quirks"; the IdP half is
-  `ORG_ID=1` and **no** role-attribute mapping (`grafana.container:52-55`), so
+  re-applied by `scripts/authentik-grafana-sso-fix.sh` (idempotent — re-run after
-  users land as Viewer with `login = <sub hex>`; no email/groups/role claims
+  an ak-shell re-creation of the provider or a blueprint import, both of which
-  reach Grafana until the evaluator bug is fixed — do not write role mappings
+  revert it silently). The Grafana half is the unit's own env (`grafana.container`):
-  that depend on those claims. The local admin account stays the only Admin.
+  `SCOPES="openid email profile"` (a GitLab-style `user:email` scope crashes the
-  (The Viewer-default outcome and the `<sub hex>` login are not readable from
+  authorize view), the standard userinfo `API_URL` with **no `?api=1`**, the
-  the host without the database or an admin password. *(unverified 2026-10-18)*)
+  attribute paths `email`/`sub`/`preferred_username`, and `ALLOW_SIGN_UP=true`
  (safe: the guard fronts the callback). Grafana v13 only GETs `<api_url>/emails`
  when the token carries no `email` claim — so it is the email claim itself (an
  IdP fix, above) that makes the login land, not any Grafana toggle.
 - **What a new login becomes:** a Viewer in org 1 (`ALLOW_ASSIGN_ORG=true`,
  `ORG_ID=1`, no role-attribute mapping), with `login = <sub hex>` but the real
  `email`. The local admin account stays the only Admin. The landing that the
  edge-only 13-sso spec never checked is now asserted by
  `tests/e2e/specs/20-grafana-sso.spec.ts` (RP round-trip → `/api/user` 200 with
  an email).
 - **Never point this unit's `EnvironmentFile` at the repo-root `.env`.** That
  file is a gluetun env file and it carries `HOME=/root`; systemd imports it
  for the service, podman then looks for its configuration under `/root`
@@ -239,45 +239,92 @@ prefill is over. `preemptions` separates the third shape: a preempted-and-recomp
 sequence stops producing entirely and then re-prefills from token 0, which is a
 stall, not sludge.
-Knobs, in the order worth trying them (radiance GUIDE §4.5):
+**Split the step ledger before choosing a knob.** Three counters decide which mechanism you are
 looking at, and none is a rate: `engine_step_seconds_decode_total / engine_steps_decode_total` is
 the mean **decode-only** step, and `(step_seconds_total − step_seconds_decode_total) /
 (steps_total − steps_decode_total)` is the mean **chunk step** (a chunk alone, or a chunk with
 decode rows beside it). On frank-archon's live 1.8.1 shape that pair is **15.5 ms against 166 ms**,
 with 4.7 % of the steps holding 34 % of the step time. Decode-only steps themselves move only
 14.1 → 16.4 ms between a prefill-free minute and a heavy one, and the lagged buckets stay flat: so
 a decode-rate collapse on that box is **step time sharing**, and the eviction/residency story is
 worth ~10 % of one step — not a lever.
- `--prefill-interleave N` — after a chunk step taken beside decoders, the next N
+**Four traps in the PromQL above.** Learn them before trusting a bucket:
-  steps carry decode rows only and admit nothing. Decoders keep their cadence; the
+- **Latency is `vllm:`, cadence is `radiance:`.** The TTFT / inter-token / per-output-token / e2e
-  prompt pays that in TTFT. Counts as `engine_steps_held_total`.
+  histograms and the running/waiting gauges exist only under the vLLM compatibility prefix; preemption
- `--busy-prefill-chunk MODE` — the chunk's share of a step that has other work:
+  is `vllm:num_preemptions_total` (the counter) while `radiance:num_requests_preempted` is the
-  `auto` (default) reserves a row plus the draft window per decode row behind it and
+  *instant* depth. A `radiance:`-only query finds no latency and no preemption, and reads as "the
-  an eighth of the budget per waiting prefill/request, `off` lets the chunk take the
+  endpoint never stalls anyone".
-  whole step (admission never runs while it does), `N` pins the share. Never under
+- **The tps gauges are 1 s decayed windows, batch-pooled.** Per-stream rate is
-  the prefill stager's smallest pass — the startup line names it, and below it every
+  `decode_tokens_per_second / vllm:num_requests_running`. With a 15 s scrape, a 120 s step samples one
-  routed layer reads its pooled experts synchronously, which costs the prefill far
+  scrape in eight — use step ≤ 30 s for a bucket table.
-  more than sharing saves. Counts as `engine_prefill_chunks_capped_total`.
+- **Every counter resets at a restart and the metric set changes with the build.** Prove the window is
- `--max-num-batched-tokens` — the blunt rung, and the only one older pins have. The vendor's own
+  one shape (`up`, or the first sample of a counter that only the current build exports) before
-  curve prices **halving the shared chunk at roughly a fifth of prefill throughput** (a 48-layer MoE
+  ranking buckets; frank-archon's 48 h window straddled two containers, and one bucket's "collapse"
-  at 200k on two cards — knob pricing, not this endpoint's number, so treat it as the order of
+  turned out to belong entirely to the older one.
-  magnitude a chunk decision costs). Raising it past the checkpoint interval buys nothing and loses
+- **A 2-minute mean step time dilutes the chunk.** The sharp variable is the phase ratio in the query
-  expert residency to the arena.
+  above, never `step_seconds/steps` — 4.7 % of steps cannot move a mean that is 95 % decode.
 - `--prefill-span N` — the inverse trade: a *lone* prefill carries N intervals a step while every
  shared step is planned at a smaller shared budget, which on an expert-tiered box lends that arena
  back to the expert slab. The vendor's own measurement says the win is **long-prompt-only**: nothing
  at short prompts, a large TTFT gain at the far end of the window.
-Two traps. **The arena/slab coupling makes a shared step doubly expensive on an
+Three instruments the PromQL above cannot reach:
 expert-tiered shape**: the full-plan arena takes VRAM the resident expert slab would have had, so the
 decode rows inside that step also read experts over the link — a smaller shared budget helps twice.
 And **re-check
 `prefix_hit_rate` / `prefix_evictions` after any chunk change**: the arena and the KV
 pool are the same memory, and the failure looks like turn two taking as long as turn
 one.
-Availability is a version question, not a config question: `--busy-prefill-chunk` and
+- **The engine's own step-detail histograms (`prefill_steps_duration`, `prefill_step_tokens`,
-`--prefill-interleave` are absent from older pins — prove it on the running container
+  `token_queue_wait`) are exported but reach no target**, and the arena / expert slab / heat engine /
-(`podman exec <c> <binary> --help | grep -iE "prefill|chunk"`), never from the vendored
+  prefill stager have **no metric at all**. Per-request forensics mean `curl localhost:7770/stats` on
-tree, and check the source clone's depth before dating a flag from `git log -S`
+  the endpoint's own host — same phase counters plus `preemptions` (the count), the chunk geometry,
-(a `--depth 1` clone reports its own tip for every search).
+  and the expert/link byte totals. `/diagnostics` answers "is the *box* the limit" in one call:
  configuration drift, the VRAM split, each card's link chain and clocks, findings with fixes.
 - **`prefix_hit_cut_tokens_total` measures prefill load you created for yourself.** On a hybrid
  (linear + paged) model a prefix hit reuses only as far as the last **linear-state** checkpoint, so
  cached tokens are replayed as prefill and stall decoders for it; frank-archon ran 38 % of all its
  prefill tokens that way, every `prefix_hit_cuts_total` labelled `reason="no_snapshot"`. A high
  prefill rate is not always a client that sends new tokens. Retention is `--checkpoint-policy` -
  exactly three names, `geometric-backoff` (default), `tip-only`, `keep-all`, one owner:
  [[frank-archon]].
 - **Bucket the arriving request too** — `histogram_quantile` over
  `vllm:time_to_first_token_seconds_bucket` by the same prefill series. The decode gauge only sees
  sequences that are already running; the one waiting for its first token pays the same steps and
  shows in no decode number.
-Run on: [[frank-archon]] — the census was taken on its 3-slot radiance 1.0.13 era (budget 2048 =
+Knobs, in the order worth trying them (radiance GUIDE §4.5; the pinned source is the referee):
-interval 2048); the live 1.8.1 shape carries the two chunk knobs. The measured census and the verdict
+
-for this fleet's shape are that page's.
+- `--prefill-interleave N` — after a chunk step taken beside decoders, the next N steps carry decode
  rows only and admit nothing. **The knob that acts on the measured mechanism.** Decoders keep their
  cadence; the prefilling prompt pays in TTFT and an arriving request waits out the held steps.
  Counts as `engine_steps_held_total` — if that counter is zero, nothing is protecting anyone.
 - `--busy-prefill-chunk N` — pins a shared chunk's share of the step. **`auto` is a prefill-speed
  setting, not decoder protection:** it reserves `1 + spec` rows per decoding sequence plus an eighth
  of the budget per *waiting* request or prefill, and the source refuses to halve for "decode rows
  alone", so a shape can run `auto` and still lose every chunk step. Prove engagement from
  `engine_prefill_chunks_capped_total` against the chunk-step count. The floor is the prefill
  stager's smallest pass — the largest arena level that lends, plus one, named on the startup line
  `prefill staging: … a step of more than N tokens copies the pooled experts of …` — and under it
  every routed layer reads its pooled experts synchronously: the vendor prices the whole job (four
  decoders plus a long prompt) at **29.5 s with a 2048 chunk and 31.9 s at 512**, so shrinking the
  chunk blindly buys latency in both directions.
 - `--prefill-span N` — needs `--max-num-batched-tokens` of N intervals. A *lone* prefill then
  carries N intervals a step (on an expert-tiered box only from 8 intervals, where the vendor's own
  measurement says a span starts to win), while **every other step is planned at the shared
  budget** — a lending arena level that also keeps the stager's ring, the one combination the chunk
  knob cannot express. Priced by the vendor at 320 MiB a card of expert residency, ~3 % off the KV
  pool, and TTFT-neutral on one of his two 2×R9700 boxes: measure your own prompt lengths first.
 - `--max-num-batched-tokens` alone — the blunt rung, and the only one older pins have. It raises the
  arena the expert slab loses, and on this payload its resonance is violent (MNBT 2560 collapses
  decode — [[frank-archon]]).
 Two traps. **The arena/slab coupling makes a shared step expensive twice:** the full-plan arena is
 reclaimed from the expert slab *synchronously before the step*, and the loan regrows only after a
 quiet interval, so every step of a prefill run — the decode-only ones included — runs on the smaller
 slab. And **re-check `prefix_hit_rate` / `prefix_evictions` after any chunk or span change**: the
 arena, the staging ring and the KV pool are the same memory, and the failure looks like turn two
 taking exactly as long as turn one.
 Availability is a version question, not a config question: `--busy-prefill-chunk`,
 `--prefill-interleave` and `--prefill-span` are absent from older pins — prove it on the running
 container (`podman exec <c> <binary> --help | grep -iE "prefill|chunk"`) or from `/diagnostics`'s
 `configuration` block, never from a vendored tree, and check a source clone's depth before dating a
 flag from `git log -S` (a `--depth 1` clone reports its own tip for every search).
 Run on: [[frank-archon]] — owner of the live census, the measured step pair, and the arms.
 ## Related
@@ -88,25 +88,30 @@ on stop. Boot budget 1800 s (the publisher's compose allows a 15-minute `start_p
 | Knob (env name, `FRANK_ARCHON_` dropped) | Live value |
 |---|---|
 | TP / wire | `TP=2`, `TP_WIRE=wht6` — libr4d one-shot all-reduce, not RCCL; `min-kb` stays the engine default |
-| Slots / window | `MAX_SEQS=2`, `MAX_MODEL_LEN=200000` (native 262,144; longer is refused, never truncated) — this pair is the **cache bound**, not a KV allocation |
+| Slots / window | `MAX_SEQS=4`, `MAX_MODEL_LEN=200000` (native 262,144; longer is refused, never truncated) — this pair is the **cache bound**: the resolver sizes the KV carve against `slots × window`, and no slot count funds itself (Laws) |
 | KV dtype | `KV_DTYPE=fp8` — the only dtype that funds the slot count (Laws) |
 | VRAM split | **derived at boot** from `GPU_HEADROOM_MIB=96` + the cache bound + `HOST_POOL_MIB`; naming `VRAM_WEIGHTS_MIB`/`VRAM_KV_MIB` is a boot refusal (removed at 1.3.0) |
-| Expert tier | `PLACEMENT=expert_tiered`, `HOST_POOL_MIB=44928` **per rank** |
+| Expert tier | `PLACEMENT=expert_tiered`, `HOST_POOL_MIB=49152` **per rank** — the pinned host pool is also the **KV lever**: the expert floor is `plane + 2 units − host room`, so every MiB pinned there lowers what the experts claim on the card one-for-one and the cache takes it (Laws) |
 | PLE | `NGRAM_PLACEMENT=ram` (47.7 GiB pinned; refuses boot on a short memlock) |
 | Spec / batch | `SPEC_TOKENS=2` — MTP depth 2, the `.rad` carries the model's own `mtp.*` head (`mtp.fc_hidden`/`fc_embedding`/`draft_head`); `MAX_BATCHED_TOKENS=2048` (MNBT — 2560 collapses decode, Laws) |
 | Prefix cache | `PREFIX_CACHE_MIB=8192`, pinned **host tier only** — the 64 GiB disk tier is gone, so an evicted prefix is re-prefilled; `STATE_DIR=/home/dank/radiance-state` now exists only to hold the rad-tune cache |
 | Template | `TEMPLATE=scripts/qwen38-froggeric-v22.5.jinja` (28,234 B) via `--override-chat-template` — froggeric's fixed Qwen3.8 template (the repo root's `chat_template.jinja`, vendored); the checkpoint's own template is baked into the `.rad` by rad-convert and this overrides it, and radiance derives the reasoning and tool-call reply format from it |
-| Step sharing | `BUSY_PREFILL_CHUNK=auto` (a shared chunk reserves a row plus the draft window per waiting decoder) · `PREFILL_INTERLEAVE=0` — held at zero until the decode-vs-prefill buckets ask for it |
+| Step sharing | `BUSY_PREFILL_CHUNK=auto` — prefill-speed first: it reserves 3 tokens per waiting decoder and ⅛ of the budget per waiting request, and does **not** shrink the chunk for decoders alone · `PREFILL_INTERLEAVE=1` — the decoder dial, **armed 2026-10-20**: one decode-only step after each chunk step taken beside decoders; meter `engine_steps_held_total`, price is the prefilling prompt's TTFT |
-| TTM gate | `TTM_MIN_MIB=150000`, `TTM_TARGET_MIB=204800` |
+| TTM gate | `TTM_MIN_MIB=162000`, `TTM_TARGET_MIB=204800` |
 | Boot | `RAD_MIN_MIB=110000` deployer floor; the listener binds only after the full load (health ~100 s, budget 1800 s) |
-Boot ledger for the live shape (2026-10-19 11:10, pin 1.8.1, `MAX_SEQS=2`, `HOST_POOL_MIB=44928`):
+Boot ledger for the live shape (2026-10-20, pin 1.8.1, `MAX_SEQS=4`, `HOST_POOL_MIB=49152`):
-the resolver prints `elastic 26.73 GiB` → **experts 23.73 GiB (89 %) / kv cache 2.76 GiB (10 %)**,
+the resolver prints `elastic 26.73 GiB` → **experts 20.72 GiB (78 %) / kv cache 5.64 GiB (21 %)**,
-`vram_weights 27.31 GiB`, `host_pool 44.47 GiB`. `kv_attn` (13 layers, 4-token blocks) pages
+`prefill staging 373.65 MiB` out of the cache, `vram_weights 24.30 GiB`, `host_pool 48.59 GiB`.
-**26.00 KiB** → 91,301 blocks = **365,204 tokens = 1.8 full-length sessions = 91 % of the
+The referee line: `kv: the cache can grow to 5.64 GiB a card — 747,492 tokens of context, 3.7
-2 × 200,000 worst case** (the resolver never funds 100 %, see Laws); `mover` lends 2,146 slots
+sessions at 200,000 — with every block it lends the experts taken back; 440.04 MiB of that is
-(2.54 GiB) out of the cache to the expert slab; sampler **9 rows**, template 28,234 B, vision live
+per-sequence state, and up to 5.20 GiB is lent to the experts while no request holds it`.
-(≤ 4,194,304 px, 16,384 patches/pass).
+`--checkpoint-slots 40` is auto (`max_num_seqs × 10` slots of 111.98 MiB = 4,479 MiB of pinned
 host), `/health` reports `queue_cap 32` (`max(8, seqs × 8)`), sampler **12 rows**
 (= `seqs × (1 + speculative depth)`), 4 hardware queues a card, template 28,234 B, vision live
 (≤ 4,194,304 px, 16,384 patches/pass), P2P confirmed (`the 2 cards reach each other's memory
 directly`). Experts living in the pinned host pool: **8.99 GiB a rank** (was 5.99) — the price of
 the bigger cache, paid over PCIe when those experts are hit.
 The boot **ledger**, not this arithmetic, is the referee: radiance states every budget before
 anything is allocated (weights vs KV, the host pool's coverage of the worst case, the PLE pin,
@@ -145,6 +150,7 @@ marked "re-fetch" is gone from `/home/dank/models/`, which today holds only `Sti
 | Window | Payload | Runtime | Shape | Replaced because | Rollback |
 |---|---|---|---|---|---|
 | 2026-10-19 → 10-20 | same `qwen3.8-next-flash-fp8-iq4r-moe.rad` (payload unchanged) | radiance **1.8.1**, `sha256:d590e300…` (same digest) | 2 × 200,000 fp8 KV — resolver cache 2.76 GiB = 365,204 tokens (91 % backing); host pool 44,928 MiB a rank, `TTM_MIN_MIB=150000`, `queue_cap 16`, sampler 9 rows; `--prefill-interleave 1` armed on 10-20 | Frank ordered **four** concurrent conversations over a ≥ 700,000-token pool; the resolver funds that only by lowering the expert floor with more pinned host room (Laws) | revert `1bb6399` and redeploy `scripts/deploy.sh dankserver --service frank-archon` — trio back to `MAX_SEQS=2` / `HOST_POOL_MIB=44928` / `TTM_MIN_MIB=150000` |
 | 2026-10-15 → 10-19 | same `qwen3.8-next-flash-fp8-iq4r-moe.rad` (payload unchanged) | radiance **1.0.13**, `sha256:4c3f4845…` | 3 × 200,000 fp8 KV bought with the **explicit** pair `--vram-weights-mib 25600 --vram-kv-mib 5120`; host pool 44,928 → 45,568 MiB a rank; prefix cache 8 GiB host + 64 GiB disk | decode turned to sludge beside any long prompt — step contention, 18.7 ms decode-only steps against 317 ms steps carrying a prefill chunk (Laws) — and the only two knobs that fix it, `--busy-prefill-chunk` and `--prefill-interleave`, exist only after this pin | `FRANK_ARCHON_IMAGE` back to `sha256:4c3f4845…` **plus a trio rewrite**: restore the pair and the host+disk prefix knobs, delete `BUSY_PREFILL_CHUNK`/`PREFILL_INTERLEAVE`. A digest swap alone is a boot refusal (Runtime) |
 | 2026-10-08 → 10-15 | in-house `qwen4exp-swift15-w4dn8-i8-hc8m.rad`, 116,416 MiB (Swift1.5 abliterated BF16, GPTQ w4 experts + 24 bf16 protected, KLD 0.0131 / top-1 97.92 %) | radiance 1.0.13, same digest | 3 × 200,000 fp8 KV, slab 25,600 / pool 5,120 — the shape the live file still carries | the engine author's own container is the supported path and his quality ceiling; the finetune's refusal-bending left the endpoint | `.rad` purged 2026-10-17 (trash-staged then emptied, Frank's approval). Re-download `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700` → `qwen4exp-w4nl64-i8-hc8m.rad`, 122,071,973,888 B, LFS sha256 `09dd9306…`, via the iq4r fetch script's hfget/verify pattern retargeted at that repo; point `MODEL_FILE` at it, redeploy. Re-**convert** is dead: the BF16 source was `rm`'d 2026-10-15 (`kld-ref`, grams, corpora and `/home/dank/logs/swift15_*.log` survive) |
 | 2026-10-05 → 10-08 | `qwen4exp-fp8-native.rad`, 177,109 MiB (173.0 GiB) — native modelopt block-FP8, no requantisation | radiance 1.0.7 line, same digest | — | native FP8 experts (56.4 GiB/rank) ate a 40 GiB pool and left KV at 3,543 MiB = 386,904 tokens, **1.93 sessions** — context economics, not quality; and the host pool could not be trimmed to pay for it (36,864 refused the boot) | rebuild-only: `scripts/download-frank-archon.sh` (official `Qwen/Qwen3.8-Flash-Next-FP8`, 131 shards, 172.8 GB, ungated) + `scripts/build-frank-archon-rad.sh` (no-recipe convert under the descriptor law); raw **and** `.rad` were `rm`'d 2026-10-15 (Frank's call, to free the volume for the iq4r download) |
@@ -211,27 +217,44 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
  never `--kv-cache-size-mib`; with the disk tier engaged the engine also forced ~9.6 GiB of weight
  buffers + arena onto the weights side (~29.6 GiB/card effective). **Neither flag exists at 1.8.1:**
  the same ceiling is enforced by the resolver, and passing the pair is a boot refusal.
- **At 1.8.1 the split belongs to the resolver, and it funds ~90 % of the worst case, never 100 %.**
+- **At 1.8.1 the split belongs to the resolver, and the source fixes its order.**
-  The engine takes `--gpu-headroom-mib`, the cache bound (`--max-model-len` × `--max-num-seqs`) and
+  `core/mem/vram_budget.cpp` claims the card bottom-up (`capacity → already held → drift → arena →
-  `--host-pool-mib`, prints `claimable → elastic → experts / kv cache` before allocating anything, and
+  staging → queues → prefetch → headroom = claimable`), takes the **static weights off the top as a
-  sizes the expert plane from the model's own weight footprint plus slack — not from a number typed by
+  floor, not a share**, and divides the `elastic` remainder in three lines (146-156):
  hand. Three boots of evidence:
-  | shape | demand | cache given | backing |
+  ```cpp
  exp = std::min(b.expert_floor, b.elastic);
  kv  = std::min(kv_cap, std::max(b.elastic - exp, kv_one));
  exp = std::min(std::max(b.expert_ceiling, b.expert_floor), b.elastic - kv);
  ```
  `kv_cap` is `KVManager::ceiling()` = `state + Σ_paged max_seqs × ceil(max_ctx / block) × bytes`
  (`core/mem/kv.cpp:266`), so `--max-num-seqs` and `--max-model-len` set the cache's **demand**, never
  its funding. The funding is `elastic − expert_floor`, and the floor is
  `min(plane + spare, plane + spare + 2·unit_max − host_room)` (`core/place/planner.cpp:100-104`, with
  `host_room = host_pool − file_stage_reserve` at `core/engine_bringup.cpp:1418-1422`). **That is the
  cache's lever: every MiB added to `--host-pool-mib` drops the expert floor one-for-one and the cache
  takes it** — minus `stage`, the prefill stager's ring (two layers of the experts left off the card),
  which is charged back out of the cache whenever the cache lends (`vram_budget.cpp:193-203`). Boots of
  evidence, all on 1.8.1:
  | shape | demand | cache given | engine's `backs %` |
  |---|---|---|---|
-  | 3 × 200,000, host pool 44,928 | 600,000 tok | 4.20 GiB | 93 % (2.8 sessions) |
+  | 3 × 200,000, host pool 44,928 | 600,000 tok | 4.20 GiB | 93 % |
  | 3 × 200,000, host pool 45,568 | 600,000 tok | **4.20 GiB — byte-identical** | 93 % |
-  | 2 × 200,000, host pool 44,928 (live) | 400,000 tok | 2.76 GiB = 365,204 tok | 91 % (1.8 sessions) |
+  | 2 × 200,000, host pool 44,928 | 400,000 tok | 2.76 GiB | 91 % |
  | 4 × 200,000, host pool 49,152 (live) | 800,000 tok | **5.64 GiB = 747,492 tok** | 93 % |
-  So `--max-num-seqs` is a **demand signal** for the resolver and an admission cap for the scheduler —
+  The 45,568 row is the ceiling case, not a mystery: at 3 slots the cache already held its whole
-  never a KV allocation — and no slot count fully funds the worst case. Growing host RAM by less than a
+  `ceiling`, so extra host room could not grow it and shows up as `unclaimed`. **The referee is the
-  slab step (640 MiB) moved nothing: the expert plane's claim decides what is left for KV, so cache is
+  ledger line `kv: the pool backs N % … the cache can grow to … N tokens`, never arithmetic.** To buy
-  grown through slots × window or the dtype, and experts are grown by not leaving elastic pool idle.
+  window: raise `--host-pool-mib` (cost: pinned host RAM, and experts evicted to host cross PCIe —
-  `--host-pool-mib` is still load-bearing the other way — it is where the expert rows the cards cannot
+  `experts-from-host` was 5.99 GiB a rank at 44,928 and is 8.99 at 49,152) or cut the per-token cost
-  hold get pinned, and under-sizing it has always been a boot failure (36,864 refused a boot in the fp8
+  (`fp8` KV). Raising slots raises demand **and** costs 10 × 111.98 MiB of pinned checkpoint-store RAM
-  era). The two-slot boot's own behavioural check: three concurrent conversations all answer `200` —
+  per slot. Under-sizing the pool the other way has always been a boot failure (36,864 refused a boot in
-  the third **queues** (`queue_cap: 16`), it is not refused; the 400 class is a single request over
+  the fp8 era: the host cannot hold what the cards will not). Behavioural check on the live shape:
-  `--max-model-len`, or three concurrent full-height conversations against a 1.8-session cache.
+  `/health` reports `queue_cap 32`; a request beyond the slot count queues rather than being refused,
  and the 400 class is a single request over `--max-model-len`.
 - **Idle KV VRAM is the expert cache's spillover — the price is paid twice.** `mover: 2275
  slots can be lent out of the KV cache (2.69 GiB)`, `arena: lends 740 MiB while steps are <=
  256 tokens`; the lend grows with the pool (2.69 GiB from 3.46, 6.08 from 6.912, 4.22 in the pair era)
@@ -280,41 +303,104 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
 - **MNBT 2048.** 2560 collapses decode to 2.6-7.7 t/s (MTP resonance); the w4 file tier streams
  in place only at MNBT ≤ 2048 and the activation arena scales with the chunk (32768 → 33.3
  GiB, refused).
- **Decode pays for the chunk it shares a step with.** Census of the live engine taken at the
+- **Decode pays for the chunk it shares a step with, and `--busy-prefill-chunk auto` does not
-  3-slot shape on 1.0.13 (`/stats` + `/metrics`, cumulative over the 2026-10-15 boot):
+  protect it.** Re-measured on the live 1.8.1 shape (34.8 h boot, Prometheus at 60 s,
-  1,435,585 decode-only steps at **18.7 ms** each against 127,991 prefill+mixed steps
+  `engine_step_seconds{,_decode}_total` / `engine_steps{,_decode,_mixed,_prefill}_total` /
-  at **317 ms** — 60 % of engine wall-clock spent inside 8 % of the steps, because the
+  `prefill_tokens_total`): decode-only steps run at **15.5 ms**, chunk steps (a chunk alone, or a
-  chunk is `min(remaining, checkpoint interval 2048, MNBT 2048)` = 2048 rows and every
+  chunk with decode rows beside it) at **166 ms** — 4.7 % of the steps hold **34 % of the engine's
-  decode row in that step waits the whole chunk. Day of Prometheus (120 s step, 721
+  step time**, so a session decoding while another prompts advances one token per 166 ms. Bucket the
-  samples): `decode_tokens_per_second` median **13.3 t/s** in the 56 % of samples where
+  harness's `radiance:decode_tokens_per_second` by `radiance:prefill_tokens_per_second` (15 s
-  `prefill_tokens_per_second` ≥ 1000, against 106-240 t/s in the light-prefill buckets;
+  scrapes; per-stream rate = the gauge ÷ `vllm:num_requests_running`): median **166 t/s** under light
-  bucketed by mean step time it is 128.7 t/s at 0.02-0.05 s steps and 12.0 t/s at
+  prefill, 81 t/s at 1-5k, **13 t/s above 5k** — a **12.7×** collapse, and at 13 t/s with two
-  0.15-0.4 s. The stall is **during**, not after — decode in windows 2-14 min after a
+  sessions running each stream sits at 6.7 t/s = 149 ms a token, one token per shared step. Nothing
-  busy-prefill window sits at the idle baseline (58-64 vs 77-85 t/s, the residual
+  else moves: decode-*only* steps stay 14-18 ms in every bucket (14.1 ms in a prefill-free minute,
-  confounded by co-scheduling), so no lingering expert-residency penalty at 2-minute
+  17.9 ms in the deep one) and the relationship is lag-0 only (r = −0.74 at lag 0, |r| ≤ 0.19 from
-  resolution, and `preemptions: 0` rules out preempt-and-re-prefill. Method:
+  30 s on, decode back at baseline by 5 min) — residency is not the mechanism either. The second
-  [[LLM Performance Diagnosis]] §9. The law is shape-independent; the fix lives at 1.8.1 (next law),
+  victim is the **arriving** request: through a high-prefill window TTFT p90 goes 2.4 → **18.0 s**
-  and nobody has re-run the census since the migration — so the fix is ordered, **not verified**.
+  and 15.5 % of requests wait over 10 s for their first token, with `http_queue_depth` at 2-3, while
- **The knobs for it arrived with 1.8.1, and only one of them is on.** `--busy-prefill-chunk` caps the
+  median TTFT (~1.9 s) and per-token latency barely move. Preemption is real but burst-shaped: 264
-  shared chunk so a step with waiting decoders takes a smaller piece (live: `auto`, the engine picks;
+  `vllm:num_preemptions_total` in the boot, 108 of them inside 105 s of a 6.5k-tok/s prefill. The
-  the explicit value is 512), and `--prefill-interleave N` inserts N decode-only steps after a chunk
+  chunk is `min(remaining, checkpoint interval, MNBT, busy_cap)`; the first three read 2048 here,
-  step taken beside decoders. `interleave` is deliberately **0** — the env comment holds it there until
+  and `busy_cap` is the fourth law.
-  the decode-vs-prefill buckets ask for it, and those buckets have not been read since the migration.
+- **`auto` is a prefill-speed dial, not a decoder-protection dial.** `Scheduler::busy_cap` reserves
-  The vendor prices a 1024 chunk at about a fifth off prefill throughput to halve the stall; that is a
+  `1 + spec` rows per decoding sequence behind the chunk (3 tokens at MTP 2) plus one eighth of the
-  knob for choosing tomorrow's chunk, not a measurement of this shape. MNBT remains an arm and not an
+  budget per *waiting request or waiting prefill*, and the code refuses to do more for decoders:
-  edit: 2560 collapses decode through MTP resonance. Any chunk change re-scores draft acceptance before
+  "not for decode rows alone: their reserve is a few tokens, so the chunk ends a quantum short and
-  it is trusted.
+  the next step finishes the interval… Halving would hold a long prompt to half-interval chunks while
- **Slots cost KV, rows, and window — never speed.** `--max-num-seqs` carries no radiance constraint: it
+  anything decodes behind it, which costs more" (`core/sched/scheduler.cpp:874-878`, 1.8.1). Live
-  adds ≈110 MiB of recurrent state per slot (the live `kv_full_backing` arithmetic prices them exactly),
+  proof: `prefill_chunks_capped` 5,018 against 62,386 chunk steps — 8 %, and those are the minutes
-  declares the fused TP all-reduce over `seqs × (1 + spec)` rows, and `wht6` is bit-exact up to 25 rows
+  with a waiter, not the minutes with a decoder. An explicit `N` is the only value that applies
-  (single-sequence decode and any step of ≤ 25 rows is exact; only busier steps are lossy) — 2 × 3 = 6
+  "whenever the chunk shares the step".
-  today. Measured 2026-10-06: seqs 2 and seqs 3 are statistically tied (30.9-33.1 t/s, inside the ±3 %
+- **At MNBT 2048 no chunk value keeps both the arena loan and the staged copies.** The arena is a
-  floor). Measured 2026-10-19 at 1.8.1: the third slot's real price is **KV, not rows** — dropping it
+  ladder of levels (256 and 1024 rows here) and a step's rows are its **whole** token count —
-  moved the resolver's cache from 4.20 GiB to 2.76 GiB and the backing from 2.8 to 1.8 sessions, while
+  `max(n_tok, n_seq × (1 + spec))`, `core/engine.cpp:1142` — so a 1024-token chunk with two decode
-  the ≈4 GiB of host RAM it freed came back unbudgeted, landing in elastic pool that turned into a bigger
+  rows is a 1030-row step: it falls back to the full plan, which "lends nothing", and it arms the
-  mover lend instead of cache (resolver law above). The old "seqs 2 cannot boot" law belonged to the
+  prefill stager. The stager's floor is the largest lending level plus one, 1025 rows here — named on
-  retired tcclaviger fork's `torch.compile` shape specialization, and radiance booted seqs 2 on its first
+  the boot line `prefill staging: … a step of more than 1024 tokens copies the pooled experts of 48
-  fp8-native start.
+  routed layers onto the card up to 3 layers ahead` — and below it "every routed layer reads its
  pooled experts synchronously and the prefill slows far more than sharing saves" (GUIDE §4.5). The
  vendor's own number for naive shrinking, on this architecture: four decoders with a long prompt
  beside them finished in **29.5 s at a 2048 chunk and 31.9 s at 512**.
 - **A big step takes the arena from the expert slab and keeps it for the whole prefill.** The arena
  lends 987.77 MiB (steps ≤ 256 rows) or 670.26 MiB (≤ 1024) to the slab; a full-plan step reclaims
  it **synchronously before the step** — units revert to their host copies (an address change, not a
  copy) and both read streams drain — and `arena_tick` regrows the loan only **2 s after the last big
  step** (`core/engine.cpp:1129-1202`), so every step of the prefill run, decode-only included, runs
  with a slab ~780 expert units smaller over a link also carrying the staged copies. `/diagnostics`
  says the binding constraint in one line: "a prefill step streams every routed layer's pooled
  experts onto the card, so its speed is bounded by the link; decode reads the ones it routes to".
  Measured size of the whole effect: the ~10 % above. Real, small, not worth chasing.
 - **`--prefill-interleave` is the dial that buys decode its cadence; armed at 1 on 2026-10-20.**
  N decode-only steps follow any chunk step taken beside decoders; held steps admit nothing, so it
  costs the prefilling prompt its TTFT and an arriving request its first step. Rate arithmetic at the
  two measured step times and 0.657 acceptance (≈2.3 tokens a step): N=0 → ~14 t/s, **N=1 (armed) →
  ~23**, N=2 → ~33, N=4 → ~48, N=8 → ~69, with the prefill slowed by (166 + 15.5N)/166. **Arithmetic, not
  measurement** — the arm is chosen by the §9 bucket pair plus `steps_held`, and `auto` is not a
  substitute for it.
 - **`--prefill-span` is the shape that keeps the loan, and it is priced.** It needs MNBT = N
  intervals, so `--prefill-span 2 --max-num-batched-tokens 4096`: a *lone* prefill carries 2
  intervals a step (and on an expert-tiered box only from 8 intervals / 16k tokens, where the span
  starts to win), while **every other step is planned at the shared budget of 2048** — which is a
  lending level *and* keeps the stager's ring, the one combination `--busy-prefill-chunk` cannot
  express. Costs per GUIDE §4.5: 320 MiB a card of expert residency, ~3 % off the KV pool (§4.3's
  cliff is sharp), a decode step uploads twice the per-step arrays at 4096, and checkpoints inside a
  span are not written, so a partially-shared prompt resumes one interval earlier. On the vendor's
  second box (2×R9700, w4nl64, TP2) it was TTFT-neutral — 2.61 vs 2.56 s at 18k, 10.72 vs 10.55 at
  75k — hence their own rule: measure your own prompts at two ranks before turning it on. MNBT 4096
  does not boot at 3 slots; this shape is 2.
 - **Most of this endpoint's prefill work is replay of tokens the prefix cache already had.** Over
  the same window `prefill_tokens_total` 65.2 M against `prefix_hit_cut_tokens_total` 24.9 M = **38 %
  replayed**, and it is not an eviction: a hit on a hybrid model reuses only as far as the last
  **linear-state** checkpoint (`reuse = min(h.n_attn_tokens, h.n_linear_tokens)`,
  `core/sched/scheduler.cpp:953-978`), and the retention default keeps a **geometric ladder** - about
  `log2(depth/interval) + 2` snapshots - so a turn that diverges `d` behind the tip can resume as
  deep as `tip − 3d`. The live ladder for a 200k session at interval 2048 is 2048 · 131072 · 163840
  · … · 198656: diverge at 120k and 21.7k tokens replay, diverge at 50k and 48k. Every
  `prefix_hit_cuts_total` carries `reason="no_snapshot"` (the other two reasons stay dead), so the
  cause object agrees and `hit_cut_tokens` is the cost. `--checkpoint-policy` admits exactly
  **`geometric-backoff` | `tip-only` | `keep-all`** and refuses anything else at boot. It governs
  retention only - the writes happen at every interval whatever the policy says (only `--prefill-span`
  skips them) - so `keep-all` buys a replay bounded by one interval at any depth for **pinned host RAM
  alone**: 97 snapshots a 200k session, `--checkpoint-slots auto` asking 196 × 111.98 MiB = 21.4 GiB
  against a store budget of an eighth of host RAM (31.5 GiB here, so `share` admits it) with `room` -
  the pinning room left beside the expert pool at boot - as the hard bound the ledger line names.
  Retention also spares a snapshot a later turn resumed from (`core/mem/prefix.cpp:709-723`), so a
  chain that keeps resuming at one deep point stops paying for it. Shortening
  `--checkpoint-interval` does **not** buy the deep case: the coarsest ladder scale is the largest
  `interval × 2^k` under the depth, so the deepest gap stays ≈ half the prompt (129,024 tokens at
  interval 2048 against 130,816 at 256) while every cut multiplies the snapshot writes.
 - **Slots cost KV, host RAM, rows, and pin — never speed.** `--max-num-seqs` carries no radiance
  constraint: it adds ≈110 MiB of recurrent state per slot, 10 × 111.98 MiB of pinned checkpoint-store
  RAM per slot (`--checkpoint-slots auto`), and declares the fused TP all-reduce over
  `seqs × (1 + spec)` rows — `wht6` is bit-exact up to 25 rows, and the live shape is 4 × 3 = **12**, so
  single-sequence decode stays exact and so does any step up to that many rows. Raising it also raises
  the cache's *demand* (`ceiling = slots × window`), which the resolver funds from `elastic −
  expert_floor` — so a slot count without matching host room is paid for by expert residency. Measured
  2026-10-06: seqs 2 and 3 statistically tied (30.9-33.1 t/s, inside the ±3 % floor). Measured
  2026-10-19 → 10-20 at 1.8.1: 2 → 4 slots cost **+4,224 MiB of pinned host pool** and **−3.01 GiB of
  expert residency a card** and bought 2.76 → 5.64 GiB of cache (365,204 → 747,492 tokens). The old
  "seqs 2 cannot boot" law belonged to the retired tcclaviger fork's `torch.compile` shape
  specialization, and radiance booted seqs 2 on its first fp8-native start.
 - **A dequantize-to-compute path bills host RAM twice.** On the EMULATION backend an offload request
  of **118 GiB** landed as **~238 GiB** of real host RAM — the dequantized copy rides on top of the
  pinned weights, and the doubling is invisible in the weight count. Read the engine log for
@@ -396,12 +482,35 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
  4600 MT/s ended it, with zero `gdn_hip` faults either side; the prefix-caching / GDN re-seed
  theory was confounded on both arms. Consumer non-ECC raises no MCE for a silent flip and a
  60 s / 8 GiB pattern check covers 0.5 % of 246 GiB — silent RAM corruption outranks every
-  software theory.
+  software theory. **That mitigation is a BIOS setting and it is not in effect now:** `sudo
  dmidecode -t memory | grep -i configured` answers 4800 MT/s on 2026-10-20, and the only event
  between the fix and today is the 2026-10-12 power failure that also reset TTM. Check the speed
  before crediting any software revert with a fix — and before believing a clean run of the arm
  you just tested, because the confounder is still in the slot.
 - **TP=2 here** is legal since the 2026-09-20 ProArt B850 swap (x8/x8 on CPU root ports,
  `Routing+`/`ReqEn+` end to end) only because radiance's wire is `libr4d`'s own one-shot
  all-reduce; RCCL needs atomics on the whole chain and fails `hipErrorIllegalState` without
  them (). `wht6` buys +5.6-6.3 % prefill for ~1.3 % of 8-stream
  decode, `exact` is the bit-exact alternative.
 - **The link is already at the board's ceiling — do not chase the `link-below-max` warning.**
  Both root ports (`00:01.1`, `00:01.3`) are **x8 max** at 32 GT/s, so PCIe 5.0 x8 ≈ 31.5 GB/s is
  the whole budget per card and the cards' own `pcie_dpm` tables expose only x8 states.
  `/diagnostics` still warns that the on-card switch hop (`01:00.0`, `05:00.0`) runs "x8 of its
  x16 while the card is 100 % busy" and suggests a reboot or a reseat — pointless when the host
  side of the chain is x8 max, which the same endpoint reports two findings later as
  `link-capped`. The width that matters is the minimum of the chain, and it is x8 either way, so
  the only lever on a link-bound prefill is **less traffic over the link** (more expert
  residency, fewer pooled experts), never a wider link. P2P resolves **on** (`auto` asks the
  driver, the two cards reach each other's memory), so the collectives do not hairpin through host
  RAM, and the engine's copies ride one DMA queue that "give[s] way to the collectives" — the
  all-reduce wins the link, the staged experts wait.
 - **Eight cores is the whole host, media stack included.** Dankserver is a Ryzen 7 7700X (8
  cores / 8 threads, one NUMA node, `nodes: -1` for both cards) running Jellyfin, Plex, Calibre
  and RomM beside the engine, and `--http-threads` defaults to one thread per core. Measured
  during a heavy prefill: load average ≈ 1.0, `jellyfin` 11 % CPU, `radiance` 7 % — **CPU
  contention is not the current decode culprit**, so do not spend an experiment window on it;
  re-read `uptime` and `ps -eo pcpu,comm --sort=-pcpu | head` first if a future collapse looks
  different from the step-sharing one.
 - `--prefix-cache-dir` creation is **not recursive** — the serve script `mkdir -p`s the path or
  the disk tier silently never opens. The tier does survive a payload swap when arch and
  tokenizer match (it recovered 900,176 blocks on the 2026-10-15 swap; `cache_n` proves a hit).
@@ -445,11 +554,19 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
 ## Contract (the wire)
 - **Live surface:** `/health` (== `/ping`), `/v1/models`, `/v1/chat/completions`
-  `/v1/completions`, `/metrics`, `/stats`, `/sessions`, `/dashboard`. `/health` answers 200
+  `/v1/completions`, `/metrics`, `/stats`, `/sessions`, `/dashboard`, **`/diagnostics`**. `/health`
-  JSON (model id, `in_flight`, `queue_depth`) while serving and **503 only while stopping**
+  answers 200 JSON (model id, `in_flight`, `queue_depth`) while serving and **503 only while
-  (`core/server/admin.cpp:385`), and the listener binds *after* the full load (`main.cpp`:
+  stopping** (`core/server/admin.cpp:385`), and the listener binds *after* the full load
-  `engine.start()` then `engine.serve()`) — during the load the port refuses connections, so a
+  (`main.cpp`: `engine.start()` then `engine.serve()`) — during the load the port refuses
-  bare `curl -fsS …/health` gate is honest and needs no body grep.
+  connections, so a bare `curl -fsS …/health` gate is honest and needs no body grep.
 - **`/diagnostics` is the instrument for "is the box, not the config, the limit"** — one call
  returns `configuration` (every flag, so drift from the env file is checkable without the host),
  `environment`, `derived` (the VRAM split as the ledger prints it), `hardware` (host, per-card
  `link_chain` + `pcie_dpm` + clocks + power cap + DMA engine choice + P2P answer) and `findings`
  (level / what / why / fix). On this box it reports `experts-from-host` (5.99 GiB a rank crosses
  PCIe when used), `link-capped` (root ports are x8 max — the board), `swap-in-use`,
  `tp-wire-lossy`, and **`link-below-max` as a `warn` at 100 % card busy on both cards** — which is
  structural on this board; see the x8-ceiling law under *This host, and the deploy path*.
 - **Served id.** `FRANK_ARCHON_SERVED_NAME=frank-archon` (the wire rename of 2026-10-09 moved
  only the alias; container, unit, scripts and env keep the service name). The env's contract
  line is that **radiance validates the requested model name, so every client — harness, bench
@@ -476,12 +593,28 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
  and 16,384 patches per pass. Probe with a ≥ 256 px solid-colour PNG and a ≥ 256 token budget
  — at 64 tokens you get empty `content` and the colour only inside `reasoning_content`.
 - **Capacity behaviour:** a prompt over `--max-model-len` is refused, never truncated; a request
-  longer than the KV pool's token count is accepted then refused as too large to fit; under the
+  longer than the KV pool's token count is accepted then refused as too large to fit; a request
-  3-slot contract a **fourth** full-height conversation is refused, not paged or squeezed.
+  beyond `--max-num-seqs` is held by the HTTP port (`queue_cap` = `max(8, 8 × slots)` — **32** here)
  and answered 503 only past that, never paged or squeezed. The cache funds **3.7** full-height
  sessions, so four conversations can be resident but a fourth near-full-height one can still
  **preempt a running sequence from the back** to free its blocks
  (`core/sched/scheduler.cpp:1291-1322`); the victim is recomputed, so its session halts and then
  re-prefills. Meter: `vllm:num_preemptions_total` (cumulative for the boot, `/stats preemptions`);
  `radiance:num_requests_preempted` is only the instant depth.
 - **Observability:** `/metrics` carries vLLM-compatible names plus `radiance:*`, scraped by the
-  Dankclaw Prometheus job at `192.168.0.235:7770` over plain LAN routing (the netavark
+  Dankclaw Prometheus job at `192.168.0.235:7770` (15 s interval) over plain LAN routing (the
-  published-port quirk is same-host only). The boot ledger — the referee for every shape number
+  netavark published-port quirk is same-host only). **Latency lives under `vllm:`, cadence under
-  above — is in the container log (`podman logs frank-archon`)
+  `radiance:`** — TTFT / inter-token / per-output-token / e2e / decode-time histograms and the
  running/waiting gauges are `vllm:*` with no radiance-native equivalent, so a `radiance:`-only
  query misses every latency answer. The step-phase counters are all scraped:
  `engine_steps{,_decode,_mixed,_prefill,_held}_total`, `engine_step_seconds{,_decode}_total`,
  `engine_prefill_chunks_capped_total`, `prefill_tokens_total`, `decode_tokens_per_second`,
  `prefill_tokens_per_second`, `prefix_hit_cut_tokens_total`, `http_queue_depth`. Two holes: the
  engine's own step-detail histograms (`prefill_steps_duration`, `prefill_step_tokens`,
  `token_queue_wait`) are exported but reach no target, and the **arena / expert slab / heat engine /
  prefill stager have no metric at all** — that half of every law above is read from the boot ledger,
  `/stats` and `/diagnostics`, never from a graph. The boot ledger — the referee for every shape
  number here — is in the container log (`podman logs frank-archon`);
  `/home/dank/logs/frank_archon.log` is where the serve script redirects `podman run`.
 ## Files
@@ -0,0 +1,144 @@
 {
  "uid": "host-system",
  "title": "Hosts \u2014 Dankclaw & Dankserver",
  "description": "node_exporter for both servers. Starter set \u2014 throw-at-the-wall breadth; refine the windows and drop the noise. The picker selects the host; CPU/load/memory/filesystem/network are per-node.",
  "tags": ["host", "system"],
  "timezone": "browser",
  "schemaVersion": 39,
  "refresh": "30s",
  "time": { "from": "now-6h", "to": "now" },
  "templating": {
    "list": [
      {
        "name": "node",
        "label": "host",
        "type": "query",
        "datasource": { "type": "prometheus", "uid": "prometheus" },
        "query": { "query": "label_values(up{job=~\"node-.*\"}, job)", "refId": "A" },
        "includeAll": true,
        "multi": true,
        "current": { "text": "All", "value": "$__all" }
      }
    ]
  },
  "panels": [
    {
      "id": 1, "type": "stat", "title": "Uptime",
      "description": "Time since boot. A drop means a reboot.",
      "gridPos": { "h": 4, "w": 6, "x": 0, "y": 0 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "fieldConfig": { "defaults": { "unit": "s" }, "overrides": [] },
      "targets": [{ "expr": "time() - node_boot_time_seconds{job=~\"$node\"}", "legendFormat": "{{job}}", "refId": "A" }]
    },
    {
      "id": 2, "type": "stat", "title": "Memory used",
      "description": "1 - MemAvailable/MemTotal. The honest \"in use\" figure (excludes reclaimable cache).",
      "gridPos": { "h": 4, "w": 6, "x": 6, "y": 0 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1, "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 0.8 }, { "color": "red", "value": 0.92 }] } }, "overrides": [] },
      "targets": [{ "expr": "1 - node_memory_MemAvailable_bytes{job=~\"$node\"} / node_memory_MemTotal_bytes{job=~\"$node\"}", "legendFormat": "{{job}}", "refId": "A" }]
    },
    {
      "id": 3, "type": "stat", "title": "Load 5m / core",
      "description": "Per-core load5. >1 sustained means runnable tasks are queuing behind cores.",
      "gridPos": { "h": 4, "w": 6, "x": 12, "y": 0 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 0.7 }, { "color": "red", "value": 1 }] } }, "overrides": [] },
      "targets": [{ "expr": "node_load5{job=~\"$node\"} / count by (job) (node_cpu_seconds_total{job=~\"$node\",mode=\"idle\"})", "legendFormat": "{{job}}", "refId": "A" }]
    },
    {
      "id": 4, "type": "stat", "title": "Open file handles",
      "description": "FDs allocated vs the kernel maximum. Rising toward the max precedes EMFILE across the host.",
      "gridPos": { "h": 4, "w": 6, "x": 18, "y": 0 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "fieldConfig": { "defaults": { "unit": "percentunit", "max": 1, "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 0.7 }, { "color": "red", "value": 0.9 }] } }, "overrides": [] },
      "targets": [{ "expr": "node_filefd_allocated{job=~\"$node\"} / node_filefd_maximum{job=~\"$node\"}", "legendFormat": "{{job}}", "refId": "A" }]
    },
    {
      "id": 5, "type": "timeseries", "title": "CPU busy %",
      "description": "CPU seconds/s in each mode = cores busy per mode; the stack sums to total cores in use (16/host here). A user or system spike points at what is loading the box; steal > 0 means the hypervisor is throttling.",
      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 4 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "fieldConfig": { "defaults": { "unit": "short", "min": 0, "custom": { "fillOpacity": 10, "lineWidth": 1, "stacking": { "mode": "normal" } } }, "overrides": [] },
      "targets": [
        { "expr": "sum by (job,mode) (rate(node_cpu_seconds_total{job=~\"$node\",mode!=\"idle\",mode!=\"iowait\"}[5m]))", "legendFormat": "{{job}} {{mode}}", "refId": "A" }
      ]
    },
    {
      "id": 6, "type": "timeseries", "title": "Load average",
      "description": "load1/5/15 per host.",
      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 4 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] },
      "targets": [
        { "expr": "node_load1{job=~\"$node\"}", "legendFormat": "{{job}} load1", "refId": "A" },
        { "expr": "node_load5{job=~\"$node\"}", "legendFormat": "{{job}} load5", "refId": "B" },
        { "expr": "node_load15{job=~\"$node\"}", "legendFormat": "{{job}} load15", "refId": "C" }
      ]
    },
    {
      "id": 7, "type": "timeseries", "title": "Memory breakdown",
      "description": "Total, free, buffers+cache. Reclaimable cache is not pressure; MemAvailable is the real headroom.",
      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 12 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "fieldConfig": { "defaults": { "unit": "bytes" }, "overrides": [] },
      "targets": [
        { "expr": "node_memory_MemTotal_bytes{job=~\"$node\"} - node_memory_MemAvailable_bytes{job=~\"$node\"}", "legendFormat": "{{job}} used", "refId": "A" },
        { "expr": "node_memory_MemAvailable_bytes{job=~\"$node\"}", "legendFormat": "{{job}} available", "refId": "B" }
      ]
    },
    {
      "id": 8, "type": "timeseries", "title": "Filesystem free %",
      "description": "Per mountpoint. The media/library stacks fill disks.",
      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 12 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1, "thresholds": { "mode": "absolute", "steps": [{ "color": "red", "value": null }, { "color": "green", "value": 0.15 }] } }, "overrides": [] },
      "targets": [
        { "expr": "node_filesystem_avail_bytes{job=~\"$node\",fstype!~\"tmpfs|overlay|squashfs\"} / node_filesystem_size_bytes{job=~\"$node\",fstype!~\"tmpfs|overlay|squashfs\"}", "legendFormat": "{{job}} {{mountpoint}}", "refId": "A" }
      ]
    },
    {
      "id": 9, "type": "timeseries", "title": "Disk throughput",
      "description": "Read/write bytes/s per device.",
      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 20 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "fieldConfig": { "defaults": { "unit": "Bps" }, "overrides": [] },
      "targets": [
        { "expr": "sum by (job,device) (rate(node_disk_read_bytes_total{job=~\"$node\"}[5m]))", "legendFormat": "{{job}} {{device}} R", "refId": "A" },
        { "expr": "-sum by (job,device) (rate(node_disk_written_bytes_total{job=~\"$node\"}[5m]))", "legendFormat": "{{job}} {{device}} W", "refId": "B" }
      ]
    },
    {
      "id": 10, "type": "timeseries", "title": "Disk busy %",
      "description": "Fraction of time the device had I/O in flight. Near 1 = the device is the bottleneck.",
      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 20 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1 }, "overrides": [] },
      "targets": [
        { "expr": "rate(node_disk_io_time_seconds_total{job=~\"$node\"}[5m])", "legendFormat": "{{job}} {{device}}", "refId": "A" }
      ]
    },
    {
      "id": 11, "type": "timeseries", "title": "Network throughput",
      "description": "Receive (up) / transmit (down) bytes/s per interface.",
      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 28 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "fieldConfig": { "defaults": { "unit": "Bps" }, "overrides": [] },
      "targets": [
        { "expr": "sum by (job,device) (rate(node_network_receive_bytes_total{job=~\"$node\",device!~\"lo|veth.*|br.*|docker.*\"}[5m]))", "legendFormat": "{{job}} {{device}} RX", "refId": "A" },
        { "expr": "-sum by (job,device) (rate(node_network_transmit_bytes_total{job=~\"$node\",device!~\"lo|veth.*|br.*|docker.*\"}[5m]))", "legendFormat": "{{job}} {{device}} TX", "refId": "B" }
      ]
    },
    {
      "id": 12, "type": "timeseries", "title": "Network errors / drops",
      "description": "Interface errors and drops \u2014 the signature of a saturated link or a bad NIC/cable.",
      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 28 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "fieldConfig": { "defaults": { "unit": "ops" }, "overrides": [] },
      "targets": [
        { "expr": "sum by (job,device) (rate(node_network_receive_errs_total{job=~\"$node\"}[5m]))", "legendFormat": "{{job}} {{device}} RX err", "refId": "A" },
        { "expr": "sum by (job,device) (rate(node_network_transmit_errs_total{job=~\"$node\"}[5m]))", "legendFormat": "{{job}} {{device}} TX err", "refId": "B" }
      ]
    }
  ]
 }
@@ -0,0 +1,75 @@
 {
  "uid": "logs-overview",
  "title": "Logs \u2014 Loki (systemd journal)",
  "description": "Everything in Loki arrives as the journal (job=systemd-journal); the container is a JSON field parsed at query time, not a stream label. Starter breadth \u2014 the volume/error panels group by the parsed container_name. Loki keeps ~7 days; wide ranges are cheap counts, not the raw log panel.",
  "tags": ["logs", "loki"],
  "timezone": "browser",
  "schemaVersion": 39,
  "refresh": "30s",
  "time": { "from": "now-1h", "to": "now" },
  "templating": {
    "list": [
      {
        "name": "container",
        "label": "container",
        "type": "query",
        "datasource": { "type": "loki", "uid": "loki" },
        "query": { "query": "{job=\"systemd-journal\"} | json", "label": "container_name", "refId": "A" },
        "includeAll": true,
        "multi": true,
        "refresh": 2,
        "current": { "text": "All", "value": "$__all" }
      }
    ]
  },
  "panels": [
    {
      "id": 1, "type": "timeseries", "title": "Log volume by container (lines/s)",
      "description": "Parse the journal JSON, count lines per container. Spikes = a chatty or crashing unit.",
      "gridPos": { "h": 9, "w": 24, "x": 0, "y": 0 },
      "datasource": { "type": "loki", "uid": "loki" },
      "fieldConfig": { "defaults": { "unit": "reqps", "custom": { "fillOpacity": 20, "stacking": { "mode": "normal" } } }, "overrides": [] },
      "targets": [{ "expr": "sum by (container_name) (count_over_time({job=\"systemd-journal\"} | json [$__auto]))", "legendFormat": "{{container_name}}", "refId": "A" }]
    },
    {
      "id": 2, "type": "timeseries", "title": "Total log rate",
      "description": "All journal lines per second.",
      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 9 },
      "datasource": { "type": "loki", "uid": "loki" },
      "fieldConfig": { "defaults": { "unit": "reqps" }, "overrides": [] },
      "targets": [{ "expr": "sum(count_over_time({job=\"systemd-journal\"} [$__auto]))", "legendFormat": "all lines/s", "refId": "A" }]
    },
    {
      "id": 3, "type": "timeseries", "title": "Error-level line rate",
      "description": "Lines matching error/panic/fatal/fail. A blunt text match \u2014 the journal carries no structured level for all units.",
      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 9 },
      "datasource": { "type": "loki", "uid": "loki" },
      "fieldConfig": { "defaults": { "unit": "reqps", "color": { "mode": "fixed", "fixedColor": "red" } }, "overrides": [] },
      "targets": [{ "expr": "sum(count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal|fail)\" [$__auto]))", "legendFormat": "error lines/s", "refId": "A" }]
    },
    {
      "id": 4, "type": "timeseries", "title": "Errors by container",
      "description": "Which unit is throwing. Rank by the top offenders.",
      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 17 },
      "datasource": { "type": "loki", "uid": "loki" },
      "fieldConfig": { "defaults": { "unit": "reqps" }, "overrides": [] },
      "targets": [{ "expr": "sum by (container_name) (count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal)\" | json [$__auto]))", "legendFormat": "{{container_name}}", "refId": "A" }]
    },
    {
      "id": 5, "type": "stat", "title": "Error lines (last 15m)",
      "description": "Absolute count over the window, not a rate.",
      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 17 },
      "datasource": { "type": "loki", "uid": "loki" },
      "fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 50 }, { "color": "red", "value": 500 }] } }, "overrides": [] },
      "targets": [{ "expr": "sum(count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal)\" [15m]))", "instant": true, "legendFormat": "errors/15m", "refId": "A" }]
    },
    {
      "id": 6, "type": "logs", "title": "Journal (filtered)",
      "description": "Raw lines for the selected containers. Set the container picker; keep the window tight.",
      "gridPos": { "h": 12, "w": 24, "x": 0, "y": 25 },
      "datasource": { "type": "loki", "uid": "loki" },
      "options": { "showTime": true, "wrapLogMessage": true, "sortOrder": "Descending", "enableLogDetails": true },
      "targets": [{ "expr": "{job=\"systemd-journal\"} | json | container_name=~\"$container\"", "refId": "A" }]
    }
  ]
 }
@@ -0,0 +1,96 @@
 {
  "uid": "observability",
  "title": "Observability \u2014 self-monitoring",
  "description": "The health of the stack that watches the fleet: Prometheus scrape/TSDB, Loki ingest, Grafana. If this is red the dashboards lie. Note: Alertmanager is running but NOT scraped by Prometheus (no job) \u2014 its metrics are absent by design gap, not fault.",
  "tags": ["observability", "meta"],
  "timezone": "browser",
  "schemaVersion": 39,
  "refresh": "30s",
  "time": { "from": "now-6h", "to": "now" },
  "panels": [
    {
      "id": 1, "type": "stat", "title": "Targets up",
      "gridPos": { "h": 4, "w": 4, "x": 0, "y": 0 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }] } }, "overrides": [] },
      "targets": [{ "expr": "count(up == 1)", "instant": true, "refId": "A" }]
    },
    {
      "id": 2, "type": "stat", "title": "Targets down",
      "description": "Known structural downs: comfyui (no metrics endpoint), gitea (metrics off), and the homepage 'services' probes. Real regressions ride up from here too \u2014 check the table before dismissing.",
      "gridPos": { "h": 4, "w": 4, "x": 4, "y": 0 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "red", "value": 1 }] } }, "overrides": [] },
      "targets": [{ "expr": "count(up == 0)", "instant": true, "refId": "A" }]
    },
    {
      "id": 3, "type": "stat", "title": "TSDB head series",
      "description": "Active series in memory. A jump = a new exporter or a label-cardinality leak.",
      "gridPos": { "h": 4, "w": 5, "x": 8, "y": 0 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] },
      "targets": [{ "expr": "prometheus_tsdb_head_series", "instant": true, "refId": "A" }]
    },
    {
      "id": 4, "type": "stat", "title": "Discovered targets",
      "gridPos": { "h": 4, "w": 5, "x": 13, "y": 0 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] },
      "targets": [{ "expr": "sum(prometheus_sd_discovered_targets)", "instant": true, "refId": "A" }]
    },
    {
      "id": 5, "type": "stat", "title": "Prometheus mem",
      "gridPos": { "h": 4, "w": 3, "x": 18, "y": 0 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "fieldConfig": { "defaults": { "unit": "bytes" }, "overrides": [] },
      "targets": [{ "expr": "process_resident_memory_bytes{job=\"prometheus\"}", "instant": true, "refId": "A" }]
    },
    {
      "id": 6, "type": "stat", "title": "Grafana up",
      "gridPos": { "h": 4, "w": 3, "x": 21, "y": 0 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "fieldConfig": { "defaults": { "mappings": [{ "type": "value", "options": { "1": { "text": "UP", "color": "green" }, "0": { "text": "DOWN", "color": "red" } } }], "thresholds": { "mode": "absolute", "steps": [{ "color": "red", "value": null }, { "color": "green", "value": 1 }] } }, "overrides": [] },
      "targets": [{ "expr": "up{job=\"grafana\"}", "instant": true, "refId": "A" }]
    },
    {
      "id": 7, "type": "table", "title": "Target health",
      "description": "Every scrape target and its current up. Sort by Value to surface regressions.",
      "gridPos": { "h": 9, "w": 12, "x": 0, "y": 4 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "options": { "sortBy": [{ "displayName": "Value", "sortOrder": "Asc" }] },
      "targets": [{ "expr": "up", "format": "table", "instant": true, "refId": "A" }]
    },
    {
      "id": 8, "type": "timeseries", "title": "Scrape interval drift (p99 actual)",
      "description": "Actual seconds between scrapes (summary quantile). Above the 15s scrape_interval = Prometheus is falling behind its own cadence (slow endpoints or overload).",
      "gridPos": { "h": 9, "w": 12, "x": 12, "y": 4 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "fieldConfig": { "defaults": { "unit": "s" }, "overrides": [] },
      "targets": [{ "expr": "prometheus_target_interval_length_seconds{quantile=\"0.99\"}", "legendFormat": "{{instance}}", "refId": "A" }]
    },
    {
      "id": 9, "type": "timeseries", "title": "Scrape samples/s",
      "description": "Total samples ingested across all jobs.",
      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 13 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] },
      "targets": [{ "expr": "sum(rate(scrape_samples_scraped[5m]))", "legendFormat": "samples/s", "refId": "A" }]
    },
    {
      "id": 10, "type": "timeseries", "title": "Loki request rate",
      "description": "Loki served requests/s (push, query). A drop to zero means ingest or query has stopped.",
      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 13 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "fieldConfig": { "defaults": { "unit": "reqps" }, "overrides": [] },
      "targets": [{ "expr": "sum by (route) (rate(loki_request_duration_seconds_count[5m]))", "legendFormat": "{{route}}", "refId": "A" }]
    },
    {
      "id": 11, "type": "timeseries", "title": "Exporter process CPU",
      "description": "CPU by monitored process \u2014 spot a runaway exporter or the Prometheus process itself.",
      "gridPos": { "h": 8, "w": 24, "x": 0, "y": 21 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "fieldConfig": { "defaults": { "unit": "percentunit" }, "overrides": [] },
      "targets": [{ "expr": "sum by (job) (rate(process_cpu_seconds_total[5m]))", "legendFormat": "{{job}}", "refId": "A" }]
    }
  ]
 }
@@ -29,8 +29,14 @@ export default defineConfig({
      // green suite with a test that does not exist. 17- joined on 2026-09-16
      // (the hermes vhost, whose upstream is a host address rather than a
      // container name, so it needs its own pass condition). 18- joined on
-      // 2026-09-24 (OpenWebUI SSO through authentik).
+      // 2026-09-24 (OpenWebUI SSO through authentik). 20- joined on 2026-10-21
-      testMatch: /(10|12|13|16|17|18)-.*\.spec\.ts/,
+      // (Grafana's OAuth2 relying-party login, which the edge-only 13- never
      // follows). 21- joined 2026-10-21 (the open-notebook vhost, deployed
      // 2026-10-19 - the newest guarded app, in neither GUARDED list nor any
      // spec until now: the 17-hermes "uncovered vhost broke silently" class).
      // 22- joins Prometheus's app-is-really-there proof (its own API), closing
      // the "not a login wall but maybe a 404" hole 20- closed for Grafana.
      testMatch: /(10|12|13|16|17|18|20|21|22)-.*\.spec\.ts/,
      use: { storageState: ".auth/qa-e2e.json" },
      dependencies: ["setup"],
    },
@@ -15,7 +15,7 @@ const BASE = "https://www.franksdank.cloud";
 // Every vhost that Caddy routes through anubis:8923 (see dankclaw/Caddyfile).
 // Keep this list in lockstep with the Caddyfile: a new guarded vhost must be
 // added here or the guard-chain audit is incomplete.
-const GUARDED = ["www", "search", "romm", "files", "seerr", "grafana", "prometheus", "loki"];
+const GUARDED = ["www", "search", "romm", "files", "seerr", "grafana", "prometheus", "loki", "notebook"];
 // Markers that prove app content leaked past the guard to an anonymous user.
 const LEAK_MARKERS: Record<string, string[]> = {
@@ -25,6 +25,7 @@ const LEAK_MARKERS: Record<string, string[]> = {
  "seerr.franksdank.cloud": ["Request Movie"],
  "grafana.franksdank.cloud": ["Dashboards"],
  "prometheus.franksdank.cloud": ["Expression Browser", "Graph"],
  "notebook.franksdank.cloud": ["Open Notebook"],
 };
 test("homepage root (/) is guarded for unauthenticated users", async ({ page }) => {
@@ -0,0 +1,56 @@
 import { expect, test } from "@playwright/test";
 // Grafana (grafana.franksdank.cloud) is an OAuth2 RELYING PARTY, not an
 // outpost-fronted proxy: the guard chain (13-sso) only proves an authenticated
 // session reaches Grafana's login page. The login itself happens through
 // Grafana's own "Sign in with Authentik" button, which the edge test never
 // follows — that is exactly how the RP login could be dead (500 on authorize,
 // or an email-claim gap bouncing back to /login) while 13- still passed green.
 // This spec drives the real RP round-trip and asserts a live Grafana session.
 const HOST = "grafana.franksdank.cloud";
 const CHALLENGE = "Making sure you're not a bot";
 test("SSO: Grafana relying-party login lands a real session", async ({ page, request }) => {
  test.setTimeout(180_000);
  // Warm the guard: the vhost runs its own Anubis proof-of-work on first visit.
  await page.goto(`https://${HOST}/`);
  await expect
    .poll(
      async () => !(await page.locator("body").innerText()).includes(CHALLENGE),
      { timeout: 90_000 },
    )
    .toBe(true);
  // The "Sign in with Authentik" button targets this RP start URL exactly.
  // Going straight there is the same code path without depending on the SPA's
  // button markup.
  await page.goto(`https://${HOST}/login/generic_oauth`);
  // The chain: grafana -> auth (implicit consent, the session is already
  // there) -> back to grafana with the code -> grafana provisions the user and
  // redirects to "/". A dead provider/claims config bounces back to /login.
  await expect
    .poll(
      async () => {
        const u = new URL(page.url());
        return u.hostname === HOST && u.pathname !== "/login";
      },
      { timeout: 90_000 },
    )
    .toBe(true);
  const body = await page.locator("body").innerText();
  expect(body).not.toContain("Login failed");
  expect(body.toLowerCase()).not.toContain("powered by authentik"); // IdP flow shell
  expect(body).not.toContain(CHALLENGE);
  // A genuine logged-in session exists: /api/user answers 200 with an email,
  // not 401. This is the assertion the edge-only spec could never make.
  const resp = await page.request.get(`https://${HOST}/api/user`);
  expect(resp.status()).toBe(200);
  const user = await resp.json();
  expect(typeof user.email).toBe("string");
  expect(user.email.length).toBeGreaterThan(0);
 });
@@ -0,0 +1,52 @@
 import { expect, test } from "@playwright/test";
 // open-notebook (notebook.franksdank.cloud), deployed 2026-10-19, is a guarded
 // vhost whose outpost (notebook-proxy) fronts an app with its OWN password
 // login - the same shape as hermes (17-) and OpenWebUI (18-): the guard chain
 // (Anubis PoW -> authentik outpost) must hand off to the app, and the app's own
 // login page is the expected landing. Until now this host appeared in neither
 // the 13-/14- GUARDED lists nor any dedicated spec, so a broken upstream would
 // have stayed invisible - the exact 17-hermes "uncovered vhost" failure class.
 // 14- now covers the unauthenticated gate; this spec owns authenticated
 // reachability with a POSITIVE app marker, not just "not a login wall".
 const VHOST = "https://notebook.franksdank.cloud/";
 const CHALLENGE = "Making sure you're not a bot";
 test("notebook vhost reaches the open-notebook app through the guard chain", async ({ page }) => {
  test.setTimeout(180_000);
  await page.goto(VHOST);
  // The Anubis proof-of-work clears in a real browser and the outpost hands off
  // to the app; the user must never be parked on an authentik flow.
  await expect
    .poll(async () => !(await page.locator("body").innerText()).includes(CHALLENGE), { timeout: 90_000 })
    .toBe(true);
  await expect
    .poll(() => new URL(page.url()).hostname, { timeout: 60_000 })
    .toBe("notebook.franksdank.cloud");
  expect(
    new URL(page.url()).pathname.startsWith("/if/"),
    "must not be parked on an authentik flow",
  ).toBe(false);
  expect(
    await page.locator('input[name="uidField"]').count(),
    "must not show the authentik login form",
  ).toBe(0);
  // Positive identity: the page really is open-notebook's own login, not a
  // Caddy error or the IdP shell. The app answering its own login is the pass.
  await expect(page, "destination title must be Open Notebook").toHaveTitle(/open notebook/i, {
    timeout: 30_000,
  });
  await expect(page.locator("body"), "open-notebook login marker").toContainText(
    /enter your password/i,
    { timeout: 30_000 },
  );
  const body = await page.locator("body").innerText();
  expect(body, "must not leak the IdP flow shell").not.toContain("Powered by authentik");
  expect(body, "must not be a Caddy upstream error").not.toContain("Bad Gateway");
  const resp = await page.request.get(VHOST);
  expect(resp.status(), "open-notebook must answer, not error").toBeLessThan(500);
 });
@@ -0,0 +1,33 @@
 import { expect, test } from "@playwright/test";
 // Prometheus (prometheus.franksdank.cloud) sits behind the same embedded outpost
 // as Grafana. 13-sso and 11-admin only ever assert it is "not a login wall and
 // under 500" - which a Prometheus serving a 404 or a blank page would ALSO pass.
 // This is the app-is-really-there proof 20- provides for Grafana: with the
 // standard session the outpost hands off silently to the Prometheus UI, and the
 // app's own API answers success. A guard-route or upstream regression that lets
 // any non-Prometheus answer through fails here.
 test("SSO: Prometheus app answers its own API through authentik", async ({ page, request }) => {
  test.setTimeout(180_000);
  await page.goto("https://prometheus.franksdank.cloud/");
  // Challenge clears, outpost hands off silently to the app (no credential prompt).
  await expect
    .poll(
      async () => !(await page.locator("body").innerText()).includes("Making sure you're not a bot"),
      { timeout: 90_000 },
    )
    .toBe(true);
  await expect
    .poll(() => new URL(page.url()).hostname, { timeout: 60_000 })
    .toBe("prometheus.franksdank.cloud");
  expect(await page.locator('input[name="uidField"]').count(), "no authentik form").toBe(0);
  await expect(page, "must land on the Prometheus UI").toHaveTitle(/prometheus/i, { timeout: 30_000 });
  // Load-bearing assertion: the Prometheus API itself answers, authenticated,
  // through the outpost. An IdP bounce, a 404, or a raw upstream error fails here.
  const api = await request.get("https://prometheus.franksdank.cloud/api/v1/status/buildinfo");
  expect(api.status(), "Prometheus API must answer 200 through the guard").toBe(200);
  expect((await api.json()).status, "Prometheus API must report success").toBe("success");
 });
@@ -145,7 +145,7 @@ nosudo_bin() {
    # (torch.compile query_start_loc constraint, 2026-10-02); radiance booted
    # seqs 2 in the fp8-native era and the 2026-10-06 sweep had seqs 2 and 3
    # tied inside the +-3% noise floor — slots buy concurrency, not speed.
-    grep -q '^FRANK_ARCHON_MAX_SEQS=2$' "$ENV_FILE"
+    grep -q '^FRANK_ARCHON_MAX_SEQS=4$' "$ENV_FILE"
    grep -q '^FRANK_ARCHON_TP=2$' "$ENV_FILE"
    grep -q '^FRANK_ARCHON_TP_WIRE=wht6$' "$ENV_FILE"
    grep -q '^FRANK_ARCHON_SPEC_TOKENS=2$' "$ENV_FILE"
@@ -158,7 +158,7 @@ nosudo_bin() {
    # so the pool's token count is a boot-ledger line — verified live by the
    # capacity leg of scripts/probe-frank-archon-kv.py at every cutover.
    grep -q '^FRANK_ARCHON_PLACEMENT=expert_tiered$' "$ENV_FILE"
-    grep -q '^FRANK_ARCHON_HOST_POOL_MIB=44928$' "$ENV_FILE"
+    grep -q '^FRANK_ARCHON_HOST_POOL_MIB=49152$' "$ENV_FILE"
    grep -q '^FRANK_ARCHON_KV_DTYPE=fp8$' "$ENV_FILE"
    grep -q '^FRANK_ARCHON_NGRAM_PLACEMENT=ram$' "$ENV_FILE"
    # Prefix cache: pinned host RAM only (the disk tier was removed), and the
@@ -177,8 +177,11 @@ nosudo_bin() {
    [ "$ttmt" -gt "$ttmm" ]
    pool=$(awk -F= '/^FRANK_ARCHON_HOST_POOL_MIB=/{print $2}' "$ENV_FILE")
    ph=$(awk -F= '/^FRANK_ARCHON_PREFIX_CACHE_MIB=/{print $2}' "$ENV_FILE")
-    # 2 pools + PLE 47.7 GiB (48,824 MiB) + prefix host, rounded up.
+    # 2 pools + PLE 47.7 GiB (48,824 MiB) + prefix host + MNBT + the auto
-    ledger=$(( 2 * pool + 48824 + ph + 2048 ))
+    # checkpoint store (MAX_SEQS x 10 slots of 112 MiB -- the store is pinned host
    # memory too, and it grows with the slot count).
    seqs=$(awk -F= '/^FRANK_ARCHON_MAX_SEQS=/{print $2}' "$ENV_FILE")
    ledger=$(( 2 * pool + 48824 + ph + 2048 + seqs * 10 * 112 ))
    [ "$ttmm" -ge "$ledger" ]
    # The .rad size floor is positive (a truncated convert must fail fast).
    grep -q '^FRANK_ARCHON_RAD_MIN_MIB=' "$ENV_FILE"
Author	SHA1	Message	Date
Frank	1aeb7af6df	docs(probe): price the corpus chars/token ratio at run time The stored 3.28 chars/token (measured 2026-10-11) had drifted to 3.20 as the repo's docs grew, so leg B sent ~201,900-token prompts and took a legitimate 400 from the engine's own window check — which reads as a capacity-contract failure on a healthy endpoint. One ~4k-token request now prices the ratio, and the leg sizes to 97 % of the deployed window.	2026-10-20 23:26:53 -07:00
Frank	cd0be97b39	docs(vault): the four-slot ledger, and the resolver's funding law from the source The shape block and boot ledger now quote the live 1.8.1 boot: experts 20.72 GiB (78 %) / kv cache 5.64 GiB (21 %) = 747,492 tokens, 3.7 full-length sessions, `--checkpoint-slots 40`, `queue_cap 32`, sampler 12 rows, `experts-from-host` 8.99 GiB a rank. The KV Capacity law is rewritten from the pinned source instead of inference: `core/mem/vram_budget.cpp:146-156` funds the cache as `elastic - expert_floor`, `core/mem/kv.cpp:266` sets its ceiling from `max_seqs x max_ctx` (demand, never funding), and `core/place/planner.cpp:100-104` makes the expert floor fall one-for-one with `--host-pool-mib` — which is why +4,224 MiB of pinned host pool bought the pool and cost 3.01 GiB of residency a card. The old "funds ~90 %, never 100 %" phrasing was inference; the byte-identical 45,568 row is the ceiling case, not a mystery. Slot-cost law carries the four-slot price (state + checkpoint store + rows, still under the 25-row exact-wire bound). The two-slot era is one row in Eras with its rollback commit. Date stamps in the trio comments corrected to 2026-10-20.	2026-10-20 23:26:53 -07:00
Frank	1bb6399524	feat(frank-archon): four decode slots over a >=700k-token KV pool Frank's order: 4 slots, a 700k pool, round UP if the resolver cannot land it exactly. The cache target is MAX_SEQS x MAX_MODEL_LEN, so the demand is 800,000 fp8 tokens; the resolver funds ~91-93 % of demand (three boots of evidence on the endpoint page), which clears 700k, and the VRAM has to come from the expert plane -- which has to go somewhere, so HOST_POOL_MIB moves with it. expert_vram_floor (core/place/planner.cpp:100-104) is exactly plane + 2 units - host_room, and the live boot ledger pins that plane at 67.60 GiB a rank (23.73 GiB floor at 44,928 of room), so 49,152 drops the floor to 19.61 GiB and leaves the cache ~6.9 GiB: enough for the whole target, with the leftover returned to the experts. Costs, both real: pinned host RAM 87.75 -> 96.0 GiB (the pool is pinned once per rank) and ~3.2 GiB a card of expert residency, read over the x8 Gen5 link when the router picks a host-resident expert. The boot ledger's vram_kv line is the referee; if it comes back under 700,000 tokens HOST_POOL_MIB goes up 1024 and this redeploys. TTM_MIN_MIB 150,000 -> 162,000 for the new pin ledger (2 x pool + PLE 48,824 + prefix 8,192 + MNBT 2,048 + the auto checkpoint store, now 40 slots of 112 MiB = 161,848 MiB), and the bats ledger guard now counts the checkpoint store because it grows with the slot count. The two env-value assertions in that seam are bumped with the values they guard.	2026-10-20 23:10:05 -07:00
Frank	aa75b45543	feat(frank-archon): arm --prefill-interleave at 1 A prefill chunk that shares a step with decoding sequences holds the device for the chunk's whole time: measured on this shape, decode-only steps 15.5 ms against chunk steps 166 ms, and 4.7 % of the steps carry 34 % of the engine's step time. Harness-visible decode falls 166 -> 13 tok/s (per stream 6.7 = one token per shared step) once concurrent prefill passes 5k tok/s, and TTFT p90 for an arriving request goes 2.4 -> 18.0 s. BUSY_PREFILL_CHUNK stays `auto`: it is a prefill-speed dial, and the source refuses to halve the chunk for decode rows alone -- which is why engine_steps_held_total read zero for the whole 34.7 h boot. PREFILL_INTERLEAVE is the knob that acts on the measured mechanism: N decode-only steps follow any chunk step taken beside decoders. N=1 is the conservative first arm -- a stream takes its token from the chunk step and one from the held step (~2 tokens per 182 ms instead of one per 166 ms); the cost lands on the prefilling prompt's TTFT and on requests still queued, not on anything decoding. Rollback: PREFILL_INTERLEAVE=0. Verification: the flag must reach argv (the serve script only passes it when > 0), engine_steps_held_total must leave zero, then the §9 bucket pair must move before N rises.	2026-10-20 22:52:31 -07:00
Frank	b1fc1113c6	docs(vault): replace an invented --checkpoint-policy name with the three that exist The replay law named `--checkpoint-policy floor`; config.cpp admits exactly geometric-backoff (default), tip-only, keep-all and refuses anything else at boot, so the recorded arm could never be deployed. Replaced with the real semantics, priced from core/mem/prefix.{h,cpp}: the default keeps a geometric ladder (9 snapshots for a 200k session at interval 2048 -- exactly the live ledger line -- so a turn diverging d behind the tip can resume at tip-3d: diverge at 120k, replay 21.7k); retention is not the write path, so keep-all buys a bounded replay for pinned host RAM alone (196 slots x 111.98 MiB = 21.4 GiB against the store's eighth-of-host budget of 31.5 GiB, hard bound = room); and shortening --checkpoint-interval cannot fix the deep case -- the coarsest ladder scale is interval x 2^k under the depth, so the deepest gap stays about half the prompt at every interval. §9 keeps only the three-name locator.	2026-10-20 22:34:13 -07:00
Frank	9f7457e39e	docs(vault): answer "what are we doing wrong" on decode-vs-prefill frank-archon Laws: add the measured interference census (decode-only step 15.5 ms vs 166 ms chunk steps; 4.7% of steps hold 34% of step time; median decode 166 -> 13 t/s above 5k prefill tok/s, per-stream 6.7 t/s = one token per shared step; TTFT p90 2.4 -> 18.0 s for arriving requests), the lag test that kills the residency theory, the busy_cap arithmetic, and the correction that --prefill-interleave is the decoder dial and auto is not. Facts that change how an operator reads this box: /diagnostics exists and reports link-below-max at 100% busy; latency metrics live under vllm:, cadence under radiance:; preemption is vllm:num_preemptions_total (264 this boot, all bursts) while the radiance gauge is instant depth; both root ports are x8 Gen5 max by board (31.5 GB/s ceiling, not 64); DDR is back at 4800 MT/s while the 4600 corruption fix is recorded -- verify before blaming garbage tokens. §9 (LLM Performance Diagnosis): method corrected to the phase-ratio counters, dead 1.0.13 numbers destroyed, four PromQL traps added (metric prefix, 1 s EMA gauges vs 15 s scrape, counter resets across builds, 2-minute mean diluting the chunk), instruments that cannot be reached from Prometheus, and the knob list priced from the pinned 1.8.1 source instead of the stale guide.	2026-10-20 22:08:00 -07:00
Frank	687dbfbabe	docs(scripts): correct frank-archon's step-sharing and slot-count comments The env comment claimed BUSY_PREFILL_CHUNK=auto protects decode rows; the 1.8.1 source (core/sched/scheduler.cpp busy_cap) reserves 1+spec rows per decoding sequence and 1/8 of the budget per waiting request, and refuses to shrink the chunk for decoders alone -- auto is a prefill-speed dial. The interleave comment carried a 512 that is not this build's auto value (0 until the held-step counter asks; 64 in help mode). The serve header still described three decode slots; MAX_SEQS has been 2 since the 2026-10-19 reslice. No behavior change.	2026-10-20 22:07:55 -07:00
dank	961713e92d	Merge pull request 'test(e2e): cover notebook + prometheus guard-surface gaps (full-suite audit)' (#56 ) from feat/e2e-audit-coverage into main	2026-10-20 21:21:44 -07:00
dank	01db29e754	Merge pull request 'feat(grafana): starter dashboards (host/system, Loki logs, observability self-monitoring)' (#55 ) from feat/grafana-starter-dashboards into main	2026-10-20 21:20:53 -07:00
Frank	ab06e4c221	test(e2e): cover the two guard-surface gaps found in the full-suite audit Audit of all 12 e2e specs for the 'green-but-vacuous' class that broke Grafana's SSO. The suite is otherwise sound (incident-driven, leakage-based); two real coverage gaps surfaced: - notebook.franksdank.cloud (open-notebook, deployed 2026-10-19) was a guarded vhost in NEITHER the 13-/14- GUARDED lists NOR any dedicated spec. A broken upstream would stay invisible - the exact 17-hermes class. Added to the 14- guard audit (unauth -> authentik gate, 'Open Notebook' leak marker) plus a new 21- that proves authenticated reachability to open-notebook's own login through the guard chain, with a positive app marker. - prometheus.franksdank.cloud was only ever asserted '<500 and not a login wall' (13-/11-), which a 404 also passes. New 22- adds the app-is-really- there proof 20- gives Grafana: SSO handoff + its own /api/v1/status/buildinfo answers {status:success}. Both specs wired into the standard project (playwright testMatch), which the e2e_scaffold bats gate enforces. Verified live against the fleet; full bats suite green.	2026-10-20 21:16:23 -07:00
Frank	1006e96312	feat(grafana): starter dashboards for host/system, Loki logs, and observability self-monitoring llm-fleet.json already owns LLM capacity+spec-decode. These three cover the domains that had NO dashboard: - host-system: node_exporter CPU/Load/Mem/Disk/Net/Uptime/FS/FDs for both Dankclaw and Dankserver (templated by job node-dankclaw\|node-dankserver) - logs-overview: Loki volume + per-container error rate + a sample log panel; container is parsed from the JSON line at query time (journal streams have no container_name stream label) - observability: the stack's own health - Prometheus target up/down + scrape interval drift + samples/s + TSDB, Loki ingest/discard, Grafana CPU/mem. Panel queries verified live against Prometheus/Loki; provisioned dashboards confirmed via /api/search. Alertmanager is running but NOT scraped by Prometheus - noted in the dashboard, no empty panels built on it.	2026-10-20 21:10:06 -07:00
dank	7d85fed1c9	Merge pull request 'Fix Grafana Authentik SSO (500 + missing email claim) + RP landing test' (#54 ) from fix/grafana-authentik-sso into main	2026-10-20 21:04:02 -07:00
Frank	d8196ce499	docs: Grafana reads claims from userinfo, not USE_ID_TOKEN The login lands once authentik emits the email claim (the IdP fix); Grafana needs no USE_ID_TOKEN toggle, and dropping it also avoids a *_TOKEN false-positive in the literal-secret seam. Correct the two SSO notes accordingly.	2026-10-20 20:41:57 -07:00
Frank	6f442a14a3	fix(observability): repair Grafana Authentik OAuth2 relying-party login The Authentik SSO button returned "Server Error" (500): the grafana provider had a NULL authorization_flow and the authorize view crashed building a None flow planner. After that, the login failed with "InternalError" — no email claim reached Grafana, so it fell back to GET <api_url>/emails, which authentik 404s. Unit env: request authentik real scopes (openid email profile, not the GitLab user:email that crashes the authorize view); standard userinfo API_URL with no ?api=1 (Grafana v13 turns ?api=1 into a /emails 403); email/sub/preferred_username attribute paths; ALLOW_SIGN_UP=true since the Anubis->Authentik guard fronts the callback. scripts/authentik-grafana-sso-fix.sh idempotently re-applies the two IdP-side causes (NULL authorization_flow; grafana-email-scope returning a bare string instead of {"email":..}). Verified end to end: RP login lands on / and /api/user answers 200 with the real email, user provisioned as Viewer.	2026-10-20 20:41:36 -07:00
Frank	afe72781bc	docs: correct Grafana Authentik SSO facts The vault claimed the fork property-mapping evaluator was broken and userinfo returned only sub, so email could never reach Grafana (outcome unverified). That was a misdiagnosis: the real causes were the NULL authorization_flow (the NoneType.slug crash) and the email scope mapping returning a bare string rather than a dict. Record the four verified causes and fixes, point to the repair script and the new landing spec, and drop the unverified caveat.	2026-10-20 20:40:15 -07:00
Frank	200039d589	test(e2e): cover Grafana relying-party SSO landing (spec 20) 13-sso proves an authenticated session reaches Grafana login page but never follows the OAuth button, so the relying-party login could be dead (500, or an email-claim gap bouncing to /login) while the suite stayed green. Spec 20 drives /login/generic_oauth through the real Anubis+authentik round-trip and asserts a live session (/api/user 200 with a non-empty email). Wire 20- into the standard project (qa-e2e session).	2026-10-20 20:40:15 -07:00
Frank	91d9864bb0	fix(observability): repair Grafana Authentik OAuth2 relying-party login The Authentik SSO button returned "Server Error" (500) because the grafana provider authorization_flow was NULL and the authorize view crashed building a None flow planner. After that, the login failed with "InternalError": no email claim reached Grafana, so it fell back to GET <api_url>/emails, which authentik 404s. Unit env: request authentik real scopes (openid email profile, not the GitLab user:email that crashes the authorize view); read claims from the id_token (USE_ID_TOKEN=true) and drop ?api=1 from the API URL (Grafana v13 turns it into a /emails 404); pin email/sub/preferred_username attribute paths; allow first-login provisioning since the Anubis->Authentik guard fronts the callback. scripts/authentik-grafana-sso-fix.sh idempotently re-applies the two IdP-side causes (NULL authorization_flow; the grafana-email-scope mapping that returned a bare string instead of {"email":..}). Verified end to end: RP login lands on / and /api/user answers 200 with the real email, user provisioned as Viewer.	2026-10-20 20:40:15 -07:00
dank	980491668c	Merge pull request 'Alloy ships logs: root host-network service so it can read the journal' (#53 ) from feat/alloy-root-journal into main	2026-10-20 20:05:00 -07:00
dank	2fe68e69af	Merge pull request 'Observability: vllm rule group for frank-archon + Loki datasource; correct stale log-pipeline facts' (#52 ) from fix/observability-vllm-loki into main	2026-10-20 19:39:38 -07:00