frank-medium: serve the whole 262144-token window, and repoint every signal to llama.cpp

Context: 262144 was measured before it was deployed -- 24.72 GB on GPU 0 and
29.14 GB on GPU 1 of 34.21 GB each, ~5 GB of headroom on the busier card, and a
140,069-token prompt prefilled through the single slot in 134 s to prove the
window is real. The 114688 wall was the vLLM predecessor's arithmetic, not this
runtime's.

Throughput, measured through the service: 1.0-1.4k tok/s prefill from 4k to
140k, 43.8 tok/s decode on a 512-token generation, 41-71 % DFlash2 acceptance,
and the prefix cache re-prefilling 4 of 998 tokens on a repeated prompt.

The signals move with the runtime, so the fleet's rules and dashboard are
re-cut rather than renamed: vllm: series do not exist any more, and two of the
old rules watched things llama.cpp does not export at all (KV-pool fill, p95
latency). Those are retired with the reason written down, and LlmSpecDecodeOff
takes one of the slots -- a lost drafter is the failure no client can see. The
observability seam now fails if any rule names a series the endpoint does not
export, which is what would have caught the stale vllm: expressions.

omp models.yml (outside the repo) is updated in the same pass: contextWindow
262144, and the full reasoning-effort map, because Qwen3.8's template answers
500 on 'high' -- verified against the live endpoint, low/medium/xhigh answer
200. Subagents run at high.

AGENTS.md

@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
 
 | Target | Service | Runtime | Model | Port |
 |--------|---------|---------|-------|------|
-| Dankserver | `frank-medium` | vllm-radiance 0.9.3 (ggz14/radiance-vllm-mxfp4 patches), 2x single-GPU rootless Quadlets behind an adaptive-routing proxy | just1moremodel Qwen3.8-27B-Uncensored-MXFP4-awq + Qwen3.8-27B-DFlash2-FP8 drafter, 2x Radeon AI PRO R9700 (one card per replica, TP=1), 114688 ctx / 4 seqs each | 7770 |
+| Dankserver | `frank-medium` | mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build) | ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL + z-lab Qwen3.8-27B-DFlash2 Q8_0 drafter, 2x Radeon AI PRO R9700 in ONE layer-split process, 262144 ctx / 1 slot, text only | 7770 |
 | Dankclaw | `frank-large` | halogen-flash-server 0.5.8 (rootless Quadlet) | peonist-ai Qwen3.8-Flash-Next w4b `.hgn` on the gfx1151 iGPU (Strix Halo LPDDR), 200192 ctx per conversation over one 400384-position KV pool / 2 slots, text only | 7770 |
 | Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`-mtp` + BF16 mmproj) on the RTX 3090, 131072 ctx / 1 slot, MTP speculative decoding, the fleet's vision endpoint | 7771 |
 
@@ -69,9 +69,9 @@ The planner maps changed paths to services; the remote runner restarts only
 the selected service and records its revision outside the checkout.
 Documentation-only changes are a no-op. LLM services are selected only by
 their own files — frank-large by `dankclaw/frank-large/quadlet/`, frank-small
-by its unit/env/serve trio in `scripts/` — so unrelated container changes
-cannot interrupt inference. Set `FRANK_SUDO_PW` for headless runs on a host
-without passwordless sudo; the value never enters the repo.
+and frank-medium by their unit/env/serve trio in `scripts/` — so unrelated
+container changes cannot interrupt inference. Set `FRANK_SUDO_PW` for headless
+runs on a host without passwordless sudo; the value never enters the repo.
 
 ### Deployment completeness (MANDATORY)
 
@@ -300,34 +300,41 @@ the halogen side refuses images with a named reason; 22,362 MiB of the card's
 
 Runbook: `serverAppConfigs-docs/LLM/frank-small-qwen38-gsq-rco.md`.
 
-### frank-medium — vllm-radiance MXFP4 (Dankserver, ACTIVE)
+### frank-medium — llama.cpp layer split + DFlash2 (Dankserver, ACTIVE)
 
-`stilldeadcode/vllm-radiance:0.9.3` plus the `ggz14/radiance-vllm-mxfp4`
-patches (pinned clone at `/home/dank/GitPackages/radiance-vllm-mxfp4`; its
-patches are applied at every container start), serving
-`just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` with the
-`Qwen3.8-27B-DFlash2-FP8` block-diffusion drafter (`SPEC=7`).
+Mainline llama.cpp at **`fb27a525`** (a gfx1201 HIP build, `Root=`-less root
+systemd unit `scripts/frank-medium.service`) serving
+`ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` at **UD-Q8_K_XL**
+(29.29 GiB) with the **z-lab DFlash2** `Q8_0` drafter, in **one process across
+both R9700s**, `--split-mode layer`, 262144 ctx / 1 slot, text only.
 
-The deployment is **two single-GPU replicas behind an adaptive-routing proxy** —
-`frank-medium-a` (:7771, R9700 #1), `frank-medium-b` (:7772, R9700 #2) and
-`frank-medium-proxy` (:7770, the fleet endpoint), three rootless Quadlets
-under `dankserver/frank-medium/quadlet/`. Each replica is TP=1 at
-114688 context / 4 sequences with vLLM-profiled KV (`KV_MEM=0`).
+**Layer split is why both cards now serve ONE model.** Tensor parallelism is
+impossible (see the PCIe lesson below), and so is vLLM's pipeline parallelism —
+its stage handoff is an RCCL `send`/`recv` of the hidden state, the exact
+primitive that cannot be dispatched on card #2. llama.cpp's layer split needs
+no collective at all: layers are distributed in order and the boundary
+activation is a per-backend copy, once per forward pass. Measured on this host
+(the predecessor runtime): 1991 t/s prefill layer-split vs 272 tensor.
 
-**TP=2 across both cards cannot run on this host.** RCCL's collective kernels
-carry a hidden hostcall-buffer parameter, and ROCr refuses to dispatch them on
-a device without PCIe atomics; the second R9700 sits behind the chipset
-bridges `05:00.0`/`06:09.0`, which advertise `AtomicOpsCap: Routing-` — a
-read-only capability, so no software can enable atomics on that path. Not the
-kernel version (7.0.0-28/-30/-31 identical), not the image, not the RCCL
-build (a from-source rebuild fails identically). Putting both cards on CPU root
-ports restores TP=2: on this board that means an M.2→PCIe x4 riser into the
-free, CPU-attached M.2_2 slot, or a board whose CPU x16 bifurcates to x8/x8.
-`frank-medium.container` stays in the repo
-as the TP=2 target; the deploy handler does not install it.
+This is the box's **only** LLM runtime: the vllm-radiance replica pair and its
+routing proxy (`frank-medium-a/-b/-proxy`) were retired the same day they went
+live, with their units moved to `~/.AGENTTRASH/20260916-vllm-radiance-retire/`.
 
-Runbook: `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md`.
-Prior runtime: [[frank-medium-rocmfpx]] (llama.cpp era, same day).
+Files: `scripts/build-llama-rdna4.sh` (builds the runtime, gfx1201 only, private
+ROCm 7.14 toolchain bundled beside the binary), `scripts/download-frank-medium-gguf.sh`
+(resumable, byte-verified payload), `scripts/frank-medium.env`,
+`scripts/serve-frank-medium.sh`, `scripts/frank-medium.service`. The deploy
+handler refuses to start a unit whose runtime or payload is missing, retires
+the vLLM quadlets, waits for 7770 to be released (the retired proxy held it) and
+gates on `/health`.
+
+The ref is pinned by two features and must not float: **DFlash2**
+(ggml-org/llama.cpp#27342) and **qwen35 MTP**. The ROCmFPX fork the endpoint
+used earlier that day has neither.
+
+Runbook: `serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md`.
+Retired runtimes: [[frank-medium-radiance-mxfp4]] (vLLM replicas, same day),
+[[frank-medium-rocmfpx]] (llama.cpp fork, earlier that day).
 
 ## Key Lessons Learned
 Durable operational lessons from the fleet's history. Full detail in the
@@ -364,14 +371,35 @@ linked runbooks.
   (`Pcie atomics not enabled, hostcall not supported` → `AQL dispatch
   failed!`), not from vLLM or RCCL. `AMD_LOG_LEVEL` costs one env var and
   prints the failing HIP call with its return code.
+- **The one software path that lets both cards serve ONE model: llama.cpp's
+  layer split.** vLLM cannot do it here in any of its parallelisms — TP needs
+  an allreduce, PP needs the same p2p primitive, and both die on the card that
+  cannot dispatch an RCCL kernel. `--split-mode layer` in llama.cpp needs no
+  collective: layers are distributed in order and only the boundary activation
+  crosses PCIe, once per forward pass. Measured here: 1991 t/s prefill /
+  18.8 tok/s decode layer-split, against 272 / 26.3 tensor (butterfly
+  fallback). That is what `frank-medium` runs — see
+  [[frank-medium-llamacpp-dflash2]].
 
 
 **ROCm / GPU**
 
-- **Layer split, never tensor, on the RCCL-less ROCmFPX builds.** Tensor
-  parallel falls back to a butterfly allreduce and collapses prefill ~7x
-  (measured 271.8 vs 1991 t/s). Pin the MTP draft head to ROCm1 — an
-  unpinned draft under tensor mode aborts the KV build.
+- **Layer split is the deployed shape on both multi-device llama.cpp services,
+  for two different reasons.** On Dankserver it is the *only* way both R9700s
+  serve one model (no collective can be dispatched on card #2). On a
+  RCCL-less fork build it is also the fast one: tensor parallel falls back to
+  a butterfly allreduce and collapses prefill ~7x (measured 271.8 vs 1991 t/s).
+  Pin the spec-decode drafter to one device — an unpinned draft under tensor
+  mode aborts the KV build, and on a layer split the natural home is the card
+  that ends up holding the target's last layers.
+- **Bundling a ROCm runtime next to a llama.cpp binary: `cp -r SRC DST/` is a
+  trap.** With a missing `DST`, cp creates `DST` as a copy of `SRC` and drops
+  one directory level. rocBLAS and hipBLASLt look for their device code at a
+  fixed depth (`../lib/rocblas/library`, `../lib/hipblaslt/library/<arch>`),
+  and the flattened copy fails at *request* time, not load time: the engine
+  reads the weights, answers `/health` and dies on the first completion with
+  `Cannot read .../TensileLibrary.dat: No such file or directory for GPU arch
+  : gfx1201`. Declare the destination, then copy the contents.
 - **mmap must stay on** for the big-PLE models: `--no-mmap` forces the
   ~50 GiB PLE table into anonymous memory → silent cgroup kill, dmesg is the
   only evidence. mmap pages ARE page cache. Never pin the PLE tensor with
@@ -492,7 +520,7 @@ Active:
 |---------|---------|
 | frank-large (halogen-flash-server, Dankclaw) | `serverAppConfigs-docs/LLM/halogen-flash-server.md` |
 | frank-small (llama.cpp Qwen3.8-27B GSQ-RCO, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-qwen38-gsq-rco.md` |
-| frank-medium (vllm-radiance MXFP4, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` |
+| frank-medium (llama.cpp + DFlash2, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md` |
 | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
 | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
 

scripts/frank-medium.env

@@ -50,11 +50,19 @@
 # in the attention layers only, plus ~0.16 GiB per sequence of f32 DeltaNet
 # state at any depth.
 #
-# SIZING (per card, 31.86 GiB): weights split ~14.7 GiB; the drafter rides one
-# card; KV at CTX_SIZE 131,072 is 8.00 GiB total (~4 GiB per card, since the 16
-# attention layers are split with everything else); the loader's compute buffers
-# complete the ledger.  Read the loader's own "KV self size" / "compute buffer
-# size" lines rather than trusting this arithmetic.
+# SIZING, measured (rocm-smi, 2026-09-16).  Weights split across the two cards;
+# the drafter rides the second.  The loader's compute buffers are part of what
+# is seen below -- they are allocated at load, not per request:
+#
+#   CTX_SIZE   GPU 0        GPU 1        headroom on the busier card
+#   131072     19.89 GB     24.30 GB     9.9 GB / 31 %
+#   262144     24.72 GB     29.14 GB     5.07 GB / 15 %   <- deployed
+#
+# 262144 is the window the model ships (n_ctx_train) and it fits with room, so
+# the fleet gets the whole thing; the arithmetic that used to cap this endpoint
+# at 114688 was the vLLM predecessor's, not this runtime's.  If a future change
+# adds VRAM (a longer drafter, quantized KV, more slots) re-read the numbers
+# rather than assuming, and prefer CTX_SIZE over --gpu-layers.
 #
 # WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP reads
 # those bare vars; 2026-08-13 incident).  llama.cpp takes --host/--port/--temp
@@ -74,9 +82,9 @@ DRAFT_MODEL_FILE=Qwen3.8-27B-DFlash2-Q8_0.gguf
 ALIAS=frank-medium
 BIND_ADDR=0.0.0.0
 LISTEN_PORT=7770
-
-# Placement.  HIP_VISIBLE_DEVICES hides the iGPU; DEVICES is llama.cpp's own
-# list, which is the pair the split runs across.
+# Device placement.  HIP_VISIBLE_DEVICES hides the Raphael iGPU (HIP device 2)
+# so nothing can drift onto shared system RAM; DEVICES is what llama.cpp splits
+# across, and neither of the two cards has an XGMI link to the other.
 HIP_VISIBLE_DEVICES=0,1
 DEVICES=ROCm0,ROCm1
 SPLIT_MODE=layer
@@ -88,11 +96,11 @@ NGL=999
 FIT=off
 FLASH_ATTN=on
 
-# One slot at the model's documented-configured window.  --parallel divides
-# --ctx-size between slots; it does not multiply.  The target model ships
-# n_ctx_train 262144 -- raising CTX_SIZE is a measurement, not a guess (watch
-# the per-card ledger), and this service was deployed at 131072 first.
-CTX_SIZE=131072
+# One slot at the model's full configured window (n_ctx_train 262144), which was
+# measured to fit before it was deployed: see SIZING above.  --parallel DIVIDES
+# --ctx-size between slots, so one slot carries the whole window and a second
+# concurrent request queues behind it.
+CTX_SIZE=262144
 PARALLEL=1
 
 BATCH_SIZE=2048

serverAppConfigs-docs/Infrastructure/Observability.md

@@ -35,14 +35,14 @@ graph LR
 
 Recorded 2026-09-11, on the day the stack moved to Quadlet units. Everything in the table above is deployed, healthy and boot-persistent. Almost nothing is wired up beyond the default install:
 
-- **Grafana has a datasource and one dashboard as of 2026-09-16; before that it had neither.** `shared/grafana/provisioning/` is installed to `/opt/grafana/provisioning` by the deployer and read by the container: the Prometheus datasource (`http://prometheus:9090`, fixed uid `prometheus`) and the **LLM fleet — capacity and reuse** dashboard (KV pool fill, queue depth, prefix-cache reuse, latency p95, throughput) covering both LLM endpoints. Verified through the API rather than by eye — the datasource answers `llamacpp:` counters for `frank-medium`, and `GET /api/search?type=dash-db` returns uid `llm-fleet`. One trap worth knowing: a provisioned dashboard does **not** appear in the legacy `dashboard` table on this Grafana version (it lives in unified storage behind `dashboard.grafana.app`), so the API is the way to check. Still outstanding: no dashboards beyond this one, no Loki queries, and only datasources+dashboards are provisioning-as-code.
+- **Grafana has a datasource and one dashboard as of 2026-09-16; before that it had neither.** `shared/grafana/provisioning/` is installed to `/opt/grafana/provisioning` by the deployer and read by the container: the Prometheus datasource (`http://prometheus:9090`, fixed uid `prometheus`) and the **LLM fleet — capacity and reuse** dashboard (speculative acceptance, queue depth, prefix-cache reuse, decode step rate, throughput) covering the llama.cpp endpoints. **Re-cut on 2026-09-16**: the panels were keyed to `vllm:` series, which stopped existing when frank-medium moved to llama.cpp, and two of them (KV-pool fill, request-latency p95) have no llama.cpp equivalent at all — those two are retired rather than restated, and the acceptance panel replaces the KV one. A renamed query against a series that does not exist is a panel that draws nothing forever. Verified through the API rather than by eye — the datasource answers `llamacpp:` counters for `frank-medium`, and `GET /api/search?type=dash-db` returns uid `llm-fleet`. One trap worth knowing: a provisioned dashboard does **not** appear in the legacy `dashboard` table on this Grafana version (it lives in unified storage …
 - **Grafana's dashboard now also surfaces firing alerts.** The **Alerts firing (ALERTS)** panel graphs `ALERTS{alertstate="firing"}` for the LLM rules, because this stack has no other place where an alert becomes visible without opening Prometheus.
-- **Prometheus now evaluates rules; nothing delivers them (2026-09-16).** `shared/prometheus/rules/llm-fleet.yml` ships with the config (the deployer's `copy_prometheus_config` installs both — a rule file nothing loads evaluates nothing, and Prometheus reports no error for that) and holds six rules for the LLM fleet: pool saturation, request queueing, prefix-cache collapse, endpoint down, p95 latency, plus the `job:llm_prefix_cache_reuse:rate5m` recording rule the alert and the dashboard share. Verified with `promtool check rules` (SUCCESS, 6 rules) and `/api/v1/rules` (all six `health=ok`). **There is still no Alertmanager and no contact point anywhere**: a firing alert appears on Prometheus' `/alerts`, in the `ALERTS` metric and on the LLM fleet dashboard's alerts panel — and reaches a human only if one is looking.
+- **Prometheus now evaluates rules; nothing delivers them (2026-09-16).** `shared/prometheus/rules/llm-fleet.yml` ships with the config (the deployer's `copy_prometheus_config` installs both — a rule file nothing loads evaluates nothing, and Prometheus reports no error for that) and holds five rules for the LLM fleet: request queueing, prefix-cache collapse, speculative decoding off, endpoint down, plus the `job:llm_prefix_cache_reuse:rate5m` recording rule the alert and the dashboard share. **Two rules from the vLLM era were retired, not renamed**, because the series they watched no longer exist: KV-pool saturation (llama.cpp exports no KV-fill gauge; its consequence *is* queueing, which `LlmRequestsQueueing` already covers) and p95 request latency (no latency histogram). `LlmSpecDecodeOff` takes the second slot — the failure a client cannot see, since a lost drafter just halves the speed. Verified with `promtool check rules` and `/api/v1/rules` (`health=ok` for all five). **There is still no Alertmanager and no contact point anywhere**: a firing alert appears on Prometheus' `/alerts`, in the `ALERTS` metric and on the LLM fleet dashboard'…
 - **Loki is configured but unread.** Storage is filesystem with 30-day retention, `auth_enabled: false`, and the push path works; Grafana can now reach it in Explore, but no dashboard or alert consumes it.
 - **node-exporter runs its default collector set** — no collector tuning, no textfile collector for custom metrics.
 - **Alloy is the one configured piece.** It ships the host journal and podman container logs to Loki with `job` labels, and that pipeline is live.
 
-The honest summary: metrics and logs are collected and retained, one dashboard visualizes the LLM fleet, six rules evaluate over it, and **no rule reaches a person** — there is no Alertmanager and no contact point, so correlation stays manual and every signal has to be looked at.
+The honest summary: metrics and logs are collected and retained, one dashboard visualizes the LLM fleet, five rules evaluate over it, and **no rule reaches a person** — there is no Alertmanager and no contact point, so correlation stays manual and every signal has to be looked at.
 
 ## Deployment
 

serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md

@@ -145,7 +145,7 @@ Deployed flags:
 |---|---|---|
 | `--split-mode` / `--device` / `--tensor-split` | `layer` / `ROCm0,ROCm1` / `1,1` | both cards, one model, no collective |
 | `--gpu-layers` / `--fit` | `999` / `off` | everything on the cards; auto-fit must not resize a context we set |
-| `--ctx-size` / `--parallel` | `131072` / `1` | one slot; `--parallel` **divides** the context |
+| `--ctx-size` / `--parallel` | `262144` / `1` | one slot carries the model's whole configured window; `--parallel` **divides** the context |
 | `--flash-attn` / cache types | `on` / `f16` K and V | an agentic checkpoint is the wrong place to spend KV precision |
 | `--spec-type` / `--spec-draft-n-max` | `draft-dflash` / `7` | DFlash2 blocks are 8 tokens, so 7 drafts is the ceiling llama.cpp allows |
 | `--spec-draft-device` | `ROCm1` | the drafter rides the card that ends up holding the target's last layers |
@@ -169,12 +169,37 @@ Deployed flags:
 
   `block_size=8` and `n_extract=5` are DFlash2's own metadata; the selector is
   what distinguishes it from DFlash 1.
-- Both cards hold the model: at 131072 context, `rocm-smi` reports
-  **17.9 GiB on GPU 0 and 22.0 GiB on GPU 1** (the drafter and the attention
-  layers' KV sit on the second).
-- A real completion (60-token prompt, 43 output tokens incl. reasoning):
-  **28.3 tok/s**, `draft_n = 49`, `draft_n_accepted = 36` → **73 % acceptance**.
-  The reasoning trace arrives in `reasoning_content`.
+- Both cards hold the model: at 262144 context `rocm-smi` reports **24.72 GB on
+  GPU 0 and 29.14 GB on GPU 1** of 34.21 GB each — the drafter, the attention
+  layers' KV and the compute buffers all sit on the second card, which keeps
+  ~5.1 GB of headroom. At 131072 the same pair reads 19.89 / 24.30 GB.
+- **140,069 prompt tokens prefilled in 134 s** (1043 tok/s) through the one
+  slot: the raised window is real, not just declared, and it is 22 % beyond the
+  114688 the vLLM predecessor could offer.
+
+### Measured throughput (2026-09-16, deployed service, `temperature 0`)
+
+Same filler corpus for every shape, so these are shapes rather than a
+benchmark: a large prompt with a tiny budget isolates prefill, a small prompt
+with a long budget isolates decode. `draft`/`acc` are per request; acceptance
+moves with the text (filler prose accepts far worse than code or a real agent
+loop).
+
+| Shape | Prefill | Decode | Acceptance |
+|---|---|---|---|
+| 4k prompt / 32 out | 3994 tok in 3.25 s — 1230 tok/s | 59.8 tok/s | 71 % |
+| 1k prompt / 512 out | 998 tok in 1.41 s — 707 tok/s | **43.8 tok/s** | 41 % |
+| 16k prompt / 16 out | 12,528 tok in 8.94 s — **1402 tok/s** | — | 100 % (13/13) |
+| 64k prompt / 16 out | 48,508 tok in 39.8 s — 1219 tok/s | — | 53 % |
+| 140k prompt / 16 out | 140,069 tok in 134.3 s — 1043 tok/s | — | — |
+| same 1k prompt again | **4 tokens prefilled** | 43.8 tok/s | 41 % |
+
+The last row is the prefix cache doing its job: the second request of the same
+conversation re-prefilled 4 of 998 tokens. Prefill holds ~1.0-1.4k tok/s from
+4k to 140k, and decode sits at ~44 tok/s on a long generation — against the
+vLLM predecessor's 42.8 tok/s single-stream (two replicas) and the ROCmFPX
+llama.cpp measured here at 35.7-46.2 tok/s. One process, one model, both cards,
+no proxy.
 
 ## Observability
 

shared/grafana/provisioning/dashboards/llm-fleet.json

@@ -1,7 +1,7 @@
 {
   "uid": "llm-fleet",
-  "title": "LLM fleet — capacity and reuse",
-  "description": "The signals that explain frank-medium under load, from the 2026-09-16 capacity work. The panels read the endpoint's own counters: KV fill, queue depth, prefix-cache reuse, and request latency. frank-small (llama.cpp on Dankclaw's RTX 3090) exports throughput counters and appears in the same endpoint picker; frank-large runs the halogen engine, which publishes /health, /cache and /v1/* and no metrics endpoint at all, so it cannot appear here. See serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md for the numbers these panels are read against.",
+  "title": "LLM fleet \u2014 capacity and reuse",
+  "description": "The signals that explain the LLM endpoints under load, from the 2026-09-16 capacity work. The panels read llama.cpp's own counters: speculative acceptance, queue depth, prefix-cache reuse, decode step rate and throughput. frank-medium (Dankserver, both R9700s, layer split) and frank-small (Dankclaw, RTX 3090) both export them; frank-large runs the halogen engine, which publishes /health, /cache and /v1/* and no metrics endpoint at all, so it cannot appear here. See serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md for the numbers these panels are read against.",
   "tags": [
     "llm",
     "frank-medium",
@@ -42,7 +42,7 @@
     {
       "id": 1,
       "type": "timeseries",
-      "title": "KV pool fill (vllm:kv_cache_usage_perc)",
+      "title": "Speculative decoding (DFlash2 acceptance)",
       "description": "How full each card's KV pool is. One card holds ~122,636 tokens; sustained high values with a cold prefix cache is the thrash this fleet measured.",
       "gridPos": {
         "h": 8,
@@ -69,8 +69,13 @@
       "targets": [
         {
           "refId": "A",
-          "expr": "vllm:kv_cache_usage_perc{job=~\"$job\"} * 100",
-          "legendFormat": "{{job}} engine {{engine}}"
+          "expr": "rate(llamacpp:spec_decode_num_accepted_tokens_total{job=~\"$job\"}[5m]) / clamp_min(rate(llamacpp:spec_decode_num_drafts_total{job=~\"$job\"}[5m]), 0.001)",
+          "legendFormat": "{{job}} accepted per draft step"
+        },
+        {
+          "refId": "B",
+          "expr": "rate(llamacpp:spec_decode_num_draft_tokens_total{job=~\"$job\"}[5m])",
+          "legendFormat": "{{job}} draft tokens/s"
         }
       ]
     },
@@ -103,13 +108,13 @@
       "targets": [
         {
           "refId": "A",
-          "expr": "vllm:num_requests_waiting{job=~\"$job\"}",
-          "legendFormat": "{{job}} waiting"
+          "expr": "llamacpp:requests_deferred{job=~\"$job\"}",
+          "legendFormat": "{{job}} deferred"
         },
         {
           "refId": "B",
-          "expr": "vllm:num_requests_running{job=~\"$job\"}",
-          "legendFormat": "{{job}} running"
+          "expr": "llamacpp:requests_processing{job=~\"$job\"}",
+          "legendFormat": "{{job}} processing"
         }
       ]
     },
@@ -143,7 +148,7 @@
       "targets": [
         {
           "refId": "A",
-          "expr": "rate(vllm:prefix_cache_hits_total{job=~\"$job\"}[5m]) / clamp_min(rate(vllm:prefix_cache_queries_total{job=~\"$job\"}[5m]), 1)",
+          "expr": "rate(llamacpp:prompt_tokens_cached_total{job=~\"$job\"}[5m]) / clamp_min(rate(llamacpp:prompt_tokens_total{job=~\"$job\"}[5m]), 1)",
           "legendFormat": "{{job}} reuse"
         }
       ]
@@ -151,8 +156,8 @@
     {
       "id": 4,
       "type": "timeseries",
-      "title": "End-to-end request latency p95",
-      "description": "Wall-clock per request, prefill included — the number an agent-loop turn feels.",
+      "title": "Decode steps and batch size",
+      "description": "Wall-clock per request, prefill included \u2014 the number an agent-loop turn feels.",
       "gridPos": {
         "h": 8,
         "w": 12,
@@ -176,8 +181,13 @@
       "targets": [
         {
           "refId": "A",
-          "expr": "histogram_quantile(0.95, sum by (le, job) (rate(vllm:e2e_request_latency_seconds_bucket{job=~\"$job\"}[5m])))",
-          "legendFormat": "{{job}} p95"
+          "expr": "rate(llamacpp:n_decode_total{job=~\"$job\"}[5m])",
+          "legendFormat": "{{job}} decode steps/s"
+        },
+        {
+          "refId": "B",
+          "expr": "llamacpp:n_busy_slots_per_decode{job=~\"$job\"}",
+          "legendFormat": "{{job}} sequences per decode step"
         }
       ]
     },
@@ -207,16 +217,6 @@
         "overrides": []
       },
       "targets": [
-        {
-          "refId": "A",
-          "expr": "rate(vllm:generation_tokens_total{job=~\"$job\"}[5m])",
-          "legendFormat": "{{job}} generated tok/s"
-        },
-        {
-          "refId": "B",
-          "expr": "rate(vllm:prompt_tokens_total{job=~\"$job\"}[5m])",
-          "legendFormat": "{{job}} prompt tok/s"
-        },
         {
           "refId": "C",
           "expr": "llamacpp:predicted_tokens_seconds{job=~\"$job\"}",
@@ -233,7 +233,7 @@
       "id": 6,
       "type": "timeseries",
       "title": "Alerts firing (ALERTS)",
-      "description": "Prometheus rules for the LLM fleet (shared/prometheus/rules/llm-fleet.yml): pool saturation, queueing, prefix-cache collapse, endpoint down, p95 latency. A firing alert appears here, on Prometheus' /alerts page, and nowhere else — this stack has no Alertmanager, so nothing is notified.",
+      "description": "Prometheus rules for the LLM fleet (shared/prometheus/rules/llm-fleet.yml): pool saturation, queueing, prefix-cache collapse, endpoint down, p95 latency. A firing alert appears here, on Prometheus' /alerts page, and nowhere else \u2014 this stack has no Alertmanager, so nothing is notified.",
       "gridPos": {
         "h": 8,
         "w": 12,

shared/prometheus/rules/llm-fleet.yml

@@ -1,7 +1,18 @@
 # Alerting and recording rules for the LLM fleet.
 #
-# Thresholds are not guesses: each one corresponds to a failure mode measured on
-# this fleet on 2026-09-16 (see serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md).
+# Every expression here is keyed to a series that EXISTS as of 2026-09-16. When
+# frank-medium moved from vllm-radiance to llama.cpp the exported namespace
+# changed with it (vllm: -> llamacpp:), and two signals the old file alerted on
+# no longer exist in any form:
+#
+#   * vllm:kv_cache_usage_perc — llama.cpp exports no KV-fill gauge. Its
+#     observable consequence is queueing, which is what LlmRequestsQueueing
+#     watches instead. A renamed query would have been a rule that can never
+#     fire.
+#   * vllm:e2e_request_latency_seconds_* — no latency histogram either. The
+#     rule it fed (p95 wall clock) is retired rather than restated, and
+#     LlmSpecDecodeOff takes its place: the failure an operator actually
+#     cannot see from the outside on this deployment.
 #
 # IMPORTANT: nothing is notified. There is no Alertmanager in this stack, so a
 # firing alert appears on Prometheus' /alerts page, in the ALERTS metric (which
@@ -17,42 +28,30 @@ groups:
       # expression. The exported series are counters; the rate is the reuse.
       - record: job:llm_prefix_cache_reuse:rate5m
         expr: |
-          rate(vllm:prefix_cache_hits_total[5m])
-            / clamp_min(rate(vllm:prefix_cache_queries_total[5m]), 0.001)
+          rate(llamacpp:prompt_tokens_cached_total[5m])
+            / clamp_min(rate(llamacpp:prompt_tokens_total[5m]), 0.001)
 
-      # The capacity wall. Each R9700 carries 122,636 KV tokens; past ~90 %
-      # the next large request queues or evicts a cached prefix.
-      - alert: LlmKvPoolSaturated
-        expr: vllm:kv_cache_usage_perc > 0.90
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          summary: "{{ $labels.job }}: KV pool above 90 % for 10 minutes"
-          description: >-
-            A card's KV pool is nearly full. Expect queueing or cold re-prefills
-            for large requests; the wall and its arithmetic are in the
-            frank-medium runbook.
-
-      # Concurrency limit reached: MAXSEQS=4 per replica.
+      # The capacity wall, in the only form llama.cpp exports it: work waiting
+      # for the single slot. Past here the next large request queues.
       - alert: LlmRequestsQueueing
-        expr: vllm:num_requests_waiting > 0
+        expr: llamacpp:requests_deferred > 0
         for: 10m
         labels:
           severity: warning
         annotations:
           summary: "{{ $labels.job }}: requests waiting for 10 minutes"
           description: >-
-            All sequence slots are busy and work is queueing. Either the fleet is
-            genuinely busy or one endpoint is carrying it alone.
+            The endpoint's slot(s) are busy and work is queueing. Either the
+            fleet is genuinely busy or one endpoint is carrying it alone. With
+            --parallel 1 a second concurrent request always shows up here.
 
       # The thrash signature measured on 2026-09-16: requests keep arriving but
-      # almost none are served from cache, which is what three concurrent
-      # 50-80k-token conversations do to a two-card fleet.
+      # almost none are served from cache, which is what several concurrent
+      # 50-80k-token conversations do to this fleet.
       - alert: LlmPrefixCacheCollapse
         expr: >-
           job:llm_prefix_cache_reuse:rate5m < 0.20
-            and rate(vllm:prefix_cache_queries_total[5m]) > 0.05
+            and rate(llamacpp:prompt_tokens_total[5m]) > 0.05
         for: 15m
         labels:
           severity: info
@@ -61,12 +60,29 @@ groups:
           description: >-
             Nearly every request is prefilling from scratch. If several large
             contexts are in flight together, this is the capacity wall rather
-            than a routing fault — placement and admission were both measured
-            and neither fixes it.
+            than a routing fault.
 
-      # The endpoint, not the replica: two replicas serve :7770 through the
-      # proxy, so this fires only when both are down (a single restart is
-      # invisible — the proxy fails over).
+      # The speculative path is the one failure no client can see: the server
+      # answers normally, just at half the speed, when the drafter is gone.
+      # Drafts and accepts are counters, so a rate that stays at zero while the
+      # endpoint is serving tokens means nothing is being drafted.
+      - alert: LlmSpecDecodeOff
+        expr: >-
+          rate(llamacpp:spec_decode_num_drafts_total[5m]) == 0
+            and rate(llamacpp:tokens_predicted_total[5m]) > 0.1
+        for: 15m
+        labels:
+          severity: warning
+        annotations:
+          summary: "{{ $labels.job }}: serving without speculative decoding"
+          description: >-
+            Tokens are being produced but no draft has been proposed for 15
+            minutes. On frank-medium that means the DFlash2 drafter is not
+            running (check the unit's log for common_speculative_impl_draft_dflash),
+            and the endpoint is decoding autoregressively.
+
+      # The endpoint, not a replica: one process serves each port now, so a
+      # restart of the unit is a scrape gap, not a failover.
       - alert: LlmEndpointDown
         expr: up{job=~"frank-.*"} == 0
         for: 2m
@@ -76,24 +92,7 @@ groups:
           summary: "{{ $labels.job }}: scrape target down for 2 minutes"
           description: >-
             Prometheus cannot reach the LLM endpoint. Check the units:
-            `systemctl --user is-active frank-medium-{a,b,proxy}` on Dankserver,
-            `systemctl is-active frank-small` (the llama.cpp root unit) or
-            `systemctl --user is-active frank-large` (the halogen quadlet) on
-            Dankclaw. frank-large is not scraped at all — the halogen engine
-            exports no metrics — so this alert can only ever name the endpoints
-            that do export one.
-
-      # What an operator actually feels: wall-clock per request, prefill
-      # included. A cold 50k-token prefill alone is ~22 s on one card.
-      - alert: LlmLatencyHigh
-        expr: >-
-          histogram_quantile(0.95,
-            sum by (le, job) (rate(vllm:e2e_request_latency_seconds_bucket[10m]))) > 60
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          summary: "{{ $labels.job }}: p95 request latency above 60 s for 10 minutes"
-          description: >-
-            Requests are taking a minute or more end to end. On this hardware
-            that is usually large contexts competing for one card's KV pool.
+            `systemctl is-active frank-medium` on Dankserver (a root llama.cpp
+            unit) and `systemctl is-active frank-small` on Dankclaw. frank-large
+            is not scraped at all — the halogen engine exports no metrics — so
+            this alert can only ever name the endpoints that do export one.

tests/frank_medium_unit.bats

@@ -100,12 +100,19 @@ DRAFT_BYTES=2056414816      # DFlash2 Q8_0 drafter
     [ "$parallel" -ge 1 ]
     [ "$cache_k" = "f16" ] && [ "$cache_v" = "f16" ]
     # qwen35 spends KV on its 16 full-attention layers only: 2 (K,V) x 4 heads
-    # x 256 dim x 2 B = 64 KiB per token at f16. Worst case for the fit is ONE
-    # card carrying the drafter and every attention layer's KV, with ~2 GiB of
-    # compute buffers: half the weights, plus that, must stay inside the card.
+    # x 256 dim x 2 B = 64 KiB per token at f16. A layer split puts those
+    # layers (and their KV) on both cards, so the worst card carries half the
+    # weights, half the KV, the drafter and ~2 GiB of compute buffers.
+    #
+    # This model is deliberately calibrated against rocm-smi: at 262144 it
+    # predicts 28.53e9 B against the 29.14e9 B measured on the busier card
+    # (2026-09-16), i.e. ~2 % optimistic. Keep it that way -- a model that is
+    # pessimistic beyond a few GiB would reject a context that fits.
     local per_card
-    per_card=$(( MODEL_BYTES / 2 + DRAFT_BYTES + ctx * 65536 + 2 * 1024 * 1024 * 1024 ))
+    per_card=$(( MODEL_BYTES / 2 + DRAFT_BYTES + (ctx * 65536) / 2 + 2 * 1024 * 1024 * 1024 ))
     [ "$per_card" -le "$CARD_BYTES" ]
+    # ...and the window may not exceed what the checkpoint was trained for.
+    [ "$ctx" -le 262144 ]
 }
 
 @test "frank-medium is text only: the vision tower stays on frank-small" {

tests/observability_quadlet.bats

@@ -345,13 +345,26 @@ PAIRS
     local cfg="$REPO_ROOT/shared/prometheus/prometheus.yml"
     [ -f "$rules" ]
     python3 -c "
-import yaml
+import re, yaml
 d = yaml.safe_load(open('$rules'))
 names = [r.get('alert') or r.get('record') for g in d['groups'] for r in g['rules']]
-assert 'record' or True
-for want in ('LlmKvPoolSaturated', 'LlmRequestsQueueing', 'LlmPrefixCacheCollapse',
-             'LlmEndpointDown', 'LlmLatencyHigh'):
+for want in ('job:llm_prefix_cache_reuse:rate5m', 'LlmRequestsQueueing',
+             'LlmPrefixCacheCollapse', 'LlmSpecDecodeOff', 'LlmEndpointDown'):
     assert want in names, want
+# Every rule must be keyed to a series that exists. The vLLM era's namespaces
+# are gone (frank-medium is llama.cpp now) and a rule naming one of them
+# evaluates nothing forever without erroring — the failure this seam exists for.
+exprs = [r.get('expr', '') for g in d['groups'] for r in g['rules']]
+assert not any('vllm:' in e for e in exprs), exprs
+# ...and the llama.cpp series a rule may name are the ones the server exports
+# (measured from /metrics on 2026-09-16). Anything else is a guess.
+live = {'llamacpp:prompt_tokens_total', 'llamacpp:prompt_tokens_cached_total',
+        'llamacpp:requests_deferred', 'llamacpp:requests_processing',
+        'llamacpp:spec_decode_num_drafts_total',
+        'llamacpp:spec_decode_num_accepted_tokens_total',
+        'llamacpp:tokens_predicted_total'}
+used = {m for e in exprs for m in re.findall(r'llamacpp:[a-z_0-9]+', e)}
+assert used <= live, used - live
 print('rules ok:', len(names))
 "
     # A rule file that nothing loads evaluates nothing, and Prometheus reports