From dc76dc5dc43629ceee57d0d0a463d0ea8d426ad1 Mon Sep 17 00:00:00 2001 From: Frank Date: Wed, 16 Sep 2026 15:29:40 -0700 Subject: [PATCH] frank-medium: serve the whole 262144-token window, and repoint every signal to llama.cpp Context: 262144 was measured before it was deployed -- 24.72 GB on GPU 0 and 29.14 GB on GPU 1 of 34.21 GB each, ~5 GB of headroom on the busier card, and a 140,069-token prompt prefilled through the single slot in 134 s to prove the window is real. The 114688 wall was the vLLM predecessor's arithmetic, not this runtime's. Throughput, measured through the service: 1.0-1.4k tok/s prefill from 4k to 140k, 43.8 tok/s decode on a 512-token generation, 41-71 % DFlash2 acceptance, and the prefix cache re-prefilling 4 of 998 tokens on a repeated prompt. The signals move with the runtime, so the fleet's rules and dashboard are re-cut rather than renamed: vllm: series do not exist any more, and two of the old rules watched things llama.cpp does not export at all (KV-pool fill, p95 latency). Those are retired with the reason written down, and LlmSpecDecodeOff takes one of the slots -- a lost drafter is the failure no client can see. The observability seam now fails if any rule names a series the endpoint does not export, which is what would have caught the stale vllm: expressions. omp models.yml (outside the repo) is updated in the same pass: contextWindow 262144, and the full reasoning-effort map, because Qwen3.8's template answers 500 on 'high' -- verified against the live endpoint, low/medium/xhigh answer 200. Subagents run at high. --- AGENTS.md | 94 ++++++++++------ scripts/frank-medium.env | 34 +++--- .../Infrastructure/Observability.md | 6 +- .../LLM/frank-medium-llamacpp-dflash2.md | 39 +++++-- .../provisioning/dashboards/llm-fleet.json | 50 ++++----- shared/prometheus/rules/llm-fleet.yml | 101 +++++++++--------- tests/frank_medium_unit.bats | 15 ++- tests/observability_quadlet.bats | 21 +++- 8 files changed, 220 insertions(+), 140 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index d9bfcf9..c0c08c1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| -| Dankserver | `frank-medium` | vllm-radiance 0.9.3 (ggz14/radiance-vllm-mxfp4 patches), 2x single-GPU rootless Quadlets behind an adaptive-routing proxy | just1moremodel Qwen3.8-27B-Uncensored-MXFP4-awq + Qwen3.8-27B-DFlash2-FP8 drafter, 2x Radeon AI PRO R9700 (one card per replica, TP=1), 114688 ctx / 4 seqs each | 7770 | +| Dankserver | `frank-medium` | mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build) | ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL + z-lab Qwen3.8-27B-DFlash2 Q8_0 drafter, 2x Radeon AI PRO R9700 in ONE layer-split process, 262144 ctx / 1 slot, text only | 7770 | | Dankclaw | `frank-large` | halogen-flash-server 0.5.8 (rootless Quadlet) | peonist-ai Qwen3.8-Flash-Next w4b `.hgn` on the gfx1151 iGPU (Strix Halo LPDDR), 200192 ctx per conversation over one 400384-position KV pool / 2 slots, text only | 7770 | | Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`-mtp` + BF16 mmproj) on the RTX 3090, 131072 ctx / 1 slot, MTP speculative decoding, the fleet's vision endpoint | 7771 | @@ -69,9 +69,9 @@ The planner maps changed paths to services; the remote runner restarts only the selected service and records its revision outside the checkout. Documentation-only changes are a no-op. LLM services are selected only by their own files — frank-large by `dankclaw/frank-large/quadlet/`, frank-small -by its unit/env/serve trio in `scripts/` — so unrelated container changes -cannot interrupt inference. Set `FRANK_SUDO_PW` for headless runs on a host -without passwordless sudo; the value never enters the repo. +and frank-medium by their unit/env/serve trio in `scripts/` — so unrelated +container changes cannot interrupt inference. Set `FRANK_SUDO_PW` for headless +runs on a host without passwordless sudo; the value never enters the repo. ### Deployment completeness (MANDATORY) @@ -300,34 +300,41 @@ the halogen side refuses images with a named reason; 22,362 MiB of the card's Runbook: `serverAppConfigs-docs/LLM/frank-small-qwen38-gsq-rco.md`. -### frank-medium — vllm-radiance MXFP4 (Dankserver, ACTIVE) +### frank-medium — llama.cpp layer split + DFlash2 (Dankserver, ACTIVE) -`stilldeadcode/vllm-radiance:0.9.3` plus the `ggz14/radiance-vllm-mxfp4` -patches (pinned clone at `/home/dank/GitPackages/radiance-vllm-mxfp4`; its -patches are applied at every container start), serving -`just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` with the -`Qwen3.8-27B-DFlash2-FP8` block-diffusion drafter (`SPEC=7`). +Mainline llama.cpp at **`fb27a525`** (a gfx1201 HIP build, `Root=`-less root +systemd unit `scripts/frank-medium.service`) serving +`ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` at **UD-Q8_K_XL** +(29.29 GiB) with the **z-lab DFlash2** `Q8_0` drafter, in **one process across +both R9700s**, `--split-mode layer`, 262144 ctx / 1 slot, text only. -The deployment is **two single-GPU replicas behind an adaptive-routing proxy** — -`frank-medium-a` (:7771, R9700 #1), `frank-medium-b` (:7772, R9700 #2) and -`frank-medium-proxy` (:7770, the fleet endpoint), three rootless Quadlets -under `dankserver/frank-medium/quadlet/`. Each replica is TP=1 at -114688 context / 4 sequences with vLLM-profiled KV (`KV_MEM=0`). +**Layer split is why both cards now serve ONE model.** Tensor parallelism is +impossible (see the PCIe lesson below), and so is vLLM's pipeline parallelism — +its stage handoff is an RCCL `send`/`recv` of the hidden state, the exact +primitive that cannot be dispatched on card #2. llama.cpp's layer split needs +no collective at all: layers are distributed in order and the boundary +activation is a per-backend copy, once per forward pass. Measured on this host +(the predecessor runtime): 1991 t/s prefill layer-split vs 272 tensor. -**TP=2 across both cards cannot run on this host.** RCCL's collective kernels -carry a hidden hostcall-buffer parameter, and ROCr refuses to dispatch them on -a device without PCIe atomics; the second R9700 sits behind the chipset -bridges `05:00.0`/`06:09.0`, which advertise `AtomicOpsCap: Routing-` — a -read-only capability, so no software can enable atomics on that path. Not the -kernel version (7.0.0-28/-30/-31 identical), not the image, not the RCCL -build (a from-source rebuild fails identically). Putting both cards on CPU root -ports restores TP=2: on this board that means an M.2→PCIe x4 riser into the -free, CPU-attached M.2_2 slot, or a board whose CPU x16 bifurcates to x8/x8. -`frank-medium.container` stays in the repo -as the TP=2 target; the deploy handler does not install it. +This is the box's **only** LLM runtime: the vllm-radiance replica pair and its +routing proxy (`frank-medium-a/-b/-proxy`) were retired the same day they went +live, with their units moved to `~/.AGENTTRASH/20260916-vllm-radiance-retire/`. -Runbook: `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md`. -Prior runtime: [[frank-medium-rocmfpx]] (llama.cpp era, same day). +Files: `scripts/build-llama-rdna4.sh` (builds the runtime, gfx1201 only, private +ROCm 7.14 toolchain bundled beside the binary), `scripts/download-frank-medium-gguf.sh` +(resumable, byte-verified payload), `scripts/frank-medium.env`, +`scripts/serve-frank-medium.sh`, `scripts/frank-medium.service`. The deploy +handler refuses to start a unit whose runtime or payload is missing, retires +the vLLM quadlets, waits for 7770 to be released (the retired proxy held it) and +gates on `/health`. + +The ref is pinned by two features and must not float: **DFlash2** +(ggml-org/llama.cpp#27342) and **qwen35 MTP**. The ROCmFPX fork the endpoint +used earlier that day has neither. + +Runbook: `serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md`. +Retired runtimes: [[frank-medium-radiance-mxfp4]] (vLLM replicas, same day), +[[frank-medium-rocmfpx]] (llama.cpp fork, earlier that day). ## Key Lessons Learned Durable operational lessons from the fleet's history. Full detail in the @@ -364,14 +371,35 @@ linked runbooks. (`Pcie atomics not enabled, hostcall not supported` → `AQL dispatch failed!`), not from vLLM or RCCL. `AMD_LOG_LEVEL` costs one env var and prints the failing HIP call with its return code. +- **The one software path that lets both cards serve ONE model: llama.cpp's + layer split.** vLLM cannot do it here in any of its parallelisms — TP needs + an allreduce, PP needs the same p2p primitive, and both die on the card that + cannot dispatch an RCCL kernel. `--split-mode layer` in llama.cpp needs no + collective: layers are distributed in order and only the boundary activation + crosses PCIe, once per forward pass. Measured here: 1991 t/s prefill / + 18.8 tok/s decode layer-split, against 272 / 26.3 tensor (butterfly + fallback). That is what `frank-medium` runs — see + [[frank-medium-llamacpp-dflash2]]. **ROCm / GPU** -- **Layer split, never tensor, on the RCCL-less ROCmFPX builds.** Tensor - parallel falls back to a butterfly allreduce and collapses prefill ~7x - (measured 271.8 vs 1991 t/s). Pin the MTP draft head to ROCm1 — an - unpinned draft under tensor mode aborts the KV build. +- **Layer split is the deployed shape on both multi-device llama.cpp services, + for two different reasons.** On Dankserver it is the *only* way both R9700s + serve one model (no collective can be dispatched on card #2). On a + RCCL-less fork build it is also the fast one: tensor parallel falls back to + a butterfly allreduce and collapses prefill ~7x (measured 271.8 vs 1991 t/s). + Pin the spec-decode drafter to one device — an unpinned draft under tensor + mode aborts the KV build, and on a layer split the natural home is the card + that ends up holding the target's last layers. +- **Bundling a ROCm runtime next to a llama.cpp binary: `cp -r SRC DST/` is a + trap.** With a missing `DST`, cp creates `DST` as a copy of `SRC` and drops + one directory level. rocBLAS and hipBLASLt look for their device code at a + fixed depth (`../lib/rocblas/library`, `../lib/hipblaslt/library/`), + and the flattened copy fails at *request* time, not load time: the engine + reads the weights, answers `/health` and dies on the first completion with + `Cannot read .../TensileLibrary.dat: No such file or directory for GPU arch + : gfx1201`. Declare the destination, then copy the contents. - **mmap must stay on** for the big-PLE models: `--no-mmap` forces the ~50 GiB PLE table into anonymous memory → silent cgroup kill, dmesg is the only evidence. mmap pages ARE page cache. Never pin the PLE tensor with @@ -492,7 +520,7 @@ Active: |---------|---------| | frank-large (halogen-flash-server, Dankclaw) | `serverAppConfigs-docs/LLM/halogen-flash-server.md` | | frank-small (llama.cpp Qwen3.8-27B GSQ-RCO, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-qwen38-gsq-rco.md` | -| frank-medium (vllm-radiance MXFP4, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` | +| frank-medium (llama.cpp + DFlash2, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md` | | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` | | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` | diff --git a/scripts/frank-medium.env b/scripts/frank-medium.env index 5d683e9..7ff6a10 100644 --- a/scripts/frank-medium.env +++ b/scripts/frank-medium.env @@ -50,11 +50,19 @@ # in the attention layers only, plus ~0.16 GiB per sequence of f32 DeltaNet # state at any depth. # -# SIZING (per card, 31.86 GiB): weights split ~14.7 GiB; the drafter rides one -# card; KV at CTX_SIZE 131,072 is 8.00 GiB total (~4 GiB per card, since the 16 -# attention layers are split with everything else); the loader's compute buffers -# complete the ledger. Read the loader's own "KV self size" / "compute buffer -# size" lines rather than trusting this arithmetic. +# SIZING, measured (rocm-smi, 2026-09-16). Weights split across the two cards; +# the drafter rides the second. The loader's compute buffers are part of what +# is seen below -- they are allocated at load, not per request: +# +# CTX_SIZE GPU 0 GPU 1 headroom on the busier card +# 131072 19.89 GB 24.30 GB 9.9 GB / 31 % +# 262144 24.72 GB 29.14 GB 5.07 GB / 15 % <- deployed +# +# 262144 is the window the model ships (n_ctx_train) and it fits with room, so +# the fleet gets the whole thing; the arithmetic that used to cap this endpoint +# at 114688 was the vLLM predecessor's, not this runtime's. If a future change +# adds VRAM (a longer drafter, quantized KV, more slots) re-read the numbers +# rather than assuming, and prefer CTX_SIZE over --gpu-layers. # # WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP reads # those bare vars; 2026-08-13 incident). llama.cpp takes --host/--port/--temp @@ -74,9 +82,9 @@ DRAFT_MODEL_FILE=Qwen3.8-27B-DFlash2-Q8_0.gguf ALIAS=frank-medium BIND_ADDR=0.0.0.0 LISTEN_PORT=7770 - -# Placement. HIP_VISIBLE_DEVICES hides the iGPU; DEVICES is llama.cpp's own -# list, which is the pair the split runs across. +# Device placement. HIP_VISIBLE_DEVICES hides the Raphael iGPU (HIP device 2) +# so nothing can drift onto shared system RAM; DEVICES is what llama.cpp splits +# across, and neither of the two cards has an XGMI link to the other. HIP_VISIBLE_DEVICES=0,1 DEVICES=ROCm0,ROCm1 SPLIT_MODE=layer @@ -88,11 +96,11 @@ NGL=999 FIT=off FLASH_ATTN=on -# One slot at the model's documented-configured window. --parallel divides -# --ctx-size between slots; it does not multiply. The target model ships -# n_ctx_train 262144 -- raising CTX_SIZE is a measurement, not a guess (watch -# the per-card ledger), and this service was deployed at 131072 first. -CTX_SIZE=131072 +# One slot at the model's full configured window (n_ctx_train 262144), which was +# measured to fit before it was deployed: see SIZING above. --parallel DIVIDES +# --ctx-size between slots, so one slot carries the whole window and a second +# concurrent request queues behind it. +CTX_SIZE=262144 PARALLEL=1 BATCH_SIZE=2048 diff --git a/serverAppConfigs-docs/Infrastructure/Observability.md b/serverAppConfigs-docs/Infrastructure/Observability.md index 6cceb03..17fba8c 100644 --- a/serverAppConfigs-docs/Infrastructure/Observability.md +++ b/serverAppConfigs-docs/Infrastructure/Observability.md @@ -35,14 +35,14 @@ graph LR Recorded 2026-09-11, on the day the stack moved to Quadlet units. Everything in the table above is deployed, healthy and boot-persistent. Almost nothing is wired up beyond the default install: -- **Grafana has a datasource and one dashboard as of 2026-09-16; before that it had neither.** `shared/grafana/provisioning/` is installed to `/opt/grafana/provisioning` by the deployer and read by the container: the Prometheus datasource (`http://prometheus:9090`, fixed uid `prometheus`) and the **LLM fleet — capacity and reuse** dashboard (KV pool fill, queue depth, prefix-cache reuse, latency p95, throughput) covering both LLM endpoints. Verified through the API rather than by eye — the datasource answers `llamacpp:` counters for `frank-medium`, and `GET /api/search?type=dash-db` returns uid `llm-fleet`. One trap worth knowing: a provisioned dashboard does **not** appear in the legacy `dashboard` table on this Grafana version (it lives in unified storage behind `dashboard.grafana.app`), so the API is the way to check. Still outstanding: no dashboards beyond this one, no Loki queries, and only datasources+dashboards are provisioning-as-code. +- **Grafana has a datasource and one dashboard as of 2026-09-16; before that it had neither.** `shared/grafana/provisioning/` is installed to `/opt/grafana/provisioning` by the deployer and read by the container: the Prometheus datasource (`http://prometheus:9090`, fixed uid `prometheus`) and the **LLM fleet — capacity and reuse** dashboard (speculative acceptance, queue depth, prefix-cache reuse, decode step rate, throughput) covering the llama.cpp endpoints. **Re-cut on 2026-09-16**: the panels were keyed to `vllm:` series, which stopped existing when frank-medium moved to llama.cpp, and two of them (KV-pool fill, request-latency p95) have no llama.cpp equivalent at all — those two are retired rather than restated, and the acceptance panel replaces the KV one. A renamed query against a series that does not exist is a panel that draws nothing forever. Verified through the API rather than by eye — the datasource answers `llamacpp:` counters for `frank-medium`, and `GET /api/search?type=dash-db` returns uid `llm-fleet`. One trap worth knowing: a provisioned dashboard does **not** appear in the legacy `dashboard` table on this Grafana version (it lives in unified storage … - **Grafana's dashboard now also surfaces firing alerts.** The **Alerts firing (ALERTS)** panel graphs `ALERTS{alertstate="firing"}` for the LLM rules, because this stack has no other place where an alert becomes visible without opening Prometheus. -- **Prometheus now evaluates rules; nothing delivers them (2026-09-16).** `shared/prometheus/rules/llm-fleet.yml` ships with the config (the deployer's `copy_prometheus_config` installs both — a rule file nothing loads evaluates nothing, and Prometheus reports no error for that) and holds six rules for the LLM fleet: pool saturation, request queueing, prefix-cache collapse, endpoint down, p95 latency, plus the `job:llm_prefix_cache_reuse:rate5m` recording rule the alert and the dashboard share. Verified with `promtool check rules` (SUCCESS, 6 rules) and `/api/v1/rules` (all six `health=ok`). **There is still no Alertmanager and no contact point anywhere**: a firing alert appears on Prometheus' `/alerts`, in the `ALERTS` metric and on the LLM fleet dashboard's alerts panel — and reaches a human only if one is looking. +- **Prometheus now evaluates rules; nothing delivers them (2026-09-16).** `shared/prometheus/rules/llm-fleet.yml` ships with the config (the deployer's `copy_prometheus_config` installs both — a rule file nothing loads evaluates nothing, and Prometheus reports no error for that) and holds five rules for the LLM fleet: request queueing, prefix-cache collapse, speculative decoding off, endpoint down, plus the `job:llm_prefix_cache_reuse:rate5m` recording rule the alert and the dashboard share. **Two rules from the vLLM era were retired, not renamed**, because the series they watched no longer exist: KV-pool saturation (llama.cpp exports no KV-fill gauge; its consequence *is* queueing, which `LlmRequestsQueueing` already covers) and p95 request latency (no latency histogram). `LlmSpecDecodeOff` takes the second slot — the failure a client cannot see, since a lost drafter just halves the speed. Verified with `promtool check rules` and `/api/v1/rules` (`health=ok` for all five). **There is still no Alertmanager and no contact point anywhere**: a firing alert appears on Prometheus' `/alerts`, in the `ALERTS` metric and on the LLM fleet dashboard'… - **Loki is configured but unread.** Storage is filesystem with 30-day retention, `auth_enabled: false`, and the push path works; Grafana can now reach it in Explore, but no dashboard or alert consumes it. - **node-exporter runs its default collector set** — no collector tuning, no textfile collector for custom metrics. - **Alloy is the one configured piece.** It ships the host journal and podman container logs to Loki with `job` labels, and that pipeline is live. -The honest summary: metrics and logs are collected and retained, one dashboard visualizes the LLM fleet, six rules evaluate over it, and **no rule reaches a person** — there is no Alertmanager and no contact point, so correlation stays manual and every signal has to be looked at. +The honest summary: metrics and logs are collected and retained, one dashboard visualizes the LLM fleet, five rules evaluate over it, and **no rule reaches a person** — there is no Alertmanager and no contact point, so correlation stays manual and every signal has to be looked at. ## Deployment diff --git a/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md b/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md index 21a81f7..2313edc 100644 --- a/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md +++ b/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md @@ -145,7 +145,7 @@ Deployed flags: |---|---|---| | `--split-mode` / `--device` / `--tensor-split` | `layer` / `ROCm0,ROCm1` / `1,1` | both cards, one model, no collective | | `--gpu-layers` / `--fit` | `999` / `off` | everything on the cards; auto-fit must not resize a context we set | -| `--ctx-size` / `--parallel` | `131072` / `1` | one slot; `--parallel` **divides** the context | +| `--ctx-size` / `--parallel` | `262144` / `1` | one slot carries the model's whole configured window; `--parallel` **divides** the context | | `--flash-attn` / cache types | `on` / `f16` K and V | an agentic checkpoint is the wrong place to spend KV precision | | `--spec-type` / `--spec-draft-n-max` | `draft-dflash` / `7` | DFlash2 blocks are 8 tokens, so 7 drafts is the ceiling llama.cpp allows | | `--spec-draft-device` | `ROCm1` | the drafter rides the card that ends up holding the target's last layers | @@ -169,12 +169,37 @@ Deployed flags: `block_size=8` and `n_extract=5` are DFlash2's own metadata; the selector is what distinguishes it from DFlash 1. -- Both cards hold the model: at 131072 context, `rocm-smi` reports - **17.9 GiB on GPU 0 and 22.0 GiB on GPU 1** (the drafter and the attention - layers' KV sit on the second). -- A real completion (60-token prompt, 43 output tokens incl. reasoning): - **28.3 tok/s**, `draft_n = 49`, `draft_n_accepted = 36` → **73 % acceptance**. - The reasoning trace arrives in `reasoning_content`. +- Both cards hold the model: at 262144 context `rocm-smi` reports **24.72 GB on + GPU 0 and 29.14 GB on GPU 1** of 34.21 GB each — the drafter, the attention + layers' KV and the compute buffers all sit on the second card, which keeps + ~5.1 GB of headroom. At 131072 the same pair reads 19.89 / 24.30 GB. +- **140,069 prompt tokens prefilled in 134 s** (1043 tok/s) through the one + slot: the raised window is real, not just declared, and it is 22 % beyond the + 114688 the vLLM predecessor could offer. + +### Measured throughput (2026-09-16, deployed service, `temperature 0`) + +Same filler corpus for every shape, so these are shapes rather than a +benchmark: a large prompt with a tiny budget isolates prefill, a small prompt +with a long budget isolates decode. `draft`/`acc` are per request; acceptance +moves with the text (filler prose accepts far worse than code or a real agent +loop). + +| Shape | Prefill | Decode | Acceptance | +|---|---|---|---| +| 4k prompt / 32 out | 3994 tok in 3.25 s — 1230 tok/s | 59.8 tok/s | 71 % | +| 1k prompt / 512 out | 998 tok in 1.41 s — 707 tok/s | **43.8 tok/s** | 41 % | +| 16k prompt / 16 out | 12,528 tok in 8.94 s — **1402 tok/s** | — | 100 % (13/13) | +| 64k prompt / 16 out | 48,508 tok in 39.8 s — 1219 tok/s | — | 53 % | +| 140k prompt / 16 out | 140,069 tok in 134.3 s — 1043 tok/s | — | — | +| same 1k prompt again | **4 tokens prefilled** | 43.8 tok/s | 41 % | + +The last row is the prefix cache doing its job: the second request of the same +conversation re-prefilled 4 of 998 tokens. Prefill holds ~1.0-1.4k tok/s from +4k to 140k, and decode sits at ~44 tok/s on a long generation — against the +vLLM predecessor's 42.8 tok/s single-stream (two replicas) and the ROCmFPX +llama.cpp measured here at 35.7-46.2 tok/s. One process, one model, both cards, +no proxy. ## Observability diff --git a/shared/grafana/provisioning/dashboards/llm-fleet.json b/shared/grafana/provisioning/dashboards/llm-fleet.json index 2b4508d..f03a2fa 100644 --- a/shared/grafana/provisioning/dashboards/llm-fleet.json +++ b/shared/grafana/provisioning/dashboards/llm-fleet.json @@ -1,7 +1,7 @@ { "uid": "llm-fleet", - "title": "LLM fleet — capacity and reuse", - "description": "The signals that explain frank-medium under load, from the 2026-09-16 capacity work. The panels read the endpoint's own counters: KV fill, queue depth, prefix-cache reuse, and request latency. frank-small (llama.cpp on Dankclaw's RTX 3090) exports throughput counters and appears in the same endpoint picker; frank-large runs the halogen engine, which publishes /health, /cache and /v1/* and no metrics endpoint at all, so it cannot appear here. See serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md for the numbers these panels are read against.", + "title": "LLM fleet \u2014 capacity and reuse", + "description": "The signals that explain the LLM endpoints under load, from the 2026-09-16 capacity work. The panels read llama.cpp's own counters: speculative acceptance, queue depth, prefix-cache reuse, decode step rate and throughput. frank-medium (Dankserver, both R9700s, layer split) and frank-small (Dankclaw, RTX 3090) both export them; frank-large runs the halogen engine, which publishes /health, /cache and /v1/* and no metrics endpoint at all, so it cannot appear here. See serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md for the numbers these panels are read against.", "tags": [ "llm", "frank-medium", @@ -42,7 +42,7 @@ { "id": 1, "type": "timeseries", - "title": "KV pool fill (vllm:kv_cache_usage_perc)", + "title": "Speculative decoding (DFlash2 acceptance)", "description": "How full each card's KV pool is. One card holds ~122,636 tokens; sustained high values with a cold prefix cache is the thrash this fleet measured.", "gridPos": { "h": 8, @@ -69,8 +69,13 @@ "targets": [ { "refId": "A", - "expr": "vllm:kv_cache_usage_perc{job=~\"$job\"} * 100", - "legendFormat": "{{job}} engine {{engine}}" + "expr": "rate(llamacpp:spec_decode_num_accepted_tokens_total{job=~\"$job\"}[5m]) / clamp_min(rate(llamacpp:spec_decode_num_drafts_total{job=~\"$job\"}[5m]), 0.001)", + "legendFormat": "{{job}} accepted per draft step" + }, + { + "refId": "B", + "expr": "rate(llamacpp:spec_decode_num_draft_tokens_total{job=~\"$job\"}[5m])", + "legendFormat": "{{job}} draft tokens/s" } ] }, @@ -103,13 +108,13 @@ "targets": [ { "refId": "A", - "expr": "vllm:num_requests_waiting{job=~\"$job\"}", - "legendFormat": "{{job}} waiting" + "expr": "llamacpp:requests_deferred{job=~\"$job\"}", + "legendFormat": "{{job}} deferred" }, { "refId": "B", - "expr": "vllm:num_requests_running{job=~\"$job\"}", - "legendFormat": "{{job}} running" + "expr": "llamacpp:requests_processing{job=~\"$job\"}", + "legendFormat": "{{job}} processing" } ] }, @@ -143,7 +148,7 @@ "targets": [ { "refId": "A", - "expr": "rate(vllm:prefix_cache_hits_total{job=~\"$job\"}[5m]) / clamp_min(rate(vllm:prefix_cache_queries_total{job=~\"$job\"}[5m]), 1)", + "expr": "rate(llamacpp:prompt_tokens_cached_total{job=~\"$job\"}[5m]) / clamp_min(rate(llamacpp:prompt_tokens_total{job=~\"$job\"}[5m]), 1)", "legendFormat": "{{job}} reuse" } ] @@ -151,8 +156,8 @@ { "id": 4, "type": "timeseries", - "title": "End-to-end request latency p95", - "description": "Wall-clock per request, prefill included — the number an agent-loop turn feels.", + "title": "Decode steps and batch size", + "description": "Wall-clock per request, prefill included \u2014 the number an agent-loop turn feels.", "gridPos": { "h": 8, "w": 12, @@ -176,8 +181,13 @@ "targets": [ { "refId": "A", - "expr": "histogram_quantile(0.95, sum by (le, job) (rate(vllm:e2e_request_latency_seconds_bucket{job=~\"$job\"}[5m])))", - "legendFormat": "{{job}} p95" + "expr": "rate(llamacpp:n_decode_total{job=~\"$job\"}[5m])", + "legendFormat": "{{job}} decode steps/s" + }, + { + "refId": "B", + "expr": "llamacpp:n_busy_slots_per_decode{job=~\"$job\"}", + "legendFormat": "{{job}} sequences per decode step" } ] }, @@ -207,16 +217,6 @@ "overrides": [] }, "targets": [ - { - "refId": "A", - "expr": "rate(vllm:generation_tokens_total{job=~\"$job\"}[5m])", - "legendFormat": "{{job}} generated tok/s" - }, - { - "refId": "B", - "expr": "rate(vllm:prompt_tokens_total{job=~\"$job\"}[5m])", - "legendFormat": "{{job}} prompt tok/s" - }, { "refId": "C", "expr": "llamacpp:predicted_tokens_seconds{job=~\"$job\"}", @@ -233,7 +233,7 @@ "id": 6, "type": "timeseries", "title": "Alerts firing (ALERTS)", - "description": "Prometheus rules for the LLM fleet (shared/prometheus/rules/llm-fleet.yml): pool saturation, queueing, prefix-cache collapse, endpoint down, p95 latency. A firing alert appears here, on Prometheus' /alerts page, and nowhere else — this stack has no Alertmanager, so nothing is notified.", + "description": "Prometheus rules for the LLM fleet (shared/prometheus/rules/llm-fleet.yml): pool saturation, queueing, prefix-cache collapse, endpoint down, p95 latency. A firing alert appears here, on Prometheus' /alerts page, and nowhere else \u2014 this stack has no Alertmanager, so nothing is notified.", "gridPos": { "h": 8, "w": 12, diff --git a/shared/prometheus/rules/llm-fleet.yml b/shared/prometheus/rules/llm-fleet.yml index e507ad5..75d19bf 100644 --- a/shared/prometheus/rules/llm-fleet.yml +++ b/shared/prometheus/rules/llm-fleet.yml @@ -1,7 +1,18 @@ # Alerting and recording rules for the LLM fleet. # -# Thresholds are not guesses: each one corresponds to a failure mode measured on -# this fleet on 2026-09-16 (see serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md). +# Every expression here is keyed to a series that EXISTS as of 2026-09-16. When +# frank-medium moved from vllm-radiance to llama.cpp the exported namespace +# changed with it (vllm: -> llamacpp:), and two signals the old file alerted on +# no longer exist in any form: +# +# * vllm:kv_cache_usage_perc — llama.cpp exports no KV-fill gauge. Its +# observable consequence is queueing, which is what LlmRequestsQueueing +# watches instead. A renamed query would have been a rule that can never +# fire. +# * vllm:e2e_request_latency_seconds_* — no latency histogram either. The +# rule it fed (p95 wall clock) is retired rather than restated, and +# LlmSpecDecodeOff takes its place: the failure an operator actually +# cannot see from the outside on this deployment. # # IMPORTANT: nothing is notified. There is no Alertmanager in this stack, so a # firing alert appears on Prometheus' /alerts page, in the ALERTS metric (which @@ -17,42 +28,30 @@ groups: # expression. The exported series are counters; the rate is the reuse. - record: job:llm_prefix_cache_reuse:rate5m expr: | - rate(vllm:prefix_cache_hits_total[5m]) - / clamp_min(rate(vllm:prefix_cache_queries_total[5m]), 0.001) + rate(llamacpp:prompt_tokens_cached_total[5m]) + / clamp_min(rate(llamacpp:prompt_tokens_total[5m]), 0.001) - # The capacity wall. Each R9700 carries 122,636 KV tokens; past ~90 % - # the next large request queues or evicts a cached prefix. - - alert: LlmKvPoolSaturated - expr: vllm:kv_cache_usage_perc > 0.90 - for: 10m - labels: - severity: warning - annotations: - summary: "{{ $labels.job }}: KV pool above 90 % for 10 minutes" - description: >- - A card's KV pool is nearly full. Expect queueing or cold re-prefills - for large requests; the wall and its arithmetic are in the - frank-medium runbook. - - # Concurrency limit reached: MAXSEQS=4 per replica. + # The capacity wall, in the only form llama.cpp exports it: work waiting + # for the single slot. Past here the next large request queues. - alert: LlmRequestsQueueing - expr: vllm:num_requests_waiting > 0 + expr: llamacpp:requests_deferred > 0 for: 10m labels: severity: warning annotations: summary: "{{ $labels.job }}: requests waiting for 10 minutes" description: >- - All sequence slots are busy and work is queueing. Either the fleet is - genuinely busy or one endpoint is carrying it alone. + The endpoint's slot(s) are busy and work is queueing. Either the + fleet is genuinely busy or one endpoint is carrying it alone. With + --parallel 1 a second concurrent request always shows up here. # The thrash signature measured on 2026-09-16: requests keep arriving but - # almost none are served from cache, which is what three concurrent - # 50-80k-token conversations do to a two-card fleet. + # almost none are served from cache, which is what several concurrent + # 50-80k-token conversations do to this fleet. - alert: LlmPrefixCacheCollapse expr: >- job:llm_prefix_cache_reuse:rate5m < 0.20 - and rate(vllm:prefix_cache_queries_total[5m]) > 0.05 + and rate(llamacpp:prompt_tokens_total[5m]) > 0.05 for: 15m labels: severity: info @@ -61,12 +60,29 @@ groups: description: >- Nearly every request is prefilling from scratch. If several large contexts are in flight together, this is the capacity wall rather - than a routing fault — placement and admission were both measured - and neither fixes it. + than a routing fault. - # The endpoint, not the replica: two replicas serve :7770 through the - # proxy, so this fires only when both are down (a single restart is - # invisible — the proxy fails over). + # The speculative path is the one failure no client can see: the server + # answers normally, just at half the speed, when the drafter is gone. + # Drafts and accepts are counters, so a rate that stays at zero while the + # endpoint is serving tokens means nothing is being drafted. + - alert: LlmSpecDecodeOff + expr: >- + rate(llamacpp:spec_decode_num_drafts_total[5m]) == 0 + and rate(llamacpp:tokens_predicted_total[5m]) > 0.1 + for: 15m + labels: + severity: warning + annotations: + summary: "{{ $labels.job }}: serving without speculative decoding" + description: >- + Tokens are being produced but no draft has been proposed for 15 + minutes. On frank-medium that means the DFlash2 drafter is not + running (check the unit's log for common_speculative_impl_draft_dflash), + and the endpoint is decoding autoregressively. + + # The endpoint, not a replica: one process serves each port now, so a + # restart of the unit is a scrape gap, not a failover. - alert: LlmEndpointDown expr: up{job=~"frank-.*"} == 0 for: 2m @@ -76,24 +92,7 @@ groups: summary: "{{ $labels.job }}: scrape target down for 2 minutes" description: >- Prometheus cannot reach the LLM endpoint. Check the units: - `systemctl --user is-active frank-medium-{a,b,proxy}` on Dankserver, - `systemctl is-active frank-small` (the llama.cpp root unit) or - `systemctl --user is-active frank-large` (the halogen quadlet) on - Dankclaw. frank-large is not scraped at all — the halogen engine - exports no metrics — so this alert can only ever name the endpoints - that do export one. - - # What an operator actually feels: wall-clock per request, prefill - # included. A cold 50k-token prefill alone is ~22 s on one card. - - alert: LlmLatencyHigh - expr: >- - histogram_quantile(0.95, - sum by (le, job) (rate(vllm:e2e_request_latency_seconds_bucket[10m]))) > 60 - for: 10m - labels: - severity: warning - annotations: - summary: "{{ $labels.job }}: p95 request latency above 60 s for 10 minutes" - description: >- - Requests are taking a minute or more end to end. On this hardware - that is usually large contexts competing for one card's KV pool. + `systemctl is-active frank-medium` on Dankserver (a root llama.cpp + unit) and `systemctl is-active frank-small` on Dankclaw. frank-large + is not scraped at all — the halogen engine exports no metrics — so + this alert can only ever name the endpoints that do export one. diff --git a/tests/frank_medium_unit.bats b/tests/frank_medium_unit.bats index eceb9c0..fc8f4bf 100755 --- a/tests/frank_medium_unit.bats +++ b/tests/frank_medium_unit.bats @@ -100,12 +100,19 @@ DRAFT_BYTES=2056414816 # DFlash2 Q8_0 drafter [ "$parallel" -ge 1 ] [ "$cache_k" = "f16" ] && [ "$cache_v" = "f16" ] # qwen35 spends KV on its 16 full-attention layers only: 2 (K,V) x 4 heads - # x 256 dim x 2 B = 64 KiB per token at f16. Worst case for the fit is ONE - # card carrying the drafter and every attention layer's KV, with ~2 GiB of - # compute buffers: half the weights, plus that, must stay inside the card. + # x 256 dim x 2 B = 64 KiB per token at f16. A layer split puts those + # layers (and their KV) on both cards, so the worst card carries half the + # weights, half the KV, the drafter and ~2 GiB of compute buffers. + # + # This model is deliberately calibrated against rocm-smi: at 262144 it + # predicts 28.53e9 B against the 29.14e9 B measured on the busier card + # (2026-09-16), i.e. ~2 % optimistic. Keep it that way -- a model that is + # pessimistic beyond a few GiB would reject a context that fits. local per_card - per_card=$(( MODEL_BYTES / 2 + DRAFT_BYTES + ctx * 65536 + 2 * 1024 * 1024 * 1024 )) + per_card=$(( MODEL_BYTES / 2 + DRAFT_BYTES + (ctx * 65536) / 2 + 2 * 1024 * 1024 * 1024 )) [ "$per_card" -le "$CARD_BYTES" ] + # ...and the window may not exceed what the checkpoint was trained for. + [ "$ctx" -le 262144 ] } @test "frank-medium is text only: the vision tower stays on frank-small" { diff --git a/tests/observability_quadlet.bats b/tests/observability_quadlet.bats index 7189013..9919532 100755 --- a/tests/observability_quadlet.bats +++ b/tests/observability_quadlet.bats @@ -345,13 +345,26 @@ PAIRS local cfg="$REPO_ROOT/shared/prometheus/prometheus.yml" [ -f "$rules" ] python3 -c " -import yaml +import re, yaml d = yaml.safe_load(open('$rules')) names = [r.get('alert') or r.get('record') for g in d['groups'] for r in g['rules']] -assert 'record' or True -for want in ('LlmKvPoolSaturated', 'LlmRequestsQueueing', 'LlmPrefixCacheCollapse', - 'LlmEndpointDown', 'LlmLatencyHigh'): +for want in ('job:llm_prefix_cache_reuse:rate5m', 'LlmRequestsQueueing', + 'LlmPrefixCacheCollapse', 'LlmSpecDecodeOff', 'LlmEndpointDown'): assert want in names, want +# Every rule must be keyed to a series that exists. The vLLM era's namespaces +# are gone (frank-medium is llama.cpp now) and a rule naming one of them +# evaluates nothing forever without erroring — the failure this seam exists for. +exprs = [r.get('expr', '') for g in d['groups'] for r in g['rules']] +assert not any('vllm:' in e for e in exprs), exprs +# ...and the llama.cpp series a rule may name are the ones the server exports +# (measured from /metrics on 2026-09-16). Anything else is a guess. +live = {'llamacpp:prompt_tokens_total', 'llamacpp:prompt_tokens_cached_total', + 'llamacpp:requests_deferred', 'llamacpp:requests_processing', + 'llamacpp:spec_decode_num_drafts_total', + 'llamacpp:spec_decode_num_accepted_tokens_total', + 'llamacpp:tokens_predicted_total'} +used = {m for e in exprs for m in re.findall(r'llamacpp:[a-z_0-9]+', e)} +assert used <= live, used - live print('rules ok:', len(names)) " # A rule file that nothing loads evaluates nothing, and Prometheus reports