frank-medium: serve the whole 262144-token window, and repoint every signal to llama.cpp

Context: 262144 was measured before it was deployed -- 24.72 GB on GPU 0 and 29.14 GB on GPU 1 of 34.21 GB each, ~5 GB of headroom on the busier card, and a 140,069-token prompt prefilled through the single slot in 134 s to prove the window is real. The 114688 wall was the vLLM predecessor's arithmetic, not this runtime's. Throughput, measured through the service: 1.0-1.4k tok/s prefill from 4k to 140k, 43.8 tok/s decode on a 512-token generation, 41-71 % DFlash2 acceptance, and the prefix cache re-prefilling 4 of 998 tokens on a repeated prompt. The signals move with the runtime, so the fleet's rules and dashboard are re-cut rather than renamed: vllm: series do not exist any more, and two of the old rules watched things llama.cpp does not export at all (KV-pool fill, p95 latency). Those are retired with the reason written down, and LlmSpecDecodeOff takes one of the slots -- a lost drafter is the failure no client can see. The observability seam now fails if any rule names a series the endpoint does not export, which is what would have caught the stale vllm: expressions. omp models.yml (outside the repo) is updated in the same pass: contextWindow 262144, and the full reasoning-effort map, because Qwen3.8's template answers 500 on 'high' -- verified against the live endpoint, low/medium/xhigh answer 200. Subagents run at high.
2026-09-16 15:29:40 -07:00
parent ab51d61198
commit dc76dc5dc4
8 changed files with 220 additions and 140 deletions
@@ -35,14 +35,14 @@ graph LR

 Recorded 2026-09-11, on the day the stack moved to Quadlet units. Everything in the table above is deployed, healthy and boot-persistent. Almost nothing is wired up beyond the default install:

- **Grafana has a datasource and one dashboard as of 2026-09-16; before that it had neither.** `shared/grafana/provisioning/` is installed to `/opt/grafana/provisioning` by the deployer and read by the container: the Prometheus datasource (`http://prometheus:9090`, fixed uid `prometheus`) and the **LLM fleet — capacity and reuse** dashboard (KV pool fill, queue depth, prefix-cache reuse, latency p95, throughput) covering both LLM endpoints. Verified through the API rather than by eye — the datasource answers `llamacpp:` counters for `frank-medium`, and `GET /api/search?type=dash-db` returns uid `llm-fleet`. One trap worth knowing: a provisioned dashboard does **not** appear in the legacy `dashboard` table on this Grafana version (it lives in unified storage behind `dashboard.grafana.app`), so the API is the way to check. Still outstanding: no dashboards beyond this one, no Loki queries, and only datasources+dashboards are provisioning-as-code.
+- **Grafana has a datasource and one dashboard as of 2026-09-16; before that it had neither.** `shared/grafana/provisioning/` is installed to `/opt/grafana/provisioning` by the deployer and read by the container: the Prometheus datasource (`http://prometheus:9090`, fixed uid `prometheus`) and the **LLM fleet — capacity and reuse** dashboard (speculative acceptance, queue depth, prefix-cache reuse, decode step rate, throughput) covering the llama.cpp endpoints. **Re-cut on 2026-09-16**: the panels were keyed to `vllm:` series, which stopped existing when frank-medium moved to llama.cpp, and two of them (KV-pool fill, request-latency p95) have no llama.cpp equivalent at all — those two are retired rather than restated, and the acceptance panel replaces the KV one. A renamed query against a series that does not exist is a panel that draws nothing forever. Verified through the API rather than by eye — the datasource answers `llamacpp:` counters for `frank-medium`, and `GET /api/search?type=dash-db` returns uid `llm-fleet`. One trap worth knowing: a provisioned dashboard does **not** appear in the legacy `dashboard` table on this Grafana version (it lives in unified storage …
 - **Grafana's dashboard now also surfaces firing alerts.** The **Alerts firing (ALERTS)** panel graphs `ALERTS{alertstate="firing"}` for the LLM rules, because this stack has no other place where an alert becomes visible without opening Prometheus.
- **Prometheus now evaluates rules; nothing delivers them (2026-09-16).** `shared/prometheus/rules/llm-fleet.yml` ships with the config (the deployer's `copy_prometheus_config` installs both — a rule file nothing loads evaluates nothing, and Prometheus reports no error for that) and holds six rules for the LLM fleet: pool saturation, request queueing, prefix-cache collapse, endpoint down, p95 latency, plus the `job:llm_prefix_cache_reuse:rate5m` recording rule the alert and the dashboard share. Verified with `promtool check rules` (SUCCESS, 6 rules) and `/api/v1/rules` (all six `health=ok`). **There is still no Alertmanager and no contact point anywhere**: a firing alert appears on Prometheus' `/alerts`, in the `ALERTS` metric and on the LLM fleet dashboard's alerts panel — and reaches a human only if one is looking.
+- **Prometheus now evaluates rules; nothing delivers them (2026-09-16).** `shared/prometheus/rules/llm-fleet.yml` ships with the config (the deployer's `copy_prometheus_config` installs both — a rule file nothing loads evaluates nothing, and Prometheus reports no error for that) and holds five rules for the LLM fleet: request queueing, prefix-cache collapse, speculative decoding off, endpoint down, plus the `job:llm_prefix_cache_reuse:rate5m` recording rule the alert and the dashboard share. **Two rules from the vLLM era were retired, not renamed**, because the series they watched no longer exist: KV-pool saturation (llama.cpp exports no KV-fill gauge; its consequence *is* queueing, which `LlmRequestsQueueing` already covers) and p95 request latency (no latency histogram). `LlmSpecDecodeOff` takes the second slot — the failure a client cannot see, since a lost drafter just halves the speed. Verified with `promtool check rules` and `/api/v1/rules` (`health=ok` for all five). **There is still no Alertmanager and no contact point anywhere**: a firing alert appears on Prometheus' `/alerts`, in the `ALERTS` metric and on the LLM fleet dashboard'…
 - **Loki is configured but unread.** Storage is filesystem with 30-day retention, `auth_enabled: false`, and the push path works; Grafana can now reach it in Explore, but no dashboard or alert consumes it.
 - **node-exporter runs its default collector set** — no collector tuning, no textfile collector for custom metrics.
 - **Alloy is the one configured piece.** It ships the host journal and podman container logs to Loki with `job` labels, and that pipeline is live.

-The honest summary: metrics and logs are collected and retained, one dashboard visualizes the LLM fleet, six rules evaluate over it, and **no rule reaches a person** — there is no Alertmanager and no contact point, so correlation stays manual and every signal has to be looked at.
+The honest summary: metrics and logs are collected and retained, one dashboard visualizes the LLM fleet, five rules evaluate over it, and **no rule reaches a person** — there is no Alertmanager and no contact point, so correlation stays manual and every signal has to be looked at.

 ## Deployment

@@ -145,7 +145,7 @@ Deployed flags:
 |---|---|---|
 | `--split-mode` / `--device` / `--tensor-split` | `layer` / `ROCm0,ROCm1` / `1,1` | both cards, one model, no collective |
 | `--gpu-layers` / `--fit` | `999` / `off` | everything on the cards; auto-fit must not resize a context we set |
-| `--ctx-size` / `--parallel` | `131072` / `1` | one slot; `--parallel` **divides** the context |
+| `--ctx-size` / `--parallel` | `262144` / `1` | one slot carries the model's whole configured window; `--parallel` **divides** the context |
 | `--flash-attn` / cache types | `on` / `f16` K and V | an agentic checkpoint is the wrong place to spend KV precision |
 | `--spec-type` / `--spec-draft-n-max` | `draft-dflash` / `7` | DFlash2 blocks are 8 tokens, so 7 drafts is the ceiling llama.cpp allows |
 | `--spec-draft-device` | `ROCm1` | the drafter rides the card that ends up holding the target's last layers |
@@ -169,12 +169,37 @@ Deployed flags:

  `block_size=8` and `n_extract=5` are DFlash2's own metadata; the selector is
  what distinguishes it from DFlash 1.
- Both cards hold the model: at 131072 context, `rocm-smi` reports
-  **17.9 GiB on GPU 0 and 22.0 GiB on GPU 1** (the drafter and the attention
-  layers' KV sit on the second).
- A real completion (60-token prompt, 43 output tokens incl. reasoning):
-  **28.3 tok/s**, `draft_n = 49`, `draft_n_accepted = 36` → **73 % acceptance**.
-  The reasoning trace arrives in `reasoning_content`.
+- Both cards hold the model: at 262144 context `rocm-smi` reports **24.72 GB on
+  GPU 0 and 29.14 GB on GPU 1** of 34.21 GB each — the drafter, the attention
+  layers' KV and the compute buffers all sit on the second card, which keeps
+  ~5.1 GB of headroom. At 131072 the same pair reads 19.89 / 24.30 GB.
+- **140,069 prompt tokens prefilled in 134 s** (1043 tok/s) through the one
+  slot: the raised window is real, not just declared, and it is 22 % beyond the
+  114688 the vLLM predecessor could offer.
+
+### Measured throughput (2026-09-16, deployed service, `temperature 0`)
+
+Same filler corpus for every shape, so these are shapes rather than a
+benchmark: a large prompt with a tiny budget isolates prefill, a small prompt
+with a long budget isolates decode. `draft`/`acc` are per request; acceptance
+moves with the text (filler prose accepts far worse than code or a real agent
+loop).
+
+| Shape | Prefill | Decode | Acceptance |
+|---|---|---|---|
+| 4k prompt / 32 out | 3994 tok in 3.25 s — 1230 tok/s | 59.8 tok/s | 71 % |
+| 1k prompt / 512 out | 998 tok in 1.41 s — 707 tok/s | **43.8 tok/s** | 41 % |
+| 16k prompt / 16 out | 12,528 tok in 8.94 s — **1402 tok/s** | — | 100 % (13/13) |
+| 64k prompt / 16 out | 48,508 tok in 39.8 s — 1219 tok/s | — | 53 % |
+| 140k prompt / 16 out | 140,069 tok in 134.3 s — 1043 tok/s | — | — |
+| same 1k prompt again | **4 tokens prefilled** | 43.8 tok/s | 41 % |
+
+The last row is the prefix cache doing its job: the second request of the same
+conversation re-prefilled 4 of 998 tokens. Prefill holds ~1.0-1.4k tok/s from
+4k to 140k, and decode sits at ~44 tok/s on a long generation — against the
+vLLM predecessor's 42.8 tok/s single-stream (two replicas) and the ROCmFPX
+llama.cpp measured here at 35.7-46.2 tok/s. One process, one model, both cards,
+no proxy.

 ## Observability