docs: frank-medium is llama.cpp with DFlash2 on both cards

The runbook for the new deployment, plus the sweep of everything that still
described the vllm-radiance replica pair: the vault index and its ACTIVE
marker, the superseded banner on the radiance page, the pointer from the
ROCmFPX page, the homepage card, the Grafana dashboard description and the
observability page's scrape table.

The new page carries the two facts a future agent needs and the flags do not
say: llama.cpp's layer split is what lets BOTH cards serve ONE model without
any collective (vLLM's PP handoff is an RCCL send/recv, which is the primitive
this host cannot dispatch), and the rocBLAS Tensile-path trap that makes a
healthy-looking build fail on the first request.
This commit is contained in:
Frank committed 2026-09-16 15:15:02 -07:00
1 parent c9c42e7489
commit ab51d61198
7 files changed
+266 -21

No files matched your search

@@ -35,7 +35,7 @@ graph LR
Recorded 2026-09-11, on the day the stack moved to Quadlet units. Everything in the table above is deployed, healthy and boot-persistent. Almost nothing is wired up beyond the default install:
- **Grafana has a datasource and one dashboard as of 2026-09-16; before that it had neither.** `shared/grafana/provisioning/` is installed to `/opt/grafana/provisioning` by the deployer and read by the container: the Prometheus datasource (`http://prometheus:9090`, fixed uid `prometheus`) and the **LLM fleet — capacity and reuse** dashboard (KV pool fill, queue depth, prefix-cache reuse, latency p95, throughput) covering both LLM endpoints. Verified through the API rather than by eye — the datasource answers `vllm:kv_cache_usage_perc` for `frank-medium`, and `GET /api/search?type=dash-db` returns uid `llm-fleet`. One trap worth knowing: a provisioned dashboard does **not** appear in the legacy `dashboard` table on this Grafana version (it lives in unified storage behind `dashboard.grafana.app`), so the API is the way to check. Still outstanding: no dashboards beyond this one, no Loki queries, and only datasources+dashboards are provisioning-as-code.
- **Grafana has a datasource and one dashboard as of 2026-09-16; before that it had neither.** `shared/grafana/provisioning/` is installed to `/opt/grafana/provisioning` by the deployer and read by the container: the Prometheus datasource (`http://prometheus:9090`, fixed uid `prometheus`) and the **LLM fleet — capacity and reuse** dashboard (KV pool fill, queue depth, prefix-cache reuse, latency p95, throughput) covering both LLM endpoints. Verified through the API rather than by eye — the datasource answers `llamacpp:` counters for `frank-medium`, and `GET /api/search?type=dash-db` returns uid `llm-fleet`. One trap worth knowing: a provisioned dashboard does **not** appear in the legacy `dashboard` table on this Grafana version (it lives in unified storage behind `dashboard.grafana.app`), so the API is the way to check. Still outstanding: no dashboards beyond this one, no Loki queries, and only datasources+dashboards are provisioning-as-code.
- **Grafana's dashboard now also surfaces firing alerts.** The **Alerts firing (ALERTS)** panel graphs `ALERTS{alertstate="firing"}` for the LLM rules, because this stack has no other place where an alert becomes visible without opening Prometheus.
- **Prometheus now evaluates rules; nothing delivers them (2026-09-16).** `shared/prometheus/rules/llm-fleet.yml` ships with the config (the deployer's `copy_prometheus_config` installs both — a rule file nothing loads evaluates nothing, and Prometheus reports no error for that) and holds six rules for the LLM fleet: pool saturation, request queueing, prefix-cache collapse, endpoint down, p95 latency, plus the `job:llm_prefix_cache_reuse:rate5m` recording rule the alert and the dashboard share. Verified with `promtool check rules` (SUCCESS, 6 rules) and `/api/v1/rules` (all six `health=ok`). **There is still no Alertmanager and no contact point anywhere**: a firing alert appears on Prometheus' `/alerts`, in the `ALERTS` metric and on the LLM fleet dashboard's alerts panel — and reaches a human only if one is looking.
- **Loki is configured but unread.** Storage is filesystem with 30-day retention, `auth_enabled: false`, and the push path works; Grafana can now reach it in Explore, but no dashboard or alert consumes it.
@@ -85,21 +85,17 @@ The rule is the same as for every other service: the only off-host path is a gua
| `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on `dankclaw_default` |
| `comfyui` | `comfyui:8188` | UI probe, no native exporter |
| `services` | `192.168.0.143:8081`, `:3005` | SearXNG, OpenWebUI up/down probes |
| `frank-medium` | `192.168.0.235:7770` | vLLM on Dankserver — 418 `vllm:` series |
| `frank-medium` | `192.168.0.235:7770` | llama.cpp on Dankserver — `llamacpp:` counters, including the spec-decode pair |
| `frank-small` | `192.168.0.143:7771` | llama.cpp on Dankclaw's RTX 3090 — `llamacpp:` counters, including the spec-decode pair |
The LLM endpoints were added on 2026-09-16, after a night spent diagnosing
frank-medium by hand, and re-cut the same day when Dankclaw split into two
endpoints. vLLM exports exactly what that diagnosis needed:
`vllm:kv_cache_usage_perc` (how full the per-card pool is),
`vllm:num_requests_waiting` (queue depth), the request-latency histograms, and
the prefix-cache counters `vllm:prefix_cache_hits_total` /
`vllm:prefix_cache_queries_total`, whose rates give the reuse rate that says
whether a pinned conversation is being served warm or re-prefilled cold. What
those numbers mean on this hardware is in [[frank-medium-radiance-mxfp4]];
llama.cpp's counters (prompt and decode throughput, KV fill, and the
`spec_decode_*` pair that proves MTP is running) are read against
[[frank-small-qwen38-gsq-rco]].
endpoints. The vLLM runtime behind that diagnosis is retired; frank-medium
now runs mainline llama.cpp, and the endpoint exports llama.cpp's own
counters — prompt and decode throughput, KV fill, and the `spec_decode_*`
pair that proves the DFlash2 drafter is running. What those numbers mean on
this hardware is in [[frank-medium-llamacpp-dflash2]]; the same counter
family on frank-small is read against [[frank-small-qwen38-gsq-rco]].
**`frank-large` is deliberately NOT scraped.** The halogen engine publishes
`/health`, `/cache` and `/v1/*` and no Prometheus text endpoint at all, so a