dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(monitoring,harness): follow frank-medium onto vllm:* metrics and fix red-by-construction targets

Browse Source 
The radiance TP=2 cutover changed the metric namespace frank-medium exports
(llamacpp:* -> vllm:*) and left two scrape targets addressing host processes
by an address a container cannot reach.

- shared/prometheus/rules/llm-fleet.yml: two namespaces side by side. Added
  LlmRequestsWaiting, LlmKvCacheSaturated (vllm-only gauges llama.cpp never
  had), the job:llm_vllm_prefix_cache_reuse:rate5m and
  job:llm_spec_accept_length:rate5m recording rules, and re-cut
  LlmPrefixCacheCollapse / LlmSpecDecodeOff to OR both families with
  per-family traffic floors so an idle endpoint cannot fire them. Every
  expression validated against live Prometheus.
- shared/prometheus/prometheus.yml: frank-small -> host.containers.internal:7771
  and frank-large scraped at host.containers.internal:7770. Both are HOST
  processes on Dankclaw and netavark DNATs published container ports only, so
  the LAN-IP targets were red by construction; frank-large's CIRU llama.cpp
  build does export the llamacpp:* family, contra the old comment.
- tests/observability_quadlet.bats: the two seams now encode the new contract
  (per-endpoint reachability by process type; every named series must be on the
  measured list for its engine; engine-independent rules must name both).
- homepage docs.html/services.json: frank-medium sees pictures, frank-large is
  CIRU Orca with a single seat (was halogen w4b, two seats, 200k).
- docs: radiance runbook is ACTIVE (deployed flags, three-arm battery, verified
  capabilities, rollback, replica-era sections labelled); llama.cpp runbook
  carries the SUPERSEDED banner; Dual-R9700-Cooperation states the measured
  verdict; Observability records the namespace split and the DNAT lesson;
  AGENTS.md topology/frank-medium/Model Config/Benchmarking Policy/runbook
  index flipped; both indexes updated; work-log entry added.

Also repaired agent-pass damage: prose prepended above frontmatter openers,
duplicated runbook bodies, a literal tool-call token pasted into a runbook, and
an index destroyed by diff artifacts (restored from HEAD, flips re-applied).
Vault link check: 128 pages, 0 unresolved. ./tests/run.sh: 325 pass.
This commit is contained in:
Frank
2026-09-20 19:35:56 -07:00
parent c2116658f8
commit 08f1c7ad09
13 changed files with 687 additions and 286 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/Infrastructure/Observability.md
+21 -15
View File
@@ -41,6 +41,7 @@ Recorded 2026-09-11, on the day the stack moved to Quadlet units. Everything in
- **Grafana has a datasource and one dashboard as of 2026-09-16; before that it had neither.** `shared/grafana/provisioning/` is installed to `/opt/grafana/provisioning` by the deployer and read by the container: the Prometheus datasource (`http://prometheus:9090`, fixed uid `prometheus`) and the **LLM fleet — capacity and reuse** dashboard (speculative acceptance, queue depth, prefix-cache reuse, decode step rate, throughput) covering the llama.cpp endpoints. **Re-cut on 2026-09-16**: the panels were keyed to `vllm:` series, which stopped existing when frank-medium moved to llama.cpp, and two of them (KV-pool fill, request-latency p95) have no llama.cpp equivalent at all — those two are retired rather than restated, and the acceptance panel replaces the KV one. A renamed query against a series that does not exist is a panel that draws nothing forever. Verified through the API rather than by eye — the datasource answers `llamacpp:` counters for `frank-medium`, and `GET /api/search?type=dash-db` returns uid `llm-fleet`. One trap worth knowing: a provisioned dashboard does **not** appear in the legacy `dashboard` table on this Grafana version (it lives in unified storage …
- **Grafana's dashboard now also surfaces firing alerts.** The **Alerts firing (ALERTS)** panel graphs `ALERTS{alertstate="firing"}` for the LLM rules, because this stack has no other place where an alert becomes visible without opening Prometheus.
- **Prometheus now evaluates rules, and they are delivered (2026-09-17).** `shared/prometheus/rules/llm-fleet.yml` ships with the config (the deployer's `copy_prometheus_config` installs both — a rule file nothing loads evaluates nothing, and Prometheus reports no error for that) and holds five rules for the LLM fleet: request queueing, prefix-cache collapse, speculative decoding off, endpoint down, plus the `job:llm_prefix_cache_reuse:rate5m` recording rule the alert and the dashboard share. **Two rules from the vLLM era were retired, not renamed**, because the series they watched no longer exist: KV-pool saturation (llama.cpp exports no KV-fill gauge; its consequence *is* queueing, which `LlmRequestsQueueing` already covers) and p95 request latency (no latency histogram). `LlmSpecDecodeOff` takes the second slot — the failure a client cannot see, since a lost drafter just halves the speed. Verified with `promtool check rules` and `/api/v1/rules` (`health=ok` for all five). **Delivery went live 2026-09-17**: the `alerting:` block in `prometheus.yml` points at `alertmanager:9093` (container name on `dankclaw_default` — the published port is loopback-only and would be wrong from inside Prometheus). Alertmanager groups by `alertname` (30 s group wait, 5 m interval, 4 h repeat) and webhooks the host-side **Signal relay** at `host.containers.internal:18093/alertmanager` — the relay binds the host's LAN IP (192.168.0.143:18093, the one address netavark delivers that dial to), with an iptables set admitting loopback, the host and the podman bridge subnet only. The relay (`scripts/signal-relay.py`, a systemd *user* unit, not a quadlet — it runs python3 from the checkout and can be enabled) dedupes per group for 15 minutes and sends through **signal-cli account 318311** (`/usr/local/bin/signal-cli`, state in `~/.local/share/signal-cli`), Frank's existing account. The Grafana dashboard's ALERTS panel and Prometheus' `/alerts` remain the second and third surfaces.
- **2026-09-20: the rule file now spans two metric namespaces.** frank-medium cut over to radiance/vLLM, so `llm-fleet.yml` carries `vllm:*` rules beside the `llamacpp:*` ones — seven alerts and two recording rules in total. KV-pool saturation and preemption pressure came back (they were retired 2026-09-16 for want of a llama.cpp gauge; vLLM exports both), and the two engine-independent rules — `LlmSpecDecodeOff`, `LlmPrefixCacheCollapse` — OR the two families and are floored on each family's own traffic counter so an idle endpoint cannot fire them. **The provisioned dashboard is the known gap:** its panels are still keyed to `llamacpp:*` series, so frank-medium's row went blank at the cutover while frank-small's and frank-large's still draw. Re-cut pending — the KV-fill and latency panels that were retired for llama.cpp have vLLM equivalents again (`vllm:kv_cache_usage_perc`, `vllm:inter_token_latency_seconds`).
- **Loki is configured but unread.** Storage is filesystem with 30-day retention, `auth_enabled: false`, and the push path works; Grafana can now reach it in Explore, but no dashboard or alert consumes it.
- **node-exporter runs its default collector set** — no collector tuning, no textfile collector for custom metrics.
- **Alloy is the one configured piece.** It ships the host journal and podman container logs to Loki with `job` labels, and that pipeline is live.
@@ -88,26 +89,31 @@ The rule is the same as for every other service: the only off-host path is a gua
| `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on `dankclaw_default` |
| `comfyui` | `comfyui:8188` | UI probe, no native exporter |
| `services` | `192.168.0.143:8081`, `:3005` | SearXNG, OpenWebUI up/down probes |
| `frank-medium` | `192.168.0.235:7770` | llama.cpp on Dankserver — `llamacpp:` counters, including the spec-decode pair |
| `frank-small` | `192.168.0.143:7771` | llama.cpp on Dankclaw's RTX 3090 — `llamacpp:` counters, including the spec-decode pair |
| `frank-medium` | `192.168.0.235:7770` | radiance/vLLM on Dankserver — `vllm:` counters: KV fill, queue depth, preemptions, prefix-cache reuse, and the `spec_decode_*` trio |
| `frank-small` | `host.containers.internal:7771` | llama.cpp on Dankclaw's RTX 3090 — `llamacpp:` counters, including the spec-decode pair |
| `frank-large` | `host.containers.internal:7770` | CIRU llama.cpp on Dankclaw's iGPU — 19 `llamacpp:` families |
The LLM endpoints were added on 2026-09-16, after a night spent diagnosing
frank-medium by hand, and re-cut the same day when Dankclaw split into two
endpoints. The vLLM runtime behind that diagnosis is retired; frank-medium
now runs mainline llama.cpp, and the endpoint exports llama.cpp's own
counters — prompt and decode throughput, KV fill, and the `spec_decode_*`
pair that proves the drafter is running (draft-mtp on both endpoints as of 2026-09-17).
this hardware is in [[frank-medium-llamacpp-dflash2]]; the same counter
family on frank-small is read against [[frank-small-qwen38-gsq-rco]].
endpoints. frank-medium has since changed runtime twice (vLLM → mainline
llama.cpp → radiance/vLLM on 2026-09-20), and the metric family it exports
changed with it: `llamacpp:*` while it ran llama.cpp, `vllm:*` now.
**`frank-large` is deliberately NOT scraped.** The halogen engine publishes
`/health`, `/cache` and `/v1/*` and no Prometheus text endpoint at all, so a
target there would be red by construction and would fire `LlmEndpointDown`
forever — the same class of defect as the page-only probes below. Its liveness
is on the homepage (a loopback probe) and in the engine's own boot ledger.
**Both Dankclaw targets are addressed as `host.containers.internal`
(169.254.1.2), never the LAN IP.** frank-large and frank-small are *host
processes* on Dankclaw, and netavark DNATs only *published* container ports —
from the Prometheus container `192.168.0.143:7770/:7771` is refused even
though it answers from every other vantage on the LAN. Both targets were red
by construction until a live `/targets` audit caught it on 2026-09-20.
`frank-medium` keeps its LAN address: it is a rootless Quadlet on another host
that publishes 7770, so its published-port DNAT does apply.
Both scraped endpoints are reached on their LAN addresses, never through the
guarded vhosts: Prometheus cannot solve an Anubis challenge.
**`frank-large` IS scraped as of 2026-09-20.** The earlier claim that it had
"no Prometheus text endpoint at all" was true of the halogen engine it ran
then; the CIRU llama.cpp build that replaced it exports the full
`llamacpp:*` family, which is what makes the target addressable at all. Its
liveness is additionally on the homepage (a loopback probe) and in the
engine's own boot ledger.
For services without a native exporter, use the Prometheus **blackbox exporter** in probe mode to assert up/down and HTTP latency. The homepage status timer probes the same endpoints from the host.