diff --git a/serverAppConfigs-docs/Infrastructure/Observability.md b/serverAppConfigs-docs/Infrastructure/Observability.md index eeed3b1..0f71a97 100644 --- a/serverAppConfigs-docs/Infrastructure/Observability.md +++ b/serverAppConfigs-docs/Infrastructure/Observability.md @@ -36,12 +36,13 @@ graph LR Recorded 2026-09-11, on the day the stack moved to Quadlet units. Everything in the table above is deployed, healthy and boot-persistent. Almost nothing is wired up beyond the default install: - **Grafana has a datasource and one dashboard as of 2026-09-16; before that it had neither.** `shared/grafana/provisioning/` is installed to `/opt/grafana/provisioning` by the deployer and read by the container: the Prometheus datasource (`http://prometheus:9090`, fixed uid `prometheus`) and the **LLM fleet — capacity and reuse** dashboard (KV pool fill, queue depth, prefix-cache reuse, latency p95, throughput) covering both LLM endpoints. Verified through the API rather than by eye — the datasource answers `vllm:kv_cache_usage_perc` for `frank-medium`, and `GET /api/search?type=dash-db` returns uid `llm-fleet`. One trap worth knowing: a provisioned dashboard does **not** appear in the legacy `dashboard` table on this Grafana version (it lives in unified storage behind `dashboard.grafana.app`), so the API is the way to check. Still outstanding: no dashboards beyond this one, no Loki queries, and only datasources+dashboards are provisioning-as-code. -- **Prometheus has no alerting at all.** `/opt/prometheus/config/` contains `prometheus.yml` and nothing else — no rule files, no recording rules — and there is no Alertmanager anywhere in the stack. It scrapes and retains; it never notifies anyone. +- **Grafana's dashboard now also surfaces firing alerts.** The **Alerts firing (ALERTS)** panel graphs `ALERTS{alertstate="firing"}` for the LLM rules, because this stack has no other place where an alert becomes visible without opening Prometheus. +- **Prometheus now evaluates rules; nothing delivers them (2026-09-16).** `shared/prometheus/rules/llm-fleet.yml` ships with the config (the deployer's `copy_prometheus_config` installs both — a rule file nothing loads evaluates nothing, and Prometheus reports no error for that) and holds six rules for the LLM fleet: pool saturation, request queueing, prefix-cache collapse, endpoint down, p95 latency, plus the `job:llm_prefix_cache_reuse:rate5m` recording rule the alert and the dashboard share. Verified with `promtool check rules` (SUCCESS, 6 rules) and `/api/v1/rules` (all six `health=ok`). **There is still no Alertmanager and no contact point anywhere**: a firing alert appears on Prometheus' `/alerts`, in the `ALERTS` metric and on the LLM fleet dashboard's alerts panel — and reaches a human only if one is looking. - **Loki is configured but unread.** Storage is filesystem with 30-day retention, `auth_enabled: false`, and the push path works; Grafana can now reach it in Explore, but no dashboard or alert consumes it. - **node-exporter runs its default collector set** — no collector tuning, no textfile collector for custom metrics. - **Alloy is the one configured piece.** It ships the host journal and podman container logs to Loki with `job` labels, and that pipeline is live. -The honest summary: metrics and logs are collected and retained, one dashboard now visualizes the LLM fleet, and **nothing is alerted on** — there is no rules file and no Alertmanager anywhere, so every graph in this stack is read after the fact by a human, and correlating metrics with logs is still manual. +The honest summary: metrics and logs are collected and retained, one dashboard visualizes the LLM fleet, six rules evaluate over it, and **no rule reaches a person** — there is no Alertmanager and no contact point, so correlation stays manual and every signal has to be looked at. ## Deployment diff --git a/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md b/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md index 9d2d2ee..38cbd5b 100644 --- a/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md +++ b/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md @@ -418,8 +418,16 @@ scraped — Prometheus has `frank-medium` and `frank-large` jobs (queue depth), `rate(vllm:prefix_cache_hits_total[5m]) / rate(vllm:prefix_cache_queries_total[5m])` (whether pinned conversations are served warm), and the request-latency histograms. See -[[Observability on Dankclaw]]. There is still **no alerting** in this stack, so -a saturated pool will not page anyone — it shows up on a graph after the fact. +[[Observability on Dankclaw]], and one Grafana dashboard reads them +(**LLM fleet — capacity and reuse**). + +Five rules watch this service (`shared/prometheus/rules/llm-fleet.yml`): pool +saturation above 90 %, requests queueing for 10 minutes, prefix reuse collapsing +below 20 % with traffic, the scrape target down, and p95 latency above 60 s — +each threshold tied to a measurement on this page rather than to a guess. They +evaluate and show up on the dashboard's **Alerts firing** panel, but **nothing +is notified**: this stack has no Alertmanager, so an alert reaches a person only +if that person is looking. That is the honest state of "alerting" here. ## Citations