docs: record the alerting rules and what they still cannot do

Both pages claimed there was no alerting; there are six rules now, so they say
which rules exist, that promtool and /api/v1/rules both confirm them, and — in
the same breath — that no rule reaches a person, because this stack has no
Alertmanager and no contact point. The dashboard's ALERTS panel is named as the
only place an alert becomes visible without opening Prometheus.

serverAppConfigs-docs/Infrastructure/Observability.md

@@ -36,12 +36,13 @@ graph LR
 Recorded 2026-09-11, on the day the stack moved to Quadlet units. Everything in the table above is deployed, healthy and boot-persistent. Almost nothing is wired up beyond the default install:
 
 - **Grafana has a datasource and one dashboard as of 2026-09-16; before that it had neither.** `shared/grafana/provisioning/` is installed to `/opt/grafana/provisioning` by the deployer and read by the container: the Prometheus datasource (`http://prometheus:9090`, fixed uid `prometheus`) and the **LLM fleet — capacity and reuse** dashboard (KV pool fill, queue depth, prefix-cache reuse, latency p95, throughput) covering both LLM endpoints. Verified through the API rather than by eye — the datasource answers `vllm:kv_cache_usage_perc` for `frank-medium`, and `GET /api/search?type=dash-db` returns uid `llm-fleet`. One trap worth knowing: a provisioned dashboard does **not** appear in the legacy `dashboard` table on this Grafana version (it lives in unified storage behind `dashboard.grafana.app`), so the API is the way to check. Still outstanding: no dashboards beyond this one, no Loki queries, and only datasources+dashboards are provisioning-as-code.
-- **Prometheus has no alerting at all.** `/opt/prometheus/config/` contains `prometheus.yml` and nothing else — no rule files, no recording rules — and there is no Alertmanager anywhere in the stack. It scrapes and retains; it never notifies anyone.
+- **Grafana's dashboard now also surfaces firing alerts.** The **Alerts firing (ALERTS)** panel graphs `ALERTS{alertstate="firing"}` for the LLM rules, because this stack has no other place where an alert becomes visible without opening Prometheus.
+- **Prometheus now evaluates rules; nothing delivers them (2026-09-16).** `shared/prometheus/rules/llm-fleet.yml` ships with the config (the deployer's `copy_prometheus_config` installs both — a rule file nothing loads evaluates nothing, and Prometheus reports no error for that) and holds six rules for the LLM fleet: pool saturation, request queueing, prefix-cache collapse, endpoint down, p95 latency, plus the `job:llm_prefix_cache_reuse:rate5m` recording rule the alert and the dashboard share. Verified with `promtool check rules` (SUCCESS, 6 rules) and `/api/v1/rules` (all six `health=ok`). **There is still no Alertmanager and no contact point anywhere**: a firing alert appears on Prometheus' `/alerts`, in the `ALERTS` metric and on the LLM fleet dashboard's alerts panel — and reaches a human only if one is looking.
 - **Loki is configured but unread.** Storage is filesystem with 30-day retention, `auth_enabled: false`, and the push path works; Grafana can now reach it in Explore, but no dashboard or alert consumes it.
 - **node-exporter runs its default collector set** — no collector tuning, no textfile collector for custom metrics.
 - **Alloy is the one configured piece.** It ships the host journal and podman container logs to Loki with `job` labels, and that pipeline is live.
 
-The honest summary: metrics and logs are collected and retained, one dashboard now visualizes the LLM fleet, and **nothing is alerted on** — there is no rules file and no Alertmanager anywhere, so every graph in this stack is read after the fact by a human, and correlating metrics with logs is still manual.
+The honest summary: metrics and logs are collected and retained, one dashboard visualizes the LLM fleet, six rules evaluate over it, and **no rule reaches a person** — there is no Alertmanager and no contact point, so correlation stays manual and every signal has to be looked at.
 
 ## Deployment
 

serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md

@@ -418,8 +418,16 @@ scraped — Prometheus has `frank-medium` and `frank-large` jobs
 (queue depth), `rate(vllm:prefix_cache_hits_total[5m]) /
  rate(vllm:prefix_cache_queries_total[5m])` (whether pinned conversations are
 served warm), and the request-latency histograms. See
-[[Observability on Dankclaw]]. There is still **no alerting** in this stack, so
-a saturated pool will not page anyone — it shows up on a graph after the fact.
+[[Observability on Dankclaw]], and one Grafana dashboard reads them
+(**LLM fleet — capacity and reuse**).
+
+Five rules watch this service (`shared/prometheus/rules/llm-fleet.yml`): pool
+saturation above 90 %, requests queueing for 10 minutes, prefix reuse collapsing
+below 20 % with traffic, the scrape target down, and p95 latency above 60 s —
+each threshold tied to a measurement on this page rather than to a guess. They
+evaluate and show up on the dashboard's **Alerts firing** panel, but **nothing
+is notified**: this stack has no Alertmanager, so an alert reaches a person only
+if that person is looking. That is the honest state of "alerting" here.
 
 ## Citations