observability: LLM alerting rules, evaluated but not yet delivered
The scraped metrics have no rules over them, so every graph was read after the
fact by a human. This adds shared/prometheus/rules/llm-fleet.yml, loaded via
rule_files and installed by the same copier that ships prometheus.yml (a rule
file nothing loads evaluates nothing, and Prometheus reports no error for it):
LlmKvPoolSaturated kv pool > 90% for 10m — the 122,636-token wall
LlmRequestsQueueing waiting > 0 for 10m — MAXSEQS=4 reached
LlmPrefixCacheCollapse reuse < 20% for 15m with traffic — the thrash signature
LlmEndpointDown scrape target down 2m — both replicas gone
LlmLatencyHigh p95 end-to-end > 60s for 10m — what an operator feels
job:llm_prefix_cache_reuse:rate5m — recording rule shared by the alert and the
dashboard, so the two cannot drift
Every threshold is tied to something measured on this fleet, not to a guess,
and the file says plainly at the top that **nothing is notified**: there is no
Alertmanager here, so a firing alert reaches Prometheus' /alerts page, the
ALERTS metric and the dashboard panel added alongside it — and nowhere else.
Choosing how Frank actually gets paged is his call, not a missing config line.
Seam covers the silent failure mode: rules exist, the config references the
directory, the deployer installs it, and the dashboard graphs ALERTS.
This commit is contained in:
@@ -201,8 +201,13 @@ copy_searxng_config() {
|
||||
}
|
||||
|
||||
copy_prometheus_config() {
|
||||
run_privileged mkdir -p /opt/prometheus/config
|
||||
run_privileged mkdir -p /opt/prometheus/config /opt/prometheus/config/rules
|
||||
run_privileged cp shared/prometheus/prometheus.yml /opt/prometheus/config/prometheus.yml
|
||||
# Rule files are read from the same mount (--config.file sits beside them),
|
||||
# so they deploy with the config that references them: a rule file without
|
||||
# its rule_files entry evaluates nothing, silently.
|
||||
run_privileged cp shared/prometheus/rules/llm-fleet.yml \
|
||||
/opt/prometheus/config/rules/llm-fleet.yml
|
||||
}
|
||||
|
||||
copy_loki_config() {
|
||||
|
||||
Reference in New Issue
Block a user