alerting: Alertmanager + Signal relay deliver firing rules to Frank's phone
- dankclaw/observability/quadlet/alertmanager.container: loopback :9093, config from shared/prometheus/alertmanager.yml, family conventions - shared/prometheus/alertmanager.yml: single signal receiver, group_by alertname, 30s/5m/4h intervals, webhook to host.containers.internal:18093 - scripts/signal-relay.py + .service: host-side user unit, loopback :18093, 15-min dedupe, sends via signal-cli account 318311 - prometheus.yml: alerting block to alertmanager:9093; stale comments fixed - llm-fleet.yml: delivery header; LlmSpecDecodeOff annotation now drafter-agnostic - deploy.sh/deploy-remote.sh/deploy-plan.py: alertmanager service, config copy, relay install, planner routing for config + relay files - homepage: alertmanager status card (loopback probe) - tests: alertmanager in UNITS/loopback/health/PAIRS/planner + five new delivery-path seams (webhook<->relay contract, prometheus target, relay unit, signal-cli account, planner routing) - docs: Alertmanager.md component page, Observability.md, Prometheus.md, index.md, AGENTS.md delivery line
This commit is contained in:
1 parent
17450dd172
commit
590853c133
17 files changed
+513
-51
No files matched your search
@@ -14,12 +14,13 @@
|
||||
# LlmSpecDecodeOff takes its place: the failure an operator actually
|
||||
# cannot see from the outside on this deployment.
|
||||
#
|
||||
# IMPORTANT: nothing is notified. There is no Alertmanager in this stack, so a
|
||||
# firing alert appears on Prometheus' /alerts page, in the ALERTS metric (which
|
||||
# the LLM fleet dashboard graphs) and nowhere else. Turning these into something
|
||||
# that reaches a person means adding an Alertmanager or using Grafana's own
|
||||
# alerting with a contact point — a decision about how Frank wants to be paged,
|
||||
# not a missing line of configuration.
|
||||
# IMPORTANT: delivery. Since 2026-09-17 a firing alert reaches Frank on
|
||||
# Signal: Prometheus pushes to Alertmanager (dankclaw/observability/quadlet/
|
||||
# alertmanager.container), whose single receiver webhooks the host-side
|
||||
# signal-relay user unit, which hands the text to signal-cli under the
|
||||
# existing account on this host. Before that date a firing alert appeared on
|
||||
# Prometheus' /alerts page, in the ALERTS metric (which the LLM fleet
|
||||
# dashboard graphs) and nowhere else.
|
||||
groups:
|
||||
- name: llm-fleet
|
||||
interval: 30s
|
||||
@@ -77,9 +78,11 @@ groups:
|
||||
summary: "{{ $labels.job }}: serving without speculative decoding"
|
||||
description: >-
|
||||
Tokens are being produced but no draft has been proposed for 15
|
||||
minutes. On frank-medium that means the DFlash2 drafter is not
|
||||
running (check the unit's log for common_speculative_impl_draft_dflash),
|
||||
and the endpoint is decoding autoregressively.
|
||||
minutes. On the llama.cpp endpoints that means the drafter is not
|
||||
running (check the unit's log for the
|
||||
"adding speculative implementation" line — draft-mtp on both
|
||||
endpoints as of 2026-09-17), and the endpoint is decoding
|
||||
autoregressively.
|
||||
|
||||
# The endpoint, not a replica: one process serves each port now, so a
|
||||
# restart of the unit is a scrape gap, not a failover.
|
||||
|
||||
Reference in new issue
Block a user