Files

T

Frank a4fe5f6839 fix(frank-zealot): serve /metrics so Prometheus can see the endpoint

The scrape has never worked: lastError is "server returned HTTP status 501 Not
Implemented" on http://host.containers.internal:7771/metrics, and the oldest
retained sample of up{job="frank-zealot"} (2026-10-11) is already 0 -- the target
has been red across four payload generations while /health answered 200 the whole
time, keeping the critical LlmEndpointDown alert permanently active. llama.cpp
gates /metrics behind --metrics; halogen serves it natively, which is why the
asymmetry looked like a 3090-only problem.

METRICS=1 in the shipped env, --metrics reaches the argv through EXTRA_ARGS, and
the seam asserts both directions (default ships the flag, METRICS=0 strips it).
Diagnosed via /api/v1/targets lastError rather than the alert list, which is also
the lesson: Alertmanager here has no persistence, so a reboot re-stamps startsAt
and a weeks-old fault looks fresh.

2026-10-18 01:52:19 -07:00

5.5 KiB

Raw Blame History

type, title, description, tags, timestamp

type

title

description

Observability Provisioning

The monitoring stack is provisioned as code: dashboards, datasources, scrape jobs and alert rules live in the repo and ride the normal deploy flow. No hand-edits inside containers — they are wiped on recreate and never committed.

The config triad

Repo path	What it holds
`shared/prometheus/prometheus.yml`	scrape jobs (global + per-host)
`shared/prometheus/rules/llm-fleet.yml`	alert rules + recording rules
`shared/grafana/provisioning/`	datasources + dashboards (provisioned tree)

Targets and datasources reference container names (loki:3100, alloy:12345), never loopback-published ports; the LLM endpoints are reached by LAN IP (192.168.0.143:7770, :7771, 192.168.0.235:7770) because they are host-netns services, not containers. Prometheus 3 refuses HTML responses as metrics — HTML-scraped "probes" are structurally inert and need a blackbox exporter (see Stale Signal Audit).

The deploy path

scripts/deploy-remote.sh carries two copier functions:

copy_prometheus_config → /opt/prometheus/config/ on the target
copy_grafana_provisioning → /opt/grafana/provisioning/

scripts/deploy-plan.py maps changes under shared/prometheus/** and shared/grafana/** to the prometheus/grafana services, so deploy.sh dankclaw --service grafana (etc.) restarts only what the change touched. Seam tests in tests/observability_quadlet.bats pin: unit files exist, the copier functions exist, and the provisioned paths match the units' mounts.

Entry points

Add a scrape target — edit shared/prometheus/prometheus.yml. Decide the vantage first: container name if the exporter is a container on the same netns; host LAN IP if it is a host process (LLM endpoints); a blackbox module if it is an HTTP probe, not a metrics endpoint.

Add an alert/recording rule — edit shared/prometheus/rules/llm-fleet.yml. Current delivered set: 5 alert rules + 1 recording rule for the LLM fleet. Rules reload with the prometheus service.

Add a datasource — drop a YAML file into shared/grafana/provisioning/datasources/. Point it at the container name of the backend (http://prometheus:9090), not loopback.

Add a dashboard — drop the JSON into shared/grafana/provisioning/dashboards/. Provisioned dashboards live in unified storage, not (only) the SQLite table — grep the API, not the DB, when verifying (see the verify section of Fleet Audit and Posture).

Verify through the APIs, not the filesystem

# prometheus: rules and targets actually loaded
curl -s http://127.0.0.1:9090/api/v1/rules | python3 -m json.tool | head
curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | \
  python3 -c "import json,sys; [print(t['labels'].get('job'), t['health'], (t['lastError'] or '')[:60]) for t in json.load(sys.stdin)['data']['activeTargets']]"
# grafana: datasource + dashboard provisioned
curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources | python3 -m json.tool | grep -E '"name"|"url"'

Health answers from a stale process and provisioned resources can hide in unified storage — a file existing on disk is not proof the running server loaded it. See Stale Signal Audit for the never-green-target decision table (refused = outage, 404/HTML = config defect).

Current delivered state (2026-09-17)

Prometheus → Alertmanager (dankclaw/observability/quadlet/alertmanager.container, loopback :9093) → host-side Signal relay (scripts/signal-relay.py, user unit, 192.168.0.143:18093 — the LAN IP, the one bind a container can reach via host.containers.internal) → signal-cli. Some targets read down for structural reasons (HTML-scraped "probes", Gitea metrics off); the remedy is the blackbox exporter. See Observability.

A llama.cpp endpoint is invisible unless it opts in. up{job="frank-zealot"} has been 0 for the entire retained history of the metric — the oldest sample, 2026-10-11, is already 0 — while the service itself answered /health 200 the whole time. Cause: llama.cpp answers 501 Not Implemented on /metrics unless the server is started with --metrics, so the scrape was refused at the protocol layer and LlmEndpointDown (critical) sat active across four payload generations. Fixed 2026-10-18 by the METRICS=1 knob in scripts/frank-zealot.env. Two laws fall out of it: a target can read down beside a healthy service, so read lastError from /api/v1/targets before believing an outage; and Alertmanager runs here without persistence, so every restart re-stamps startsAt on a long-standing alert — a "fresh" alert after a reboot is not evidence of a fresh fault. Remaining known downs are the structural ones above (HTML-scraped probes, Gitea metrics, ComfyUI), and halogen serves /metrics natively, which is why frank-dragoon has always been green.

Observability — stack topology and the alert delivery chain
Stale Signal Audit — classifying down targets before "fixing" them
Quadlet Lifecycle — the units that host these services
Deploy State Triage — how the planner selects these services

5.5 KiB Raw Blame History