Frank
a4fe5f6839
The scrape has never worked: lastError is "server returned HTTP status 501 Not Implemented" on http://host.containers.internal:7771/metrics, and the oldest retained sample of up{job="frank-zealot"} (2026-10-11) is already 0 -- the target has been red across four payload generations while /health answered 200 the whole time, keeping the critical LlmEndpointDown alert permanently active. llama.cpp gates /metrics behind --metrics; halogen serves it natively, which is why the asymmetry looked like a 3090-only problem. METRICS=1 in the shipped env, --metrics reaches the argv through EXTRA_ARGS, and the seam asserts both directions (default ships the flag, METRICS=0 strips it). Diagnosed via /api/v1/targets lastError rather than the alert list, which is also the lesson: Alertmanager here has no persistence, so a reboot re-stamps startsAt and a weeks-old fault looks fresh.
111 lines
5.5 KiB
Markdown
111 lines
5.5 KiB
Markdown
---
|
|
type: reference
|
|
title: Observability Provisioning
|
|
description: How Prometheus config/rules and Grafana datasources/dashboards are provisioned as code in serverAppConfigs — the config triad, deploy copiers, planner mappings, seam tests, entry points for each change type, and API verification.
|
|
tags: [observability, prometheus, grafana, provisioning, deployment]
|
|
timestamp: 2026-10-05T00:00:00-07:00
|
|
---
|
|
|
|
# Observability Provisioning
|
|
|
|
The monitoring stack is **provisioned as code**: dashboards, datasources,
|
|
scrape jobs and alert rules live in the repo and ride the normal deploy flow.
|
|
No hand-edits inside containers — they are wiped on recreate and never
|
|
committed.
|
|
|
|
## The config triad
|
|
|
|
| Repo path | What it holds |
|
|
|---|---|
|
|
| `shared/prometheus/prometheus.yml` | scrape jobs (global + per-host) |
|
|
| `shared/prometheus/rules/llm-fleet.yml` | alert rules + recording rules |
|
|
| `shared/grafana/provisioning/` | datasources + dashboards (provisioned tree) |
|
|
|
|
Targets and datasources reference **container names** (`loki:3100`,
|
|
`alloy:12345`), never loopback-published ports; the LLM endpoints are reached
|
|
by LAN IP (`192.168.0.143:7770`, `:7771`, `192.168.0.235:7770`) because they
|
|
are host-netns services, not containers. Prometheus 3 refuses HTML responses
|
|
as metrics — HTML-scraped "probes" are structurally inert and need a blackbox
|
|
exporter (see [[Stale Signal Audit]]).
|
|
|
|
## The deploy path
|
|
|
|
`scripts/deploy-remote.sh` carries two copier functions:
|
|
|
|
- `copy_prometheus_config` → `/opt/prometheus/config/` on the target
|
|
- `copy_grafana_provisioning` → `/opt/grafana/provisioning/`
|
|
|
|
`scripts/deploy-plan.py` maps changes under `shared/prometheus/**` and
|
|
`shared/grafana/**` to the `prometheus`/`grafana` services, so
|
|
`deploy.sh dankclaw --service grafana` (etc.) restarts only what the change
|
|
touched. Seam tests in `tests/observability_quadlet.bats` pin: unit files
|
|
exist, the copier functions exist, and the provisioned paths match the units'
|
|
mounts.
|
|
|
|
## Entry points
|
|
|
|
**Add a scrape target** — edit `shared/prometheus/prometheus.yml`. Decide the
|
|
vantage first: container name if the exporter is a container on the same
|
|
netns; host LAN IP if it is a host process (LLM endpoints); a blackbox module
|
|
if it is an HTTP probe, not a metrics endpoint.
|
|
|
|
**Add an alert/recording rule** — edit `shared/prometheus/rules/llm-fleet.yml`.
|
|
Current delivered set: 5 alert rules + 1 recording rule for the LLM fleet.
|
|
Rules reload with the prometheus service.
|
|
|
|
**Add a datasource** — drop a YAML file into
|
|
`shared/grafana/provisioning/datasources/`. Point it at the container name of
|
|
the backend (`http://prometheus:9090`), not loopback.
|
|
|
|
**Add a dashboard** — drop the JSON into
|
|
`shared/grafana/provisioning/dashboards/`. Provisioned dashboards live in
|
|
unified storage, not (only) the SQLite table — grep the API, not the DB, when
|
|
verifying (see the verify section of [[Fleet Audit and Posture]]).
|
|
|
|
## Verify through the APIs, not the filesystem
|
|
|
|
```bash
|
|
# prometheus: rules and targets actually loaded
|
|
curl -s http://127.0.0.1:9090/api/v1/rules | python3 -m json.tool | head
|
|
curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | \
|
|
python3 -c "import json,sys; [print(t['labels'].get('job'), t['health'], (t['lastError'] or '')[:60]) for t in json.load(sys.stdin)['data']['activeTargets']]"
|
|
# grafana: datasource + dashboard provisioned
|
|
curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources | python3 -m json.tool | grep -E '"name"|"url"'
|
|
```
|
|
|
|
Health answers from a stale process and provisioned resources can hide in
|
|
unified storage — a file existing on disk is not proof the running server
|
|
loaded it. See [[Stale Signal Audit]] for the never-green-target decision
|
|
table (refused = outage, 404/HTML = config defect).
|
|
|
|
## Current delivered state (2026-09-17)
|
|
|
|
Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container`,
|
|
loopback `:9093`) → host-side Signal relay (`scripts/signal-relay.py`, user
|
|
unit, `192.168.0.143:18093` — the LAN IP, the one bind a container can reach
|
|
via `host.containers.internal`) → signal-cli. Some targets read down for
|
|
structural reasons (HTML-scraped "probes", Gitea metrics off); the remedy is
|
|
the blackbox exporter. See [[Observability]].
|
|
|
|
**A llama.cpp endpoint is invisible unless it opts in.** `up{job="frank-zealot"}`
|
|
has been 0 for the entire retained history of the metric — the oldest sample,
|
|
2026-10-11, is already 0 — while the service itself answered `/health` 200 the
|
|
whole time. Cause: llama.cpp answers **501 Not Implemented** on `/metrics` unless
|
|
the server is started with `--metrics`, so the scrape was refused at the protocol
|
|
layer and `LlmEndpointDown` (critical) sat active across four payload generations.
|
|
Fixed 2026-10-18 by the `METRICS=1` knob in `scripts/frank-zealot.env`. Two laws
|
|
fall out of it: a target can read down beside a healthy service, so read
|
|
`lastError` from `/api/v1/targets` before believing an outage; and Alertmanager
|
|
runs here without persistence, so every restart re-stamps `startsAt` on a
|
|
long-standing alert — a "fresh" alert after a reboot is not evidence of a fresh
|
|
fault. Remaining known downs are the structural ones above (HTML-scraped probes,
|
|
Gitea metrics, ComfyUI), and halogen serves `/metrics` natively, which is why
|
|
`frank-dragoon` has always been green.
|
|
|
|
## Related
|
|
|
|
- [[Observability]] — stack topology and the alert delivery chain
|
|
- [[Stale Signal Audit]] — classifying down targets before "fixing" them
|
|
- [[Quadlet Lifecycle]] — the units that host these services
|
|
- [[Deploy State Triage]] — how the planner selects these services
|