fix(frank-zealot): serve /metrics so Prometheus can see the endpoint
The scrape has never worked: lastError is "server returned HTTP status 501 Not Implemented" on http://host.containers.internal:7771/metrics, and the oldest retained sample of up{job="frank-zealot"} (2026-10-11) is already 0 -- the target has been red across four payload generations while /health answered 200 the whole time, keeping the critical LlmEndpointDown alert permanently active. llama.cpp gates /metrics behind --metrics; halogen serves it natively, which is why the asymmetry looked like a 3090-only problem. METRICS=1 in the shipped env, --metrics reaches the argv through EXTRA_ARGS, and the seam asserts both directions (default ships the flag, METRICS=0 strips it). Diagnosed via /api/v1/targets lastError rather than the alert list, which is also the lesson: Alertmanager here has no persistence, so a reboot re-stamps startsAt and a weeks-old fault looks fresh.
This commit is contained in:
@@ -87,6 +87,21 @@ via `host.containers.internal`) → signal-cli. Some targets read down for
|
||||
structural reasons (HTML-scraped "probes", Gitea metrics off); the remedy is
|
||||
the blackbox exporter. See [[Observability]].
|
||||
|
||||
**A llama.cpp endpoint is invisible unless it opts in.** `up{job="frank-zealot"}`
|
||||
has been 0 for the entire retained history of the metric — the oldest sample,
|
||||
2026-10-11, is already 0 — while the service itself answered `/health` 200 the
|
||||
whole time. Cause: llama.cpp answers **501 Not Implemented** on `/metrics` unless
|
||||
the server is started with `--metrics`, so the scrape was refused at the protocol
|
||||
layer and `LlmEndpointDown` (critical) sat active across four payload generations.
|
||||
Fixed 2026-10-18 by the `METRICS=1` knob in `scripts/frank-zealot.env`. Two laws
|
||||
fall out of it: a target can read down beside a healthy service, so read
|
||||
`lastError` from `/api/v1/targets` before believing an outage; and Alertmanager
|
||||
runs here without persistence, so every restart re-stamps `startsAt` on a
|
||||
long-standing alert — a "fresh" alert after a reboot is not evidence of a fresh
|
||||
fault. Remaining known downs are the structural ones above (HTML-scraped probes,
|
||||
Gitea metrics, ComfyUI), and halogen serves `/metrics` natively, which is why
|
||||
`frank-dragoon` has always been green.
|
||||
|
||||
## Related
|
||||
|
||||
- [[Observability]] — stack topology and the alert delivery chain
|
||||
|
||||
Reference in New Issue
Block a user