fix(frank-zealot): serve /metrics so Prometheus can see the endpoint
The scrape has never worked: lastError is "server returned HTTP status 501 Not Implemented" on http://host.containers.internal:7771/metrics, and the oldest retained sample of up{job="frank-zealot"} (2026-10-11) is already 0 -- the target has been red across four payload generations while /health answered 200 the whole time, keeping the critical LlmEndpointDown alert permanently active. llama.cpp gates /metrics behind --metrics; halogen serves it natively, which is why the asymmetry looked like a 3090-only problem. METRICS=1 in the shipped env, --metrics reaches the argv through EXTRA_ARGS, and the seam asserts both directions (default ships the flag, METRICS=0 strips it). Diagnosed via /api/v1/targets lastError rather than the alert list, which is also the lesson: Alertmanager here has no persistence, so a reboot re-stamps startsAt and a weeks-old fault looks fresh.
This commit is contained in:
@@ -153,7 +153,9 @@ alertmanager.container`, loopback :9093) → host-side Signal relay
|
||||
IP, the one bind a container can reach as `host.containers.internal`;
|
||||
iptables admits loopback, the host and the podman bridge subnet only) → signal-cli account 318311.
|
||||
Prometheus targets read down for structural reasons (HTML-scraped "probes",
|
||||
Gitea metrics off); the remedy is a blackbox exporter. See
|
||||
Gitea metrics off); the remedy is a blackbox exporter. `frank-zealot` was down for
|
||||
a different reason until 2026-10-18: llama.cpp will not serve `/metrics` at all
|
||||
without `--metrics`. See
|
||||
`serverAppConfigs-docs/Infrastructure/Observability.md`.
|
||||
|
||||
## Testing
|
||||
@@ -1091,6 +1093,15 @@ linked runbooks.
|
||||
B/token (the `GGML_CUDA_FA_ALL_QUANTS` door), so a build decision must be
|
||||
proven by the KV buffer line in the boot ledger, not by the build flags you
|
||||
remember passing.
|
||||
- **A llama.cpp endpoint is invisible to Prometheus unless `--metrics` is passed.**
|
||||
Without it the engine answers **501 Not Implemented** on `/metrics`, so the scrape
|
||||
fails while `/health` answers 200 and `LlmEndpointDown` fires beside a healthy
|
||||
service — that is how `frank-zealot` sat red across four payload generations
|
||||
(the oldest retained sample of `up{job="frank-zealot"}`, 2026-10-11, is already
|
||||
0). halogen serves `/metrics` natively, so the asymmetry reads as "only the 3090
|
||||
is broken". Diagnose with `lastError` from `/api/v1/targets`, not the alert list,
|
||||
and note Alertmanager has no persistence: a reboot re-stamps `startsAt` on an
|
||||
alert that has been firing for weeks.
|
||||
|
||||
**Ops**
|
||||
|
||||
|
||||
Reference in New Issue
Block a user