fix(frank-zealot): serve /metrics so Prometheus can see the endpoint

The scrape has never worked: lastError is "server returned HTTP status 501 Not
Implemented" on http://host.containers.internal:7771/metrics, and the oldest
retained sample of up{job="frank-zealot"} (2026-10-11) is already 0 -- the target
has been red across four payload generations while /health answered 200 the whole
time, keeping the critical LlmEndpointDown alert permanently active. llama.cpp
gates /metrics behind --metrics; halogen serves it natively, which is why the
asymmetry looked like a 3090-only problem.

METRICS=1 in the shipped env, --metrics reaches the argv through EXTRA_ARGS, and
the seam asserts both directions (default ships the flag, METRICS=0 strips it).
Diagnosed via /api/v1/targets lastError rather than the alert list, which is also
the lesson: Alertmanager here has no persistence, so a reboot re-stamps startsAt
and a weeks-old fault looks fresh.
This commit is contained in:
Frank
2026-10-18 01:52:19 -07:00
parent 34a30d8505
commit a4fe5f6839
6 changed files with 75 additions and 1 deletions
+12 -1
View File
@@ -153,7 +153,9 @@ alertmanager.container`, loopback :9093) → host-side Signal relay
IP, the one bind a container can reach as `host.containers.internal`;
iptables admits loopback, the host and the podman bridge subnet only) → signal-cli account 318311.
Prometheus targets read down for structural reasons (HTML-scraped "probes",
Gitea metrics off); the remedy is a blackbox exporter. See
Gitea metrics off); the remedy is a blackbox exporter. `frank-zealot` was down for
a different reason until 2026-10-18: llama.cpp will not serve `/metrics` at all
without `--metrics`. See
`serverAppConfigs-docs/Infrastructure/Observability.md`.
## Testing
@@ -1091,6 +1093,15 @@ linked runbooks.
B/token (the `GGML_CUDA_FA_ALL_QUANTS` door), so a build decision must be
proven by the KV buffer line in the boot ledger, not by the build flags you
remember passing.
- **A llama.cpp endpoint is invisible to Prometheus unless `--metrics` is passed.**
Without it the engine answers **501 Not Implemented** on `/metrics`, so the scrape
fails while `/health` answers 200 and `LlmEndpointDown` fires beside a healthy
service — that is how `frank-zealot` sat red across four payload generations
(the oldest retained sample of `up{job="frank-zealot"}`, 2026-10-11, is already
0). halogen serves `/metrics` natively, so the asymmetry reads as "only the 3090
is broken". Diagnose with `lastError` from `/api/v1/targets`, not the alert list,
and note Alertmanager has no persistence: a reboot re-stamps `startsAt` on an
alert that has been firing for weeks.
**Ops**