observability: the LLM surfaces follow the two-endpoint split
frank-large has no /metrics to scrape any more, and scraping it would be red by construction: the halogen engine publishes /health, /cache and /v1/* only. That job is removed with the reason written down (the remedy is the blackbox exporter this fleet already wants for the comfyui/services probes), and frank-small's llama.cpp endpoint takes the slot at 192.168.0.143:7771. Homepage: frank-large's card still described the ik runtime, and the new endpoint had no card at all. Both fixed, with a loopback probe -- the checker runs on Dankclaw, which is also what makes the card green-able. Dashboard: the endpoint picker was sourced from a vLLM-only metric, so a llama.cpp job could never be selected, and the prose named frank-large as the llama.cpp endpoint. The picker now matches both runtimes' series and the llama.cpp targets carry the job filter every other panel already used. The seams assert the new contract including the ABSENCE of a 7770 target, so a well-meaning edit cannot re-add a permanently-red probe.
This commit is contained in:
@@ -75,9 +75,13 @@ groups:
|
||||
annotations:
|
||||
summary: "{{ $labels.job }}: scrape target down for 2 minutes"
|
||||
description: >-
|
||||
Prometheus cannot reach the LLM endpoint. Check the host's units:
|
||||
systemctl --user is-active frank-medium-{a,b,pro} on Dankclaw's
|
||||
peers (frank-large) or Dankserver (frank-medium).
|
||||
Prometheus cannot reach the LLM endpoint. Check the units:
|
||||
`systemctl --user is-active frank-medium-{a,b,proxy}` on Dankserver,
|
||||
`systemctl is-active frank-small` (the llama.cpp root unit) or
|
||||
`systemctl --user is-active frank-large` (the halogen quadlet) on
|
||||
Dankclaw. frank-large is not scraped at all — the halogen engine
|
||||
exports no metrics — so this alert can only ever name the endpoints
|
||||
that do export one.
|
||||
|
||||
# What an operator actually feels: wall-clock per request, prefill
|
||||
# included. A cold 50k-token prefill alone is ~22 s on one card.
|
||||
|
||||
Reference in New Issue
Block a user