observability: the LLM surfaces follow the two-endpoint split

frank-large has no /metrics to scrape any more, and scraping it would be red by
construction: the halogen engine publishes /health, /cache and /v1/* only. That
job is removed with the reason written down (the remedy is the blackbox
exporter this fleet already wants for the comfyui/services probes), and
frank-small's llama.cpp endpoint takes the slot at 192.168.0.143:7771.

Homepage: frank-large's card still described the ik runtime, and the new
endpoint had no card at all. Both fixed, with a loopback probe -- the checker
runs on Dankclaw, which is also what makes the card green-able.

Dashboard: the endpoint picker was sourced from a vLLM-only metric, so a
llama.cpp job could never be selected, and the prose named frank-large as the
llama.cpp endpoint. The picker now matches both runtimes' series and the
llama.cpp targets carry the job filter every other panel already used.

The seams assert the new contract including the ABSENCE of a 7770 target, so a
well-meaning edit cannot re-add a permanently-red probe.
This commit is contained in:
Frank
2026-09-16 13:59:49 -07:00
parent 43ec77fee6
commit 79cca19919
6 changed files with 57 additions and 22 deletions
+8 -7
View File
@@ -41,18 +41,19 @@ PY
}
@test "the LLM status cards track the fleet, not retired service names" {
# This drifted: the card probing Dankserver's :7770 was still named
# frank-small long after frank-small was retired and frank-medium took that
# endpoint, so the homepage labelled the fleet's busiest service with a dead
# name. The probe target was right; only the identity was wrong.
# This drifted twice: the card probing Dankserver's :7770 was still named
# frank-small long after frank-medium took that endpoint, and Dankclaw's
# second endpoint now exists again as a real service on its own port. The
# probe target is what matters; the name has to follow it.
python3 - "$REPO_ROOT/shared/homepage/services.json" <<'PY'
import json, sys
services = json.load(open(sys.argv[1]))
by_probe = {s["probe"]: s["name"] for s in services}
medium = by_probe.get("192.168.0.235:7770")
assert medium == "frank-medium", f"Dankserver :7770 card is named {medium!r}"
assert "frank-small" not in {s["name"] for s in services}, "retired frank-small still listed"
# The big endpoint keeps its card too, so a fleet overview shows both.
assert by_probe.get("127.0.0.1:7770") == "frank-large"
# Dankclaw serves two endpoints on two ports: frank-large (halogen, 7770) and
# frank-small (llama.cpp on the RTX 3090, 7771).
assert by_probe.get("127.0.0.1:7770") == "frank-large", by_probe.get("127.0.0.1:7770")
assert by_probe.get("127.0.0.1:7771") == "frank-small", by_probe.get("127.0.0.1:7771")
PY
}
+8 -2
View File
@@ -295,17 +295,23 @@ PAIRS
done
}
@test "prometheus scrapes both LLM endpoints on the fleet's :7770" {
@test "prometheus scrapes every LLM endpoint that exports metrics" {
# The LLM endpoints are the services whose behaviour this fleet argues about
# most (KV pressure, prefix-cache reuse, queue depth), and vLLM exports all
# of it. If a scrape target drifts from the endpoint the units publish,
# nobody notices until someone needs the graph during an incident.
local cfg="$REPO_ROOT/shared/prometheus/prometheus.yml"
grep -q '"192.168.0.235:7770"' "$cfg" || { echo "frank-medium not scraped"; return 1; }
grep -q '"192.168.0.143:7770"' "$cfg" || { echo "frank-large not scraped"; return 1; }
grep -q '"192.168.0.143:7771"' "$cfg" || { echo "frank-small not scraped"; return 1; }
# The medium endpoint is exported by the proxy on the host network; the
# quadlet's own contract test pins that port, so name the job too.
grep -q 'job_name: "frank-medium"' "$cfg"
grep -q 'job_name: "frank-small"' "$cfg"
# frank-large is NOT scraped on purpose: the halogen engine serves
# /health, /cache and /v1/* and no metrics endpoint, so a target here would
# be red by construction and would fire LlmEndpointDown forever. Assert the
# absence, so a well-meaning edit cannot re-add a permanently-red probe.
! grep -q '192.168.0.143:7770' "$cfg"
! grep -qE '^\s*-\s*"?https?://[^"]*franksdank' "$cfg"
}