observability: scrape the LLM fleet into Prometheus

Neither LLM endpoint was scraped, and frank-medium exports 418 vllm: series —
including the ones this fleet argues about under load: kv_cache_usage_perc (how
full the per-card pool is), prefix_cache_hit_rate (whether pinned conversations
are being reused), num_requests_waiting (queue depth) and the request-latency
histograms. Everything diagnosed tonight by hand — 122,636 KV tokens per card,
thrash when three large contexts overlap, cold-vs-warm prefix reuse — is a
graph now, on the LAN addresses rather than the guarded vhosts.

frank-large comes along in the same change: llama.cpp exposes throughput
counters under llamacpp:, and watching one LLM endpoint without the other was
arbitrary.

The seam added to observability_quadlet.bats pins the cross-file contract that
breaks silently: if a scrape target drifts from the :7770 the units publish,
the graphs vanish with no error anywhere.
This commit is contained in:
Frank
2026-09-16 06:32:33 -07:00
parent 4f2f49d2d0
commit 42b47063ec
2 changed files with 35 additions and 0 deletions
+14
View File
@@ -285,3 +285,17 @@ PAIRS
[ "$output" = "$svc" ] || { echo "$svc planned as '$output'"; return 1; }
done
}
@test "prometheus scrapes both LLM endpoints on the fleet's :7770" {
# The LLM endpoints are the services whose behaviour this fleet argues about
# most (KV pressure, prefix-cache reuse, queue depth), and vLLM exports all
# of it. If a scrape target drifts from the endpoint the units publish,
# nobody notices until someone needs the graph during an incident.
local cfg="$REPO_ROOT/shared/prometheus/prometheus.yml"
grep -q '"192.168.0.235:7770"' "$cfg" || { echo "frank-medium not scraped"; return 1; }
grep -q '"192.168.0.143:7770"' "$cfg" || { echo "frank-large not scraped"; return 1; }
# The medium endpoint is exported by the proxy on the host network; the
# quadlet's own contract test pins that port, so name the job too.
grep -q 'job_name: "frank-medium"' "$cfg"
! grep -qE '^\s*-\s*"?https?://[^"]*franksdank' "$cfg"
}