observability: scrape the LLM fleet into Prometheus

Neither LLM endpoint was scraped, and frank-medium exports 418 vllm: series —
including the ones this fleet argues about under load: kv_cache_usage_perc (how
full the per-card pool is), prefix_cache_hit_rate (whether pinned conversations
are being reused), num_requests_waiting (queue depth) and the request-latency
histograms. Everything diagnosed tonight by hand — 122,636 KV tokens per card,
thrash when three large contexts overlap, cold-vs-warm prefix reuse — is a
graph now, on the LAN addresses rather than the guarded vhosts.

frank-large comes along in the same change: llama.cpp exposes throughput
counters under llamacpp:, and watching one LLM endpoint without the other was
arbitrary.

The seam added to observability_quadlet.bats pins the cross-file contract that
breaks silently: if a scrape target drifts from the :7770 the units publish,
the graphs vanish with no error anywhere.
This commit is contained in:
Frank
2026-09-16 06:32:33 -07:00
parent 4f2f49d2d0
commit 42b47063ec
2 changed files with 35 additions and 0 deletions
+21
View File
@@ -53,3 +53,24 @@ scrape_configs:
- targets:
- "192.168.0.143:8081" # SearXNG (via the Gluetun tunnel)
- "192.168.0.143:3005" # OpenWebUI
# LLM fleet. Scraped on the LAN addresses, not the guarded vhosts: these are
# machine clients and Prometheus cannot solve an Anubis challenge.
# frank-medium is vLLM, and its series are the ones that explain behaviour
# under load — vllm:kv_cache_usage_perc (how full the pool is),
# vllm:prefix_cache_hit_rate (whether pinned conversations are being reused),
# vllm:num_requests_waiting (queue depth), and the request-latency histograms.
# frank-large is llama.cpp and exposes throughput counters under llamacpp:.
- job_name: "frank-medium"
metrics_path: "/metrics"
static_configs:
- targets: ["192.168.0.235:7770"]
labels:
service: frank-medium
- job_name: "frank-large"
metrics_path: "/metrics"
static_configs:
- targets: ["192.168.0.143:7770"]
labels:
service: frank-large