observability: scrape the LLM fleet into Prometheus
Neither LLM endpoint was scraped, and frank-medium exports 418 vllm: series — including the ones this fleet argues about under load: kv_cache_usage_perc (how full the per-card pool is), prefix_cache_hit_rate (whether pinned conversations are being reused), num_requests_waiting (queue depth) and the request-latency histograms. Everything diagnosed tonight by hand — 122,636 KV tokens per card, thrash when three large contexts overlap, cold-vs-warm prefix reuse — is a graph now, on the LAN addresses rather than the guarded vhosts. frank-large comes along in the same change: llama.cpp exposes throughput counters under llamacpp:, and watching one LLM endpoint without the other was arbitrary. The seam added to observability_quadlet.bats pins the cross-file contract that breaks silently: if a scrape target drifts from the :7770 the units publish, the graphs vanish with no error anywhere.
This commit is contained in:
@@ -53,3 +53,24 @@ scrape_configs:
|
||||
- targets:
|
||||
- "192.168.0.143:8081" # SearXNG (via the Gluetun tunnel)
|
||||
- "192.168.0.143:3005" # OpenWebUI
|
||||
|
||||
# LLM fleet. Scraped on the LAN addresses, not the guarded vhosts: these are
|
||||
# machine clients and Prometheus cannot solve an Anubis challenge.
|
||||
# frank-medium is vLLM, and its series are the ones that explain behaviour
|
||||
# under load — vllm:kv_cache_usage_perc (how full the pool is),
|
||||
# vllm:prefix_cache_hit_rate (whether pinned conversations are being reused),
|
||||
# vllm:num_requests_waiting (queue depth), and the request-latency histograms.
|
||||
# frank-large is llama.cpp and exposes throughput counters under llamacpp:.
|
||||
- job_name: "frank-medium"
|
||||
metrics_path: "/metrics"
|
||||
static_configs:
|
||||
- targets: ["192.168.0.235:7770"]
|
||||
labels:
|
||||
service: frank-medium
|
||||
|
||||
- job_name: "frank-large"
|
||||
metrics_path: "/metrics"
|
||||
static_configs:
|
||||
- targets: ["192.168.0.143:7770"]
|
||||
labels:
|
||||
service: frank-large
|
||||
|
||||
Reference in New Issue
Block a user