Frank
42b47063ec
Neither LLM endpoint was scraped, and frank-medium exports 418 vllm: series — including the ones this fleet argues about under load: kv_cache_usage_perc (how full the per-card pool is), prefix_cache_hit_rate (whether pinned conversations are being reused), num_requests_waiting (queue depth) and the request-latency histograms. Everything diagnosed tonight by hand — 122,636 KV tokens per card, thrash when three large contexts overlap, cold-vs-warm prefix reuse — is a graph now, on the LAN addresses rather than the guarded vhosts. frank-large comes along in the same change: llama.cpp exposes throughput counters under llamacpp:, and watching one LLM endpoint without the other was arbitrary. The seam added to observability_quadlet.bats pins the cross-file contract that breaks silently: if a scrape target drifts from the :7770 the units publish, the graphs vanish with no error anywhere.
77 lines
2.3 KiB
YAML
77 lines
2.3 KiB
YAML
# Prometheus configuration for the homelab.
|
|
# Deployed to /opt/prometheus/config/prometheus.yml on Dankclaw.
|
|
|
|
global:
|
|
scrape_interval: 15s
|
|
scrape_timeout: 10s
|
|
evaluation_interval: 15s
|
|
|
|
scrape_configs:
|
|
# Prometheus itself
|
|
- job_name: "prometheus"
|
|
static_configs:
|
|
- targets: ["localhost:9090"]
|
|
|
|
# Host metrics — both hosts
|
|
- job_name: "node-dankclaw"
|
|
static_configs:
|
|
- targets: ["192.168.0.143:9100"]
|
|
|
|
- job_name: "node-dankserver"
|
|
static_configs:
|
|
- targets: ["192.168.0.235:9100"]
|
|
|
|
# Gitea metrics (quadlet on an isolated network; use the published LAN port)
|
|
- job_name: "gitea"
|
|
static_configs:
|
|
- targets: ["192.168.0.143:3000"]
|
|
metrics_path: "/metrics"
|
|
|
|
# Grafana metrics
|
|
- job_name: "grafana"
|
|
static_configs:
|
|
- targets: ["grafana:3000"]
|
|
metrics_path: "/metrics"
|
|
|
|
# Loki metrics
|
|
- job_name: "loki"
|
|
static_configs:
|
|
- targets: ["loki:3100"]
|
|
metrics_path: "/metrics"
|
|
|
|
# ComfyUI (no native /metrics; probe the UI)
|
|
- job_name: "comfyui"
|
|
metrics_path: "/"
|
|
static_configs:
|
|
- targets: ["comfyui:8188"]
|
|
|
|
# Blackbox-style probes for service up/down (published LAN ports;
|
|
# gitea and comfyui are covered by their own jobs above)
|
|
- job_name: "services"
|
|
metrics_path: "/"
|
|
static_configs:
|
|
- targets:
|
|
- "192.168.0.143:8081" # SearXNG (via the Gluetun tunnel)
|
|
- "192.168.0.143:3005" # OpenWebUI
|
|
|
|
# LLM fleet. Scraped on the LAN addresses, not the guarded vhosts: these are
|
|
# machine clients and Prometheus cannot solve an Anubis challenge.
|
|
# frank-medium is vLLM, and its series are the ones that explain behaviour
|
|
# under load — vllm:kv_cache_usage_perc (how full the pool is),
|
|
# vllm:prefix_cache_hit_rate (whether pinned conversations are being reused),
|
|
# vllm:num_requests_waiting (queue depth), and the request-latency histograms.
|
|
# frank-large is llama.cpp and exposes throughput counters under llamacpp:.
|
|
- job_name: "frank-medium"
|
|
metrics_path: "/metrics"
|
|
static_configs:
|
|
- targets: ["192.168.0.235:7770"]
|
|
labels:
|
|
service: frank-medium
|
|
|
|
- job_name: "frank-large"
|
|
metrics_path: "/metrics"
|
|
static_configs:
|
|
- targets: ["192.168.0.143:7770"]
|
|
labels:
|
|
service: frank-large
|