Frank
139a3cd4fa
The deploy died at 'Unit blackbox-exporter.service not found': a .container without Image= is not generated at all, and the header argued for Network=host it never declared. Completes the unit (digest pinned prom/blackbox-exporter, Network=host) and fixes the bind topology the first cut got wrong: - Prometheus is bridge-networked and the netavark hairpin gap bars it from its own host's LAN IP, so a LAN-only prober bind is unscrapable and a loopback-only bind was claimed to ride the gateway on faith. The prober now binds 9115 on the podman gateway (scrape path) AND the LAN (healthcheck/diagnostics); the listen_address line leaves blackbox.yml so config and Exec cannot disagree. - comfyui probe target http://comfyui:8188 -> http://127.0.0.1:8188: bridge container names do not resolve in the host netns; the published 0.0.0.0:8188 answers on loopback there. - New seam: every observability unit must declare Image= (red on the shipped file by name); copier test pinned module presence instead of the removed listen line. Docs updated to the real topology.
232 lines
9.8 KiB
YAML
232 lines
9.8 KiB
YAML
# Prometheus configuration for the homelab.
|
|
# Deployed to /opt/prometheus/config/prometheus.yml on Dankclaw.
|
|
|
|
global:
|
|
scrape_interval: 15s
|
|
scrape_timeout: 10s
|
|
evaluation_interval: 15s
|
|
|
|
# Alerting and recording rules for the LLM fleet. They evaluate, expose
|
|
# state through /alerts and the ALERTS metric, and — since 2026-09-17 —
|
|
# deliver to Signal through Alertmanager (see the alerting section below).
|
|
rule_files:
|
|
- /etc/prometheus/rules/*.yml
|
|
|
|
# The delivery half. Alertmanager runs on the same dankclaw_default network,
|
|
# so it is addressed by container name, like grafana and loki below. Its
|
|
# single receiver webhooks the host-side signal-relay (192.168.0.143:18093,
|
|
# reached from the container as host.containers.internal).
|
|
alerting:
|
|
alertmanagers:
|
|
- static_configs:
|
|
- targets: ["alertmanager:9093"]
|
|
|
|
scrape_configs:
|
|
# Prometheus itself
|
|
- job_name: "prometheus"
|
|
static_configs:
|
|
- targets: ["localhost:9090"]
|
|
|
|
# Host metrics — both hosts. node-dankclaw is scraped over
|
|
# host.containers.internal, not its own LAN address: this Prometheus is a
|
|
# bridge container, and on this netavark host a container can only reach the
|
|
# host's LAN IP at a PUBLISHED port — node-exporter's net=host bind on
|
|
# 192.168.0.143:9100 is refused by construction (the same rule that sends
|
|
# frank-dragoon and frank-zealot to the gateway below). node-dankserver is a
|
|
# different machine, so its LAN address works.
|
|
- job_name: "node-dankclaw"
|
|
static_configs:
|
|
- targets: ["host.containers.internal:9100"]
|
|
|
|
- job_name: "node-dankserver"
|
|
static_configs:
|
|
- targets: ["192.168.0.235:9100"]
|
|
|
|
# Gitea metrics (quadlet on an isolated network; use the published LAN port)
|
|
- job_name: "gitea"
|
|
static_configs:
|
|
- targets: ["192.168.0.143:3000"]
|
|
metrics_path: "/metrics"
|
|
|
|
# Grafana metrics
|
|
- job_name: "grafana"
|
|
static_configs:
|
|
- targets: ["grafana:3000"]
|
|
metrics_path: "/metrics"
|
|
|
|
# Loki metrics
|
|
- job_name: "loki"
|
|
static_configs:
|
|
- targets: ["loki:3100"]
|
|
metrics_path: "/metrics"
|
|
|
|
# HTTP liveness, probed honestly through the blackbox exporter. Scraping a
|
|
# web root is NOT a health check — an HTML front page fails the scrape
|
|
# protocol (`unsupported Content-Type`), which is exactly why the old
|
|
# `services` and `comfyui` jobs were red forever beside healthy services
|
|
# (Stale Signal Audit §1). The prober is a host-netns container listening on
|
|
# the podman gateway (the scrape address below - the netavark hairpin gap
|
|
# bars this container from its own host's LAN IP, cadvisor job note) and on
|
|
# the LAN; from that netns it reaches the host LAN IP, and published ports
|
|
# via 127.0.0.1 - bridge container names do NOT resolve in the host netns,
|
|
# which is why comfyui's probe is a loopback URL. comfyui's own
|
|
# metrics-scrape job is retired, one owner per service.
|
|
- job_name: "services"
|
|
metrics_path: /probe
|
|
params:
|
|
module: [http_2xx]
|
|
static_configs:
|
|
- targets:
|
|
- "http://192.168.0.143:8081" # SearXNG (via the Gluetun tunnel)
|
|
- "http://192.168.0.143:3005" # OpenWebUI
|
|
- "http://127.0.0.1:8188" # ComfyUI UI (published 0.0.0.0:8188; probed from the prober's host netns)
|
|
relabel_configs:
|
|
- source_labels: [__address__]
|
|
target_label: __param_target
|
|
- source_labels: [__param_target]
|
|
target_label: instance
|
|
- target_label: __address__
|
|
replacement: "host.containers.internal:9115"
|
|
|
|
# LLM fleet. Scraped on the LAN addresses where the endpoint is reachable from
|
|
# this container, not the guarded vhosts: these are machine clients and
|
|
# Prometheus cannot solve an Anubis challenge.
|
|
#
|
|
# One metric namespace: llamacpp:* — frank-dragoon, frank-zealot and
|
|
# frank-archon, all llama.cpp with /metrics (the family verified from
|
|
# /metrics on 2026-09-16 and 2026-09-20) and all exporting
|
|
# requests_deferred (the queue depth, the only form llama.cpp shows the
|
|
# capacity wall in), spec_decode_num_drafts_total, tokens_predicted_total,
|
|
# and the prompt_tokens_cached_total / prompt_tokens_total reuse pair.
|
|
#
|
|
# frank-dragoon and frank-zealot are scraped over host.containers.internal,
|
|
# not their own LAN addresses: this Prometheus is a bridge-networked
|
|
# container (Network=dankclaw_default), and on this netavark host a
|
|
# container can only reach the host's LAN IP at a PUBLISHED port — a
|
|
# host-process bind like llama-server's 0.0.0.0:7770 is refused at
|
|
# 192.168.0.143:7770 while it answers at 169.254.1.2:7770. That is
|
|
# why frank-zealot was red here for days (up{job="frank-zealot"} == 0,
|
|
# "dial tcp 192.168.0.143:7771") even though the endpoint was healthy and
|
|
# exporting: the target address was unreachable by construction, not the
|
|
# service down. Same host, same rule, so both local endpoints use the
|
|
# gateway.
|
|
#
|
|
# (2026-10-05) frank-archon LEFT the llamacpp family with FreeToken
|
|
# (the PR587 ROCm engine then serving it had NO /metrics — the job
|
|
# scraped 404s forever, a structural never-green) and RE-JOINED the same
|
|
# day on the radiance engine: its exporter speaks vLLM-compatible names
|
|
# plus radiance:* families, a different namespace — hence the separate
|
|
# job below, not a re-add to the llamacpp pair. The netavark quirk above
|
|
# is same-host only; plain LAN routing REACHES 192.168.0.235:7770.
|
|
- job_name: "frank-zealot"
|
|
metrics_path: "/metrics"
|
|
static_configs:
|
|
- targets: ["host.containers.internal:7771"]
|
|
labels:
|
|
service: frank-zealot
|
|
|
|
- job_name: "frank-dragoon"
|
|
metrics_path: "/metrics"
|
|
static_configs:
|
|
- targets: ["host.containers.internal:7770"]
|
|
labels:
|
|
service: frank-dragoon
|
|
|
|
# frank-archon (dankserver) — the radiance engine since 2026-10-05.
|
|
# A DIFFERENT host, so plain LAN routing is correct here (the
|
|
# host.containers.internal trick is same-host only): health answers at
|
|
# 192.168.0.235:7770 from this bridge container. Radiance exports
|
|
# vLLM-compatible names plus radiance:* families — NOT the llamacpp:*
|
|
# namespace, so llm-fleet rules must not be pointed at this target.
|
|
- job_name: "frank-archon"
|
|
metrics_path: "/metrics"
|
|
static_configs:
|
|
- targets: ["192.168.0.235:7770"]
|
|
labels:
|
|
service: frank-archon
|
|
|
|
# Per-container metrics from cadvisor (a rootless quadlet on each host).
|
|
# cadvisor keys every series by the cgroup path; for a quadlet container that
|
|
# path is …/app.slice/<name>.service[/libpod-payload-<hex>], so the unit NAME
|
|
# lives in it. The relabel below pulls that name out into a `container` label
|
|
# and keeps only the per-unit aggregate — cadvisor's raw host root,
|
|
# system.slice daemons and machine.slice scopes carry no such name and are
|
|
# dropped, as is the per-CPU-core split (only cpu="total" survives), so the
|
|
# scrape stays small. cadvisor cannot bind per-interface, so it is reached at
|
|
# the podman gateway like every same-host target: a BRIDGE CONTAINER CANNOT
|
|
# DIAL ITS OWN HOST'S LAN IP AT ALL — the netavark hairpin gap refuses the
|
|
# connection whether the port is a host-net bind or a published port (the
|
|
# original note here claimed published ports were exempt; measured 2026-10-21
|
|
# from inside this container: LAN refused, gateway answers). Cross-host
|
|
# targets (dankserver) have no hairpin problem and stay plain LAN.
|
|
- job_name: "cadvisor-dankclaw"
|
|
static_configs:
|
|
- targets: ["host.containers.internal:8080"]
|
|
labels:
|
|
host: dankclaw
|
|
metric_relabel_configs:
|
|
- source_labels: [id]
|
|
regex: '.*/app\.slice/([^.]+)\.service(?:/.*)?$'
|
|
target_label: container
|
|
replacement: '$1'
|
|
# Keep only the unit-level cgroup; drop its payload/control children so a
|
|
# container is not counted twice.
|
|
- source_labels: [id]
|
|
regex: '.*/\.service/.*'
|
|
action: drop
|
|
# Drop everything that is not a named unit (no `container` was extracted).
|
|
- source_labels: [container]
|
|
regex: '.+'
|
|
action: keep
|
|
# cpu="total" for cpu series; metrics with no cpu label (memory, network,
|
|
# filesystem) are kept.
|
|
- source_labels: [cpu]
|
|
regex: '^$|total'
|
|
action: keep
|
|
- regex: 'id|name|image|container_label.*|machine_id|boot_id'
|
|
action: labeldrop
|
|
|
|
- job_name: "cadvisor-dankserver"
|
|
static_configs:
|
|
- targets: ["192.168.0.235:8080"]
|
|
labels:
|
|
host: dankserver
|
|
metric_relabel_configs:
|
|
- source_labels: [id]
|
|
regex: '.*/app\.slice/([^.]+)\.service(?:/.*)?$'
|
|
target_label: container
|
|
replacement: '$1'
|
|
- source_labels: [id]
|
|
regex: '.*/\.service/.*'
|
|
action: drop
|
|
- source_labels: [container]
|
|
regex: '.+'
|
|
action: keep
|
|
- source_labels: [cpu]
|
|
regex: '^$|total'
|
|
action: keep
|
|
- regex: 'id|name|image|container_label.*|machine_id|boot_id'
|
|
action: labeldrop
|
|
|
|
# Per-GPU metrics (gpu_utilization_percent / gpu_memory_used_bytes /
|
|
# gpu_memory_total_bytes / gpu_temperature_celsius / gpu_power_watts) from the
|
|
# host-process gpu-exporter (scripts/gpu-exporter.py). One job per host, same
|
|
# reach rules as node/cadvisor: Dankclaw is scraped at the podman gateway
|
|
# because the netavark hairpin quirk makes Prometheus fail to dial its own
|
|
# host LAN IP; Dankserver is plain LAN. The gpu label (PCI BDF on AMD, index on
|
|
# NVIDIA) + vendor/model come from the exporter, so the GPU row on the
|
|
# host-system dashboard filters by host. Power is emitted only where readable
|
|
# (NVIDIA + the Dankclaw iGPU); the R9700s have no sysfs power, so they simply
|
|
# have no gpu_power_watts series — no fabricated zero.
|
|
- job_name: "gpu-dankclaw"
|
|
static_configs:
|
|
- targets: ["host.containers.internal:9401"]
|
|
labels:
|
|
host: dankclaw
|
|
|
|
- job_name: "gpu-dankserver"
|
|
static_configs:
|
|
- targets: ["192.168.0.235:9401"]
|
|
labels:
|
|
host: dankserver
|