Files

T

Frank 93d1181fee docs: OKF v0.1 compliance pass + fact-check + 3 research briefs

- Migrate all 127 non-reserved docs to OKF v0.1 frontmatter (type/description/tags/timestamp); fix 13 YAML issues, 2 duplicate-FM blocks, strip reserved-page FM
- Restructure log.md to OKF (## YYYY-MM-DD, newest-first)
- Fact-check against scripts/AGENTS.md ground truth: fix 6 false 'Active' banners on retired runbooks, 96->114 GiB GTT, deployment.md vs deploy-plan.py/deploy-remote.sh, Deployment Flow frank-medium=Quadlet, Calibre 9.9.0->9.15.0 + CVE-2025-7404 conflation, Copyparty v1.20.24, RomM 5.3.0 breaking change
- Move *arr docs to new FrankBook-Pro/ section (git mv, link-safe)
- Write active runbook bodies from ground truth (frank-large, frank-small)
- Add 3 research briefs + HTML report: Local TTS and Voice Cloning, AI Job Search and Resume Tools, Most Popular Self-Hosted AI Apps (Jun-Sep 2026)
- Add upstream-version rows to ComfyUI/OpenWebUI/Hermes app docs
- All 336 tests pass; 136 pages, 0 unresolved wikilinks

2026-09-22 02:09:41 -07:00

4.7 KiB

Raw Blame History

type, description, tags, timestamp

type

description

Prometheus

Prometheus collects metrics from every service on Dankclaw by scraping their /metrics endpoints, and feeds Grafana once a datasource exists. node-exporter supplies host metrics from both machines.

Host: Dankclaw — Vhost: prometheus.franksdank.cloud (guarded: Anubis → Authentik; post-auth the Caddy vhost routes straight to prometheus:9090) — Port: 9090 (loopback) — Docs: https://prometheus.io/docs

Facts

Item	Value
Image	`docker.io/prom/prometheus:latest`
Published	`127.0.0.1:9090` → 9090, loopback only (see Observability → Exposure)
Unit	`dankclaw/observability/quadlet/prometheus.container` → `prometheus.service`
Config	`/opt/prometheus/config/prometheus.yml`, installed from `shared/prometheus/prometheus.yml` on every deploy
Data	`/opt/prometheus/data` (TSDB)
Retention	`--storage.tsdb.retention.time=30d`
Alerting	5 alert rules + 1 recording rule (`shared/prometheus/rules/llm-fleet.yml`), evaluated and delivered since 2026-09-17: the `alerting:` block ships to `alertmanager:9093` → Signal relay → signal-cli (see Alertmanager)

Targets

shared/prometheus/prometheus.yml is the source of truth; the deploy copies it into the container's /etc/prometheus/. In outline:

Job	Target	Why
`prometheus`	`localhost:9090`	itself
`node-dankclaw`	`host.containers.internal:9100`	host metrics from Node-Exporter — a net=host bind, so via the gateway (the LAN address is unreachable from this bridge container)
`node-dankserver`	`192.168.0.235:9100`	host metrics from Node-Exporter — a different machine, so the LAN address works
`grafana`, `loki`	`grafana:3000`, `loki:3100`	container names on the `dankclaw_default` network; both serve real `/metrics`
`frank-medium`	`192.168.0.235:7770`	the LLM fleet — `vllm:` series; a different machine, so the LAN address works
`frank-large`, `frank-small`	`host.containers.internal:7770`, `:7771`	the LLM fleet — `llamacpp:` series; same-host net=host binds, so via the gateway

Three jobs read down, and they are not outages (audited 2026-09-16):

Job	Target	Why it never reports up
`comfyui`	`comfyui:8188`	scrapes `/`, which is HTML — "unsupported Content-Type"
`services`	`192.168.0.143:8081`, `:3005`	same: up/down intent, implemented as a scrape
`gitea`	`192.168.0.143:3000/metrics`	Gitea's exporter is off until `[metrics] ENABLED = true`

A scrape cannot express "is this HTTP service answering?" — that is what a probe is for, and the remedy is the blackbox exporter in probe mode (/probe?target=…&module=http_2xx), which also gives the guarded vhosts a real uptime signal since Prometheus cannot solve an Anubis challenge itself. Until it is deployed, these three stay red for a structural reason: deleting them would hide a real service, and fixing them by publishing databases or enabling metrics would trade a wrong signal for a real exposure. Targets are host:port, never URLs.

Volumes

Container path	Host path
`/etc/prometheus`	`/opt/prometheus/config`
`/prometheus`	`/opt/prometheus/data`

Healthcheck

Prometheus responds on /-/healthy with 200; the unit's health command is the same wget --spider the deploy waits on.

Gotchas

Keep the node-exporter address in step with the unit. node-dankclaw scrapes 192.168.0.143:9100, which is exactly the LAN address node-exporter.container binds (Network=host, --web.listen-address twice). Change one without the other and host metrics stop with no error anywhere. The tests/observability_quadlet.bats seam pins the pairing.
The container name prometheus is what Caddy's post-auth handler proxies to, and grafana/loki are addressed by name here. Renaming a container breaks both silently.
Keep scrapes modest for a small homelab; too many targets bloat the TSDB.
Set a retention policy or the data volume grows without bound (30 days here).
The lifecycle endpoint is enabled (--web.enable-lifecycle), which is why this API must never be published beyond loopback.

4.7 KiB Raw Blame History