Files

T

Frank 6bdfd2853e docs(vault): service and infrastructure pages stop restating endpoint facts

18 pages cut their per-endpoint shape, payload, port and window claims and
link [[frank-archon]] / [[frank-dragoon]] / [[frank-zealot]] instead. Several
of the cut claims had already drifted -- the stack page still described the
kingjones ROCmFP4 era on ROCmFPX, OpenWebUI carried the 27B/3-slot bench
narrative, Hermes named a retired max_model_len and the old service name, and
the OMP harness page's live-window block had drifted off frank-zealot's real
262,144 (it is now labelled a dated worked example whose authority is the
endpoint page).

Two facts gain a single owner instead of two: the up{job=~"frank-.*"} scope
lives in Infrastructure/Prometheus.md, and the blackbox/Anubis probe form in
Architecture/Stale Signal Audit.md, each cited from the other page.

Voice Packs and Frank TTS carried live-state claims that contradicted the
engine page's 2026-10-18 deploy record (nothing listening on 7772/8880, model
dir absent); they now point at the pages that own host state.

2026-10-19 00:56:02 -07:00

8.4 KiB

Raw Blame History

type, description, tags, timestamp

type

description

Prometheus

Prometheus collects metrics from the services listed below by scraping their /metrics endpoints, and it is the datasource behind the provisioned llm-fleet dashboard in Grafana (datasource uid prometheus, shared/grafana/provisioning/datasources/prometheus.yml:14-21). node-exporter supplies host metrics from both machines.

Host: Dankclaw — Vhost: prometheus.franksdank.cloud (guarded: Anubis → Authentik; post-auth the Caddy vhost routes straight to prometheus:9090) — Port: 9090 (loopback) — Docs: https://prometheus.io/docs

Facts

Item	Value
Image	`docker.io/prom/prometheus:latest`
Published	`127.0.0.1:9090` → 9090, loopback only — never a wildcard (`dankclaw/observability/quadlet/prometheus.container:50`; `ss -ltnp` 2026-10-18 shows `127.0.0.1:9090` and no other bind; `tests/observability_quadlet.bats:63` pins it)
Unit	`dankclaw/observability/quadlet/prometheus.container` → `prometheus.service`; `Restart=always`/`RestartSec=10` + `WantedBy=default.target` (`prometheus.container:59-64`), installed at `~/.config/containers/systemd/prometheus.container`, reporting `generated` + `active` (`systemctl --user is-enabled
Config	`/opt/prometheus/config/prometheus.yml`, installed from `shared/prometheus/prometheus.yml` on every deploy, together with `/opt/prometheus/config/rules/llm-fleet.yml` from `shared/prometheus/rules/llm-fleet.yml` (`scripts/deploy-remote.sh:219-227`)
Data	`/opt/prometheus/data` (TSDB, `prometheus.container:52`)
Retention	`--storage.tsdb.retention.time=30d` (`prometheus.container:49`)
Alerting	4 alert rules + 2 recording rules (`shared/prometheus/rules/llm-fleet.yml`: alerts `LlmRequestsQueueing`, `LlmPrefixCacheCollapse`, `LlmSpecDecodeOff`, `LlmEndpointDown`; recordings `job:llm_prefix_cache_reuse:rate5m`, `job:llm_spec_accept_length:rate5m`), confirmed against the running server: `curl -s http://127.0.0.1:9090/api/v1/rules` (2026-10-18) lists 4 alerting + 2 recording. `LlmEndpointDown` is scoped `up{job=~"frank-."} == 0`, so only the `frank-` LLM jobs can page on it — any other job that is structurally never-green must stay out of that family (Stale Signal Audit). Evaluated and delivered since 2026-09-17: the `alerting:` block ships to `alertmanager:9093` (`prometheus.yml:19-22`) → Signal relay → signal-cli (see Alertmanager)

Targets

shared/prometheus/prometheus.yml is the source of truth; the deploy copies it into the container's /etc/prometheus/. In outline:

Job	Target	Why
`prometheus`	`localhost:9090`	itself
`node-dankclaw`	`host.containers.internal:9100`	host metrics from Node-Exporter — a net=host bind, so via the gateway (the LAN address is unreachable from this bridge container)
`node-dankserver`	`192.168.0.235:9100`	host metrics from Node-Exporter — a different machine, so the LAN address works
`grafana`, `loki`	`grafana:3000`, `loki:3100`	container names on the `dankclaw_default` network; both serve real `/metrics`
`frank-archon`	`192.168.0.235:7770`	the LLM fleet — a different machine, so the LAN address works (`prometheus.yml:122-131`); engine, metric families and their meaning live on frank-archon
`frank-dragoon`, `frank-zealot`	`host.containers.internal:7770`, `:7771`	the LLM fleet. They are addressed through the gateway because they are same-host — a host-process bind and a published container port are equally invisible to a bridge container at the host's LAN address (`prometheus.yml:96-99`, `:108-120`); what each endpoint exports is on frank-dragoon and frank-zealot

The frank-medium job is retired: no frank-medium job exists in the config or in /api/v1/targets (2026-10-18), and the endpoint itself left the fleet — its history is on frank-archon, which holds the port today.

Three jobs read down today, and they are not outages (re-audited against curl -s http://127.0.0.1:9090/api/v1/targets, 2026-10-18; first audited 2026-09-16):

Job	Target	`health`	`lastError`, verbatim
`gitea`	`http://192.168.0.143:3000/metrics`	down	`server returned HTTP status 404 Not Found`
`comfyui`	`http://comfyui:8188/`	down	`received unsupported Content-Type "text/html" and no fallback_scrape_protocol specified for target`
`services`	`http://192.168.0.143:8081/`	down	`received unsupported Content-Type "text/html; charset=utf-8" and no fallback_scrape_protocol specified for target`
`services`	`http://192.168.0.143:3005/`	down	same `unsupported Content-Type "text/html; charset=utf-8"` error

Gitea's exporter stays off until [metrics] ENABLED = true in its own config (no metrics env is set in the container, and curl -s http://127.0.0.1:3000/metrics → 404, 2026-10-18); ComfyUI and the two services probes are HTML pages, which Prometheus 3 refuses as metrics. Everything else is green: prometheus, node-dankclaw, node-dankserver, grafana, loki, frank-zealot, frank-dragoon, frank-archon.

frank-zealot reads up since 2026-10-18, and did not before. llama.cpp only answers /metrics when started with --metrics, so up{job="frank-zealot"} was 0 across the whole retained history (the oldest sample, 2026-10-11, is already 0) while the service answered /health 200 the whole time; the flag and its opt-out are on frank-zealot. Read lastError before believing an outage: a target can read down beside a healthy service.

Alert state is a live property, not a page fact: curl -s http://127.0.0.1:9090/api/v1/alerts returned no alerts at the 2026-10-18 morning census, one firing LlmPrefixCacheCollapse (info, frank-zealot, activeAt 11:10:48Z) minutes later, and none again by 11:56Z.

The remedy for a probe-shaped question is the blackbox exporter; the classification rule and the honesty law for it live in Stale Signal Audit. Until the exporter is deployed, these three stay red for a structural reason: deleting them would hide a real service, and fixing them by publishing databases or enabling metrics would trade a wrong signal for a real exposure. Targets are host:port, never URLs.

Volumes

Container path	Host path
`/etc/prometheus`	`/opt/prometheus/config` (`prometheus.container:51`)
`/prometheus`	`/opt/prometheus/data` (`prometheus.container:52`)

Healthcheck

Prometheus responds on /-/healthy with 200; the unit's health command is the same wget --spider the deploy waits on (prometheus.container:53, scripts/deploy-remote.sh — the prometheus) case of deploy_observability waits on http://127.0.0.1:9090/-/healthy).

Gotchas

Keep the node-exporter address in step with the unit. node-dankclaw scrapes host.containers.internal:9100 (prometheus.yml:37-39), because this Prometheus is a bridge container and a host-netns bind is unreachable at the host's LAN address from inside it; node-exporter.container:39 binds 127.0.0.1:9100 + 192.168.0.143:9100. Change one side without the other and host metrics stop with no error anywhere. The tests/observability_quadlet.bats:108-131 seam pins the pairing in both directions.
The container name prometheus is what Caddy's post-auth handler proxies to, and grafana/loki are addressed by name here. Renaming a container breaks both silently.
Keep scrapes modest for a small homelab; too many targets bloat the TSDB.
Set a retention policy or the data volume grows without bound (30 days here — --storage.tsdb.retention.time=30d, prometheus.container:49).
The lifecycle endpoint is enabled (--web.enable-lifecycle, prometheus.container:49), which is why this API must never be published beyond loopback.

8.4 KiB Raw Blame History