Files

T

Frank 87721ab0fe rename frank-small → frank-secondary (Dankclaw llama.cpp :7771)

Cross-cutting rename of the current llama.cpp endpoint on the RTX 3090
(port 7771) from frank-small to frank-secondary:

- Service trio: scripts/{frank-small.env,frank-small.service,
  serve-frank-small.sh,download-frank-small.sh} → frank-secondary.*
- Test: tests/frank_small_unit.bats → tests/frank_secondary_unit.bats
- Runbooks: serverAppConfigs-docs/LLM/frank-small-*.md →
  frank-secondary-*.md (9 files); the FreeToken runbook stays
  freetoken-frank-small.md (it documents the retired CUDA-era service)
- Identifiers: FRANK_SMALL→FRANK_SECONDARY, llm_small→llm_secondary,
  LLM_SMALL_*→LLM_SECONDARY_*, deploy_llm_small→deploy_llm_secondary
- Historical references to the OLD frank-small (FreeToken/CUDA on the
  3090, and the Dankserver service that became frank-medium) are
  preserved verbatim — they describe what the service was called
  at the time.

2026-10-06 15:37:13 -07:00

4.7 KiB

Raw Blame History

type, description, tags, timestamp

type

description

Prometheus

Prometheus collects metrics from every service on Dankclaw by scraping their /metrics endpoints, and feeds Grafana once a datasource exists. node-exporter supplies host metrics from both machines.

Host: Dankclaw — Vhost: prometheus.franksdank.cloud (guarded: Anubis → Authentik; post-auth the Caddy vhost routes straight to prometheus:9090) — Port: 9090 (loopback) — Docs: https://prometheus.io/docs

Facts

Item	Value
Image	`docker.io/prom/prometheus:latest`
Published	`127.0.0.1:9090` → 9090, loopback only (see Observability → Exposure)
Unit	`dankclaw/observability/quadlet/prometheus.container` → `prometheus.service`
Config	`/opt/prometheus/config/prometheus.yml`, installed from `shared/prometheus/prometheus.yml` on every deploy
Data	`/opt/prometheus/data` (TSDB)
Retention	`--storage.tsdb.retention.time=30d`
Alerting	5 alert rules + 1 recording rule (`shared/prometheus/rules/llm-fleet.yml`), evaluated and delivered since 2026-09-17: the `alerting:` block ships to `alertmanager:9093` → Signal relay → signal-cli (see Alertmanager)

Targets

shared/prometheus/prometheus.yml is the source of truth; the deploy copies it into the container's /etc/prometheus/. In outline:

Job	Target	Why
`prometheus`	`localhost:9090`	itself
`node-dankclaw`	`host.containers.internal:9100`	host metrics from Node-Exporter — a net=host bind, so via the gateway (the LAN address is unreachable from this bridge container)
`node-dankserver`	`192.168.0.235:9100`	host metrics from Node-Exporter — a different machine, so the LAN address works
`grafana`, `loki`	`grafana:3000`, `loki:3100`	container names on the `dankclaw_default` network; both serve real `/metrics`
`frank-medium`	`192.168.0.235:7770`	the LLM fleet — `vllm:` series; a different machine, so the LAN address works
`frank-large`, `frank-secondary`	`host.containers.internal:7770`, `:7771`	the LLM fleet — `llamacpp:` series; same-host net=host binds, so via the gateway

Three jobs read down, and they are not outages (audited 2026-09-16):

Job	Target	Why it never reports up
`comfyui`	`comfyui:8188`	scrapes `/`, which is HTML — "unsupported Content-Type"
`services`	`192.168.0.143:8081`, `:3005`	same: up/down intent, implemented as a scrape
`gitea`	`192.168.0.143:3000/metrics`	Gitea's exporter is off until `[metrics] ENABLED = true`

A scrape cannot express "is this HTTP service answering?" — that is what a probe is for, and the remedy is the blackbox exporter in probe mode (/probe?target=…&module=http_2xx), which also gives the guarded vhosts a real uptime signal since Prometheus cannot solve an Anubis challenge itself. Until it is deployed, these three stay red for a structural reason: deleting them would hide a real service, and fixing them by publishing databases or enabling metrics would trade a wrong signal for a real exposure. Targets are host:port, never URLs.

Volumes

Container path	Host path
`/etc/prometheus`	`/opt/prometheus/config`
`/prometheus`	`/opt/prometheus/data`

Healthcheck

Prometheus responds on /-/healthy with 200; the unit's health command is the same wget --spider the deploy waits on.

Gotchas

Keep the node-exporter address in step with the unit. node-dankclaw scrapes 192.168.0.143:9100, which is exactly the LAN address node-exporter.container binds (Network=host, --web.listen-address twice). Change one without the other and host metrics stop with no error anywhere. The tests/observability_quadlet.bats seam pins the pairing.
The container name prometheus is what Caddy's post-auth handler proxies to, and grafana/loki are addressed by name here. Renaming a container breaks both silently.
Keep scrapes modest for a small homelab; too many targets bloat the TSDB.
Set a retention policy or the data volume grows without bound (30 days here).
The lifecycle endpoint is enabled (--web.enable-lifecycle), which is why this API must never be published beyond loopback.

4.7 KiB Raw Blame History