Frank
87721ab0fe
Cross-cutting rename of the current llama.cpp endpoint on the RTX 3090
(port 7771) from frank-small to frank-secondary:
- Service trio: scripts/{frank-small.env,frank-small.service,
serve-frank-small.sh,download-frank-small.sh} → frank-secondary.*
- Test: tests/frank_small_unit.bats → tests/frank_secondary_unit.bats
- Runbooks: serverAppConfigs-docs/LLM/frank-small-*.md →
frank-secondary-*.md (9 files); the FreeToken runbook stays
freetoken-frank-small.md (it documents the retired CUDA-era service)
- Identifiers: FRANK_SMALL→FRANK_SECONDARY, llm_small→llm_secondary,
LLM_SMALL_*→LLM_SECONDARY_*, deploy_llm_small→deploy_llm_secondary
- Historical references to the OLD frank-small (FreeToken/CUDA on the
3090, and the Dankserver service that became frank-medium) are
preserved verbatim — they describe what the service was called
at the time.
84 lines
4.7 KiB
Markdown
84 lines
4.7 KiB
Markdown
---
|
|
type: reference
|
|
description: "Prometheus collects metrics from every service on Dankclaw by scraping their `/metrics` endpoints, and feeds Grafana once a datasource exists."
|
|
tags:
|
|
- infrastructure
|
|
- prometheus
|
|
timestamp: 2026-09-21
|
|
---
|
|
|
|
# Prometheus
|
|
|
|
Prometheus collects metrics from every service on Dankclaw by scraping their `/metrics` endpoints, and feeds Grafana once a datasource exists. node-exporter supplies host metrics from both machines.
|
|
|
|
**Host:** Dankclaw — **Vhost:** `prometheus.franksdank.cloud` (guarded: Anubis →
|
|
Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
|
|
**Port:** 9090 (loopback) — **Docs:** https://prometheus.io/docs
|
|
|
|
## Facts
|
|
|
|
| Item | Value |
|
|
|------|-------|
|
|
| Image | `docker.io/prom/prometheus:latest` |
|
|
| Published | `127.0.0.1:9090` → 9090, loopback only (see [[Observability]] → Exposure) |
|
|
| Unit | `dankclaw/observability/quadlet/prometheus.container` → `prometheus.service` |
|
|
| Config | `/opt/prometheus/config/prometheus.yml`, installed from `shared/prometheus/prometheus.yml` on every deploy |
|
|
| Data | `/opt/prometheus/data` (TSDB) |
|
|
| Retention | `--storage.tsdb.retention.time=30d` |
|
|
| Alerting | 5 alert rules + 1 recording rule (`shared/prometheus/rules/llm-fleet.yml`), evaluated **and delivered** since 2026-09-17: the `alerting:` block ships to `alertmanager:9093` → Signal relay → signal-cli (see [[Alertmanager]]) |
|
|
|
|
## Targets
|
|
|
|
`shared/prometheus/prometheus.yml` is the source of truth; the deploy copies it into the container's `/etc/prometheus/`. In outline:
|
|
|
|
| Job | Target | Why |
|
|
|-----|--------|-----|
|
|
| `prometheus` | `localhost:9090` | itself |
|
|
| `node-dankclaw` | `host.containers.internal:9100` | host metrics from [[Node-Exporter]] — a net=host bind, so via the gateway (the LAN address is unreachable from this bridge container) |
|
|
| `node-dankserver` | `192.168.0.235:9100` | host metrics from [[Node-Exporter]] — a different machine, so the LAN address works |
|
|
| `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on the `dankclaw_default` network; both serve real `/metrics` |
|
|
| `frank-medium` | `192.168.0.235:7770` | the LLM fleet — `vllm:` series; a different machine, so the LAN address works |
|
|
| `frank-large`, `frank-secondary` | `host.containers.internal:7770`, `:7771` | the LLM fleet — `llamacpp:` series; same-host net=host binds, so via the gateway |
|
|
|
|
**Three jobs read down, and they are not outages** (audited 2026-09-16):
|
|
|
|
| Job | Target | Why it never reports up |
|
|
|-----|--------|-------------------------|
|
|
| `comfyui` | `comfyui:8188` | scrapes `/`, which is HTML — "unsupported Content-Type" |
|
|
| `services` | `192.168.0.143:8081`, `:3005` | same: up/down intent, implemented as a scrape |
|
|
| `gitea` | `192.168.0.143:3000/metrics` | Gitea's exporter is off until `[metrics] ENABLED = true` |
|
|
|
|
A scrape cannot express "is this HTTP service answering?" — that is what a
|
|
probe is for, and the remedy is the **blackbox exporter** in probe mode
|
|
(`/probe?target=…&module=http_2xx`), which also gives the guarded vhosts a real
|
|
uptime signal since Prometheus cannot solve an Anubis challenge itself. Until it
|
|
is deployed, these three stay red for a structural reason: deleting them would
|
|
hide a real service, and fixing them by publishing databases or enabling metrics
|
|
would trade a wrong signal for a real exposure. Targets are `host:port`, never
|
|
URLs.
|
|
|
|
## Volumes
|
|
|
|
| Container path | Host path |
|
|
|----------------|-----------|
|
|
| `/etc/prometheus` | `/opt/prometheus/config` |
|
|
| `/prometheus` | `/opt/prometheus/data` |
|
|
|
|
## Healthcheck
|
|
|
|
Prometheus responds on `/-/healthy` with 200; the unit's health command is the same `wget --spider` the deploy waits on.
|
|
|
|
## Gotchas
|
|
|
|
- **Keep the node-exporter address in step with the unit.** `node-dankclaw` scrapes `192.168.0.143:9100`, which is exactly the LAN address `node-exporter.container` binds (`Network=host`, `--web.listen-address` twice). Change one without the other and host metrics stop with no error anywhere. The `tests/observability_quadlet.bats` seam pins the pairing.
|
|
- The container name `prometheus` is what Caddy's post-auth handler proxies to, and `grafana`/`loki` are addressed by name here. Renaming a container breaks both silently.
|
|
- Keep scrapes modest for a small homelab; too many targets bloat the TSDB.
|
|
- Set a retention policy or the data volume grows without bound (30 days here).
|
|
- The lifecycle endpoint is enabled (`--web.enable-lifecycle`), which is why this API must never be published beyond loopback.
|
|
|
|
## See Also
|
|
|
|
- [[Grafana]] — the dashboard frontend (datasource and LLM dashboard provisioned as code since 2026-09-16)
|
|
- [[Node-Exporter]] — the host metrics source
|
|
- [[Observability]] — the whole stack, including what is still unconfigured
|