dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(observability): record the quadlet cutover, the exposure posture, and the unconfigured state

Browse Source 
Rewrites the [[Observability]] umbrella page and the five component pages for
what is actually deployed: rootless Quadlet units instead of compose services,
the loopback-only publish posture (node-exporter's LAN binding explained), the
loki healthcheck living in its image because the image has no shell, and the
Home=/root trap that killed the first grafana start.

Adds the note that matters most to a reader: the stack is installed, healthy
and UNCONFIGURED. Grafana reports zero datasources and zero dashboards with an
empty provisioning directory; Prometheus has no rule files and no Alertmanager;
Loki is configured but has no reader; node-exporter runs its default
collectors. Alloy is the only configured piece. Metrics and logs are collected
and retained, and nothing is visualized or alerted on.

AGENTS.md's boot-persistence gap note is replaced by the resolution.
This commit is contained in:
Frank
2026-09-11 16:10:27 -07:00
parent 25c93984ad
commit 701ab14f60
8 changed files with 220 additions and 147 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/Infrastructure/Prometheus.md
+28 -28
View File
@@ -1,59 +1,59 @@
# Prometheus
Prometheus collects metrics from every service on Dankclaw by scraping their `/metrics` endpoints. It feeds the Grafana dashboards. The `node-exporter` containers may add podman/cgroup metrics for full host visibility.
Prometheus collects metrics from every service on Dankclaw by scraping their `/metrics` endpoints, and feeds Grafana once a datasource exists. node-exporter supplies host metrics from both machines.
**Host:** Dankclaw — **Vhost:** `prometheus.franksdank.cloud` (guarded: Anubis →
Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
**Port:** 9090 — **Docs:** https://prometheus.io/docs
**Port:** 9090 (loopback) — **Docs:** https://prometheus.io/docs
## Facts
| Item | Value |
|------|-------|
| Image | `docker.io/prom/prometheus:latest` |
| Port | 9090 (host) → 9090 (container) |
| Config | `/etc/prometheus/prometheus.yml` (bind mount) |
| Data volume | TSDB |
| Published | `127.0.0.1:9090` → 9090, loopback only (see [[Observability]] → Exposure) |
| Unit | `dankclaw/observability/quadlet/prometheus.container` → `prometheus.service` |
| Config | `/opt/prometheus/config/prometheus.yml`, installed from `shared/prometheus/prometheus.yml` on every deploy |
| Data | `/opt/prometheus/data` (TSDB) |
| Retention | `--storage.tsdb.retention.time=30d` |
| Alerting | **none** — no rule files, no Alertmanager anywhere in the stack |
## prometheus.yml Targets
## Targets
```yaml
scrape_configs:
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
`shared/prometheus/prometheus.yml` is the source of truth; the deploy copies it into the container's `/etc/prometheus/`. In outline:
- job_name: "comfyui"
static_configs:
- targets: ["comfyui:8188"]
| Job | Target | Why |
|-----|--------|-----|
| `prometheus` | `localhost:9090` | itself |
| `node-dankclaw`, `node-dankserver` | `192.168.0.143:9100`, `192.168.0.235:9100` | host metrics from each machine's [[Node-Exporter]] |
| `gitea` | `192.168.0.143:3000` | Gitea's built-in exporter, reached on its published LAN port |
| `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on the `dankclaw_default` network |
| `comfyui` | `comfyui:8188` | UI probe; ComfyUI has no native `/metrics` |
| `services` | `192.168.0.143:8081`, `:3005` | up/down probes for SearXNG and OpenWebUI |
- job_name: "node"
static_configs:
- targets: ["node-exporter:9100"]
```
Allow the `/metrics` endpoint on each service where the exporter exists. For services without a native exporter, use the Prometheus `blackbox` exporter (probe mode) to check up/down and latency.
Targets are `host:port`, never URLs. For richer HTTP assertions, add the Prometheus **blackbox** exporter in probe mode rather than scraping HTML.
## Volumes
| Container path | Host path (convention) |
|----------------|------------------------|
| Container path | Host path |
|----------------|-----------|
| `/etc/prometheus` | `/opt/prometheus/config` |
| `/prometheus` | `/opt/prometheus/data` |
## Healthcheck
Prometheus responds on `/-/healthy` with 200.
Prometheus responds on `/-/healthy` with 200; the unit's health command is the same `wget --spider` the deploy waits on.
## Gotchas
- **Keep the node-exporter address in step with the unit.** `node-dankclaw` scrapes `192.168.0.143:9100`, which is exactly the LAN address `node-exporter.container` binds (`Network=host`, `--web.listen-address` twice). Change one without the other and host metrics stop with no error anywhere. The `tests/observability_quadlet.bats` seam pins the pairing.
- The container name `prometheus` is what Caddy's post-auth handler proxies to, and `grafana`/`loki` are addressed by name here. Renaming a container breaks both silently.
- Keep scrapes modest for a small homelab; too many targets bloat the TSDB.
- Use recording rules for heavy queries (see [[Grafana]]).
- Set a retention policy or the data volume grows without bound.
- Set a retention policy or the data volume grows without bound (30 days here).
- The lifecycle endpoint is enabled (`--web.enable-lifecycle`), which is why this API must never be published beyond loopback.
## See Also
- [[Dankclaw Stack]]
- [[Grafana]] — the dashboard frontend
- [[Observability]] — the whole stack
- [[Grafana]] — the dashboard frontend (no datasource configured yet)
- [[Node-Exporter]] — the host metrics source
- [[Observability]] — the whole stack, including what is still unconfigured