From 701ab14f60d05f233710d3f3f93459ed1d48574a Mon Sep 17 00:00:00 2001 From: Frank Date: Fri, 11 Sep 2026 16:10:27 -0700 Subject: [PATCH] docs(observability): record the quadlet cutover, the exposure posture, and the unconfigured state Rewrites the [[Observability]] umbrella page and the five component pages for what is actually deployed: rootless Quadlet units instead of compose services, the loopback-only publish posture (node-exporter's LAN binding explained), the loki healthcheck living in its image because the image has no shell, and the Home=/root trap that killed the first grafana start. Adds the note that matters most to a reader: the stack is installed, healthy and UNCONFIGURED. Grafana reports zero datasources and zero dashboards with an empty provisioning directory; Prometheus has no rule files and no Alertmanager; Loki is configured but has no reader; node-exporter runs its default collectors. Alloy is the only configured piece. Metrics and logs are collected and retained, and nothing is visualized or alerted on. AGENTS.md's boot-persistence gap note is replaced by the resolution. --- AGENTS.md | 21 ++-- serverAppConfigs-docs/Infrastructure/Alloy.md | 41 ++++--- .../Infrastructure/Grafana.md | 41 ++++--- serverAppConfigs-docs/Infrastructure/Loki.md | 37 ++++--- .../Infrastructure/Node-Exporter.md | 63 +++++------ .../Infrastructure/Observability.md | 102 +++++++++++++----- .../Infrastructure/Prometheus.md | 56 +++++----- serverAppConfigs-docs/index.md | 6 +- 8 files changed, 220 insertions(+), 147 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index fc18e2e..7b444ce 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -200,15 +200,18 @@ Pi-hole unit existed but crash-looped on a sysctl conflict (a later-sorted drop-in reset `net.ipv4.ip_unprivileged_port_start` to 80), taking LAN DNS down with it. -**Known gap (2026-09-11, found while rebooting Dankclaw for the `amd_iommu=off` -change):** five Dankclaw services still run only from `dankclaw/compose.yaml` -with no Quadlet unit — `prometheus`, `grafana`, `loki`, `alloy` and -`node-exporter` (all under `~/.config/containers/systemd/` absent, but present -in the compose file). They did not come back after that reboot and were -restored with `./scripts/deploy.sh dankclaw --service `; they will not -survive the next one either. Giving them units is tracked work, not a solved -problem — treat every reboot as expected to take the observability stack down -until it is done. +**Observability gap (found 2026-09-11, fixed the same day):** rebooting +Dankclaw for the `amd_iommu=off` change exposed five services that ran only +from `dankclaw/compose.yaml` with no Quadlet unit — `prometheus`, `grafana`, +`loki`, `alloy` and `node-exporter`. They did not come back and had to be +restored by hand. They are now rootless Quadlet units under +`dankclaw/observability/quadlet/` (`./scripts/deploy.sh dankclaw --service +`), and their compose definitions were removed so only one owner exists +per container name. The three HTTP UIs publish on loopback only; off-host +access is the guarded vhost. +Note that the stack is RUNNING BUT UNCONFIGURED: Grafana has no datasources +and no dashboards, and Prometheus has no rules and no Alertmanager. See +`serverAppConfigs-docs/Infrastructure/Observability.md`. ## Testing diff --git a/serverAppConfigs-docs/Infrastructure/Alloy.md b/serverAppConfigs-docs/Infrastructure/Alloy.md index 1296625..7eb4590 100644 --- a/serverAppConfigs-docs/Infrastructure/Alloy.md +++ b/serverAppConfigs-docs/Infrastructure/Alloy.md @@ -1,47 +1,54 @@ # Alloy -Alloy is Grafana's log and telemetry shipper. It replaces Promtail (EOL 2026-03-02). One Alloy instance runs per host and forwards container and journal logs to Loki, and optionally metrics to Prometheus. +Alloy is Grafana's log and telemetry shipper. It replaces Promtail (EOL 2026-03-02). It forwards container and journal logs to Loki. -**Hosts:** Dankclaw + Dankserver (one per host) — **Docs:** https://grafana.com/docs/alloy/latest +**Host:** Dankclaw (one instance per host) — **Port:** 12345 (loopback) — **Docs:** https://grafana.com/docs/alloy/latest -## Facts (from research) +## Facts | Item | Value | |------|-------| -| Image | `franksdank/alloy:latest` (local build with a static busybox healthcheck) | -| Port | none published (push-only) | -| Config | River syntax, `config.alloy` | +| Image | `franksdank/alloy:latest` (local build: upstream + a static busybox for the healthcheck) | +| Published | `127.0.0.1:12345` → 12345 (its own UI/health port) | +| Unit | `dankclaw/observability/quadlet/alloy.container` → `alloy.service` | +| Config | `shared/alloy/config.alloy`, installed to `/opt/alloy/config/` | +| Inputs | host journald socket, podman container logs, `/var/log` | +| Output | `http://loki:3100/loki/api/v1/push` | | Successor to | Promtail (EOL 2026-03-02) | -| Inputs | journald socket, container log files | -| Output | Loki push endpoint | ## Why Alloy and not Promtail -Promtail reached end-of-life on 2026-03-02. Grafana Labs directs new installs to Alloy. This stack uses Alloy from day one. +Promtail reached end-of-life on 2026-03-02 and Grafana Labs directs new installs to Alloy. This stack uses Alloy from day one. ## Configuration -River config at `shared/alloy/config.alloy`: +`shared/alloy/config.alloy` is the source of truth: -- Read journald logs from the host's `/run/systemd/journal/socket`. -- Read container logs from `/var/lib/containers/...` (podman) or the docker path. -- Forward to `http://loki:3100/loki/api/v1/push` with `host` and `job` labels. +- Reads journald logs from the host's `/run/systemd/journal/socket`. +- Reads container logs from `/var/lib/containers/storage/...` (podman). +- Forwards to Loki with `job` labels. ## Volumes -| Container path | Host path (convention) | -|----------------|------------------------| -| `/etc/alloy/config.alloy` | `shared/alloy/config.alloy` (bind mount) | +| Container path | Host path | +|----------------|-----------| +| `/etc/alloy` | `/opt/alloy/config` | | `/run/systemd/journal/socket` | host journald socket (read-only) | | `/var/log` | host logs (read-only) | +| `/var/lib/containers` | podman storage (read-only) | + +## The UI port is published on purpose + +Alloy's UI listens on the container's own `127.0.0.1:12345` by default, and Alloy 1.x refuses to configure the `ui` service from a config block ("service ui does not support being configured"). Under compose that meant **no reachable health endpoint at all**: the homepage status card for alloy read `down` continuously and a deploy had nothing to wait on. The unit therefore passes `--server.http.listen-addr=0.0.0.0:12345` and publishes `127.0.0.1:12345` — same posture as the other three: reachable from the host and from Caddy's network, and from nothing off-host. Alloy has no authentication, which is exactly why it is not published wider. ## Gotchas - The journald socket mount is required for journal scraping. - Container log paths differ between docker (`/var/lib/docker/containers`) and podman (`/var/lib/containers/storage/...`); point the config at the right one per host. - One Alloy per host; do not run a single instance that tries to reach both hosts' logs. +- The unit `Wants=`/`After=` `loki.service`: without a reachable Loki the write client fails and the pipeline backs up. ## See Also - [[Loki]] — the log backend -- [[Observability]] — the full stack +- [[Observability]] — the full stack, including what is still unconfigured diff --git a/serverAppConfigs-docs/Infrastructure/Grafana.md b/serverAppConfigs-docs/Infrastructure/Grafana.md index d2f7589..de21546 100644 --- a/serverAppConfigs-docs/Infrastructure/Grafana.md +++ b/serverAppConfigs-docs/Infrastructure/Grafana.md @@ -9,26 +9,34 @@ Grafana visualizes the metrics that Prometheus collects from every service. It r | Item | Value | |------|-------| | Image | `docker.io/grafana/grafana:latest` | -| Port | 3002 (host) → 3000 (container) | -| Data volume | dashboards, users, datasource config | -| Datasource | Prometheus (on the same host) | -| Provisioning | dashboards in `/etc/grafana/provisioning/dashboards` | +| Published | `127.0.0.1:3002` → 3000, loopback only (see [[Observability]] → Exposure) | +| Unit | `dankclaw/observability/quadlet/grafana.container` → `grafana.service` | +| Data | `/opt/grafana/data` — `grafana.db`: accounts, sessions, anything created in the UI | +| Provisioning mount | `/opt/grafana/provisioning` (read-only) — **empty as of 2026-09-11** | +| Datasource | **none configured** — see the state note in [[Observability]] | ## Configuration -```yaml -environment: - GF_SECURITY_ADMIN_USER: "${GRAFANA_ADMIN_USER}" - GF_SECURITY_ADMIN_PASSWORD: "${GRAFANA_ADMIN_PASSWORD}" - GF_SERVER_ROOT_URL: "https://grafana.franksdank.cloud" -volumes: - - /opt/grafana/data:/var/lib/grafana - - /opt/grafana/provisioning:/etc/grafana/provisioning:ro +Everything comes from the Quadlet unit. The Authentik OAuth2 client secret is +the only credential, and systemd expands it from a per-service env file: + ``` +[Service] +EnvironmentFile=/home/dank/Containers/grafana/env # systemd-side only +[Container] +Environment=GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=${GRAFANA_OAUTH_CLIENT_SECRET} +``` + +`seed_grafana_env` in `scripts/deploy-remote.sh` seeds that file from +`dankclaw/.env` with a single-variable filter and fails the deploy when the +value is missing. The container receives the one expanded value, never the file. ## Datasource -Point Grafana at the Prometheus service by name (`http://prometheus:9090`) on the shared podman network. +None is configured. Grafana cannot query anything yet; pointing it at +`http://prometheus:9090` and `http://loki:3100` (by container name, on the +`dankclaw_default` network) is outstanding work — see the state note in +[[Observability]]. ## Healthcheck @@ -49,6 +57,13 @@ Grafana responds on `/api/health` with 200 and `"database":"ok"`. claims reach Grafana until the evaluator bug is fixed — do not write role mappings that depend on those claims. The local admin account stays the only Admin. +- **Never point this unit's `EnvironmentFile` at the repo-root `.env`.** That + file is a gluetun env file and it carries `HOME=/root`; systemd imports it + for the service, podman then looks for its configuration under `/root`, + cannot read it, and the unit dies at start with + `stat /root/.config: permission denied`. The unit reads + `/home/dank/Containers/grafana/env`, which the deployer seeds with exactly + one variable. ## See Also diff --git a/serverAppConfigs-docs/Infrastructure/Loki.md b/serverAppConfigs-docs/Infrastructure/Loki.md index d7c317e..d0834a6 100644 --- a/serverAppConfigs-docs/Infrastructure/Loki.md +++ b/serverAppConfigs-docs/Infrastructure/Loki.md @@ -1,51 +1,54 @@ # Loki -Loki is the log aggregation backend of the observability stack. It stores and queries container and system logs, and it feeds the log panels in Grafana. It replaces the need for a full ELK stack. +Loki is the log aggregation backend of the observability stack. It stores and queries container and system logs. It replaces the need for a full ELK stack. -**Host:** Dankclaw — **Docs:** https://grafana.com/docs/loki/latest +**Host:** Dankclaw — **Port:** 3100 (loopback) — **Docs:** https://grafana.com/docs/loki/latest -## Facts (from research) +## Facts | Item | Value | |------|-------| -| Image | `franksdank/loki:latest` (local build with a static busybox healthcheck) | -| Port | 3100 (host) → 3100 (container) | +| Image | `franksdank/loki:latest` (local build: upstream + a static busybox for the healthcheck) | +| Published | `127.0.0.1:3100` → 3100, loopback only | +| Unit | `dankclaw/observability/quadlet/loki.container` → `loki.service` | | Mode | single-binary (all components in one process) | -| Storage | filesystem (bind mount) | +| Storage | filesystem, `/opt/loki/data` | | Auth | `auth_enabled: false` (single-tenant homelab) | | Schema | v13 / TSDB | +| Retention | 30 days, enforced by the compactor | ## Configuration -Config lives at `shared/loki/loki-config.yaml`. Key settings: +`shared/loki/loki-config.yaml` is the source of truth, installed to `/opt/loki/config/loki-config.yaml` on every deploy. Key settings: -- `schema_config` v13 with `store: tsdb` and filesystem object store. +- `schema_config` v13 with `store: tsdb` and a filesystem object store. - `compactor` with `retention_enabled: true` so logs do not grow without bound. - `replication_factor: 1` for a single instance. ## Volumes -| Container path | Host path (convention) | -|----------------|------------------------| +| Container path | Host path | +|----------------|-----------| | `/loki` | `/opt/loki/data` | -| `/etc/loki/loki-config.yaml` | `shared/loki/loki-config.yaml` (bind mount) | +| `/etc/loki` | `/opt/loki/config` | ## Healthcheck -Loki responds on `/ready` with 200. +Loki responds on `/ready` with 200. The health command is **not** in the unit: a unit `HealthCmd` reaches podman as `CMD-SHELL`, i.e. `/bin/sh -c`, and this image is distroless with no shell at all. It ships in `dankclaw/loki/Containerfile` as an exec-form `HEALTHCHECK` instead (which is why the image carries busybox, and why the build must pass `--format docker` — the OCI image format has no `HEALTHCHECK` field and silently drops it). ## Log Shipping -The [[Alloy]] agent on each host ships container and journal logs to Loki. Grafana queries Loki through the datasource in [[Observability]]. +The [[Alloy]] agent on Dankclaw ships container and journal logs to `http://loki:3100/loki/api/v1/push` — by container name on the `dankclaw_default` network, never through the vhost (a non-browser client cannot solve the Anubis challenge). ## Gotchas -- Filesystem storage grows; enable compactor retention. -- Loki and Promtail/Alloy must agree on the push URL: `http://loki:3100/loki/api/v1/push`. -- For multi-host shipping, Loki's HTTP listener binds `0.0.0.0`. +- Filesystem storage grows; keep compactor retention on. +- Loki and Alloy must agree on the push URL: `http://loki:3100/loki/api/v1/push`. +- The listener is loopback-only now. Shipping logs from a second host would need the unit's publish changed (or a shared network), not just an Alloy install — Loki has no authentication of its own, so that decision is a security one, not a config one. +- Nothing queries Loki yet: Grafana has no datasource configured. See the state note in [[Observability]]. ## See Also -- [[Observability]] - [[Alloy]] — the log shipper - [[Grafana]] — the query frontend +- [[Observability]] — the whole stack diff --git a/serverAppConfigs-docs/Infrastructure/Node-Exporter.md b/serverAppConfigs-docs/Infrastructure/Node-Exporter.md index 58121f5..4ca283d 100644 --- a/serverAppConfigs-docs/Infrastructure/Node-Exporter.md +++ b/serverAppConfigs-docs/Infrastructure/Node-Exporter.md @@ -1,55 +1,56 @@ # Node-Exporter -Node-Exporter exposes host metrics (CPU, RAM, disk, network) to Prometheus. One instance runs per host inside its own container. +Node-Exporter exposes host metrics (CPU, RAM, disk, network) to Prometheus. One instance runs per host, each in its own container. -**Hosts:** Dankserver + Dankclaw — **Docs:** https://prometheus.io/docs/guides/node-exporter +**Hosts:** Dankclaw (unit) + Dankserver (compose service) — **Docs:** https://prometheus.io/docs/guides/node-exporter -## Facts (from research) +## Facts | Item | Value | |------|-------| | Image | `prom/node-exporter:latest` | -| Port | 9100 (host) | -| Network | `host` mode | -| Config | `--path.procfs=/host/proc`, `--path.sysfs=/host/sys`, `--path.rootfs=/rootfs` | +| Listen addresses | `127.0.0.1:9100` and `192.168.0.143:9100` (Dankclaw) | +| Network | `Network=host` + `--pid=host` | +| Unit | `dankclaw/observability/quadlet/node-exporter.container` → `node-exporter.service` | +| Paths | `--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/rootfs` | +| Auth | none — it is an unauthenticated metrics endpoint by design | -## Configuration +## Why it is the one LAN listener -```yaml -services: - node-exporter: - image: prom/node-exporter:latest - restart: always - network_mode: host - pid: host - volumes: - - /proc:/host/proc:ro - - /sys:/host/sys:ro - - /:/rootfs:ro - command: - - --path.procfs=/host/proc - - --path.sysfs=/host/sys - - --path.rootfs=/rootfs -``` +Every other observability service publishes on loopback. This one cannot: + +- **Prometheus** runs in a container on the `dankclaw_default` network and cannot reach the host's loopback address, so it needs the host's LAN IP; `shared/prometheus/prometheus.yml` scrapes `192.168.0.143:9100` under the `node-dankclaw` job. +- **The homepage status timer** runs on the host and probes `127.0.0.1:9100`. + +The unit therefore binds exactly those two addresses with two `--web.listen-address` flags — never the wildcard. It has no authentication, so it is never vhosted and never published beyond the LAN. + +## Mounts + +| Container path | Host path | +|----------------|-----------| +| `/host/proc` | `/proc` (read-only) | +| `/host/sys` | `/sys` (read-only) | +| `/rootfs` | `/` (read-only) | + +The mount paths and the `--path.*` flags are a pair: changing one without the other gives an exporter that starts cleanly and reports nothing. ## Prometheus Targets -Prometheus scrapes each host's node-exporter directly: - | Host | Target | |------|--------| -| Dankserver | `192.168.0.235:9100` | | Dankclaw | `192.168.0.143:9100` | +| Dankserver | `192.168.0.235:9100` | -These scrape jobs live in the Prometheus config; see [[Prometheus]] and [[Observability]]. +The scrape jobs live in `shared/prometheus/prometheus.yml`; see [[Prometheus]]. The `tests/observability_quadlet.bats` seam pins the Dankclaw address to this unit's listen address so the two cannot drift apart. ## Gotchas -- Requires `network_mode: host` (or a published port) so Prometheus can reach 9100. -- The `/proc`, `/sys`, and root mounts must be `read-only` (`:ro`). -- Do not mount the same paths on both hosts' containers; each node-exporter reflects only its own host. +- Host networking is required (or a published port); the deploy's health wait probes `127.0.0.1:9100/metrics`. +- The `/proc`, `/sys` and root mounts must be read-only. +- Each host's instance reflects only its own host; do not try to share one. +- Collectors are the image defaults — no per-collector tuning and no textfile collector for custom metrics. ## See Also - [[Prometheus]] — the scraper -- [[Observability]] — the full stack \ No newline at end of file +- [[Observability]] — the full stack, including what is still unconfigured diff --git a/serverAppConfigs-docs/Infrastructure/Observability.md b/serverAppConfigs-docs/Infrastructure/Observability.md index 73818bb..6500bb5 100644 --- a/serverAppConfigs-docs/Infrastructure/Observability.md +++ b/serverAppConfigs-docs/Infrastructure/Observability.md @@ -1,26 +1,85 @@ +--- +type: reference +title: Observability on Dankclaw +description: The Dankclaw observability stack — Prometheus, Grafana, Loki, Alloy and node-exporter: what runs, how it is deployed, how it is exposed, and what is still unconfigured. +tags: [observability, prometheus, grafana, loki, alloy, node-exporter, quadlet, dankclaw] +timestamp: 2026-09-11 +--- + # Observability -The observability stack runs on Dankclaw: Grafana for dashboards, Prometheus for metrics, and log aggregation for container logs. This is a small homelab — no Kubernetes. +The observability stack runs on Dankclaw: Prometheus for metrics, Loki for logs, Alloy as the shipper, Grafana for dashboards and node-exporter for host metrics. This is a small homelab — no Kubernetes. ## Stack ```mermaid graph LR S[Services] -->|/metrics| P[Prometheus] - S -->|logs| T[Alloy] - T --> L[Loki] + H[Host] -->|":9100"| P + S -->|logs| A[Alloy] + J[journald] --> A + A --> L[Loki] P --> G[Grafana] L --> G ``` -| Component | Role | Image | Port | -|-----------|------|-------|------| -| [[Prometheus]] | metrics collection | `prom/prometheus` | 9090 | -| [[Grafana]] | dashboards | `grafana/grafana` | 3002 | -| Loki | log storage & query | `grafana/loki` | 3100 | -| Alloy | log shipping (per host) | `grafana/alloy` | — | +| Component | Role | Image | Published | +|-----------|------|-------|-----------| +| [[Prometheus]] | metrics collection | `prom/prometheus` | `127.0.0.1:9090` | +| [[Grafana]] | dashboards | `grafana/grafana` | `127.0.0.1:3002` | +| [[Loki]] | log storage & query | `franksdank/loki` (local build) | `127.0.0.1:3100` | +| [[Alloy]] | log shipping | `franksdank/alloy` (local build) | `127.0.0.1:12345` | +| [[Node-Exporter]] | host metrics | `prom/node-exporter` | `192.168.0.143:9100`, host netns | -## Log Aggregation Choice +## Current state: installed, healthy, and UNCONFIGURED + +Recorded 2026-09-11, on the day the stack moved to Quadlet units. Everything in the table above is deployed, healthy and boot-persistent. Almost nothing is wired up beyond the default install: + +- **Grafana has zero datasources and zero dashboards.** `grafana.db` reports 0 rows in `data_source` and 0 in `dashboard`; `/opt/grafana/provisioning/` is empty, and the only account is the local `admin`. Nothing in the UI can query Prometheus or Loki yet. Datasources, dashboards, and provisioning-as-code are all outstanding work. +- **Prometheus has no alerting at all.** `/opt/prometheus/config/` contains `prometheus.yml` and nothing else — no rule files, no recording rules — and there is no Alertmanager anywhere in the stack. It scrapes and retains; it never notifies anyone. +- **Loki is configured but has no reader.** Storage is filesystem with 30-day retention, `auth_enabled: false`, and the push path works; nothing queries it except manual Explore once Grafana is wired up. +- **node-exporter runs its default collector set** — no collector tuning, no textfile collector for custom metrics. +- **Alloy is the one configured piece.** It ships the host journal and podman container logs to Loki with `job` labels, and that pipeline is live. + +The honest summary: metrics and logs are being collected and retained, and nothing is being visualized, correlated or alerted on. Any claim of "observability" beyond raw collection is aspirational until Grafana has datasources and there is at least one alerting path. + +## Deployment + +All five are rootless **Quadlet units** on Dankclaw. The units live in `dankclaw/observability/quadlet/.container` and install to `~/.config/containers/systemd/`, generating `.service`. + +They left `dankclaw/compose.yaml` on 2026-09-11, after a reboot proved that compose-only containers do not come back on their own — the same gap that had already taken Caddy and Pi-hole down at earlier reboots. + +```bash +./scripts/deploy.sh dankclaw --service loki # or prometheus|grafana|alloy|node-exporter +``` + +`deploy_observability` in `scripts/deploy-remote.sh` installs the unit, restarts it and waits on its endpoint. Ordering lives in the units themselves (Grafana wants Prometheus, Alloy wants Loki), and `SERVICE_ORDER` in the planner puts each dependency first. + +Every unit carries a healthcheck, so a wedged process reads `unhealthy` in `podman ps` rather than merely `Up`. See [[Loki]] for the one unit that cannot declare it inline. + +## Exposure + +The rule is the same as for every other service: the only off-host path is a guarded vhost (`.franksdank.cloud` → Anubis → Authentik embedded outpost). Two consequences are easy to get wrong: + +- **Prometheus, Grafana, Loki and Alloy publish on `127.0.0.1` only.** Caddy reaches them by container name over the `dankclaw_default` network, and the host-side homepage status timer probes loopback. Publishing on `0.0.0.0` — what the compose services did — put an unauthenticated metrics API (with the lifecycle endpoint enabled), dashboard, log store and log-ingest endpoint on every host interface, which is a bypass around the guard chain. +- **node-exporter is the one deliberate LAN listener.** Prometheus runs in a container and cannot reach the host's loopback, so the unit binds exactly two addresses: `127.0.0.1:9100` for the homepage status probe and `192.168.0.143:9100` for Prometheus. It authenticates nothing, so it never gets a vhost. + +## Scrape surface + +`shared/prometheus/prometheus.yml` is the source of truth (installed to `/opt/prometheus/config/` on every deploy). It scrapes: + +| Job | Target | Notes | +|-----|--------|-------| +| `prometheus` | `localhost:9090` | itself | +| `node-dankclaw`, `node-dankserver` | `192.168.0.143:9100`, `192.168.0.235:9100` | one node-exporter per host | +| `gitea` | `192.168.0.143:3000` | published LAN port (Gitea is on an isolated network) | +| `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on `dankclaw_default` | +| `comfyui` | `comfyui:8188` | UI probe, no native exporter | +| `services` | `192.168.0.143:8081`, `:3005` | SearXNG, OpenWebUI up/down probes | + +For services without a native exporter, use the Prometheus **blackbox exporter** in probe mode to assert up/down and HTTP latency. The homepage status timer probes the same endpoints from the host. + +## Log aggregation choice Loki + Alloy is the lightweight Grafana-native option for a two-server homelab: @@ -30,24 +89,9 @@ Loki + Alloy is the lightweight Grafana-native option for a two-server homelab: Alternatives considered and rejected: ELK (too heavy), Promtail (EOL), VictoriaLogs (newer, less documentation), plain `journald` (no centralized query). -## Scrape Surface - -| Target | Source | -|--------|--------| -| Prometheus self | `localhost:9090/metrics` | -| ComfyUI | `/system_stats` (limited) or blackbox probe | -| Gitea | `/metrics` (built-in exporter) | -| Grafana | `/metrics` (built-in) | -| Host metrics | node-exporter `:9100` | - -For services without native exporters, use the Prometheus **blackbox exporter** in probe mode to assert up/down and HTTP latency. The homepage status also probes the same endpoints. - -## Healthchecks - -Every compose service gets a healthcheck so podman reports container health and Promtail/Grafana can alert on it. See [[Deployment Flow]] for the verify step. - ## See Also -- [[Grafana]], [[Prometheus]] -- [[Dankclaw Stack]] -- [[Homepage]] — the user-facing status surface +- [[Grafana]], [[Prometheus]], [[Loki]], [[Alloy]], [[Node-Exporter]] — the components +- [[Dankclaw Stack]] — the rest of the host +- [[Homepage]] — the user-facing status surface, including the five status cards +- [[Deployment Flow]] — the verify-before-declared-deployed contract diff --git a/serverAppConfigs-docs/Infrastructure/Prometheus.md b/serverAppConfigs-docs/Infrastructure/Prometheus.md index 69808a7..f01366e 100644 --- a/serverAppConfigs-docs/Infrastructure/Prometheus.md +++ b/serverAppConfigs-docs/Infrastructure/Prometheus.md @@ -1,59 +1,59 @@ # Prometheus -Prometheus collects metrics from every service on Dankclaw by scraping their `/metrics` endpoints. It feeds the Grafana dashboards. The `node-exporter` containers may add podman/cgroup metrics for full host visibility. +Prometheus collects metrics from every service on Dankclaw by scraping their `/metrics` endpoints, and feeds Grafana once a datasource exists. node-exporter supplies host metrics from both machines. **Host:** Dankclaw — **Vhost:** `prometheus.franksdank.cloud` (guarded: Anubis → Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) — -**Port:** 9090 — **Docs:** https://prometheus.io/docs +**Port:** 9090 (loopback) — **Docs:** https://prometheus.io/docs ## Facts | Item | Value | |------|-------| | Image | `docker.io/prom/prometheus:latest` | -| Port | 9090 (host) → 9090 (container) | -| Config | `/etc/prometheus/prometheus.yml` (bind mount) | -| Data volume | TSDB | +| Published | `127.0.0.1:9090` → 9090, loopback only (see [[Observability]] → Exposure) | +| Unit | `dankclaw/observability/quadlet/prometheus.container` → `prometheus.service` | +| Config | `/opt/prometheus/config/prometheus.yml`, installed from `shared/prometheus/prometheus.yml` on every deploy | +| Data | `/opt/prometheus/data` (TSDB) | | Retention | `--storage.tsdb.retention.time=30d` | +| Alerting | **none** — no rule files, no Alertmanager anywhere in the stack | -## prometheus.yml Targets +## Targets -```yaml -scrape_configs: - - job_name: "prometheus" - static_configs: - - targets: ["localhost:9090"] +`shared/prometheus/prometheus.yml` is the source of truth; the deploy copies it into the container's `/etc/prometheus/`. In outline: - - job_name: "comfyui" - static_configs: - - targets: ["comfyui:8188"] +| Job | Target | Why | +|-----|--------|-----| +| `prometheus` | `localhost:9090` | itself | +| `node-dankclaw`, `node-dankserver` | `192.168.0.143:9100`, `192.168.0.235:9100` | host metrics from each machine's [[Node-Exporter]] | +| `gitea` | `192.168.0.143:3000` | Gitea's built-in exporter, reached on its published LAN port | +| `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on the `dankclaw_default` network | +| `comfyui` | `comfyui:8188` | UI probe; ComfyUI has no native `/metrics` | +| `services` | `192.168.0.143:8081`, `:3005` | up/down probes for SearXNG and OpenWebUI | - - job_name: "node" - static_configs: - - targets: ["node-exporter:9100"] -``` - -Allow the `/metrics` endpoint on each service where the exporter exists. For services without a native exporter, use the Prometheus `blackbox` exporter (probe mode) to check up/down and latency. +Targets are `host:port`, never URLs. For richer HTTP assertions, add the Prometheus **blackbox** exporter in probe mode rather than scraping HTML. ## Volumes -| Container path | Host path (convention) | -|----------------|------------------------| +| Container path | Host path | +|----------------|-----------| | `/etc/prometheus` | `/opt/prometheus/config` | | `/prometheus` | `/opt/prometheus/data` | ## Healthcheck -Prometheus responds on `/-/healthy` with 200. +Prometheus responds on `/-/healthy` with 200; the unit's health command is the same `wget --spider` the deploy waits on. ## Gotchas +- **Keep the node-exporter address in step with the unit.** `node-dankclaw` scrapes `192.168.0.143:9100`, which is exactly the LAN address `node-exporter.container` binds (`Network=host`, `--web.listen-address` twice). Change one without the other and host metrics stop with no error anywhere. The `tests/observability_quadlet.bats` seam pins the pairing. +- The container name `prometheus` is what Caddy's post-auth handler proxies to, and `grafana`/`loki` are addressed by name here. Renaming a container breaks both silently. - Keep scrapes modest for a small homelab; too many targets bloat the TSDB. -- Use recording rules for heavy queries (see [[Grafana]]). -- Set a retention policy or the data volume grows without bound. +- Set a retention policy or the data volume grows without bound (30 days here). +- The lifecycle endpoint is enabled (`--web.enable-lifecycle`), which is why this API must never be published beyond loopback. ## See Also -- [[Dankclaw Stack]] -- [[Grafana]] — the dashboard frontend -- [[Observability]] — the whole stack \ No newline at end of file +- [[Grafana]] — the dashboard frontend (no datasource configured yet) +- [[Node-Exporter]] — the host metrics source +- [[Observability]] — the whole stack, including what is still unconfigured diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index e72b69c..28f287b 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -84,9 +84,9 @@ covers both workloads). - [[authentik-worker]] — Authentik background worker service - [[Homepage]] — password-protected service index - [[Homepage Design Grilling]] — homepage decision log and open questions -- [[Observability]] — Grafana, Prometheus, Loki, Alloy -- [[Grafana]] — dashboards -- [[Prometheus]] — metrics collection +- [[Observability]] — Grafana, Prometheus, Loki, Alloy, node-exporter: the stack, its quadlet deployment, its exposure, and **what is still unconfigured** +- [[Grafana]] — dashboards (no datasource or dashboard configured yet) +- [[Prometheus]] — metrics collection (no alerting rules or Alertmanager) - [[Node-Exporter]] — host metrics - [[Loki]] — log storage - [[Alloy]] — log shipping (replaces Promtail)