From e494617599da69b678f5ea1736c0316b7fb42bb8 Mon Sep 17 00:00:00 2001 From: Frank Date: Tue, 20 Oct 2026 19:50:15 -0700 Subject: [PATCH] docs(alloy): document the root journal-read cutover Alloy page rewritten to the shipped root design (why root, the two mounts that make it work, deploy-pending status); Observability current state marks the log pipeline fixed-in-code, awaiting deploy. --- serverAppConfigs-docs/Infrastructure/Alloy.md | 145 ++++++++++-------- .../Infrastructure/Observability.md | 21 +-- 2 files changed, 91 insertions(+), 75 deletions(-) diff --git a/serverAppConfigs-docs/Infrastructure/Alloy.md b/serverAppConfigs-docs/Infrastructure/Alloy.md index 4175a9c..6d96169 100644 --- a/serverAppConfigs-docs/Infrastructure/Alloy.md +++ b/serverAppConfigs-docs/Infrastructure/Alloy.md @@ -3,96 +3,111 @@ type: reference description: "Alloy is Grafana's log and telemetry shipper." tags: - infrastructure -timestamp: 2026-09-11 +timestamp: 2026-10-20 --- # Alloy -Alloy is Grafana's log and telemetry shipper. It replaces Promtail (end of life as of 2026-03-02, https://grafana.com/docs/loki/latest/send-data/promtail/). It is wired to forward container and journal logs to Loki, and every unit in the path is healthy — but as of 2026-10-18 it is delivering **zero** log lines. See "Delivery state" below. +Alloy is Grafana's log and telemetry shipper. It replaces Promtail (end of life +as of 2026-03-02, https://grafana.com/docs/loki/latest/send-data/promtail/). It +forwards the host journal — including the rootless Quadlet containers' logs — +to Loki. Unlike its observability siblings it runs as a **root system service on +the host network**, because a rootless container cannot read the journal (see +*Why root* below). -**Host:** Dankclaw (one instance per host) — **Port:** 12345 (loopback) — **Docs:** https://grafana.com/docs/alloy/latest +**Host:** Dankclaw (one instance per host) — **UI/health:** 127.0.0.1:12345 +(loopback) — **Docs:** https://grafana.com/docs/alloy/latest ## Facts | Item | Value | |------|-------| -| Image | `franksdank/alloy:latest` (local build: upstream + a static busybox for the healthcheck) | -| Published | `127.0.0.1:12345` → 12345 (its own UI/health port) | -| Unit | `dankclaw/observability/quadlet/alloy.container` → `alloy.service` | -| Config | `shared/alloy/config.alloy`, installed to `/opt/alloy/config/config.alloy` on every deploy (`scripts/deploy-remote.sh:281-284`) | -| Inputs | host journald socket, podman container logs, `/var/log` | -| Output | `http://loki:3100/loki/api/v1/push` (`shared/alloy/config.alloy:13-17`) | -| Successor to | Promtail (EOL 2026-03-02, https://grafana.com/docs/loki/latest/send-data/promtail/) | -| Boot persistence | `Restart=always`/`RestartSec=10` + `WantedBy=default.target` (`dankclaw/observability/quadlet/alloy.container:58-63`); installed at `~/.config/containers/systemd/alloy.container`, reporting `generated` + `active` (`systemctl --user is-enabled|is-active alloy` 2026-10-18) | -| Delivery | **Configured and healthy, currently delivering zero data** — see "Delivery state" | +| Image | `localhost/franksdank/alloy:latest` — built rootless, then mirrored into the root image store by the deployer (a root build fails on DNS in this host's build container) | +| Unit | `dankclaw/observability/quadlet/alloy.container` → system `alloy.service`, installed at `/etc/containers/systemd/` (root manager) | +| Config | `shared/alloy/config.alloy` → `/opt/alloy/config/config.alloy` on every deploy | +| Network | `host` — required so it can reach Loki's `127.0.0.1:3100` publish | +| UI | `--server.http.listen-addr=127.0.0.1:12345` (loopback; nothing off-host reaches it) | +| Inputs | host journal: `system.journal` + `user-*.journal` (root reads both) | +| Output | `http://127.0.0.1:3100/loki/api/v1/push` | +| Boot persistence | `Restart=always`/`RestartSec=10` + `WantedBy=multi-user.target`; boot persistence is the `[Install]` section (generated system units cannot be `enable`d here) | +| Successor to | Promtail (EOL 2026-03-02) | +| Deploy status | root cutover built and tested in-repo (2026-10-20); the **live** instance is still the old rootless unit until `./scripts/deploy.sh dankclaw --service alloy` runs | -## Why Alloy and not Promtail +## Why root (the load-bearing decision) -Promtail reached end-of-life on 2026-03-02 and Grafana Labs directs new installs to Alloy. This stack uses Alloy from day one: no Promtail config or unit has ever existed in this repository (`git log -S promtail` over the whole history returns no commit that added one; the compose-era service of the same name was already Alloy — `dankclaw/observability/quadlet/alloy.container:6-7`). +A rootless Alloy delivered **zero** lines, silently and with a green health +check. Two independent causes, both proven on the host 2026-10-20: + +1. **The journal is not readable by a rootless container.** The files are + `root:systemd-journal 0640` (`group:adm`, `other::---`). Even bind-mounted, a + rootless container has no supplementary groups, so opening them returns + `DENIED`. `loki.source.journal` then sees an empty view and the counter just + sits at 0 — no error logged. +2. **Container logs live in the *user* journal, not `ctr.log`.** The rootless + Quadlet containers log via the journald driver into `user-1000.journal`; the + old file globs (`/var/lib/containers/.../ctr.log`, `/var/log/containers/*.log`) + point at a root-storage layout that does not exist here. + +**Root fixes both at once.** As root the container opens every journal file +under the machine-id directory — `system.journal` *and* `user-1000.journal` — +so a single `loki.source.journal` carries host services and the rootless +containers' logs together. Validated with a throwaway root probe before writing +the unit: **31,825 journal lines read, 26,966 pushed to Loki in 20 s, zero +drops**, and Loki's label set went from empty to populated. + +### The two mounts that make it work + +- **`/etc/machine-id` (ro) — load-bearing.** `sd_journal` locates the journal + directory by the *local* machine id. Without it the container opens nothing and + the journal source reads exactly 0 lines — silently, health green. This single + mount is the whole difference between a working and an empty pipeline; do not + remove it. (A root probe without it reads 0; with it, 31,825.) +- **`Network=host`.** Loki publishes `127.0.0.1:3100` (loopback only). No bridge + container — rootless or root — can reach the host's loopback, so Alloy must + share the host network to push to `127.0.0.1:3100`. Consequence: the UI binds + `127.0.0.1` directly (no `PublishPort`), so it stays loopback-only as before. ## Configuration -`shared/alloy/config.alloy` is the source of truth: - -- Reads journald logs from the host's `/run/systemd/journal/socket`, labelled `job="systemd-journal"` (`shared/alloy/config.alloy:20-26`). -- Reads container logs from two globs: `/var/lib/containers/storage/overlay-containers/*/userdata/ctr.log` (`job="podman"`) and `/var/log/containers/*.log` (`job="containers"`) (`config.alloy:29-35`). -- Forwards everything to `http://loki:3100/loki/api/v1/push` by container name on `dankclaw_default` (`config.alloy:13-17`). +`shared/alloy/config.alloy` is the source of truth — one `loki.source.journal` +(labelled `job="systemd-journal"`) forwarding to the loopback `loki.write` +endpoint. There is deliberately **no** container-file source: container stdout is +in the journal. ## Volumes | Container path | Host path | |----------------|-----------| -| `/etc/alloy` | `/opt/alloy/config` | -| `/run/systemd/journal/socket` | host journald socket (read-only) | -| `/var/log` | host logs (read-only) | -| `/var/lib/containers` | podman storage (read-only) | +| `/etc/alloy` | `/opt/alloy/config` (ro) | +| `/etc/machine-id` | host machine id (ro) — **required** | +| `/run/systemd/journal/socket` | host journald socket (ro) | +| `/var/log/journal` | persistent journal files (ro) | +| `/run/log/journal` | runtime journal files (ro) | -## The UI port is published on purpose +## The UI stays on loopback -Alloy's UI listens on the container's own `127.0.0.1:12345` by default, and Alloy 1.x refuses to configure the `ui` service from a config block ("service ui does not support being configured", `shared/alloy/config.alloy:8-10`). Under compose that meant **no reachable health endpoint at all**: the homepage status card for alloy read `down` continuously and a deploy had nothing to wait on. The unit therefore passes `--server.http.listen-addr=0.0.0.0:12345` (`dankclaw/observability/quadlet/alloy.container:46`) and publishes `127.0.0.1:12345` (`alloy.container:47`, confirmed by `ss -ltnp` 2026-10-18) — same posture as the other three: reachable from the host and from Caddy's network, and from nothing off-host. Alloy has no authentication, which is exactly why it is not published wider. The homepage card probes that same loopback port (`shared/homepage/services.json:233`). - -## Delivery state (2026-10-20): healthy, shipping nothing - -The unit is `active`, the container reports `healthy`, the log says `Alloy is -ready`, and the pipeline carries no data at all. - -**Root cause, proven: a rootless container cannot read the host journal.** The -journal files are `root:systemd-journal 0640` (with `group:adm`, `other::---`). -`/var/log` is bind-mounted so the files are *visible*, but a rootless container -spawns with no supplementary groups, so it is denied. Proven directly: -`podman run --rm -v /var/log/journal:/var/log/journal:ro busybox cat -/var/log/journal/*/system.journal` → `DENIED`. `loki.source.journal` opens via -`sd_journal_open`, so it silently sees an empty view — no error logged, counter -frozen at `loki_source_journal_target_lines_total 0` (`system.journal` and -`user-1000.journal` are both present and non-trivial, 2026-10-20). - -**Second, independent gap:** the rootless Quadlet containers log to the *user* -journal (`user-1000.journal`), and `loki.source.journal` reads the *system* -journal only — a fixed system read would still miss them. Container stdout is -not in a `ctr.log` here either: the file globs point at root storage -(`/var/lib/containers/storage/.../ctr.log`, `/var/log/containers/*.log`) that -rootless does not use, refused at start with `stat …: permission denied` / `no -such file or directory`, `loki_source_file_files_active_total 0`. - -Downstream: Loki holds zero streams (`GET /loki/api/v1/labels` → no `data`) — -see [[Loki]]. - -**Closing it is a permission/architecture decision, not a config edit.** The -unit, config and push endpoint are correct as written; rootless Podman simply -cannot grant the journal group. The viable shapes: run Alloy as a *system* -service (root, in `adm`/`systemd-journal`) instead of a rootless Quadlet; grant -the container's mapped uid journal read via a tmpfiles/ACL rule; or have -journald forward to a path the container can read. Each trades rootless posture -for reach — decide deliberately. Re-check `loki_source_journal_target_lines_total` -and the Loki label query to confirm a fix, never `/healthy`: health is exactly -what hides this failure. +Alloy 1.x refuses to configure the `ui` service from a config block, so the UI +address is set on the command line instead. Under the old rootless unit that +meant `0.0.0.0` inside the container plus a published `127.0.0.1:12345`; under +`Network=host` the listen-addr is `127.0.0.1:12345` directly — the host's +loopback, reachable from the host and from nothing off-host. Alloy has no +authentication, which is exactly why it is never published wider. The homepage +card and the deploy health gate both dial `http://127.0.0.1:12345/-/healthy`. ## Gotchas -- The journald socket mount is required for journal scraping. -- Container log paths differ between docker (`/var/lib/docker/containers`) and podman (`/var/lib/containers/storage/...`); point the config at the right one per host. -- One Alloy per host; do not run a single instance that tries to reach both hosts' logs. -- The unit `Wants=`/`After=` `loki.service`: without a reachable Loki the write client fails and the pipeline backs up. +- **No `/etc/machine-id` mount ⇒ silent zero.** The health check still passes. + This is the failure mode that hid the broken pipeline; check + `loki_source_journal_target_lines_total` and the Loki label query, never + `/healthy`. +- **`After=loki.service` is gone.** Alloy is on the system manager; Loki is a + rootless user unit — a system unit cannot order against a user unit. Alloy + retries its push until Loki answers, so ordering is unnecessary. +- **The image must be in the ROOT store.** The deployer builds rootless and + mirrors (`mirror_image_to_root`); a `Pull=never` root quadlet with only a + rootless tag fails at create. +- One Alloy per host; do not run a single instance that tries to reach both + hosts' logs. ## See Also diff --git a/serverAppConfigs-docs/Infrastructure/Observability.md b/serverAppConfigs-docs/Infrastructure/Observability.md index 190d379..713cb2e 100644 --- a/serverAppConfigs-docs/Infrastructure/Observability.md +++ b/serverAppConfigs-docs/Infrastructure/Observability.md @@ -102,17 +102,18 @@ verify it through the APIs. installed from `shared/grafana/provisioning/`. The Loki datasource is how you read logs: Loki has no web UI of its own, so Grafana Explore is the human surface (the guarded `loki.franksdank.cloud` vhost serves only the API). -* **Logs: configured, empty.** Alloy ships zero lines and Loki holds zero - streams (`GET /loki/api/v1/labels` returns no label names; no chunk files on - disk). The cause is source-side access, now proven, not a config bug: a - rootless container cannot read the journal files (`system.journal` is - `root:systemd-journal 0640`/`adm` and a rootless read of it returns DENIED), - and the container-log globs point at a root-storage path that does not exist - under rootless. Closing it is a permission/architecture decision, not an - edit — see [[Alloy]] → Delivery state and [[Loki]] → Contents. +* **Logs: fixed in code, awaiting deploy.** The pipeline shipped nothing because + a rootless Alloy cannot read the `0640` journal files, and the container-log + globs point at a root-storage layout that does not exist here. The fix is built + and tested in-repo (2026-10-20): Alloy is now a **root system service on + `Network=host`**, reading `system.journal` + `user-*.journal` and pushing to + Loki's `127.0.0.1:3100` loopback publish — a throwaway root probe read 31,825 + lines and pushed 26,966 with zero drops. The **live** instance is still the old + rootless unit until `deploy.sh dankclaw --service alloy` runs. Detail in + [[Alloy]] → Why root; Loki's store state in [[Loki]] → Contents. * **Not configured:** no blackbox exporter (the structural downs have no real - uptime signal) and no Alertmanager vhost. The log pipeline is provisioned - but carries no data (see *Logs* above). + uptime signal) and no Alertmanager vhost. The log pipeline's fix is in code but + not yet deployed (see *Logs* above). ## Citations