dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(alloy): document the root journal-read cutover

Browse Source 
Alloy page rewritten to the shipped root design (why root, the two
mounts that make it work, deploy-pending status); Observability current
state marks the log pipeline fixed-in-code, awaiting deploy.
This commit is contained in:
Frank committed 2026-10-20 19:50:15 -07:00
1 parent 9809ed9c74
commit e494617599
2 files changed
+91 -75
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

serverAppConfigs-docs/Infrastructure/Alloy.md
+80 -65
View File
@@ -3,96 +3,111 @@ type: reference
description: "Alloy is Grafana's log and telemetry shipper." description: "Alloy is Grafana's log and telemetry shipper."
tags: tags:
- infrastructure - infrastructure
timestamp: 2026-09-11 timestamp: 2026-10-20
--- ---
# Alloy # Alloy
Alloy is Grafana's log and telemetry shipper. It replaces Promtail (end of life as of 2026-03-02, https://grafana.com/docs/loki/latest/send-data/promtail/). It is wired to forward container and journal logs to Loki, and every unit in the path is healthy — but as of 2026-10-18 it is delivering **zero** log lines. See "Delivery state" below. Alloy is Grafana's log and telemetry shipper. It replaces Promtail (end of life
as of 2026-03-02, https://grafana.com/docs/loki/latest/send-data/promtail/). It
forwards the host journal — including the rootless Quadlet containers' logs —
to Loki. Unlike its observability siblings it runs as a **root system service on
the host network**, because a rootless container cannot read the journal (see
*Why root* below).
**Host:** Dankclaw (one instance per host) — **Port:** 12345 (loopback) — **Docs:** https://grafana.com/docs/alloy/latest **Host:** Dankclaw (one instance per host) — **UI/health:** 127.0.0.1:12345
(loopback) — **Docs:** https://grafana.com/docs/alloy/latest
## Facts ## Facts
| Item | Value | | Item | Value |
|------|-------| |------|-------|
| Image | `franksdank/alloy:latest` (local build: upstream + a static busybox for the healthcheck) | | Image | `localhost/franksdank/alloy:latest` — built rootless, then mirrored into the root image store by the deployer (a root build fails on DNS in this host's build container) |
| Published | `127.0.0.1:12345` → 12345 (its own UI/health port) | | Unit | `dankclaw/observability/quadlet/alloy.container` → system `alloy.service`, installed at `/etc/containers/systemd/` (root manager) |
| Unit | `dankclaw/observability/quadlet/alloy.container` → `alloy.service` | | Config | `shared/alloy/config.alloy` → `/opt/alloy/config/config.alloy` on every deploy |
| Config | `shared/alloy/config.alloy`, installed to `/opt/alloy/config/config.alloy` on every deploy (`scripts/deploy-remote.sh:281-284`) | | Network | `host` — required so it can reach Loki's `127.0.0.1:3100` publish |
| Inputs | host journald socket, podman container logs, `/var/log` | | UI | `--server.http.listen-addr=127.0.0.1:12345` (loopback; nothing off-host reaches it) |
| Output | `http://loki:3100/loki/api/v1/push` (`shared/alloy/config.alloy:13-17`) | | Inputs | host journal: `system.journal` + `user-*.journal` (root reads both) |
| Successor to | Promtail (EOL 2026-03-02, https://grafana.com/docs/loki/latest/send-data/promtail/) | | Output | `http://127.0.0.1:3100/loki/api/v1/push` |
| Boot persistence | `Restart=always`/`RestartSec=10` + `WantedBy=default.target` (`dankclaw/observability/quadlet/alloy.container:58-63`); installed at `~/.config/containers/systemd/alloy.container`, reporting `generated` + `active` (`systemctl --user is-enabled|is-active alloy` 2026-10-18) | | Boot persistence | `Restart=always`/`RestartSec=10` + `WantedBy=multi-user.target`; boot persistence is the `[Install]` section (generated system units cannot be `enable`d here) |
| Delivery | **Configured and healthy, currently delivering zero data** — see "Delivery state" | | Successor to | Promtail (EOL 2026-03-02) |
| Deploy status | root cutover built and tested in-repo (2026-10-20); the **live** instance is still the old rootless unit until `./scripts/deploy.sh dankclaw --service alloy` runs |
## Why Alloy and not Promtail ## Why root (the load-bearing decision)
Promtail reached end-of-life on 2026-03-02 and Grafana Labs directs new installs to Alloy. This stack uses Alloy from day one: no Promtail config or unit has ever existed in this repository (`git log -S promtail` over the whole history returns no commit that added one; the compose-era service of the same name was already Alloy — `dankclaw/observability/quadlet/alloy.container:6-7`). A rootless Alloy delivered **zero** lines, silently and with a green health
check. Two independent causes, both proven on the host 2026-10-20:
1. **The journal is not readable by a rootless container.** The files are
`root:systemd-journal 0640` (`group:adm`, `other::---`). Even bind-mounted, a
rootless container has no supplementary groups, so opening them returns
`DENIED`. `loki.source.journal` then sees an empty view and the counter just
sits at 0 — no error logged.
2. **Container logs live in the *user* journal, not `ctr.log`.** The rootless
Quadlet containers log via the journald driver into `user-1000.journal`; the
old file globs (`/var/lib/containers/.../ctr.log`, `/var/log/containers/*.log`)
point at a root-storage layout that does not exist here.
**Root fixes both at once.** As root the container opens every journal file
under the machine-id directory — `system.journal` *and* `user-1000.journal` —
so a single `loki.source.journal` carries host services and the rootless
containers' logs together. Validated with a throwaway root probe before writing
the unit: **31,825 journal lines read, 26,966 pushed to Loki in 20 s, zero
drops**, and Loki's label set went from empty to populated.
### The two mounts that make it work
- **`/etc/machine-id` (ro) — load-bearing.** `sd_journal` locates the journal
directory by the *local* machine id. Without it the container opens nothing and
the journal source reads exactly 0 lines — silently, health green. This single
mount is the whole difference between a working and an empty pipeline; do not
remove it. (A root probe without it reads 0; with it, 31,825.)
- **`Network=host`.** Loki publishes `127.0.0.1:3100` (loopback only). No bridge
container — rootless or root — can reach the host's loopback, so Alloy must
share the host network to push to `127.0.0.1:3100`. Consequence: the UI binds
`127.0.0.1` directly (no `PublishPort`), so it stays loopback-only as before.
## Configuration ## Configuration
`shared/alloy/config.alloy` is the source of truth: `shared/alloy/config.alloy` is the source of truth — one `loki.source.journal`
(labelled `job="systemd-journal"`) forwarding to the loopback `loki.write`
- Reads journald logs from the host's `/run/systemd/journal/socket`, labelled `job="systemd-journal"` (`shared/alloy/config.alloy:20-26`). endpoint. There is deliberately **no** container-file source: container stdout is
- Reads container logs from two globs: `/var/lib/containers/storage/overlay-containers/*/userdata/ctr.log` (`job="podman"`) and `/var/log/containers/*.log` (`job="containers"`) (`config.alloy:29-35`). in the journal.
- Forwards everything to `http://loki:3100/loki/api/v1/push` by container name on `dankclaw_default` (`config.alloy:13-17`).
## Volumes ## Volumes
| Container path | Host path | | Container path | Host path |
|----------------|-----------| |----------------|-----------|
| `/etc/alloy` | `/opt/alloy/config` | | `/etc/alloy` | `/opt/alloy/config` (ro) |
| `/run/systemd/journal/socket` | host journald socket (read-only) | | `/etc/machine-id` | host machine id (ro) — **required** |
| `/var/log` | host logs (read-only) | | `/run/systemd/journal/socket` | host journald socket (ro) |
| `/var/lib/containers` | podman storage (read-only) | | `/var/log/journal` | persistent journal files (ro) |
| `/run/log/journal` | runtime journal files (ro) |
## The UI port is published on purpose ## The UI stays on loopback
Alloy's UI listens on the container's own `127.0.0.1:12345` by default, and Alloy 1.x refuses to configure the `ui` service from a config block ("service ui does not support being configured", `shared/alloy/config.alloy:8-10`). Under compose that meant **no reachable health endpoint at all**: the homepage status card for alloy read `down` continuously and a deploy had nothing to wait on. The unit therefore passes `--server.http.listen-addr=0.0.0.0:12345` (`dankclaw/observability/quadlet/alloy.container:46`) and publishes `127.0.0.1:12345` (`alloy.container:47`, confirmed by `ss -ltnp` 2026-10-18) — same posture as the other three: reachable from the host and from Caddy's network, and from nothing off-host. Alloy has no authentication, which is exactly why it is not published wider. The homepage card probes that same loopback port (`shared/homepage/services.json:233`). Alloy 1.x refuses to configure the `ui` service from a config block, so the UI
address is set on the command line instead. Under the old rootless unit that
## Delivery state (2026-10-20): healthy, shipping nothing meant `0.0.0.0` inside the container plus a published `127.0.0.1:12345`; under
`Network=host` the listen-addr is `127.0.0.1:12345` directly — the host's
The unit is `active`, the container reports `healthy`, the log says `Alloy is loopback, reachable from the host and from nothing off-host. Alloy has no
ready`, and the pipeline carries no data at all. authentication, which is exactly why it is never published wider. The homepage
card and the deploy health gate both dial `http://127.0.0.1:12345/-/healthy`.
**Root cause, proven: a rootless container cannot read the host journal.** The
journal files are `root:systemd-journal 0640` (with `group:adm`, `other::---`).
`/var/log` is bind-mounted so the files are *visible*, but a rootless container
spawns with no supplementary groups, so it is denied. Proven directly:
`podman run --rm -v /var/log/journal:/var/log/journal:ro busybox cat
/var/log/journal/*/system.journal` → `DENIED`. `loki.source.journal` opens via
`sd_journal_open`, so it silently sees an empty view — no error logged, counter
frozen at `loki_source_journal_target_lines_total 0` (`system.journal` and
`user-1000.journal` are both present and non-trivial, 2026-10-20).
**Second, independent gap:** the rootless Quadlet containers log to the *user*
journal (`user-1000.journal`), and `loki.source.journal` reads the *system*
journal only — a fixed system read would still miss them. Container stdout is
not in a `ctr.log` here either: the file globs point at root storage
(`/var/lib/containers/storage/.../ctr.log`, `/var/log/containers/*.log`) that
rootless does not use, refused at start with `stat …: permission denied` / `no
such file or directory`, `loki_source_file_files_active_total 0`.
Downstream: Loki holds zero streams (`GET /loki/api/v1/labels` → no `data`) —
see [[Loki]].
**Closing it is a permission/architecture decision, not a config edit.** The
unit, config and push endpoint are correct as written; rootless Podman simply
cannot grant the journal group. The viable shapes: run Alloy as a *system*
service (root, in `adm`/`systemd-journal`) instead of a rootless Quadlet; grant
the container's mapped uid journal read via a tmpfiles/ACL rule; or have
journald forward to a path the container can read. Each trades rootless posture
for reach — decide deliberately. Re-check `loki_source_journal_target_lines_total`
and the Loki label query to confirm a fix, never `/healthy`: health is exactly
what hides this failure.
## Gotchas ## Gotchas
- The journald socket mount is required for journal scraping. - **No `/etc/machine-id` mount ⇒ silent zero.** The health check still passes.
- Container log paths differ between docker (`/var/lib/docker/containers`) and podman (`/var/lib/containers/storage/...`); point the config at the right one per host. This is the failure mode that hid the broken pipeline; check
- One Alloy per host; do not run a single instance that tries to reach both hosts' logs. `loki_source_journal_target_lines_total` and the Loki label query, never
- The unit `Wants=`/`After=` `loki.service`: without a reachable Loki the write client fails and the pipeline backs up. `/healthy`.
- **`After=loki.service` is gone.** Alloy is on the system manager; Loki is a
rootless user unit — a system unit cannot order against a user unit. Alloy
retries its push until Loki answers, so ordering is unnecessary.
- **The image must be in the ROOT store.** The deployer builds rootless and
mirrors (`mirror_image_to_root`); a `Pull=never` root quadlet with only a
rootless tag fails at create.
- One Alloy per host; do not run a single instance that tries to reach both
hosts' logs.
## See Also ## See Also
serverAppConfigs-docs/Infrastructure/Observability.md
+11 -10
View File
@@ -102,17 +102,18 @@ verify it through the APIs.
installed from `shared/grafana/provisioning/`. The Loki datasource is how installed from `shared/grafana/provisioning/`. The Loki datasource is how
you read logs: Loki has no web UI of its own, so Grafana Explore is the you read logs: Loki has no web UI of its own, so Grafana Explore is the
human surface (the guarded `loki.franksdank.cloud` vhost serves only the API). human surface (the guarded `loki.franksdank.cloud` vhost serves only the API).
* **Logs: configured, empty.** Alloy ships zero lines and Loki holds zero * **Logs: fixed in code, awaiting deploy.** The pipeline shipped nothing because
streams (`GET /loki/api/v1/labels` returns no label names; no chunk files on a rootless Alloy cannot read the `0640` journal files, and the container-log
disk). The cause is source-side access, now proven, not a config bug: a globs point at a root-storage layout that does not exist here. The fix is built
rootless container cannot read the journal files (`system.journal` is and tested in-repo (2026-10-20): Alloy is now a **root system service on
`root:systemd-journal 0640`/`adm` and a rootless read of it returns DENIED), `Network=host`**, reading `system.journal` + `user-*.journal` and pushing to
and the container-log globs point at a root-storage path that does not exist Loki's `127.0.0.1:3100` loopback publish — a throwaway root probe read 31,825
under rootless. Closing it is a permission/architecture decision, not an lines and pushed 26,966 with zero drops. The **live** instance is still the old
edit — see [[Alloy]] → Delivery state and [[Loki]] → Contents. rootless unit until `deploy.sh dankclaw --service alloy` runs. Detail in
[[Alloy]] → Why root; Loki's store state in [[Loki]] → Contents.
* **Not configured:** no blackbox exporter (the structural downs have no real * **Not configured:** no blackbox exporter (the structural downs have no real
uptime signal) and no Alertmanager vhost. The log pipeline is provisioned uptime signal) and no Alertmanager vhost. The log pipeline's fix is in code but
but carries no data (see *Logs* above). not yet deployed (see *Logs* above).
## Citations ## Citations