Files

T

Frank e494617599 docs(alloy): document the root journal-read cutover

Alloy page rewritten to the shipped root design (why root, the two
mounts that make it work, deploy-pending status); Observability current
state marks the log pipeline fixed-in-code, awaiting deploy.

2026-10-20 19:50:15 -07:00

5.8 KiB

Raw Permalink Blame History

type, description, tags, timestamp

type

description

Alloy

Alloy is Grafana's log and telemetry shipper. It replaces Promtail (end of life as of 2026-03-02, https://grafana.com/docs/loki/latest/send-data/promtail/). It forwards the host journal — including the rootless Quadlet containers' logs — to Loki. Unlike its observability siblings it runs as a root system service on the host network, because a rootless container cannot read the journal (see Why root below).

Host: Dankclaw (one instance per host) — UI/health: 127.0.0.1:12345 (loopback) — Docs: https://grafana.com/docs/alloy/latest

Facts

Item	Value
Image	`localhost/franksdank/alloy:latest` — built rootless, then mirrored into the root image store by the deployer (a root build fails on DNS in this host's build container)
Unit	`dankclaw/observability/quadlet/alloy.container` → system `alloy.service`, installed at `/etc/containers/systemd/` (root manager)
Config	`shared/alloy/config.alloy` → `/opt/alloy/config/config.alloy` on every deploy
Network	`host` — required so it can reach Loki's `127.0.0.1:3100` publish
UI	`--server.http.listen-addr=127.0.0.1:12345` (loopback; nothing off-host reaches it)
Inputs	host journal: `system.journal` + `user-*.journal` (root reads both)
Output	`http://127.0.0.1:3100/loki/api/v1/push`
Boot persistence	`Restart=always`/`RestartSec=10` + `WantedBy=multi-user.target`; boot persistence is the `[Install]` section (generated system units cannot be `enable`d here)
Successor to	Promtail (EOL 2026-03-02)
Deploy status	root cutover built and tested in-repo (2026-10-20); the live instance is still the old rootless unit until `./scripts/deploy.sh dankclaw --service alloy` runs

Why root (the load-bearing decision)

A rootless Alloy delivered zero lines, silently and with a green health check. Two independent causes, both proven on the host 2026-10-20:

The journal is not readable by a rootless container. The files are root:systemd-journal 0640 (group:adm, other::---). Even bind-mounted, a rootless container has no supplementary groups, so opening them returns DENIED. loki.source.journal then sees an empty view and the counter just sits at 0 — no error logged.
Container logs live in the user journal, not ctr.log. The rootless Quadlet containers log via the journald driver into user-1000.journal; the old file globs (/var/lib/containers/.../ctr.log, /var/log/containers/*.log) point at a root-storage layout that does not exist here.

Root fixes both at once. As root the container opens every journal file under the machine-id directory — system.journal and user-1000.journal — so a single loki.source.journal carries host services and the rootless containers' logs together. Validated with a throwaway root probe before writing the unit: 31,825 journal lines read, 26,966 pushed to Loki in 20 s, zero drops, and Loki's label set went from empty to populated.

The two mounts that make it work

/etc/machine-id (ro) — load-bearing. sd_journal locates the journal directory by the local machine id. Without it the container opens nothing and the journal source reads exactly 0 lines — silently, health green. This single mount is the whole difference between a working and an empty pipeline; do not remove it. (A root probe without it reads 0; with it, 31,825.)
Network=host. Loki publishes 127.0.0.1:3100 (loopback only). No bridge container — rootless or root — can reach the host's loopback, so Alloy must share the host network to push to 127.0.0.1:3100. Consequence: the UI binds 127.0.0.1 directly (no PublishPort), so it stays loopback-only as before.

Configuration

shared/alloy/config.alloy is the source of truth — one loki.source.journal (labelled job="systemd-journal") forwarding to the loopback loki.write endpoint. There is deliberately no container-file source: container stdout is in the journal.

Volumes

Container path	Host path
`/etc/alloy`	`/opt/alloy/config` (ro)
`/etc/machine-id`	host machine id (ro) — required
`/run/systemd/journal/socket`	host journald socket (ro)
`/var/log/journal`	persistent journal files (ro)
`/run/log/journal`	runtime journal files (ro)

The UI stays on loopback

Alloy 1.x refuses to configure the ui service from a config block, so the UI address is set on the command line instead. Under the old rootless unit that meant 0.0.0.0 inside the container plus a published 127.0.0.1:12345; under Network=host the listen-addr is 127.0.0.1:12345 directly — the host's loopback, reachable from the host and from nothing off-host. Alloy has no authentication, which is exactly why it is never published wider. The homepage card and the deploy health gate both dial http://127.0.0.1:12345/-/healthy.

Gotchas

No /etc/machine-id mount ⇒ silent zero. The health check still passes. This is the failure mode that hid the broken pipeline; check loki_source_journal_target_lines_total and the Loki label query, never /healthy.
After=loki.service is gone. Alloy is on the system manager; Loki is a rootless user unit — a system unit cannot order against a user unit. Alloy retries its push until Loki answers, so ordering is unnecessary.
The image must be in the ROOT store. The deployer builds rootless and mirrors (mirror_image_to_root); a Pull=never root quadlet with only a rootless tag fails at create.
One Alloy per host; do not run a single instance that tries to reach both hosts' logs.

5.8 KiB Raw Permalink Blame History