serverAppConfigs/serverAppConfigs-docs/Infrastructure/Alloy.md

type, description, tags, timestamp

type	description	tags	timestamp
reference	Alloy is Grafana's log and telemetry shipper.	infrastructure	2026-10-20

Alloy

Alloy is Grafana's log and telemetry shipper. It replaces Promtail (end of life as of 2026-03-02, https://grafana.com/docs/loki/latest/send-data/promtail/). It forwards the host journal — including the rootless Quadlet containers' logs — to Loki. Unlike its observability siblings it runs as a root system service on the host network, because a rootless container cannot read the journal (see Why root below).

Host: Dankclaw (one instance per host) — UI/health: 127.0.0.1:12345 (loopback) — Docs: https://grafana.com/docs/alloy/latest

Facts

Item	Value
Image	localhost/franksdank/alloy:latest — built rootless, then mirrored into the root image store by the deployer (a root build fails on DNS in this host's build container)
Unit	dankclaw/observability/quadlet/alloy.container → system alloy.service, installed at /etc/containers/systemd/ (root manager)
Config	shared/alloy/config.alloy → /opt/alloy/config/config.alloy on every deploy
Network	host — required so it can reach Loki's 127.0.0.1:3100 publish
UI	--server.http.listen-addr=127.0.0.1:12345 (loopback; nothing off-host reaches it)
Inputs	host journal: system.journal + user-*.journal (root reads both)
Output	http://127.0.0.1:3100/loki/api/v1/push
Boot persistence	Restart=always/RestartSec=10 + WantedBy=multi-user.target; boot persistence is the [Install] section (generated system units cannot be enabled here)
Successor to	Promtail (EOL 2026-03-02)
Deploy status	root cutover built and tested in-repo (2026-10-20); the live instance is still the old rootless unit until ./scripts/deploy.sh dankclaw --service alloy runs

Why root (the load-bearing decision)

A rootless Alloy delivered zero lines, silently and with a green health check. Two independent causes, both proven on the host 2026-10-20:

1. The journal is not readable by a rootless container. The files are root:systemd-journal 0640 (group:adm, other::---). Even bind-mounted, a rootless container has no supplementary groups, so opening them returns DENIED. loki.source.journal then sees an empty view and the counter just sits at 0 — no error logged.
2. Container logs live in the user journal, not ctr.log. The rootless Quadlet containers log via the journald driver into user-1000.journal; the old file globs (/var/lib/containers/.../ctr.log, /var/log/containers/*.log) point at a root-storage layout that does not exist here.

Root fixes both at once. As root the container opens every journal file under the machine-id directory — system.journal and user-1000.journal — so a single loki.source.journal carries host services and the rootless containers' logs together. Validated with a throwaway root probe before writing the unit: 31,825 journal lines read, 26,966 pushed to Loki in 20 s, zero drops, and Loki's label set went from empty to populated.

The two mounts that make it work

• /etc/machine-id (ro) — load-bearing. sd_journal locates the journal directory by the local machine id. Without it the container opens nothing and the journal source reads exactly 0 lines — silently, health green. This single mount is the whole difference between a working and an empty pipeline; do not remove it. (A root probe without it reads 0; with it, 31,825.)
• Network=host. Loki publishes 127.0.0.1:3100 (loopback only). No bridge container — rootless or root — can reach the host's loopback, so Alloy must share the host network to push to 127.0.0.1:3100. Consequence: the UI binds 127.0.0.1 directly (no PublishPort), so it stays loopback-only as before.

Configuration

shared/alloy/config.alloy is the source of truth — one loki.source.journal (labelled job="systemd-journal") forwarding to the loopback loki.write endpoint. There is deliberately no container-file source: container stdout is in the journal.

Volumes

Container path	Host path
/etc/alloy	/opt/alloy/config (ro)
/etc/machine-id	host machine id (ro) — required
/run/systemd/journal/socket	host journald socket (ro)
/var/log/journal	persistent journal files (ro)
/run/log/journal	runtime journal files (ro)

The UI stays on loopback

Alloy 1.x refuses to configure the ui service from a config block, so the UI address is set on the command line instead. Under the old rootless unit that meant 0.0.0.0 inside the container plus a published 127.0.0.1:12345; under Network=host the listen-addr is 127.0.0.1:12345 directly — the host's loopback, reachable from the host and from nothing off-host. Alloy has no authentication, which is exactly why it is never published wider. The homepage card and the deploy health gate both dial http://127.0.0.1:12345/-/healthy.

Gotchas

• No /etc/machine-id mount ⇒ silent zero. The health check still passes. This is the failure mode that hid the broken pipeline; check loki_source_journal_target_lines_total and the Loki label query, never /healthy.
• After=loki.service is gone. Alloy is on the system manager; Loki is a rootless user unit — a system unit cannot order against a user unit. Alloy retries its push until Loki answers, so ordering is unnecessary.
• The image must be in the ROOT store. The deployer builds rootless and mirrors (mirror_image_to_root); a Pull=never root quadlet with only a rootless tag fails at create.
• One Alloy per host; do not run a single instance that tries to reach both hosts' logs.

See Also

• Loki — the log backend
• Observability — the full stack, including what is still unconfigured