serverAppConfigs/serverAppConfigs-docs/Infrastructure/Alloy.md

type, description, tags, timestamp

type	description	tags	timestamp
reference	Alloy is Grafana's log and telemetry shipper.	infrastructure	2026-09-11

Alloy

Alloy is Grafana's log and telemetry shipper. It replaces Promtail (end of life as of 2026-03-02, https://grafana.com/docs/loki/latest/send-data/promtail/). It is wired to forward container and journal logs to Loki, and every unit in the path is healthy — but as of 2026-10-18 it is delivering zero log lines. See "Delivery state" below.

Host: Dankclaw (one instance per host) — Port: 12345 (loopback) — Docs: https://grafana.com/docs/alloy/latest

Facts

Item	Value
Image	franksdank/alloy:latest (local build: upstream + a static busybox for the healthcheck)
Published	127.0.0.1:12345 → 12345 (its own UI/health port)
Unit	dankclaw/observability/quadlet/alloy.container → alloy.service
Config	shared/alloy/config.alloy, installed to /opt/alloy/config/config.alloy on every deploy (scripts/deploy-remote.sh:281-284)
Inputs	host journald socket, podman container logs, /var/log
Output	http://loki:3100/loki/api/v1/push (shared/alloy/config.alloy:13-17)
Successor to	Promtail (EOL 2026-03-02, https://grafana.com/docs/loki/latest/send-data/promtail/)
Boot persistence	Restart=always/RestartSec=10 + WantedBy=default.target (dankclaw/observability/quadlet/alloy.container:58-63); installed at ~/.config/containers/systemd/alloy.container, reporting generated + active (`systemctl --user is-enabled
Delivery	Configured and healthy, currently delivering zero data — see "Delivery state"

Why Alloy and not Promtail

Promtail reached end-of-life on 2026-03-02 and Grafana Labs directs new installs to Alloy. This stack uses Alloy from day one: no Promtail config or unit has ever existed in this repository (git log -S promtail over the whole history returns no commit that added one; the compose-era service of the same name was already Alloy — dankclaw/observability/quadlet/alloy.container:6-7).

Configuration

shared/alloy/config.alloy is the source of truth:

• Reads journald logs from the host's /run/systemd/journal/socket, labelled job="systemd-journal" (shared/alloy/config.alloy:20-26).
• Reads container logs from two globs: /var/lib/containers/storage/overlay-containers/*/userdata/ctr.log (job="podman") and /var/log/containers/*.log (job="containers") (config.alloy:29-35).
• Forwards everything to http://loki:3100/loki/api/v1/push by container name on dankclaw_default (config.alloy:13-17).

Volumes

Container path	Host path
/etc/alloy	/opt/alloy/config
/run/systemd/journal/socket	host journald socket (read-only)
/var/log	host logs (read-only)
/var/lib/containers	podman storage (read-only)

The UI port is published on purpose

Alloy's UI listens on the container's own 127.0.0.1:12345 by default, and Alloy 1.x refuses to configure the ui service from a config block ("service ui does not support being configured", shared/alloy/config.alloy:8-10). Under compose that meant no reachable health endpoint at all: the homepage status card for alloy read down continuously and a deploy had nothing to wait on. The unit therefore passes --server.http.listen-addr=0.0.0.0:12345 (dankclaw/observability/quadlet/alloy.container:46) and publishes 127.0.0.1:12345 (alloy.container:47, confirmed by ss -ltnp 2026-10-18) — same posture as the other three: reachable from the host and from Caddy's network, and from nothing off-host. Alloy has no authentication, which is exactly why it is not published wider. The homepage card probes that same loopback port (shared/homepage/services.json:233).

Delivery state (2026-10-20): healthy, shipping nothing

The unit is active, the container reports healthy, the log says Alloy is ready, and the pipeline carries no data at all.

Root cause, proven: a rootless container cannot read the host journal. The journal files are root:systemd-journal 0640 (with group:adm, other::---). /var/log is bind-mounted so the files are visible, but a rootless container spawns with no supplementary groups, so it is denied. Proven directly: podman run --rm -v /var/log/journal:/var/log/journal:ro busybox cat /var/log/journal/*/system.journal → DENIED. loki.source.journal opens via sd_journal_open, so it silently sees an empty view — no error logged, counter frozen at loki_source_journal_target_lines_total 0 (system.journal and user-1000.journal are both present and non-trivial, 2026-10-20).

Second, independent gap: the rootless Quadlet containers log to the user journal (user-1000.journal), and loki.source.journal reads the system journal only — a fixed system read would still miss them. Container stdout is not in a ctr.log here either: the file globs point at root storage (/var/lib/containers/storage/.../ctr.log, /var/log/containers/*.log) that rootless does not use, refused at start with stat …: permission denied / no such file or directory, loki_source_file_files_active_total 0.

Downstream: Loki holds zero streams (GET /loki/api/v1/labels → no data) — see Loki.

Closing it is a permission/architecture decision, not a config edit. The unit, config and push endpoint are correct as written; rootless Podman simply cannot grant the journal group. The viable shapes: run Alloy as a system service (root, in adm/systemd-journal) instead of a rootless Quadlet; grant the container's mapped uid journal read via a tmpfiles/ACL rule; or have journald forward to a path the container can read. Each trades rootless posture for reach — decide deliberately. Re-check loki_source_journal_target_lines_total and the Loki label query to confirm a fix, never /healthy: health is exactly what hides this failure.

Gotchas

• The journald socket mount is required for journal scraping.
• Container log paths differ between docker (/var/lib/docker/containers) and podman (/var/lib/containers/storage/...); point the config at the right one per host.
• One Alloy per host; do not run a single instance that tries to reach both hosts' logs.
• The unit Wants=/After= loki.service: without a reachable Loki the write client fails and the pipeline backs up.

See Also

• Loki — the log backend
• Observability — the full stack, including what is still unconfigured