Frank
70dd7fbe6d
AGENTS.md: the rule set is now two groups (llm-fleet + vllm-fleet) and a Loki datasource is provisioned beside Prometheus. Observability.md: current-state census refreshed (three LLM targets up, two rule groups, Loki datasource); logs still carry no data. Alloy.md: replace the 'wrong glob' theory with the proven cause - a rootless container cannot read the 0640 systemd-journal files (a rootless busybox read returns DENIED), and rootless quadlet logs land in the user journal the system-only source never reads. Closing it is a permission/architecture decision, not a config glob.
101 lines
6.4 KiB
Markdown
101 lines
6.4 KiB
Markdown
---
|
|
type: reference
|
|
description: "Alloy is Grafana's log and telemetry shipper."
|
|
tags:
|
|
- infrastructure
|
|
timestamp: 2026-09-11
|
|
---
|
|
|
|
# Alloy
|
|
|
|
Alloy is Grafana's log and telemetry shipper. It replaces Promtail (end of life as of 2026-03-02, https://grafana.com/docs/loki/latest/send-data/promtail/). It is wired to forward container and journal logs to Loki, and every unit in the path is healthy — but as of 2026-10-18 it is delivering **zero** log lines. See "Delivery state" below.
|
|
|
|
**Host:** Dankclaw (one instance per host) — **Port:** 12345 (loopback) — **Docs:** https://grafana.com/docs/alloy/latest
|
|
|
|
## Facts
|
|
|
|
| Item | Value |
|
|
|------|-------|
|
|
| Image | `franksdank/alloy:latest` (local build: upstream + a static busybox for the healthcheck) |
|
|
| Published | `127.0.0.1:12345` → 12345 (its own UI/health port) |
|
|
| Unit | `dankclaw/observability/quadlet/alloy.container` → `alloy.service` |
|
|
| Config | `shared/alloy/config.alloy`, installed to `/opt/alloy/config/config.alloy` on every deploy (`scripts/deploy-remote.sh:281-284`) |
|
|
| Inputs | host journald socket, podman container logs, `/var/log` |
|
|
| Output | `http://loki:3100/loki/api/v1/push` (`shared/alloy/config.alloy:13-17`) |
|
|
| Successor to | Promtail (EOL 2026-03-02, https://grafana.com/docs/loki/latest/send-data/promtail/) |
|
|
| Boot persistence | `Restart=always`/`RestartSec=10` + `WantedBy=default.target` (`dankclaw/observability/quadlet/alloy.container:58-63`); installed at `~/.config/containers/systemd/alloy.container`, reporting `generated` + `active` (`systemctl --user is-enabled|is-active alloy` 2026-10-18) |
|
|
| Delivery | **Configured and healthy, currently delivering zero data** — see "Delivery state" |
|
|
|
|
## Why Alloy and not Promtail
|
|
|
|
Promtail reached end-of-life on 2026-03-02 and Grafana Labs directs new installs to Alloy. This stack uses Alloy from day one: no Promtail config or unit has ever existed in this repository (`git log -S promtail` over the whole history returns no commit that added one; the compose-era service of the same name was already Alloy — `dankclaw/observability/quadlet/alloy.container:6-7`).
|
|
|
|
## Configuration
|
|
|
|
`shared/alloy/config.alloy` is the source of truth:
|
|
|
|
- Reads journald logs from the host's `/run/systemd/journal/socket`, labelled `job="systemd-journal"` (`shared/alloy/config.alloy:20-26`).
|
|
- Reads container logs from two globs: `/var/lib/containers/storage/overlay-containers/*/userdata/ctr.log` (`job="podman"`) and `/var/log/containers/*.log` (`job="containers"`) (`config.alloy:29-35`).
|
|
- Forwards everything to `http://loki:3100/loki/api/v1/push` by container name on `dankclaw_default` (`config.alloy:13-17`).
|
|
|
|
## Volumes
|
|
|
|
| Container path | Host path |
|
|
|----------------|-----------|
|
|
| `/etc/alloy` | `/opt/alloy/config` |
|
|
| `/run/systemd/journal/socket` | host journald socket (read-only) |
|
|
| `/var/log` | host logs (read-only) |
|
|
| `/var/lib/containers` | podman storage (read-only) |
|
|
|
|
## The UI port is published on purpose
|
|
|
|
Alloy's UI listens on the container's own `127.0.0.1:12345` by default, and Alloy 1.x refuses to configure the `ui` service from a config block ("service ui does not support being configured", `shared/alloy/config.alloy:8-10`). Under compose that meant **no reachable health endpoint at all**: the homepage status card for alloy read `down` continuously and a deploy had nothing to wait on. The unit therefore passes `--server.http.listen-addr=0.0.0.0:12345` (`dankclaw/observability/quadlet/alloy.container:46`) and publishes `127.0.0.1:12345` (`alloy.container:47`, confirmed by `ss -ltnp` 2026-10-18) — same posture as the other three: reachable from the host and from Caddy's network, and from nothing off-host. Alloy has no authentication, which is exactly why it is not published wider. The homepage card probes that same loopback port (`shared/homepage/services.json:233`).
|
|
|
|
## Delivery state (2026-10-20): healthy, shipping nothing
|
|
|
|
The unit is `active`, the container reports `healthy`, the log says `Alloy is
|
|
ready`, and the pipeline carries no data at all.
|
|
|
|
**Root cause, proven: a rootless container cannot read the host journal.** The
|
|
journal files are `root:systemd-journal 0640` (with `group:adm`, `other::---`).
|
|
`/var/log` is bind-mounted so the files are *visible*, but a rootless container
|
|
spawns with no supplementary groups, so it is denied. Proven directly:
|
|
`podman run --rm -v /var/log/journal:/var/log/journal:ro busybox cat
|
|
/var/log/journal/*/system.journal` → `DENIED`. `loki.source.journal` opens via
|
|
`sd_journal_open`, so it silently sees an empty view — no error logged, counter
|
|
frozen at `loki_source_journal_target_lines_total 0` (`system.journal` and
|
|
`user-1000.journal` are both present and non-trivial, 2026-10-20).
|
|
|
|
**Second, independent gap:** the rootless Quadlet containers log to the *user*
|
|
journal (`user-1000.journal`), and `loki.source.journal` reads the *system*
|
|
journal only — a fixed system read would still miss them. Container stdout is
|
|
not in a `ctr.log` here either: the file globs point at root storage
|
|
(`/var/lib/containers/storage/.../ctr.log`, `/var/log/containers/*.log`) that
|
|
rootless does not use, refused at start with `stat …: permission denied` / `no
|
|
such file or directory`, `loki_source_file_files_active_total 0`.
|
|
|
|
Downstream: Loki holds zero streams (`GET /loki/api/v1/labels` → no `data`) —
|
|
see [[Loki]].
|
|
|
|
**Closing it is a permission/architecture decision, not a config edit.** The
|
|
unit, config and push endpoint are correct as written; rootless Podman simply
|
|
cannot grant the journal group. The viable shapes: run Alloy as a *system*
|
|
service (root, in `adm`/`systemd-journal`) instead of a rootless Quadlet; grant
|
|
the container's mapped uid journal read via a tmpfiles/ACL rule; or have
|
|
journald forward to a path the container can read. Each trades rootless posture
|
|
for reach — decide deliberately. Re-check `loki_source_journal_target_lines_total`
|
|
and the Loki label query to confirm a fix, never `/healthy`: health is exactly
|
|
what hides this failure.
|
|
|
|
## Gotchas
|
|
|
|
- The journald socket mount is required for journal scraping.
|
|
- Container log paths differ between docker (`/var/lib/docker/containers`) and podman (`/var/lib/containers/storage/...`); point the config at the right one per host.
|
|
- One Alloy per host; do not run a single instance that tries to reach both hosts' logs.
|
|
- The unit `Wants=`/`After=` `loki.service`: without a reachable Loki the write client fails and the pipeline backs up.
|
|
|
|
## See Also
|
|
|
|
- [[Loki]] — the log backend
|
|
- [[Observability]] — the full stack, including what is still unconfigured
|