Frank
e494617599
Alloy page rewritten to the shipped root design (why root, the two mounts that make it work, deploy-pending status); Observability current state marks the log pipeline fixed-in-code, awaiting deploy.
116 lines
5.8 KiB
Markdown
116 lines
5.8 KiB
Markdown
---
|
|
type: reference
|
|
description: "Alloy is Grafana's log and telemetry shipper."
|
|
tags:
|
|
- infrastructure
|
|
timestamp: 2026-10-20
|
|
---
|
|
|
|
# Alloy
|
|
|
|
Alloy is Grafana's log and telemetry shipper. It replaces Promtail (end of life
|
|
as of 2026-03-02, https://grafana.com/docs/loki/latest/send-data/promtail/). It
|
|
forwards the host journal — including the rootless Quadlet containers' logs —
|
|
to Loki. Unlike its observability siblings it runs as a **root system service on
|
|
the host network**, because a rootless container cannot read the journal (see
|
|
*Why root* below).
|
|
|
|
**Host:** Dankclaw (one instance per host) — **UI/health:** 127.0.0.1:12345
|
|
(loopback) — **Docs:** https://grafana.com/docs/alloy/latest
|
|
|
|
## Facts
|
|
|
|
| Item | Value |
|
|
|------|-------|
|
|
| Image | `localhost/franksdank/alloy:latest` — built rootless, then mirrored into the root image store by the deployer (a root build fails on DNS in this host's build container) |
|
|
| Unit | `dankclaw/observability/quadlet/alloy.container` → system `alloy.service`, installed at `/etc/containers/systemd/` (root manager) |
|
|
| Config | `shared/alloy/config.alloy` → `/opt/alloy/config/config.alloy` on every deploy |
|
|
| Network | `host` — required so it can reach Loki's `127.0.0.1:3100` publish |
|
|
| UI | `--server.http.listen-addr=127.0.0.1:12345` (loopback; nothing off-host reaches it) |
|
|
| Inputs | host journal: `system.journal` + `user-*.journal` (root reads both) |
|
|
| Output | `http://127.0.0.1:3100/loki/api/v1/push` |
|
|
| Boot persistence | `Restart=always`/`RestartSec=10` + `WantedBy=multi-user.target`; boot persistence is the `[Install]` section (generated system units cannot be `enable`d here) |
|
|
| Successor to | Promtail (EOL 2026-03-02) |
|
|
| Deploy status | root cutover built and tested in-repo (2026-10-20); the **live** instance is still the old rootless unit until `./scripts/deploy.sh dankclaw --service alloy` runs |
|
|
|
|
## Why root (the load-bearing decision)
|
|
|
|
A rootless Alloy delivered **zero** lines, silently and with a green health
|
|
check. Two independent causes, both proven on the host 2026-10-20:
|
|
|
|
1. **The journal is not readable by a rootless container.** The files are
|
|
`root:systemd-journal 0640` (`group:adm`, `other::---`). Even bind-mounted, a
|
|
rootless container has no supplementary groups, so opening them returns
|
|
`DENIED`. `loki.source.journal` then sees an empty view and the counter just
|
|
sits at 0 — no error logged.
|
|
2. **Container logs live in the *user* journal, not `ctr.log`.** The rootless
|
|
Quadlet containers log via the journald driver into `user-1000.journal`; the
|
|
old file globs (`/var/lib/containers/.../ctr.log`, `/var/log/containers/*.log`)
|
|
point at a root-storage layout that does not exist here.
|
|
|
|
**Root fixes both at once.** As root the container opens every journal file
|
|
under the machine-id directory — `system.journal` *and* `user-1000.journal` —
|
|
so a single `loki.source.journal` carries host services and the rootless
|
|
containers' logs together. Validated with a throwaway root probe before writing
|
|
the unit: **31,825 journal lines read, 26,966 pushed to Loki in 20 s, zero
|
|
drops**, and Loki's label set went from empty to populated.
|
|
|
|
### The two mounts that make it work
|
|
|
|
- **`/etc/machine-id` (ro) — load-bearing.** `sd_journal` locates the journal
|
|
directory by the *local* machine id. Without it the container opens nothing and
|
|
the journal source reads exactly 0 lines — silently, health green. This single
|
|
mount is the whole difference between a working and an empty pipeline; do not
|
|
remove it. (A root probe without it reads 0; with it, 31,825.)
|
|
- **`Network=host`.** Loki publishes `127.0.0.1:3100` (loopback only). No bridge
|
|
container — rootless or root — can reach the host's loopback, so Alloy must
|
|
share the host network to push to `127.0.0.1:3100`. Consequence: the UI binds
|
|
`127.0.0.1` directly (no `PublishPort`), so it stays loopback-only as before.
|
|
|
|
## Configuration
|
|
|
|
`shared/alloy/config.alloy` is the source of truth — one `loki.source.journal`
|
|
(labelled `job="systemd-journal"`) forwarding to the loopback `loki.write`
|
|
endpoint. There is deliberately **no** container-file source: container stdout is
|
|
in the journal.
|
|
|
|
## Volumes
|
|
|
|
| Container path | Host path |
|
|
|----------------|-----------|
|
|
| `/etc/alloy` | `/opt/alloy/config` (ro) |
|
|
| `/etc/machine-id` | host machine id (ro) — **required** |
|
|
| `/run/systemd/journal/socket` | host journald socket (ro) |
|
|
| `/var/log/journal` | persistent journal files (ro) |
|
|
| `/run/log/journal` | runtime journal files (ro) |
|
|
|
|
## The UI stays on loopback
|
|
|
|
Alloy 1.x refuses to configure the `ui` service from a config block, so the UI
|
|
address is set on the command line instead. Under the old rootless unit that
|
|
meant `0.0.0.0` inside the container plus a published `127.0.0.1:12345`; under
|
|
`Network=host` the listen-addr is `127.0.0.1:12345` directly — the host's
|
|
loopback, reachable from the host and from nothing off-host. Alloy has no
|
|
authentication, which is exactly why it is never published wider. The homepage
|
|
card and the deploy health gate both dial `http://127.0.0.1:12345/-/healthy`.
|
|
|
|
## Gotchas
|
|
|
|
- **No `/etc/machine-id` mount ⇒ silent zero.** The health check still passes.
|
|
This is the failure mode that hid the broken pipeline; check
|
|
`loki_source_journal_target_lines_total` and the Loki label query, never
|
|
`/healthy`.
|
|
- **`After=loki.service` is gone.** Alloy is on the system manager; Loki is a
|
|
rootless user unit — a system unit cannot order against a user unit. Alloy
|
|
retries its push until Loki answers, so ordering is unnecessary.
|
|
- **The image must be in the ROOT store.** The deployer builds rootless and
|
|
mirrors (`mirror_image_to_root`); a `Pull=never` root quadlet with only a
|
|
rootless tag fails at create.
|
|
- One Alloy per host; do not run a single instance that tries to reach both
|
|
hosts' logs.
|
|
|
|
## See Also
|
|
|
|
- [[Loki]] — the log backend
|
|
- [[Observability]] — the full stack, including what is still unconfigured
|