feat(alloy): run it as a root host-network service so it reads the journal

A rootless Alloy shipped zero lines: the journal files are
root:systemd-journal 0640 (other::---) and a rootless container carries no
supplementary groups, so sd_journal_open sees an empty view silently; the
rootless quadlet container logs also live in user-1000.journal, not the
ctr.log paths the old config tailed.

As root, one loki.source.journal reads system.journal + user-*.journal
and carries both. Network=host because Loki publishes 127.0.0.1:3100 and
no bridge container can reach the host loopback; the UI binds loopback
directly (no PublishPort). The /etc/machine-id mount is load-bearing —
sd_journal finds the journal dir by the local machine id; without it the
source reads 0 lines, health green.

deploy_alloy routes around the rootless loop: build rootless, mirror the
image into the root store (a root build fails on DNS in the build
container), retire the old rootless alloy unit, install to
/etc/containers/systemd. Validated with a throwaway root probe: 31,825
lines read, 26,966 pushed to Loki in 20s, zero drops.
This commit is contained in:
Frank committed 2026-10-20 19:50:15 -07:00
1 parent 70dd7fbe6d
commit 634fef242b
3 files changed
+125 -67

No files matched your search

+18 -15
View File
@@ -1,22 +1,34 @@
// Alloy configuration — ships host and container logs to Loki.
// One instance per host. Deployed to /opt/alloy/config/config.alloy.
//
// Runs as a ROOT system service on Network=host (see
// dankclaw/observability/quadlet/alloy.container), so it reaches Loki on the
// host's loopback and reads every journal — host services and rootless-quadlet
// containers alike.
logging {
level = "info"
}
// The alloy UI service listens on 127.0.0.1:12345 by default; the
// in-container healthcheck probes it there. alloy 1.x refuses to
// configure the ui service from a config block, so there is no block.
// The alloy UI listens on 127.0.0.1:12345; the in-container healthcheck probes
// it there. alloy 1.x refuses to configure the ui service from a config block,
// so there is no ui block.
// Ship logs to Loki (container name on the podman network).
// Loki is published on the host's loopback (127.0.0.1:3100). Alloy shares the
// host network, so it pushes by loopback, never by container name — a root
// container is not on the rootless dankclaw_default network.
loki.write "default" {
endpoint {
url = "http://loki:3100/loki/api/v1/push"
url = "http://127.0.0.1:3100/loki/api/v1/push"
}
}
// Systemd journal from the host socket.
// The systemd journal. As root with /etc/machine-id + the journal dirs mounted,
// this one source reads system.journal AND user-*.journal, so it carries both
// host services and the rootless Quadlet containers' logs. There is deliberately
// no container-file source: podman container stdout lands in the journal, and
// the old /var/lib/containers/.../ctr.log and /var/log/containers globs pointed
// at a root-storage layout that does not exist on this host.
loki.source.journal "journal" {
format_as_json = true
labels = {
@@ -24,12 +36,3 @@ loki.source.journal "journal" {
}
forward_to = [loki.write.default.receiver]
}
// Container logs (podman storage path; adjust for docker).
loki.source.file "containers" {
targets = [
{__path__ = "/var/lib/containers/storage/overlay-containers/*/userdata/ctr.log", job = "podman"},
{__path__ = "/var/log/containers/*.log", job = "containers"},
]
forward_to = [loki.write.default.receiver]
}