feat(alloy): run it as a root host-network service so it reads the journal
A rootless Alloy shipped zero lines: the journal files are root:systemd-journal 0640 (other::---) and a rootless container carries no supplementary groups, so sd_journal_open sees an empty view silently; the rootless quadlet container logs also live in user-1000.journal, not the ctr.log paths the old config tailed. As root, one loki.source.journal reads system.journal + user-*.journal and carries both. Network=host because Loki publishes 127.0.0.1:3100 and no bridge container can reach the host loopback; the UI binds loopback directly (no PublishPort). The /etc/machine-id mount is load-bearing — sd_journal finds the journal dir by the local machine id; without it the source reads 0 lines, health green. deploy_alloy routes around the rootless loop: build rootless, mirror the image into the root store (a root build fails on DNS in the build container), retire the old rootless alloy unit, install to /etc/containers/systemd. Validated with a throwaway root probe: 31,825 lines read, 26,966 pushed to Loki in 20s, zero drops.
This commit is contained in:
1 parent
70dd7fbe6d
commit
634fef242b
3 files changed
+125
-67
No files matched your search
+18
-15
@@ -1,22 +1,34 @@
|
||||
// Alloy configuration — ships host and container logs to Loki.
|
||||
// One instance per host. Deployed to /opt/alloy/config/config.alloy.
|
||||
//
|
||||
// Runs as a ROOT system service on Network=host (see
|
||||
// dankclaw/observability/quadlet/alloy.container), so it reaches Loki on the
|
||||
// host's loopback and reads every journal — host services and rootless-quadlet
|
||||
// containers alike.
|
||||
|
||||
logging {
|
||||
level = "info"
|
||||
}
|
||||
|
||||
// The alloy UI service listens on 127.0.0.1:12345 by default; the
|
||||
// in-container healthcheck probes it there. alloy 1.x refuses to
|
||||
// configure the ui service from a config block, so there is no block.
|
||||
// The alloy UI listens on 127.0.0.1:12345; the in-container healthcheck probes
|
||||
// it there. alloy 1.x refuses to configure the ui service from a config block,
|
||||
// so there is no ui block.
|
||||
|
||||
// Ship logs to Loki (container name on the podman network).
|
||||
// Loki is published on the host's loopback (127.0.0.1:3100). Alloy shares the
|
||||
// host network, so it pushes by loopback, never by container name — a root
|
||||
// container is not on the rootless dankclaw_default network.
|
||||
loki.write "default" {
|
||||
endpoint {
|
||||
url = "http://loki:3100/loki/api/v1/push"
|
||||
url = "http://127.0.0.1:3100/loki/api/v1/push"
|
||||
}
|
||||
}
|
||||
|
||||
// Systemd journal from the host socket.
|
||||
// The systemd journal. As root with /etc/machine-id + the journal dirs mounted,
|
||||
// this one source reads system.journal AND user-*.journal, so it carries both
|
||||
// host services and the rootless Quadlet containers' logs. There is deliberately
|
||||
// no container-file source: podman container stdout lands in the journal, and
|
||||
// the old /var/lib/containers/.../ctr.log and /var/log/containers globs pointed
|
||||
// at a root-storage layout that does not exist on this host.
|
||||
loki.source.journal "journal" {
|
||||
format_as_json = true
|
||||
labels = {
|
||||
@@ -24,12 +36,3 @@ loki.source.journal "journal" {
|
||||
}
|
||||
forward_to = [loki.write.default.receiver]
|
||||
}
|
||||
|
||||
// Container logs (podman storage path; adjust for docker).
|
||||
loki.source.file "containers" {
|
||||
targets = [
|
||||
{__path__ = "/var/lib/containers/storage/overlay-containers/*/userdata/ctr.log", job = "podman"},
|
||||
{__path__ = "/var/log/containers/*.log", job = "containers"},
|
||||
]
|
||||
forward_to = [loki.write.default.receiver]
|
||||
}
|
||||
Reference in new issue
Block a user