Frank
999c4fe440
Loki metrics queries group only by indexed stream labels, so the journal-as-JSON stream made every per-container view structurally empty (measured 6.5k lines/5min flowing while the board read zero). A loki.process relabel stage now extracts CONTAINER_NAME into a 'container' label. alloy 1.21's json stage has no on_error attribute (validated against the pinned image) — format_as_json guarantees the line shape, so no error branch exists.
69 lines
2.8 KiB
Plaintext
69 lines
2.8 KiB
Plaintext
// Alloy configuration — ships host and container logs to Loki.
|
|
// One instance per host. Deployed to /opt/alloy/config/config.alloy.
|
|
//
|
|
// Runs as a ROOT system service on Network=host (see
|
|
// dankclaw/observability/quadlet/alloy.container), so it reaches Loki on the
|
|
// host's loopback and reads every journal — host services and rootless-quadlet
|
|
// containers alike.
|
|
|
|
logging {
|
|
level = "info"
|
|
}
|
|
|
|
// The alloy UI listens on 127.0.0.1:12345; the in-container healthcheck probes
|
|
// it there. alloy 1.x refuses to configure the ui service from a config block,
|
|
// so there is no ui block.
|
|
|
|
// Loki is published on the host's loopback (127.0.0.1:3100). Alloy shares the
|
|
// host network, so it pushes by loopback, never by container name — a root
|
|
// container is not on the rootless dankclaw_default network.
|
|
// GRAFANA_HOST_LABEL is replaced with this host's name by
|
|
// copy_alloy_config at deploy time; it must not appear verbatim in the
|
|
// installed config.
|
|
loki.write "default" {
|
|
endpoint {
|
|
url = "http://127.0.0.1:3100/loki/api/v1/push"
|
|
}
|
|
}
|
|
|
|
// The systemd journal. As root with /etc/machine-id + the journal dirs mounted,
|
|
// this one source reads system.journal AND user-*.journal, so it carries both
|
|
// host services and the rootless Quadlet containers' logs. There is deliberately
|
|
// no container-file source: podman container stdout lands in the journal, and
|
|
// the old /var/lib/containers/.../ctr.log and /var/log/containers globs pointed
|
|
// at a root-storage layout that does not exist on this host.
|
|
loki.source.journal "journal" {
|
|
format_as_json = true
|
|
labels = {
|
|
job = "systemd-journal",
|
|
host = "GRAFANA_HOST_LABEL",
|
|
}
|
|
forward_to = [loki.process.relabel.receiver]
|
|
}
|
|
|
|
// Promote the fields that ARE the point of the log stream to real Loki labels.
|
|
// Without this the journal arrives as one opaque JSON line per entry and every
|
|
// per-container view is structurally impossible: Loki metrics queries can only
|
|
// group by INDEXED stream labels, so `sum by (container_name)` returns a single
|
|
// anonymous series and the logs-overview board read empty beside 6.5k
|
|
// lines/5min of live traffic (measured 2026-10-21). CONTAINER_NAME is what
|
|
// podman's journald logging writes (uppercase); host units lack it and stay on
|
|
// the host-only stream. `format_as_json = true` above guarantees every line
|
|
// reaching this stage is a JSON object, so there is no non-JSON branch to
|
|
// protect — alloy 1.21's json stage has no `on_error` attribute (validated
|
|
// against the pinned image); a malformed line would drop, which is the
|
|
// correct fate for a line Loki could not index anyway.
|
|
loki.process "relabel" {
|
|
stage.json {
|
|
expressions = {
|
|
container = "CONTAINER_NAME",
|
|
}
|
|
}
|
|
stage.labels {
|
|
values = {
|
|
container = "container",
|
|
}
|
|
}
|
|
forward_to = [loki.write.default.receiver]
|
|
}
|