fix(alloy): promote CONTAINER_NAME to an indexed Loki stream label

Loki metrics queries group only by indexed stream labels, so the
journal-as-JSON stream made every per-container view structurally
empty (measured 6.5k lines/5min flowing while the board read zero).
A loki.process relabel stage now extracts CONTAINER_NAME into a
'container' label. alloy 1.21's json stage has no on_error attribute
(validated against the pinned image) — format_as_json guarantees the
line shape, so no error branch exists.
This commit is contained in:
Frank committed 2026-10-21 13:33:03 -07:00
1 parent 2af2e6ac56
commit 999c4fe440
1 file changed
+31 -1
+31 -1
View File
@@ -17,6 +17,9 @@ logging {
// Loki is published on the host's loopback (127.0.0.1:3100). Alloy shares the
// host network, so it pushes by loopback, never by container name — a root
// container is not on the rootless dankclaw_default network.
// GRAFANA_HOST_LABEL is replaced with this host's name by
// copy_alloy_config at deploy time; it must not appear verbatim in the
// installed config.
loki.write "default" {
endpoint {
url = "http://127.0.0.1:3100/loki/api/v1/push"
@@ -32,7 +35,34 @@ loki.write "default" {
loki.source.journal "journal" {
format_as_json = true
labels = {
job = "systemd-journal",
job = "systemd-journal",
host = "GRAFANA_HOST_LABEL",
}
forward_to = [loki.process.relabel.receiver]
}
// Promote the fields that ARE the point of the log stream to real Loki labels.
// Without this the journal arrives as one opaque JSON line per entry and every
// per-container view is structurally impossible: Loki metrics queries can only
// group by INDEXED stream labels, so `sum by (container_name)` returns a single
// anonymous series and the logs-overview board read empty beside 6.5k
// lines/5min of live traffic (measured 2026-10-21). CONTAINER_NAME is what
// podman's journald logging writes (uppercase); host units lack it and stay on
// the host-only stream. `format_as_json = true` above guarantees every line
// reaching this stage is a JSON object, so there is no non-JSON branch to
// protect — alloy 1.21's json stage has no `on_error` attribute (validated
// against the pinned image); a malformed line would drop, which is the
// correct fate for a line Loki could not index anyway.
loki.process "relabel" {
stage.json {
expressions = {
container = "CONTAINER_NAME",
}
}
stage.labels {
values = {
container = "container",
}
}
forward_to = [loki.write.default.receiver]
}