Files

T

Frank 701ab14f60 docs(observability): record the quadlet cutover, the exposure posture, and the unconfigured state

Rewrites the [[Observability]] umbrella page and the five component pages for
what is actually deployed: rootless Quadlet units instead of compose services,
the loopback-only publish posture (node-exporter's LAN binding explained), the
loki healthcheck living in its image because the image has no shell, and the
Home=/root trap that killed the first grafana start.

Adds the note that matters most to a reader: the stack is installed, healthy
and UNCONFIGURED. Grafana reports zero datasources and zero dashboards with an
empty provisioning directory; Prometheus has no rule files and no Alertmanager;
Loki is configured but has no reader; node-exporter runs its default
collectors. Alloy is the only configured piece. Metrics and logs are collected
and retained, and nothing is visualized or alerted on.

AGENTS.md's boot-persistence gap note is replaced by the resolution.

2026-09-11 16:10:27 -07:00

2.7 KiB

Raw Blame History

Alloy

Alloy is Grafana's log and telemetry shipper. It replaces Promtail (EOL 2026-03-02). It forwards container and journal logs to Loki.

Host: Dankclaw (one instance per host) — Port: 12345 (loopback) — Docs: https://grafana.com/docs/alloy/latest

Facts

Item	Value
Image	`franksdank/alloy:latest` (local build: upstream + a static busybox for the healthcheck)
Published	`127.0.0.1:12345` → 12345 (its own UI/health port)
Unit	`dankclaw/observability/quadlet/alloy.container` → `alloy.service`
Config	`shared/alloy/config.alloy`, installed to `/opt/alloy/config/`
Inputs	host journald socket, podman container logs, `/var/log`
Output	`http://loki:3100/loki/api/v1/push`
Successor to	Promtail (EOL 2026-03-02)

Why Alloy and not Promtail

Promtail reached end-of-life on 2026-03-02 and Grafana Labs directs new installs to Alloy. This stack uses Alloy from day one.

Configuration

shared/alloy/config.alloy is the source of truth:

Reads journald logs from the host's /run/systemd/journal/socket.
Reads container logs from /var/lib/containers/storage/... (podman).
Forwards to Loki with job labels.

Volumes

Container path	Host path
`/etc/alloy`	`/opt/alloy/config`
`/run/systemd/journal/socket`	host journald socket (read-only)
`/var/log`	host logs (read-only)
`/var/lib/containers`	podman storage (read-only)

The UI port is published on purpose

Alloy's UI listens on the container's own 127.0.0.1:12345 by default, and Alloy 1.x refuses to configure the ui service from a config block ("service ui does not support being configured"). Under compose that meant no reachable health endpoint at all: the homepage status card for alloy read down continuously and a deploy had nothing to wait on. The unit therefore passes --server.http.listen-addr=0.0.0.0:12345 and publishes 127.0.0.1:12345 — same posture as the other three: reachable from the host and from Caddy's network, and from nothing off-host. Alloy has no authentication, which is exactly why it is not published wider.

Gotchas

The journald socket mount is required for journal scraping.
Container log paths differ between docker (/var/lib/docker/containers) and podman (/var/lib/containers/storage/...); point the config at the right one per host.
One Alloy per host; do not run a single instance that tries to reach both hosts' logs.
The unit Wants=/After= loki.service: without a reachable Loki the write client fails and the pipeline backs up.

2.7 KiB Raw Blame History