Files

T

Frank e7f0d93c86 docs(vault): fleet-wide fact re-verification pass (67 pages)

The docs-fact census wave re-read the live hosts and re-keyed every page
it could: runtime/payload identities, ports, sizes, ledger figures, unit
state, and the retirement banners that drifted as the fleet moved. The
wave was interrupted mid-run, so this commit carries only what passed the
integrity gates: every touched page parses (front matter or H1+status
convention intact, balanced fences, no truncated tables, trailing
newline), the docs_completeness and link-integrity seams are green, and
bats is 384/384.

Numbers that this session proved on live hosts are keyed to the measured
value, not the page's earlier claim.

2026-10-18 13:15:43 -07:00

5.8 KiB

Raw Blame History

type, description, tags, timestamp

type

description

Alloy

Alloy is Grafana's log and telemetry shipper. It replaces Promtail (end of life as of 2026-03-02, https://grafana.com/docs/loki/latest/send-data/promtail/). It is wired to forward container and journal logs to Loki, and every unit in the path is healthy — but as of 2026-10-18 it is delivering zero log lines. See "Delivery state" below.

Host: Dankclaw (one instance per host) — Port: 12345 (loopback) — Docs: https://grafana.com/docs/alloy/latest

Facts

Item	Value
Image	`franksdank/alloy:latest` (local build: upstream + a static busybox for the healthcheck)
Published	`127.0.0.1:12345` → 12345 (its own UI/health port)
Unit	`dankclaw/observability/quadlet/alloy.container` → `alloy.service`
Config	`shared/alloy/config.alloy`, installed to `/opt/alloy/config/config.alloy` on every deploy (`scripts/deploy-remote.sh:281-284`)
Inputs	host journald socket, podman container logs, `/var/log`
Output	`http://loki:3100/loki/api/v1/push` (`shared/alloy/config.alloy:13-17`)
Successor to	Promtail (EOL 2026-03-02, https://grafana.com/docs/loki/latest/send-data/promtail/)
Boot persistence	`Restart=always`/`RestartSec=10` + `WantedBy=default.target` (`dankclaw/observability/quadlet/alloy.container:58-63`); installed at `~/.config/containers/systemd/alloy.container`, reporting `generated` + `active` (`systemctl --user is-enabled
Delivery	Configured and healthy, currently delivering zero data — see "Delivery state"

Why Alloy and not Promtail

Promtail reached end-of-life on 2026-03-02 and Grafana Labs directs new installs to Alloy. This stack uses Alloy from day one: no Promtail config or unit has ever existed in this repository (git log -S promtail over the whole history returns no commit that added one; the compose-era service of the same name was already Alloy — dankclaw/observability/quadlet/alloy.container:6-7).

Configuration

shared/alloy/config.alloy is the source of truth:

Reads journald logs from the host's /run/systemd/journal/socket, labelled job="systemd-journal" (shared/alloy/config.alloy:20-26).
Reads container logs from two globs: /var/lib/containers/storage/overlay-containers/*/userdata/ctr.log (job="podman") and /var/log/containers/*.log (job="containers") (config.alloy:29-35).
Forwards everything to http://loki:3100/loki/api/v1/push by container name on dankclaw_default (config.alloy:13-17).

Volumes

Container path	Host path
`/etc/alloy`	`/opt/alloy/config`
`/run/systemd/journal/socket`	host journald socket (read-only)
`/var/log`	host logs (read-only)
`/var/lib/containers`	podman storage (read-only)

The UI port is published on purpose

Alloy's UI listens on the container's own 127.0.0.1:12345 by default, and Alloy 1.x refuses to configure the ui service from a config block ("service ui does not support being configured", shared/alloy/config.alloy:8-10). Under compose that meant no reachable health endpoint at all: the homepage status card for alloy read down continuously and a deploy had nothing to wait on. The unit therefore passes --server.http.listen-addr=0.0.0.0:12345 (dankclaw/observability/quadlet/alloy.container:46) and publishes 127.0.0.1:12345 (alloy.container:47, confirmed by ss -ltnp 2026-10-18) — same posture as the other three: reachable from the host and from Caddy's network, and from nothing off-host. Alloy has no authentication, which is exactly why it is not published wider. The homepage card probes that same loopback port (shared/homepage/services.json:233).

Delivery state (2026-10-18): healthy, shipping nothing

The unit is active, the container reports healthy, the log says Alloy is ready, and the pipeline carries no data at all. Measured on 2026-10-18:

curl -s http://127.0.0.1:12345/metrics → loki_source_journal_target_lines_total{component_id="loki.source.journal.journal"} 0 and loki_source_file_files_active_total{component_id="loki.source.file.containers"} 0. The journal source logs no error; the counter simply never moves. The journal files it would have to read through the /var/log bind are root:systemd-journal 0640 with an ACL (ls -l /var/log/journal/64a287d7…/ 2026-10-18).
Both file globs were refused at start (journalctl --user -u alloy, 2026-10-18T08:25:27Z), verbatim: failed to create source, skipping … component_id=loki.source.file.containers error="failed to tail file, stat failed: stat /var/lib/containers/storage/overlay-containers/*/userdata/ctr.log: permission denied" and the same message for /var/log/containers/*.log: no such file or directory.
Downstream that means Loki holds zero streams — GET /loki/api/v1/labels returns {"status":"success"} with no data — see Loki.

No fix is claimed here: the unit, the config and the push endpoint are all correct as written, and the gap is source-side access (and a Kubernetes-style glob that does not exist on this host). Anyone closing it should re-check the two counters above, not the health status — health is exactly what hides this failure mode.

Gotchas

The journald socket mount is required for journal scraping.
Container log paths differ between docker (/var/lib/docker/containers) and podman (/var/lib/containers/storage/...); point the config at the right one per host.
One Alloy per host; do not run a single instance that tries to reach both hosts' logs.
The unit Wants=/After= loki.service: without a reachable Loki the write client fails and the pipeline backs up.

5.8 KiB Raw Blame History