docs(observability): correct current state and log-pipeline root cause
AGENTS.md: the rule set is now two groups (llm-fleet + vllm-fleet) and a Loki datasource is provisioned beside Prometheus. Observability.md: current-state census refreshed (three LLM targets up, two rule groups, Loki datasource); logs still carry no data. Alloy.md: replace the 'wrong glob' theory with the proven cause - a rootless container cannot read the 0640 systemd-journal files (a rootless busybox read returns DENIED), and rootless quadlet logs land in the user journal the system-only source never reads. Closing it is a permission/architecture decision, not a config glob.
This commit is contained in:
Frank
@@ -51,15 +51,41 @@ Promtail reached end-of-life on 2026-03-02 and Grafana Labs directs new installs
|
||||
|
||||
Alloy's UI listens on the container's own `127.0.0.1:12345` by default, and Alloy 1.x refuses to configure the `ui` service from a config block ("service ui does not support being configured", `shared/alloy/config.alloy:8-10`). Under compose that meant **no reachable health endpoint at all**: the homepage status card for alloy read `down` continuously and a deploy had nothing to wait on. The unit therefore passes `--server.http.listen-addr=0.0.0.0:12345` (`dankclaw/observability/quadlet/alloy.container:46`) and publishes `127.0.0.1:12345` (`alloy.container:47`, confirmed by `ss -ltnp` 2026-10-18) — same posture as the other three: reachable from the host and from Caddy's network, and from nothing off-host. Alloy has no authentication, which is exactly why it is not published wider. The homepage card probes that same loopback port (`shared/homepage/services.json:233`).
|
||||
|
||||
## Delivery state (2026-10-18): healthy, shipping nothing
|
||||
## Delivery state (2026-10-20): healthy, shipping nothing
|
||||
|
||||
The unit is `active`, the container reports `healthy`, the log says `Alloy is ready`, and the pipeline carries no data at all. Measured on 2026-10-18:
|
||||
The unit is `active`, the container reports `healthy`, the log says `Alloy is
|
||||
ready`, and the pipeline carries no data at all.
|
||||
|
||||
- `curl -s http://127.0.0.1:12345/metrics` → `loki_source_journal_target_lines_total{component_id="loki.source.journal.journal"} 0` and `loki_source_file_files_active_total{component_id="loki.source.file.containers"} 0`. The journal source logs no error; the counter simply never moves. The journal files it would have to read through the `/var/log` bind are `root:systemd-journal 0640` with an ACL (`ls -l /var/log/journal/64a287d7…/` 2026-10-18).
|
||||
- Both file globs were refused at start (`journalctl --user -u alloy`, 2026-10-18T08:25:27Z), verbatim: `failed to create source, skipping … component_id=loki.source.file.containers error="failed to tail file, stat failed: stat /var/lib/containers/storage/overlay-containers/*/userdata/ctr.log: permission denied"` and the same message for `/var/log/containers/*.log: no such file or directory`.
|
||||
- Downstream that means Loki holds zero streams — `GET /loki/api/v1/labels` returns `{"status":"success"}` with no `data` — see [[Loki]].
|
||||
**Root cause, proven: a rootless container cannot read the host journal.** The
|
||||
journal files are `root:systemd-journal 0640` (with `group:adm`, `other::---`).
|
||||
`/var/log` is bind-mounted so the files are *visible*, but a rootless container
|
||||
spawns with no supplementary groups, so it is denied. Proven directly:
|
||||
`podman run --rm -v /var/log/journal:/var/log/journal:ro busybox cat
|
||||
/var/log/journal/*/system.journal` → `DENIED`. `loki.source.journal` opens via
|
||||
`sd_journal_open`, so it silently sees an empty view — no error logged, counter
|
||||
frozen at `loki_source_journal_target_lines_total 0` (`system.journal` and
|
||||
`user-1000.journal` are both present and non-trivial, 2026-10-20).
|
||||
|
||||
No fix is claimed here: the unit, the config and the push endpoint are all correct as written, and the gap is source-side access (and a Kubernetes-style glob that does not exist on this host). Anyone closing it should re-check the two counters above, not the health status — health is exactly what hides this failure mode.
|
||||
**Second, independent gap:** the rootless Quadlet containers log to the *user*
|
||||
journal (`user-1000.journal`), and `loki.source.journal` reads the *system*
|
||||
journal only — a fixed system read would still miss them. Container stdout is
|
||||
not in a `ctr.log` here either: the file globs point at root storage
|
||||
(`/var/lib/containers/storage/.../ctr.log`, `/var/log/containers/*.log`) that
|
||||
rootless does not use, refused at start with `stat …: permission denied` / `no
|
||||
such file or directory`, `loki_source_file_files_active_total 0`.
|
||||
|
||||
Downstream: Loki holds zero streams (`GET /loki/api/v1/labels` → no `data`) —
|
||||
see [[Loki]].
|
||||
|
||||
**Closing it is a permission/architecture decision, not a config edit.** The
|
||||
unit, config and push endpoint are correct as written; rootless Podman simply
|
||||
cannot grant the journal group. The viable shapes: run Alloy as a *system*
|
||||
service (root, in `adm`/`systemd-journal`) instead of a rootless Quadlet; grant
|
||||
the container's mapped uid journal read via a tmpfiles/ACL rule; or have
|
||||
journald forward to a path the container can read. Each trades rootless posture
|
||||
for reach — decide deliberately. Re-check `loki_source_journal_target_lines_total`
|
||||
and the Loki label query to confirm a fix, never `/healthy`: health is exactly
|
||||
what hides this failure.
|
||||
|
||||
## Gotchas
|
||||
|
||||
|
||||
@@ -3,7 +3,7 @@ type: reference
|
||||
title: "Observability on Dankclaw"
|
||||
description: "The observability stack: what runs, how it is deployed, how it is exposed, and what is still unconfigured."
|
||||
tags: [observability, prometheus, grafana, loki, alloy, node-exporter, quadlet, dankclaw]
|
||||
timestamp: 2026-09-11
|
||||
timestamp: 2026-10-20
|
||||
---
|
||||
|
||||
# Observability on Dankclaw
|
||||
@@ -83,27 +83,36 @@ health endpoint (`/prometheus`: `/-/healthy`, `/alertmanager`: `/-/healthy`
|
||||
[[Observability Provisioning]] for what is provisioned as code and how to
|
||||
verify it through the APIs.
|
||||
|
||||
## Current state (2026-10-18)
|
||||
## Current state (2026-10-20)
|
||||
|
||||
* **Metrics: working.** `curl -s http://127.0.0.1:9090/api/v1/targets` reports
|
||||
eight up targets; the four downs are structural (the `comfyui` and `services`
|
||||
HTML probes, and Gitea's disabled metrics endpoint), not outages. Rule set:
|
||||
4 alert rules + 2 recording rules (`/api/v1/rules`, matching
|
||||
`shared/prometheus/rules/llm-fleet.yml`).
|
||||
eight up targets — the three LLM endpoints (frank-archon on `vllm:*`,
|
||||
frank-dragoon and frank-zealot on `llamacpp:*`), grafana, loki, prometheus
|
||||
and both node exporters. The four downs are structural (the `comfyui` and
|
||||
`services` HTML probes, and Gitea's disabled metrics endpoint), not outages.
|
||||
Two rule groups in `shared/prometheus/rules/llm-fleet.yml`: `llm-fleet`
|
||||
(2 recording + 4 alert, `llamacpp:*`) and `vllm-fleet` (1 recording + 4
|
||||
alert, `vllm:*` — archon queueing, prefix collapse, preemption, p95
|
||||
time-to-first-token), each keyed only to the namespace its runtime exports.
|
||||
* **Delivery: working.** Alertmanager's single `signal` receiver webhooks the
|
||||
relay, and `journalctl --user -u signal-relay` records successful
|
||||
`sent to +12062476336:` lines for every rule in the set.
|
||||
* **Dashboards: working.** One provisioned Prometheus datasource (uid
|
||||
`prometheus`) and the `llm-fleet` dashboard, both installed from
|
||||
`shared/grafana/provisioning/`.
|
||||
* **Logs: configured, empty.** Alloy ships zero lines (journal counter 0, both
|
||||
file globs refused) and Loki therefore holds zero streams — `/opt/loki/data`
|
||||
is a few hundred KB of WAL/compactor bookkeeping with **no chunk files**
|
||||
(8 files total by `find`, 2026-10-18) and `GET /loki/api/v1/labels` returns
|
||||
no label names.
|
||||
See [[Alloy]] → Delivery state and [[Loki]] → Contents.
|
||||
* **Not configured:** no Loki datasource in Grafana, no blackbox exporter (the
|
||||
structural downs have no real uptime signal), no Alertmanager vhost.
|
||||
* **Dashboards: working.** Provisioned datasources — Prometheus (uid
|
||||
`prometheus`) and Loki (uid `loki`) — and the `llm-fleet` dashboard, all
|
||||
installed from `shared/grafana/provisioning/`. The Loki datasource is how
|
||||
you read logs: Loki has no web UI of its own, so Grafana Explore is the
|
||||
human surface (the guarded `loki.franksdank.cloud` vhost serves only the API).
|
||||
* **Logs: configured, empty.** Alloy ships zero lines and Loki holds zero
|
||||
streams (`GET /loki/api/v1/labels` returns no label names; no chunk files on
|
||||
disk). The cause is source-side access, now proven, not a config bug: a
|
||||
rootless container cannot read the journal files (`system.journal` is
|
||||
`root:systemd-journal 0640`/`adm` and a rootless read of it returns DENIED),
|
||||
and the container-log globs point at a root-storage path that does not exist
|
||||
under rootless. Closing it is a permission/architecture decision, not an
|
||||
edit — see [[Alloy]] → Delivery state and [[Loki]] → Contents.
|
||||
* **Not configured:** no blackbox exporter (the structural downs have no real
|
||||
uptime signal) and no Alertmanager vhost. The log pipeline is provisioned
|
||||
but carries no data (see *Logs* above).
|
||||
|
||||
## Citations
|
||||
|
||||
|
||||
Reference in New Issue
Block a user