docs(observability): record the quadlet cutover, the exposure posture, and the unconfigured state

Rewrites the [[Observability]] umbrella page and the five component pages for
what is actually deployed: rootless Quadlet units instead of compose services,
the loopback-only publish posture (node-exporter's LAN binding explained), the
loki healthcheck living in its image because the image has no shell, and the
Home=/root trap that killed the first grafana start.

Adds the note that matters most to a reader: the stack is installed, healthy
and UNCONFIGURED. Grafana reports zero datasources and zero dashboards with an
empty provisioning directory; Prometheus has no rule files and no Alertmanager;
Loki is configured but has no reader; node-exporter runs its default
collectors. Alloy is the only configured piece. Metrics and logs are collected
and retained, and nothing is visualized or alerted on.

AGENTS.md's boot-persistence gap note is replaced by the resolution.
This commit is contained in:
Frank committed 2026-09-11 16:10:27 -07:00
1 parent 25c93984ad
commit 701ab14f60
8 files changed
+220 -147

No files matched your search

+12 -9
View File
@@ -200,15 +200,18 @@ Pi-hole unit existed but crash-looped on a sysctl conflict (a later-sorted
drop-in reset `net.ipv4.ip_unprivileged_port_start` to 80), taking LAN DNS
down with it.
**Known gap (2026-09-11, found while rebooting Dankclaw for the `amd_iommu=off`
change):** five Dankclaw services still run only from `dankclaw/compose.yaml`
with no Quadlet unit — `prometheus`, `grafana`, `loki`, `alloy` and
`node-exporter` (all under `~/.config/containers/systemd/` absent, but present
in the compose file). They did not come back after that reboot and were
restored with `./scripts/deploy.sh dankclaw --service <name>`; they will not
survive the next one either. Giving them units is tracked work, not a solved
problem — treat every reboot as expected to take the observability stack down
until it is done.
**Observability gap (found 2026-09-11, fixed the same day):** rebooting
Dankclaw for the `amd_iommu=off` change exposed five services that ran only
from `dankclaw/compose.yaml` with no Quadlet unit — `prometheus`, `grafana`,
`loki`, `alloy` and `node-exporter`. They did not come back and had to be
restored by hand. They are now rootless Quadlet units under
`dankclaw/observability/quadlet/` (`./scripts/deploy.sh dankclaw --service
<name>`), and their compose definitions were removed so only one owner exists
per container name. The three HTTP UIs publish on loopback only; off-host
access is the guarded vhost.
Note that the stack is RUNNING BUT UNCONFIGURED: Grafana has no datasources
and no dashboards, and Prometheus has no rules and no Alertmanager. See
`serverAppConfigs-docs/Infrastructure/Observability.md`.
## Testing