ComfyUI's page described a compose service and a systemd unit that shelled out
to `podman compose up`. It now documents the Quadlet, the keep-groups + udev
mechanism (verified inside the running container: torch sees Radeon 8060S), the
9,146-restart incident that forced the move, and the trap that caused it — a
stale container outlives the compose edit that should have replaced it.
The observability page gains the audit result: four jobs read "down" and three
of them always will, because they scrape HTML pages or a /metrics that does not
exist. The page names each error, says plainly that none is an outage, and
refuses the tempting fix (deleting the job makes an absent scrape look like a
healthy one) in favour of the remedy it already recommended: a blackbox exporter
in probe mode, and Gitea's metrics enabled or the job repointed.
The page said "all five are rootless Quadlet units on Dankclaw", which stopped
being true when Dankserver's node-exporter was restored to a unit. It now names
both hosts and both deployments, records why the second one happened (the same
compose-only reboot argument, a day and a half of missing host metrics, and a
wildcard bind), and states the exposure rule for a two-listener fleet with the
`ss` evidence that the wildcard is gone.
Both pages claimed there was no alerting; there are six rules now, so they say
which rules exist, that promtool and /api/v1/rules both confirm them, and — in
the same breath — that no rule reaches a person, because this stack has no
Alertmanager and no contact point. The dashboard's ALERTS panel is named as the
only place an alert becomes visible without opening Prometheus.
The page said Grafana has zero datasources and zero dashboards; that is now
false, so it says what exists (Prometheus datasource, LLM fleet dashboard), how
it was verified (the API — a provisioned dashboard does not show up in the
legacy dashboard table on this Grafana version, which is a trap worth recording)
and what is still missing (nothing alerted on, Loki unread, correlation manual).
The section heading drops "UNCONFIGURED" because it no longer is, entirely.
The capacity work in this runbook was done with ad-hoc probes; the same signals
now have history in Prometheus, so the page says which series to watch and
admits the honest gap — this stack has no alerting, so a saturated pool shows up
on a graph after the fact rather than paging anyone.
Rewrites the [[Observability]] umbrella page and the five component pages for
what is actually deployed: rootless Quadlet units instead of compose services,
the loopback-only publish posture (node-exporter's LAN binding explained), the
loki healthcheck living in its image because the image has no shell, and the
Home=/root trap that killed the first grafana start.
Adds the note that matters most to a reader: the stack is installed, healthy
and UNCONFIGURED. Grafana reports zero datasources and zero dashboards with an
empty provisioning directory; Prometheus has no rule files and no Alertmanager;
Loki is configured but has no reader; node-exporter runs its default
collectors. Alloy is the only configured piece. Metrics and logs are collected
and retained, and nothing is visualized or alerted on.
AGENTS.md's boot-persistence gap note is replaced by the resolution.
All in-repo path references (AGENTS.md, README, old scripts, report artifacts,
test code paths) rewritten in the same commit; focused-test map extended for
.gitignore, old/scripts/*, dankclaw/comfyui/*, and report/* per this repo's
own convention ('add one before committing'). Obsidian wikilinks are
title-based and unaffected.
Frank