docs: ComfyUI is a unit now, and three scrape jobs can never be green

ComfyUI's page described a compose service and a systemd unit that shelled out
to `podman compose up`. It now documents the Quadlet, the keep-groups + udev
mechanism (verified inside the running container: torch sees Radeon 8060S), the
9,146-restart incident that forced the move, and the trap that caused it — a
stale container outlives the compose edit that should have replaced it.

The observability page gains the audit result: four jobs read "down" and three
of them always will, because they scrape HTML pages or a /metrics that does not
exist. The page names each error, says plainly that none is an outage, and
refuses the tempting fix (deleting the job makes an absent scrape look like a
healthy one) in favour of the remedy it already recommended: a blackbox exporter
in probe mode, and Gitea's metrics enabled or the job repointed.

serverAppConfigs-docs/Dankclaw/ComfyUI.md

@@ -16,8 +16,8 @@ https://github.com/Comfy-Org/ComfyUI
 | Port | 8188 (host) → 8188 (container) |
 | Backend | ROCm GPU (gfx1151 / Radeon 8060S). No `--cpu` flag. |
 | Data | `/home/dank/comfyui-data/{models,input,output,custom_nodes}` → `/opt/ComfyUI/...` |
-| Run | `podman compose -f dankclaw/compose.yaml up -d comfyui` |
-| Boot-persistent | user systemd unit `comfyui.service` (runs the compose `up` at boot) |
+| Run | `./scripts/deploy.sh dankclaw --service comfyui` (Quadlet unit `comfyui.service`) |
+| Boot-persistent | Quadlet unit `dankclaw/comfyui/quadlet/comfyui.container` → `~/.config/containers/systemd/` |
 
 ## Why the GPU works here
 
@@ -27,11 +27,28 @@ Dankclaw is an AMD Strix Halo APU (gfx1151). Standard PyTorch ROCm wheels do not
 
 `dankclaw/Containerfile.comfyui` starts from the AMD gfx1151 base image and bakes the ComfyUI code in. It installs ComfyUI's dependencies but skips the torch family (torch, torchvision, torchaudio). The base image already ships the matching ROCm builds. Installing an unpinned `torch` from `requirements.txt` clobbers the gfx1151 kernels with a default build, so the Containerfile filters those lines out.
 
-## GPU device access (rootless)
+## GPU device access (rootless), and why the unit exists
 
-The container reaches the GPU through the `/dev/kfd` and `/dev/dri` device passthrough. The permission is checked host-side. The run user is already a member of the `render` group on the host, so the device is readable.
+The unit passes `/dev/kfd` and `/dev/dri` (`AddDevice=`) and uses
+`GroupAdd=keep-groups`. A literal `render` entry cannot work: rootless Podman
+cannot map the group name into the user namespace. The companion udev rule
+`dankclaw/udev/99-comfyui-gpu-access.rules` keeps those nodes `MODE 0666`, which
+is what lets a **user-unit spawn** — carrying no supplementary groups — open
+them anyway. Verified inside the running container: `torch.cuda.is_available()`
+is `True` and the device is `Radeon 8060S Graphics`.
 
-Only the `video` group is added to the container. The `render` group (gid 990) is not added. Rootless podman cannot map it into the container namespace (the subgid range is 100000+), so `--group-add render` fails to start the container.
+**Incident, 2026-09-16 — why this is a Quadlet now.** ComfyUI ran from
+`dankclaw/compose.yaml` through a hand-written `~/.config/systemd/user/
+comfyui.service` that called `podman compose up -d`. Its container was created
+*before* the GPU declarations were added to the compose service, and
+`restart: unless-stopped` then kept restarting that stale container: no
+`/dev/kfd`, no `/dev/dri`, no keep-groups, dead on `RuntimeError: No HIP GPUs
+are available` — **9,146 restarts** and about seven hours down. The service was
+fine; the container was old. A `--force-recreate` would have fixed it that day,
+but the class of bug needed the unit: one owner per container name, repo-tracked,
+and recreated from the source of truth on every deploy. The compose service is
+deleted; `deploy_comfyui` retires any compose-era unit into `~/.AGENTTRASH/`
+and removes a stale container before installing the unit.
 
 ## Volumes
 
@@ -60,7 +77,11 @@ ComfyUI-Manager self-installs its own pip dependencies on start (GitPython, PyGi
 
 ## Boot persistence
 
-The user systemd unit `~/.config/systemd/user/comfyui.service` runs `podman compose -f /home/dank/serverAppConfigs/dankclaw/compose.yaml up -d comfyui` at boot. It sets `WantedBy=default.target`, and user linger is on. The old CPU venv unit is preserved at `comfyui.service.venv.bak` as a fallback.
+The Quadlet unit carries `[Install] WantedBy=default.target` with
+`Restart=always`, so the generator wires it into `default.target` and it comes
+back on its own (user linger is on). Inspect it with
+`systemctl --user is-enabled comfyui.service` → `generated`; an explicit
+`systemctl --user enable` fails on generated units.
 
 ## Healthcheck
 
@@ -68,8 +89,9 @@ The web UI responds on `/`. ComfyUI also exposes `/system_stats`. On this APU, `
 
 ## Gotchas
 
-- The GPU path needs the device passthrough. Do not remove `/dev/kfd` or `/dev/dri` from the compose service.
-- Do not add the `render` group to the container. Rootless podman cannot map it. Add only `video`.
+- The GPU path needs the device passthrough. Do not remove `AddDevice=/dev/kfd` or `AddDevice=/dev/dri` from the unit, and keep `GroupAdd=keep-groups`.
+- Do not add the `render` group by name. Rootless podman cannot map it; the udev rule plus `keep-groups` is the mechanism that works.
+- **A stale container outlives a compose edit.** If the GPU declarations change, recreate the container (`deploy.sh dankclaw --service comfyui` removes the old one) — `restart: unless-stopped` will happily keep an old config alive forever.
 - Do not let pip reinstall the torch family. The Containerfile filters it out on purpose.
 - The base image is about 32 GB. A code update means a rebuild, because the code is baked in.
 - The model collection starts empty. Add models to `/home/dank/comfyui-data/models` before generating.

serverAppConfigs-docs/Infrastructure/Observability.md

@@ -102,6 +102,22 @@ Prometheus cannot solve an Anubis challenge.
 
 For services without a native exporter, use the Prometheus **blackbox exporter** in probe mode to assert up/down and HTTP latency. The homepage status timer probes the same endpoints from the host.
 
+**Four jobs read "down" and three of them always will** (found 2026-09-16 while
+auditing why the target list was not green):
+
+| job | error | what it means |
+|-----|-------|---------------|
+| `comfyui` | `unsupported Content-Type "text/html"` | the service is **up** (this is what it says since ComfyUI was fixed — before that it read `connection refused`); Prometheus refuses to parse an HTML page as metrics, so this job cannot succeed as configured |
+| `services` (SearXNG, OpenWebUI) | `unsupported Content-Type "text/html; charset=utf-8"` | same defect: "blackbox-style probes" that scrape a page instead of metrics |
+| `gitea` | `HTTP status 404 Not Found` | `/metrics` does not exist on that Gitea (metrics disabled); the service itself is fine |
+
+None of these is an outage, and none should be silenced by deleting the job —
+an absent scrape and a healthy one look identical. The fix the repository
+already names is a **blackbox exporter** in probe mode for the page-only
+targets, and either enabling Gitea's metrics or pointing the job at the right
+path. The LLM alert rules (`LlmEndpointDown`) are scoped to `frank-*`, so these
+probes generate no false pages.
+
 ## Log aggregation choice
 
 Loki + Alloy is the lightweight Grafana-native option for a two-server homelab: