docs: ComfyUI is a unit now, and three scrape jobs can never be green
ComfyUI's page described a compose service and a systemd unit that shelled out to `podman compose up`. It now documents the Quadlet, the keep-groups + udev mechanism (verified inside the running container: torch sees Radeon 8060S), the 9,146-restart incident that forced the move, and the trap that caused it — a stale container outlives the compose edit that should have replaced it. The observability page gains the audit result: four jobs read "down" and three of them always will, because they scrape HTML pages or a /metrics that does not exist. The page names each error, says plainly that none is an outage, and refuses the tempting fix (deleting the job makes an absent scrape look like a healthy one) in favour of the remedy it already recommended: a blackbox exporter in probe mode, and Gitea's metrics enabled or the job repointed.
This commit is contained in:
@@ -102,6 +102,22 @@ Prometheus cannot solve an Anubis challenge.
|
||||
|
||||
For services without a native exporter, use the Prometheus **blackbox exporter** in probe mode to assert up/down and HTTP latency. The homepage status timer probes the same endpoints from the host.
|
||||
|
||||
**Four jobs read "down" and three of them always will** (found 2026-09-16 while
|
||||
auditing why the target list was not green):
|
||||
|
||||
| job | error | what it means |
|
||||
|-----|-------|---------------|
|
||||
| `comfyui` | `unsupported Content-Type "text/html"` | the service is **up** (this is what it says since ComfyUI was fixed — before that it read `connection refused`); Prometheus refuses to parse an HTML page as metrics, so this job cannot succeed as configured |
|
||||
| `services` (SearXNG, OpenWebUI) | `unsupported Content-Type "text/html; charset=utf-8"` | same defect: "blackbox-style probes" that scrape a page instead of metrics |
|
||||
| `gitea` | `HTTP status 404 Not Found` | `/metrics` does not exist on that Gitea (metrics disabled); the service itself is fine |
|
||||
|
||||
None of these is an outage, and none should be silenced by deleting the job —
|
||||
an absent scrape and a healthy one look identical. The fix the repository
|
||||
already names is a **blackbox exporter** in probe mode for the page-only
|
||||
targets, and either enabling Gitea's metrics or pointing the job at the right
|
||||
path. The LLM alert rules (`LlmEndpointDown`) are scoped to `frank-*`, so these
|
||||
probes generate no false pages.
|
||||
|
||||
## Log aggregation choice
|
||||
|
||||
Loki + Alloy is the lightweight Grafana-native option for a two-server homelab:
|
||||
|
||||
Reference in New Issue
Block a user