Every "error" the observability stack reported was structural, not an outage. This tranche retires all of them and gates the two silent-failure classes that caused them.
Targets
New blackbox exporter (shared/blackbox/blackbox.yml + root quadlet, loopback bind): Prometheus 3 refuses HTML scrape bodies, so the old comfyui/services probe jobs were dead targets reading as outages. SearXNG, OpenWebUI and ComfyUI are now probed via __param_target.
Gitea /metrics enabled in its quadlet - its target was a permanent down beside a healthy service.
deploy-plan.py/deploy-remote.sh: planner + copier + SERVICE_ORDER wiring for blackbox-exporter (before prometheus). Result: every Prometheus job reports up.
Logs
Loki metrics queries group only by indexed stream labels - the journal-as-JSON stream made every per-container view empty (6.5k lines/5min flowing, board reading zero). A loki.process relabel stage now promotes podman's CONTAINER_NAME to a container label (plus host).
logs-overview.json and observability.json rebuilt around the real label set; the stale "structural downs" panel is fixed.
Gates (the silent-failure classes)
copy_alloy_config now validates the rendered config with the pinned alloy image before installing - alloy attributes are position-sensitive (on_error on the labels stage passed every textual seam and only exploded at boot).
The caddyfile target-shape seam learns the documented exception: URL targets are legal only inside a prober job (per-job state; the first cut leaked probe[] across jobs and passed a planted URL - caught red).
New copier sandbox tests execute both copiers for real in sandboxes, incl. red proofs that a planted URL target and an engine-invalid alloy config are refused.
Docs - Prometheus / Observability / Provisioning pages state the post-blackbox reality; AGENTS.md drops the stale "structural reasons" passage.
Verification
./tests/run.sh - ALL TESTS PASSED (incl. new seams; the two image-gated tests skip on boxes without the alloy image).
Rendered alloy config validates clean against localhost/franksdank/alloy:latest (v1.21.0).
Seam red proofs: planted http:// target outside a prober job -> named line + exit 1.
Deploy after merge: ./scripts/deploy.sh dankclaw --service blackbox-exporter --service alloy --service prometheus (no fleet drain).
## What
Every "error" the observability stack reported was structural, not an outage. This tranche retires all of them and gates the two silent-failure classes that caused them.
**Targets**
- New **blackbox exporter** (`shared/blackbox/blackbox.yml` + root quadlet, loopback bind): Prometheus 3 refuses HTML scrape bodies, so the old `comfyui`/`services` probe jobs were dead targets reading as outages. SearXNG, OpenWebUI and ComfyUI are now probed via `__param_target`.
- **Gitea `/metrics` enabled** in its quadlet - its target was a permanent down beside a healthy service.
- `deploy-plan.py`/`deploy-remote.sh`: planner + copier + `SERVICE_ORDER` wiring for `blackbox-exporter` (before `prometheus`). Result: every Prometheus job reports up.
**Logs**
- Loki metrics queries group only by **indexed stream labels** - the journal-as-JSON stream made every per-container view empty (6.5k lines/5min flowing, board reading zero). A `loki.process` relabel stage now promotes podman's `CONTAINER_NAME` to a `container` label (plus `host`).
- `logs-overview.json` and `observability.json` rebuilt around the real label set; the stale "structural downs" panel is fixed.
**Gates (the silent-failure classes)**
- `copy_alloy_config` now **validates the rendered config with the pinned alloy image** before installing - alloy attributes are position-sensitive (`on_error` on the labels stage passed every textual seam and only exploded at boot).
- The caddyfile target-shape seam learns the documented exception: URL targets are legal only inside a prober job (per-job state; the first cut leaked `probe[]` across jobs and passed a planted URL - caught red).
- New copier sandbox tests execute both copiers for real in sandboxes, incl. red proofs that a planted URL target and an engine-invalid alloy config are refused.
**Docs** - Prometheus / Observability / Provisioning pages state the post-blackbox reality; AGENTS.md drops the stale "structural reasons" passage.
## Verification
- `./tests/run.sh` - ALL TESTS PASSED (incl. new seams; the two image-gated tests skip on boxes without the alloy image).
- Rendered alloy config `validate`s clean against `localhost/franksdank/alloy:latest` (v1.21.0).
- Seam red proofs: planted `http://` target outside a prober job -> named line + exit 1.
Deploy after merge: `./scripts/deploy.sh dankclaw --service blackbox-exporter --service alloy --service prometheus` (no fleet drain).
---
cwd: /home/blank/.omp/wt/wt-obs-fixes · session: 11fbd58f-35a5-413c-a31f-16d87f7a774a
Prometheus 3 refuses HTML scrape bodies, so the comfyui/services probe
jobs were structurally dead targets reading as outages. HTTP liveness
now rides a blackbox exporter (shared/blackbox/blackbox.yml, loopback
bind, http_2xx module); prometheus.yml probes SearXNG, OpenWebUI and
ComfyUI via __param_target relabeling. The unit is a boot-persistent,
self-healing root quadlet.
observability.json: stale 'structural downs' panel description fixed,
target census matches the post-blackbox job set. logs-overview.json:
panels grouped by the promoted container label instead of the absent
container_name, so the board stops reading empty beside live traffic.
Loki metrics queries group only by indexed stream labels, so the
journal-as-JSON stream made every per-container view structurally
empty (measured 6.5k lines/5min flowing while the board read zero).
A loki.process relabel stage now extracts CONTAINER_NAME into a
'container' label. alloy 1.21's json stage has no on_error attribute
(validated against the pinned image) — format_as_json guarantees the
line shape, so no error branch exists.
copy_alloy_config now validates the rendered config with the very
alloy image that will run it, before installing: a misplaced attribute
crash-loops the unit behind a green deploy, and no textual seam can
see it. copy_blackbox_config ships the prober config; the planner maps
the new unit and shared/blackbox/** to the blackbox-exporter service,
ordered before prometheus.
- caddyfile seam learns the documented exception: URL-shaped targets
are legal only inside a prober job (state is per-job; the first cut
leaked it and passed a planted URL).
- observability seams: blackbox unit + planner entries, both copiers
executed for real in sandboxes, alloy engine-validation refusal
(image-gated), dashboard panel invariants.
- AGENTS.md: structural target downs retired.
Prometheus/Observability/Provisioning pages: every job up after the
blackbox swap, the copier list learns copy_blackbox_config and the
alloy validate gate, the config table gains alloy + blackbox rows, and
the logs bullet states the stream-label law instead of the retired
'awaiting deploy' note.
Measured 2026-10-21 with frank-dragoon live: MemAvailable 15.3 GiB,
iGPU busy 1.75 GiB. The Qwen-Image weight sets (13.3-14.2 GB) do not
fit that budget, and the page's 122 GB /system_stats figure is the
aperture, not headroom. Adds the fit table and the
stop-mask-render-verify procedure (the 3090 bench law, applied to the
APU); ComfyUI.md gotcha links it.
dank
merged commit d61c19d41f into main2026-10-21 14:53:16 -07:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
What
Every "error" the observability stack reported was structural, not an outage. This tranche retires all of them and gates the two silent-failure classes that caused them.
Targets
shared/blackbox/blackbox.yml+ root quadlet, loopback bind): Prometheus 3 refuses HTML scrape bodies, so the oldcomfyui/servicesprobe jobs were dead targets reading as outages. SearXNG, OpenWebUI and ComfyUI are now probed via__param_target./metricsenabled in its quadlet - its target was a permanent down beside a healthy service.deploy-plan.py/deploy-remote.sh: planner + copier +SERVICE_ORDERwiring forblackbox-exporter(beforeprometheus). Result: every Prometheus job reports up.Logs
loki.processrelabel stage now promotes podman'sCONTAINER_NAMEto acontainerlabel (plushost).logs-overview.jsonandobservability.jsonrebuilt around the real label set; the stale "structural downs" panel is fixed.Gates (the silent-failure classes)
copy_alloy_confignow validates the rendered config with the pinned alloy image before installing - alloy attributes are position-sensitive (on_erroron the labels stage passed every textual seam and only exploded at boot).probe[]across jobs and passed a planted URL - caught red).Docs - Prometheus / Observability / Provisioning pages state the post-blackbox reality; AGENTS.md drops the stale "structural reasons" passage.
Verification
./tests/run.sh- ALL TESTS PASSED (incl. new seams; the two image-gated tests skip on boxes without the alloy image).validates clean againstlocalhost/franksdank/alloy:latest(v1.21.0).http://target outside a prober job -> named line + exit 1.Deploy after merge:
./scripts/deploy.sh dankclaw --service blackbox-exporter --service alloy --service prometheus(no fleet drain).cwd: /home/blank/.omp/wt/wt-obs-fixes · session: 11fbd58f-35a5-413c-a31f-16d87f7a774a