From cb86ad8fd7a8acf6222977541c64fe3cc8403bfe Mon Sep 17 00:00:00 2001 From: Frank Date: Wed, 21 Oct 2026 13:33:03 -0700 Subject: [PATCH] docs(observability): post-blackbox state, copier law, label-promotion why Prometheus/Observability/Provisioning pages: every job up after the blackbox swap, the copier list learns copy_blackbox_config and the alloy validate gate, the config table gains alloy + blackbox rows, and the logs bullet states the stream-label law instead of the retired 'awaiting deploy' note. --- .../Observability Provisioning.md | 13 ++++--- .../Infrastructure/Observability.md | 37 ++++++++++--------- .../Infrastructure/Prometheus.md | 30 +++++++-------- 3 files changed, 41 insertions(+), 39 deletions(-) diff --git a/serverAppConfigs-docs/Infrastructure/Observability Provisioning.md b/serverAppConfigs-docs/Infrastructure/Observability Provisioning.md index 2771bbf..a59771a 100644 --- a/serverAppConfigs-docs/Infrastructure/Observability Provisioning.md +++ b/serverAppConfigs-docs/Infrastructure/Observability Provisioning.md @@ -20,6 +20,8 @@ committed. | `shared/prometheus/prometheus.yml` | scrape jobs (global + per-host) | | `shared/prometheus/rules/llm-fleet.yml` | alert rules + recording rules | | `shared/grafana/provisioning/` | datasources + dashboards (provisioned tree) | +| `shared/alloy/config.alloy` | the log pipeline: journal source → label promotion → Loki push | +| `shared/blackbox/blackbox.yml` | prober modules (`http_2xx`) + the loopback bind | Targets and datasources reference **container names** (`grafana:3000` `loki:3100` — `shared/prometheus/prometheus.yml:52-61`), never @@ -31,8 +33,8 @@ scraped as `host.containers.internal:` because a bridge container cannot reach a host-netns bind at the host's LAN IP (`prometheus.yml:89-99`), and only the endpoint on another machine (`192.168.0.235:7770`) is addressed by LAN IP (`prometheus.yml:128-131`). Prometheus 3 refuses HTML responses as metrics — -HTML-scraped "probes" are structurally inert and need a blackbox exporter (see -[[Stale Signal Audit]]). +HTTP liveness rides the **blackbox exporter** (`shared/blackbox/blackbox.yml`, +deployed 2026-10-21; see [[Stale Signal Audit]]). ## The deploy path @@ -43,7 +45,8 @@ from `deploy_observability`: - `copy_alertmanager_config` → `/opt/alertmanager/config/alertmanager.yml` (`:232-235`) - `install_signal_relay` → the user unit + iptables + the health wait on `192.168.0.143:18093/health` (`:243-274`) - `copy_loki_config` → `/opt/loki/config/loki-config.yaml` (`:276-279`) -- `copy_alloy_config` → `/opt/alloy/config/config.alloy` (`:281-284`) +- `copy_alloy_config` → `/opt/alloy/config/config.alloy` — substitutes the `GRAFANA_HOST_LABEL` token with the deploy target's name and **refuses to install** if the token survives; `ALLOY_CONF_ROOT` is the bats sandbox seam +- `copy_blackbox_config` → `/opt/blackbox/blackbox.yml` (single-file mount of the prober); `BLACKBOX_CONF_ROOT` is its sandbox seam - `copy_grafana_provisioning` → `/opt/grafana/provisioning/` — datasources, provider, and **every** `dashboards/*.json` (a glob, not a hand-listed subset; `GRAFANA_PROV_ROOT` is the test seam the bats case uses to run the copier into a sandbox) `scripts/deploy-plan.py` maps `shared/prometheus/**` to `prometheus` (plus @@ -109,8 +112,8 @@ curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | \ The two Prometheus queries were run on 2026-10-18 and are what every number on this page comes from: 4 alerting + 2 recording rules, and `up` for `prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki` -`frank-zealot`, `frank-dragoon`, `frank-archon` with the four structural downs -listed in [[Prometheus]]. The Grafana proof is the browser session above, run +`frank-zealot`, `frank-dragoon`, `frank-archon`. The four structural downs +that census recorded are retired (2026-10-21, [[Prometheus]]). The Grafana proof is the browser session above, run 2026-10-21: it found a provisioned-file gap the filesystem listing hid. The filesystem side (`ls /opt/grafana/provisioning/*`) is proof of installation only — see *Add a dashboard* for why a repo JSON is not a live board. diff --git a/serverAppConfigs-docs/Infrastructure/Observability.md b/serverAppConfigs-docs/Infrastructure/Observability.md index 615404f..0660283 100644 --- a/serverAppConfigs-docs/Infrastructure/Observability.md +++ b/serverAppConfigs-docs/Infrastructure/Observability.md @@ -95,13 +95,14 @@ health endpoint (`/prometheus`: `/-/healthy`, `/alertmanager`: `/-/healthy` [[Observability Provisioning]] for what is provisioned as code and how to verify it through the APIs. -## Current state (2026-10-20) +## Current state (2026-10-21) * **Metrics: working.** `curl -s http://127.0.0.1:9090/api/v1/targets` reports - eight up targets — the three LLM endpoints (frank-archon on `vllm:*`, - frank-dragoon and frank-zealot on `llamacpp:*`), grafana, loki, prometheus - and both node exporters. The four downs are structural (the `comfyui` and - `services` HTML probes, and Gitea's disabled metrics endpoint), not outages. + every job up — the three LLM endpoints (frank-archon on `vllm:*`, + frank-dragoon and frank-zealot on `llamacpp:*`), grafana, loki, prometheus, + both node exporters, cadvisor, gitea (metrics enabled in its quadlet) and + the blackbox-probed `services` (SearXNG, OpenWebUI, ComfyUI). The structural + downs are retired — see [[Prometheus]] → Targets. Two rule groups in `shared/prometheus/rules/llm-fleet.yml`: `llm-fleet` (2 recording + 4 alert, `llamacpp:*`) and `vllm-fleet` (1 recording + 4 alert, `vllm:*` — archon queueing, prefix collapse, preemption, p95 @@ -142,26 +143,26 @@ verify it through the APIs. `power1_average`, so they carry no `gpu_power_watts` line at all (only NVIDIA and the Dankclaw 8060S iGPU report power); and it binds loopback + the host LAN, like node-exporter, never the wildcard. The dashboard graphs util/VRAM/VRAM-%/temp/power. -* **Logs: fixed in code, awaiting deploy.** The pipeline shipped nothing because - a rootless Alloy cannot read the `0640` journal files, and the container-log - globs point at a root-storage layout that does not exist here. The fix is built - and tested in-repo (2026-10-20): Alloy is now a **root system service on - `Network=host`**, reading `system.journal` + `user-*.journal` and pushing to - Loki's `127.0.0.1:3100` loopback publish — a throwaway root probe read 31,825 - lines and pushed 26,966 with zero drops. The **live** instance is still the old - rootless unit until `deploy.sh dankclaw --service alloy` runs. Detail in - [[Alloy]] → Why root; Loki's store state in [[Loki]] → Contents. -* **Not configured:** no blackbox exporter (the structural downs have no real - uptime signal) and no Alertmanager vhost. The log pipeline's fix is in code but - not yet deployed (see *Logs* above). +* **Logs: flowing; container labels land with the 2026-10-21 deploy.** The + pipeline once shipped nothing (rootless Alloy cannot read the `0640` journal; + the container-log globs pointed at a root-storage layout that does not exist + here) — fixed 2026-10-20 by making Alloy a **root system service on + `Network=host`**. Journal lines now reach Loki (measured ~6.5k lines/5min), + but the stream carried only `job`/`service_name`, so every per-container + view was structurally empty: Loki metrics queries group only by INDEXED + stream labels. Alloy now promotes podman's `CONTAINER_NAME` to a `container` + label (plus a `host` label) at ingestion — `loki.process` in + `shared/alloy/config.alloy`; detail in [[Alloy]] → Why root, and the + purpose-test law in [[Observability Provisioning]]. ## Citations -- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter,cadvisor}.container` +- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter,cadvisor,blackbox-exporter}.container` - `scripts/deploy-remote.sh:219-300` (copiers + relay), `:521-536` (health waits), `:477-486` (grafana env seed) - `scripts/signal-relay.py:26-49`, `scripts/signal-relay.service` - `scripts/gpu-exporter.py`, `scripts/gpu-exporter.service` (per-GPU host exporter) - `shared/prometheus/prometheus.yml`, `shared/prometheus/alertmanager.yml`, `shared/prometheus/rules/llm-fleet.yml` - `shared/grafana/provisioning/`, `shared/loki/loki-config.yaml`, `shared/alloy/config.alloy` +- `shared/blackbox/blackbox.yml` (prober modules + loopback bind) - `tests/observability_quadlet.bats` (boot persistence, no wildcard publish, probe addresses, delivery contract) - Host evidence 2026-10-18: `ss -ltnp`, `podman ps`, `systemctl --user is-enabled|is-active`, `curl` on `127.0.0.1:9090/api/v1/{targets,rules,alerts}`, `127.0.0.1:3100/loki/api/v1/labels`, `127.0.0.1:12345/metrics`, `127.0.0.1:3002/api/health`, `192.168.0.143:18093/health`, `journalctl --user -u signal-relay` diff --git a/serverAppConfigs-docs/Infrastructure/Prometheus.md b/serverAppConfigs-docs/Infrastructure/Prometheus.md index 9b51a7a..fffed02 100644 --- a/serverAppConfigs-docs/Infrastructure/Prometheus.md +++ b/serverAppConfigs-docs/Infrastructure/Prometheus.md @@ -42,29 +42,27 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) — The `frank-medium` job is **retired**: no `frank-medium` job exists in the config or in `/api/v1/targets` (2026-10-18), and the endpoint itself left the fleet — its history is on [[frank-archon]], which holds the port today. -**Three jobs read down today, and they are not outages** (re-audited against `curl -s http://127.0.0.1:9090/api/v1/targets`, 2026-10-18; first audited 2026-09-16): +**No job reads down structurally any more** (the three permanent-reds were +retired 2026-10-21; census re-run after each deploy): -| Job | Target | `health` | `lastError`, verbatim | -|-----|--------|----------|------------------------| -| `gitea` | `http://192.168.0.143:3000/metrics` | down | `server returned HTTP status 404 Not Found` | -| `comfyui` | `http://comfyui:8188/` | down | `received unsupported Content-Type "text/html" and no fallback_scrape_protocol specified for target` | -| `services` | `http://192.168.0.143:8081/` | down | `received unsupported Content-Type "text/html; charset=utf-8" and no fallback_scrape_protocol specified for target` | -| `services` | `http://192.168.0.143:3005/` | down | same `unsupported Content-Type "text/html; charset=utf-8"` error | +| Job | Target | how | +|---|---|---| +| `gitea` | `192.168.0.143:3000/metrics` | metrics enabled in the quadlet (`GITEA__metrics__ENABLED=true`, `ENABLED_HOST=0.0.0.0`) — the page once said "stays off until `[metrics] ENABLED = true` in its own config"; the env form is what the deployer ships | +| `services` | `http://192.168.0.143:8081` (SearXNG), `:3005` (OpenWebUI), `http://comfyui:8188` | **blackbox exporter** probe, module `http_2xx` — the comfyui metrics job is retired, one owner per service | -Gitea's exporter stays off until `[metrics] ENABLED = true` in its own config (no metrics env is set in the container, and `curl -s http://127.0.0.1:3000/metrics` → 404, 2026-10-18); ComfyUI and the two `services` probes are HTML pages, which Prometheus 3 refuses as metrics. Everything else is green: `prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`, `frank-zealot`, `frank-dragoon`, `frank-archon`. +Scraping a web root is not a health check: an HTML page fails the scrape +protocol (`unsupported Content-Type`), which is what kept those targets red +beside healthy services. The classification rule and the honesty law live in +[[Stale Signal Audit]]. The prober is `blackbox-exporter.container` (host +netns, config `shared/blackbox/blackbox.yml`, bound 127.0.0.1:9115; Prometheus +reaches it at the `host.containers.internal` gateway name). Plain targets stay +`host:port`, never URLs — the probe job's URL targets are the documented +exception, relabeled onto the prober. **`frank-zealot` reads up since 2026-10-18, and did not before.** llama.cpp only answers `/metrics` when started with `--metrics`, so `up{job="frank-zealot"}` was 0 across the whole retained history (the oldest sample, 2026-10-11, is already 0) while the service answered `/health` 200 the whole time; the flag and its opt-out are on [[frank-zealot]]. Read `lastError` before believing an outage: a target can read down beside a healthy service. Alert state is a live property, not a page fact: `curl -s http://127.0.0.1:9090/api/v1/alerts` returned no alerts at the 2026-10-18 morning census, one firing `LlmPrefixCacheCollapse` (info, `frank-zealot`, `activeAt` 11:10:48Z) minutes later, and none again by 11:56Z. -The remedy for a probe-shaped question is the **blackbox exporter**; the -classification rule and the honesty law for it live in [[Stale Signal Audit]]. -Until the exporter is deployed, these three stay red for a structural reason: -deleting them would -hide a real service, and fixing them by publishing databases or enabling metrics -would trade a wrong signal for a real exposure. Targets are `host:port`, never -URLs. - ## Volumes | Container path | Host path |