dank/serverAppConfigs
1
0
Fork 0
Code Issues 4 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

docs(observability): post-blackbox state, copier law, label-promotion why

Browse Source 
Prometheus/Observability/Provisioning pages: every job up after the
blackbox swap, the copier list learns copy_blackbox_config and the
alloy validate gate, the config table gains alloy + blackbox rows, and
the logs bullet states the stream-label law instead of the retired
'awaiting deploy' note.
This commit is contained in:
Frank
2026-10-21 13:33:03 -07:00
parent 078b550bd8
commit cb86ad8fd7
3 changed files with 41 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/Infrastructure/Observability Provisioning.md
+8 -5
View File
@@ -20,6 +20,8 @@ committed.
| `shared/prometheus/prometheus.yml` | scrape jobs (global + per-host) |
| `shared/prometheus/rules/llm-fleet.yml` | alert rules + recording rules |
| `shared/grafana/provisioning/` | datasources + dashboards (provisioned tree) |
| `shared/alloy/config.alloy` | the log pipeline: journal source → label promotion → Loki push |
| `shared/blackbox/blackbox.yml` | prober modules (`http_2xx`) + the loopback bind |
Targets and datasources reference **container names** (`grafana:3000`
`loki:3100` — `shared/prometheus/prometheus.yml:52-61`), never
@@ -31,8 +33,8 @@ scraped as `host.containers.internal:<port>` because a bridge container cannot
reach a host-netns bind at the host's LAN IP (`prometheus.yml:89-99`), and only
the endpoint on another machine (`192.168.0.235:7770`) is addressed by LAN IP
(`prometheus.yml:128-131`). Prometheus 3 refuses HTML responses as metrics —
HTML-scraped "probes" are structurally inert and need a blackbox exporter (see
[[Stale Signal Audit]]).
HTTP liveness rides the **blackbox exporter** (`shared/blackbox/blackbox.yml`,
deployed 2026-10-21; see [[Stale Signal Audit]]).
## The deploy path
@@ -43,7 +45,8 @@ from `deploy_observability`:
- `copy_alertmanager_config` → `/opt/alertmanager/config/alertmanager.yml` (`:232-235`)
- `install_signal_relay` → the user unit + iptables + the health wait on `192.168.0.143:18093/health` (`:243-274`)
- `copy_loki_config` → `/opt/loki/config/loki-config.yaml` (`:276-279`)
- `copy_alloy_config` → `/opt/alloy/config/config.alloy` (`:281-284`)
- `copy_alloy_config` → `/opt/alloy/config/config.alloy` — substitutes the `GRAFANA_HOST_LABEL` token with the deploy target's name and **refuses to install** if the token survives; `ALLOY_CONF_ROOT` is the bats sandbox seam
- `copy_blackbox_config` → `/opt/blackbox/blackbox.yml` (single-file mount of the prober); `BLACKBOX_CONF_ROOT` is its sandbox seam
- `copy_grafana_provisioning` → `/opt/grafana/provisioning/` — datasources, provider, and **every** `dashboards/*.json` (a glob, not a hand-listed subset; `GRAFANA_PROV_ROOT` is the test seam the bats case uses to run the copier into a sandbox)
`scripts/deploy-plan.py` maps `shared/prometheus/**` to `prometheus` (plus
@@ -109,8 +112,8 @@ curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | \
The two Prometheus queries were run on 2026-10-18 and are what every number on
this page comes from: 4 alerting + 2 recording rules, and `up` for
`prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`
`frank-zealot`, `frank-dragoon`, `frank-archon` with the four structural downs
listed in [[Prometheus]]. The Grafana proof is the browser session above, run
`frank-zealot`, `frank-dragoon`, `frank-archon`. The four structural downs
that census recorded are retired (2026-10-21, [[Prometheus]]). The Grafana proof is the browser session above, run
2026-10-21: it found a provisioned-file gap the filesystem listing hid. The
filesystem side (`ls /opt/grafana/provisioning/*`) is proof of installation
only — see *Add a dashboard* for why a repo JSON is not a live board.
serverAppConfigs-docs/Infrastructure/Observability.md
+19 -18
View File
@@ -95,13 +95,14 @@ health endpoint (`/prometheus`: `/-/healthy`, `/alertmanager`: `/-/healthy`
[[Observability Provisioning]] for what is provisioned as code and how to
verify it through the APIs.
## Current state (2026-10-20)
## Current state (2026-10-21)
* **Metrics: working.** `curl -s http://127.0.0.1:9090/api/v1/targets` reports
eight up targets — the three LLM endpoints (frank-archon on `vllm:*`,
frank-dragoon and frank-zealot on `llamacpp:*`), grafana, loki, prometheus
and both node exporters. The four downs are structural (the `comfyui` and
`services` HTML probes, and Gitea's disabled metrics endpoint), not outages.
every job up — the three LLM endpoints (frank-archon on `vllm:*`,
frank-dragoon and frank-zealot on `llamacpp:*`), grafana, loki, prometheus,
both node exporters, cadvisor, gitea (metrics enabled in its quadlet) and
the blackbox-probed `services` (SearXNG, OpenWebUI, ComfyUI). The structural
downs are retired — see [[Prometheus]] → Targets.
Two rule groups in `shared/prometheus/rules/llm-fleet.yml`: `llm-fleet`
(2 recording + 4 alert, `llamacpp:*`) and `vllm-fleet` (1 recording + 4
alert, `vllm:*` — archon queueing, prefix collapse, preemption, p95
@@ -142,26 +143,26 @@ verify it through the APIs.
`power1_average`, so they carry no `gpu_power_watts` line at all (only NVIDIA and
the Dankclaw 8060S iGPU report power); and it binds loopback + the host LAN, like
node-exporter, never the wildcard. The dashboard graphs util/VRAM/VRAM-%/temp/power.
* **Logs: fixed in code, awaiting deploy.** The pipeline shipped nothing because
a rootless Alloy cannot read the `0640` journal files, and the container-log
globs point at a root-storage layout that does not exist here. The fix is built
and tested in-repo (2026-10-20): Alloy is now a **root system service on
`Network=host`**, reading `system.journal` + `user-*.journal` and pushing to
Loki's `127.0.0.1:3100` loopback publish — a throwaway root probe read 31,825
lines and pushed 26,966 with zero drops. The **live** instance is still the old
rootless unit until `deploy.sh dankclaw --service alloy` runs. Detail in
[[Alloy]] → Why root; Loki's store state in [[Loki]] → Contents.
* **Not configured:** no blackbox exporter (the structural downs have no real
uptime signal) and no Alertmanager vhost. The log pipeline's fix is in code but
not yet deployed (see *Logs* above).
* **Logs: flowing; container labels land with the 2026-10-21 deploy.** The
pipeline once shipped nothing (rootless Alloy cannot read the `0640` journal;
the container-log globs pointed at a root-storage layout that does not exist
here) — fixed 2026-10-20 by making Alloy a **root system service on
`Network=host`**. Journal lines now reach Loki (measured ~6.5k lines/5min),
but the stream carried only `job`/`service_name`, so every per-container
view was structurally empty: Loki metrics queries group only by INDEXED
stream labels. Alloy now promotes podman's `CONTAINER_NAME` to a `container`
label (plus a `host` label) at ingestion — `loki.process` in
`shared/alloy/config.alloy`; detail in [[Alloy]] → Why root, and the
purpose-test law in [[Observability Provisioning]].
## Citations
- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter,cadvisor}.container`
- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter,cadvisor,blackbox-exporter}.container`
- `scripts/deploy-remote.sh:219-300` (copiers + relay), `:521-536` (health waits), `:477-486` (grafana env seed)
- `scripts/signal-relay.py:26-49`, `scripts/signal-relay.service`
- `scripts/gpu-exporter.py`, `scripts/gpu-exporter.service` (per-GPU host exporter)
- `shared/prometheus/prometheus.yml`, `shared/prometheus/alertmanager.yml`, `shared/prometheus/rules/llm-fleet.yml`
- `shared/grafana/provisioning/`, `shared/loki/loki-config.yaml`, `shared/alloy/config.alloy`
- `shared/blackbox/blackbox.yml` (prober modules + loopback bind)
- `tests/observability_quadlet.bats` (boot persistence, no wildcard publish, probe addresses, delivery contract)
- Host evidence 2026-10-18: `ss -ltnp`, `podman ps`, `systemctl --user is-enabled|is-active`, `curl` on `127.0.0.1:9090/api/v1/{targets,rules,alerts}`, `127.0.0.1:3100/loki/api/v1/labels`, `127.0.0.1:12345/metrics`, `127.0.0.1:3002/api/health`, `192.168.0.143:18093/health`, `journalctl --user -u signal-relay`
serverAppConfigs-docs/Infrastructure/Prometheus.md
+14 -16
View File
@@ -42,29 +42,27 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
The `frank-medium` job is **retired**: no `frank-medium` job exists in the config or in `/api/v1/targets` (2026-10-18), and the endpoint itself left the fleet — its history is on [[frank-archon]], which holds the port today.
**Three jobs read down today, and they are not outages** (re-audited against `curl -s http://127.0.0.1:9090/api/v1/targets`, 2026-10-18; first audited 2026-09-16):
**No job reads down structurally any more** (the three permanent-reds were
retired 2026-10-21; census re-run after each deploy):
| Job | Target | `health` | `lastError`, verbatim |
|-----|--------|----------|------------------------|
| `gitea` | `http://192.168.0.143:3000/metrics` | down | `server returned HTTP status 404 Not Found` |
| `comfyui` | `http://comfyui:8188/` | down | `received unsupported Content-Type "text/html" and no fallback_scrape_protocol specified for target` |
| `services` | `http://192.168.0.143:8081/` | down | `received unsupported Content-Type "text/html; charset=utf-8" and no fallback_scrape_protocol specified for target` |
| `services` | `http://192.168.0.143:3005/` | down | same `unsupported Content-Type "text/html; charset=utf-8"` error |
| Job | Target | how |
|---|---|---|
| `gitea` | `192.168.0.143:3000/metrics` | metrics enabled in the quadlet (`GITEA__metrics__ENABLED=true`, `ENABLED_HOST=0.0.0.0`) — the page once said "stays off until `[metrics] ENABLED = true` in its own config"; the env form is what the deployer ships |
| `services` | `http://192.168.0.143:8081` (SearXNG), `:3005` (OpenWebUI), `http://comfyui:8188` | **blackbox exporter** probe, module `http_2xx` — the comfyui metrics job is retired, one owner per service |
Gitea's exporter stays off until `[metrics] ENABLED = true` in its own config (no metrics env is set in the container, and `curl -s http://127.0.0.1:3000/metrics` → 404, 2026-10-18); ComfyUI and the two `services` probes are HTML pages, which Prometheus 3 refuses as metrics. Everything else is green: `prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`, `frank-zealot`, `frank-dragoon`, `frank-archon`.
Scraping a web root is not a health check: an HTML page fails the scrape
protocol (`unsupported Content-Type`), which is what kept those targets red
beside healthy services. The classification rule and the honesty law live in
[[Stale Signal Audit]]. The prober is `blackbox-exporter.container` (host
netns, config `shared/blackbox/blackbox.yml`, bound 127.0.0.1:9115; Prometheus
reaches it at the `host.containers.internal` gateway name). Plain targets stay
`host:port`, never URLs — the probe job's URL targets are the documented
exception, relabeled onto the prober.
**`frank-zealot` reads up since 2026-10-18, and did not before.** llama.cpp only answers `/metrics` when started with `--metrics`, so `up{job="frank-zealot"}` was 0 across the whole retained history (the oldest sample, 2026-10-11, is already 0) while the service answered `/health` 200 the whole time; the flag and its opt-out are on [[frank-zealot]]. Read `lastError` before believing an outage: a target can read down beside a healthy service.
Alert state is a live property, not a page fact: `curl -s http://127.0.0.1:9090/api/v1/alerts` returned no alerts at the 2026-10-18 morning census, one firing `LlmPrefixCacheCollapse` (info, `frank-zealot`, `activeAt` 11:10:48Z) minutes later, and none again by 11:56Z.
The remedy for a probe-shaped question is the **blackbox exporter**; the
classification rule and the honesty law for it live in [[Stale Signal Audit]].
Until the exporter is deployed, these three stay red for a structural reason:
deleting them would
hide a real service, and fixing them by publishing databases or enabling metrics
would trade a wrong signal for a real exposure. Targets are `host:port`, never
URLs.
## Volumes
| Container path | Host path |