dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs+units: fix every claim my own changes made false

Browse Source 
Retiring dankclaw/compose.yaml and adding alert rules invalidated claims across
the repo, including in the rule file agents read first. Clean cutover includes
the prose.

**Tests that had gone hollow.** Four assertions grepped dankclaw/compose.yaml
for a service name. With the file gone they passed by *absence* — a gate that
can no longer fail. Each now asserts the real invariant ("no dankclaw compose
file competes with the quadlet units") and keeps a live check for a re-added
entry, so it still fails when one appears: verified by recreating a compose file
with a comfyui service and watching three tests go red, then removing it.

**AGENTS.md, which was wrong about ComfyUI.** It said ComfyUI runs CPU-only
("no ROCm wheels for gfx1151, CUDA images wrong"). That was true in the
compose era and is false now: the restored unit runs with /dev/kfd + /dev/dri and
must NOT pass --cpu. An agent trusting the rule file would have re-broken it.
Also corrected: the observability paragraph ("RUNNING BUT UNCONFIGURED, no
datasource, no rules" — all three have been configured since 2026-09-16), the
Active runbook row that still named halogen for frank-large, halogen's absence
from the Retired table, the benchmark policy naming halogen's harness, and the
seam list.

**Unit and page claims.** prometheus.container said "no alerting rules anywhere
in this stack"; node-exporter.container said Dankserver runs from its compose
stack; Prometheus.md documented three jobs as working (they scrape HTML: they
can never report up) and said no dashboard existed; index.md and the three
authentik service pages pointed at the retired file; Authentik.md's upgrade
instruction told you to bump an image tag in it.

One bad edit of my own on the way (a stale line number clobbered a See Also
entry in Prometheus.md) — caught by reading back, fixed, and the vault link
checker now proves the cross-references hold.
This commit is contained in:
Frank
2026-09-16 07:46:23 -07:00
parent 2e33ec36c1
commit f134cc8876
12 changed files with 95 additions and 43 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/Infrastructure/Prometheus.md
+20 -7
View File
@@ -16,7 +16,7 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
| Config | `/opt/prometheus/config/prometheus.yml`, installed from `shared/prometheus/prometheus.yml` on every deploy |
| Data | `/opt/prometheus/data` (TSDB) |
| Retention | `--storage.tsdb.retention.time=30d` |
| Alerting | **none** — no rule files, no Alertmanager anywhere in the stack |
| Alerting | 5 alert rules + 1 recording rule (`shared/prometheus/rules/llm-fleet.yml`). Evaluated, **not delivered**: there is no Alertmanager (see [[Observability]]) |
## Targets
@@ -26,12 +26,25 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
|-----|--------|-----|
| `prometheus` | `localhost:9090` | itself |
| `node-dankclaw`, `node-dankserver` | `192.168.0.143:9100`, `192.168.0.235:9100` | host metrics from each machine's [[Node-Exporter]] |
| `gitea` | `192.168.0.143:3000` | Gitea's built-in exporter, reached on its published LAN port |
| `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on the `dankclaw_default` network |
| `comfyui` | `comfyui:8188` | UI probe; ComfyUI has no native `/metrics` |
| `services` | `192.168.0.143:8081`, `:3005` | up/down probes for SearXNG and OpenWebUI |
| `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on the `dankclaw_default` network; both serve real `/metrics` |
| `frank-medium`, `frank-large` | `192.168.0.235:7770`, `192.168.0.143:7770` | the LLM fleet — `vllm:` and `llamacpp:` series (LAN addresses, not the guarded vhosts) |
Targets are `host:port`, never URLs. For richer HTTP assertions, add the Prometheus **blackbox** exporter in probe mode rather than scraping HTML.
**Three jobs read down, and they are not outages** (audited 2026-09-16):
| Job | Target | Why it never reports up |
|-----|--------|-------------------------|
| `comfyui` | `comfyui:8188` | scrapes `/`, which is HTML — "unsupported Content-Type" |
| `services` | `192.168.0.143:8081`, `:3005` | same: up/down intent, implemented as a scrape |
| `gitea` | `192.168.0.143:3000/metrics` | Gitea's exporter is off until `[metrics] ENABLED = true` |
A scrape cannot express "is this HTTP service answering?" — that is what a
probe is for, and the remedy is the **blackbox exporter** in probe mode
(`/probe?target=…&module=http_2xx`), which also gives the guarded vhosts a real
uptime signal since Prometheus cannot solve an Anubis challenge itself. Until it
is deployed, these three stay red for a structural reason: deleting them would
hide a real service, and fixing them by publishing databases or enabling metrics
would trade a wrong signal for a real exposure. Targets are `host:port`, never
URLs.
## Volumes
@@ -54,6 +67,6 @@ Prometheus responds on `/-/healthy` with 200; the unit's health command is the s
## See Also
- [[Grafana]] — the dashboard frontend (no datasource configured yet)
- [[Grafana]] — the dashboard frontend (datasource and LLM dashboard provisioned as code since 2026-09-16)
- [[Node-Exporter]] — the host metrics source
- [[Observability]] — the whole stack, including what is still unconfigured