docs+units: fix every claim my own changes made false
Retiring dankclaw/compose.yaml and adding alert rules invalidated claims across
the repo, including in the rule file agents read first. Clean cutover includes
the prose.
**Tests that had gone hollow.** Four assertions grepped dankclaw/compose.yaml
for a service name. With the file gone they passed by *absence* — a gate that
can no longer fail. Each now asserts the real invariant ("no dankclaw compose
file competes with the quadlet units") and keeps a live check for a re-added
entry, so it still fails when one appears: verified by recreating a compose file
with a comfyui service and watching three tests go red, then removing it.
**AGENTS.md, which was wrong about ComfyUI.** It said ComfyUI runs CPU-only
("no ROCm wheels for gfx1151, CUDA images wrong"). That was true in the
compose era and is false now: the restored unit runs with /dev/kfd + /dev/dri and
must NOT pass --cpu. An agent trusting the rule file would have re-broken it.
Also corrected: the observability paragraph ("RUNNING BUT UNCONFIGURED, no
datasource, no rules" — all three have been configured since 2026-09-16), the
Active runbook row that still named halogen for frank-large, halogen's absence
from the Retired table, the benchmark policy naming halogen's harness, and the
seam list.
**Unit and page claims.** prometheus.container said "no alerting rules anywhere
in this stack"; node-exporter.container said Dankserver runs from its compose
stack; Prometheus.md documented three jobs as working (they scrape HTML: they
can never report up) and said no dashboard existed; index.md and the three
authentik service pages pointed at the retired file; Authentik.md's upgrade
instruction told you to bump an image tag in it.
One bad edit of my own on the way (a stale line number clobbered a See Also
entry in Prometheus.md) — caught by reading back, fixed, and the vault link
checker now proves the cross-references hold.
This commit is contained in:
1 parent
2e33ec36c1
commit
f134cc8876
12 files changed
+95
-43
No files matched your search
@@ -259,8 +259,9 @@ Two secrets live in the host `.env` (git-ignored; placeholders in
|
||||
- **Logs:** `podman logs authentik-server` / `podman logs authentik-worker`.
|
||||
- **Backups:** `pg_dump` the `authentik-postgres` database (the single source of
|
||||
truth). The `/home/dank/Containers/authentik/data` dir holds media + static files.
|
||||
- **Upgrades:** bump the image tag in `dankclaw/compose.yaml`, run
|
||||
`./scripts/deploy.sh dankclaw`. Authentik migrates the schema on startup.
|
||||
- **Upgrades:** bump the image tag in the three units under
|
||||
`dankclaw/authentik/quadlet/`, run `./scripts/deploy.sh dankclaw`. Authentik
|
||||
migrates the schema on startup.
|
||||
|
||||
## Outpost claim cache (`proxysession`) — 403 after a group add
|
||||
|
||||
|
||||
@@ -16,7 +16,7 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
|
||||
| Config | `/opt/prometheus/config/prometheus.yml`, installed from `shared/prometheus/prometheus.yml` on every deploy |
|
||||
| Data | `/opt/prometheus/data` (TSDB) |
|
||||
| Retention | `--storage.tsdb.retention.time=30d` |
|
||||
| Alerting | **none** — no rule files, no Alertmanager anywhere in the stack |
|
||||
| Alerting | 5 alert rules + 1 recording rule (`shared/prometheus/rules/llm-fleet.yml`). Evaluated, **not delivered**: there is no Alertmanager (see [[Observability]]) |
|
||||
|
||||
## Targets
|
||||
|
||||
@@ -26,12 +26,25 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
|
||||
|-----|--------|-----|
|
||||
| `prometheus` | `localhost:9090` | itself |
|
||||
| `node-dankclaw`, `node-dankserver` | `192.168.0.143:9100`, `192.168.0.235:9100` | host metrics from each machine's [[Node-Exporter]] |
|
||||
| `gitea` | `192.168.0.143:3000` | Gitea's built-in exporter, reached on its published LAN port |
|
||||
| `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on the `dankclaw_default` network |
|
||||
| `comfyui` | `comfyui:8188` | UI probe; ComfyUI has no native `/metrics` |
|
||||
| `services` | `192.168.0.143:8081`, `:3005` | up/down probes for SearXNG and OpenWebUI |
|
||||
| `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on the `dankclaw_default` network; both serve real `/metrics` |
|
||||
| `frank-medium`, `frank-large` | `192.168.0.235:7770`, `192.168.0.143:7770` | the LLM fleet — `vllm:` and `llamacpp:` series (LAN addresses, not the guarded vhosts) |
|
||||
|
||||
Targets are `host:port`, never URLs. For richer HTTP assertions, add the Prometheus **blackbox** exporter in probe mode rather than scraping HTML.
|
||||
**Three jobs read down, and they are not outages** (audited 2026-09-16):
|
||||
|
||||
| Job | Target | Why it never reports up |
|
||||
|-----|--------|-------------------------|
|
||||
| `comfyui` | `comfyui:8188` | scrapes `/`, which is HTML — "unsupported Content-Type" |
|
||||
| `services` | `192.168.0.143:8081`, `:3005` | same: up/down intent, implemented as a scrape |
|
||||
| `gitea` | `192.168.0.143:3000/metrics` | Gitea's exporter is off until `[metrics] ENABLED = true` |
|
||||
|
||||
A scrape cannot express "is this HTTP service answering?" — that is what a
|
||||
probe is for, and the remedy is the **blackbox exporter** in probe mode
|
||||
(`/probe?target=…&module=http_2xx`), which also gives the guarded vhosts a real
|
||||
uptime signal since Prometheus cannot solve an Anubis challenge itself. Until it
|
||||
is deployed, these three stay red for a structural reason: deleting them would
|
||||
hide a real service, and fixing them by publishing databases or enabling metrics
|
||||
would trade a wrong signal for a real exposure. Targets are `host:port`, never
|
||||
URLs.
|
||||
|
||||
## Volumes
|
||||
|
||||
@@ -54,6 +67,6 @@ Prometheus responds on `/-/healthy` with 200; the unit's health command is the s
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Grafana]] — the dashboard frontend (no datasource configured yet)
|
||||
- [[Grafana]] — the dashboard frontend (datasource and LLM dashboard provisioned as code since 2026-09-16)
|
||||
- [[Node-Exporter]] — the host metrics source
|
||||
- [[Observability]] — the whole stack, including what is still unconfigured
|
||||
@@ -5,7 +5,7 @@ This is the service the Caddy `forward_auth` vhosts proxy to
|
||||
(`authentik-server:9000`). It also serves the login page at
|
||||
`https://auth.franksdank.cloud`.
|
||||
|
||||
- **Compose service:** `authentik-server` (in `dankclaw/compose.yaml`)
|
||||
- **Unit:** `dankclaw/authentik/quadlet/authentik-server.container` (`authentik-server.service`)
|
||||
- **Image:** `ghcr.io/goauthentik/server:2026.8.0` (command: `server`)
|
||||
- **Port:** `9000` (published; the outpost + admin + login all live here)
|
||||
- **Data:** `/home/dank/authentik/data` (bind mount, shared with the worker)
|
||||
|
||||
@@ -4,7 +4,7 @@ The Authentik worker — background tasks: event processing, provider
|
||||
synchronization, scheduled flows, and certificate renewal. It runs the same
|
||||
image as the server; the `command: worker` selects the role.
|
||||
|
||||
- **Compose service:** `authentik-worker` (in `dankclaw/compose.yaml`)
|
||||
- **Unit:** `dankclaw/authentik/quadlet/authentik-worker.container` (`authentik-worker.service`)
|
||||
- **Image:** `ghcr.io/goauthentik/server:2026.8.0` (command: `worker`)
|
||||
- **Runs as:** `0:0` (rootless: container root maps to the host user that owns the data dir)
|
||||
- **Data:** `/home/dank/authentik/data` (bind mount, shared with the server)
|
||||
|
||||
Reference in new issue
Block a user