diff --git a/AGENTS.md b/AGENTS.md index 447038d..b76af0a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -119,9 +119,14 @@ This rule exists because reboots found gaps twice (2026-08-26: Caddy as a bare compose container never returned; 2026-09-11: five observability services had no units). The observability services now have units under `dankclaw/observability/quadlet/` and their compose definitions were removed — -one owner per container name. The stack is RUNNING BUT UNCONFIGURED (no -Grafana datasources/dashboards, no Prometheus rules); see -`serverAppConfigs-docs/Infrastructure/Observability.md`. +one owner per container name. The stack is **configured as of 2026-09-16**: a +Grafana datasource and the `llm-fleet` dashboard are provisioned from +`shared/grafana/provisioning/`, and `shared/prometheus/rules/llm-fleet.yml` +loads 5 alert rules + 1 recording rule. Nothing is *delivered* — there is no +Alertmanager, so a firing alert reaches nobody (an operator decision, not an +oversight). Five Prometheus targets read down for structural reasons +(HTML-scraped "probes", Gitea metrics off); the remedy is a blackbox exporter. +See `serverAppConfigs-docs/Infrastructure/Observability.md`. ## Testing @@ -132,8 +137,11 @@ Tests are mandatory before commit (`bats` required on the dev machine): ``` Seams: compose validity, env consistency (no leaked secrets), Caddyfile -validity, quadlet units, docs completeness, script syntax, LLM unit -integrity, LLM secret hygiene, deployment flow, service-scoped planning. +validity, quadlet units, docs completeness **including vault link integrity**, +script syntax (shell + python), LLM unit integrity, LLM secret hygiene, +deployment flow, service-scoped planning, homepage config/pages/deploy/checker, +and `e2e_scaffold` — which fails if a Playwright spec file's number prefix is in +no project, because such a spec silently never runs. See `serverAppConfigs-docs/Architecture/Testing Policy.md`. ## Secret Rules @@ -186,8 +194,9 @@ archive. **STOP the production service BEFORE benchmarking** — these servers serve real users, and a bench beside a live service starves the GPU and corrupts results. Use the runtime's own harness: `llama-bench` (llama.cpp), `vllm -bench` (vLLM), halogen's `bench`/`sweep` modes for frank-large. Rolling your -own bench requires asking first. Scratch ports are not used on LLM hosts — a +bench` (vLLM), llama-bench for ik_llama.cpp (the halogen `bench`/`sweep` modes +belong to a retired runtime). Rolling your own bench requires asking first. +Scratch ports are not used on LLM hosts — a test process cannot share the single GPU anyway; validate against the service itself. @@ -219,8 +228,7 @@ When renaming a service, retiring an endpoint, or moving a model: update the provider block, model `id`, `contextWindow`, `maxTokens`, `input` in `models.yml`, then fix every `modelRoles` entry that named the old endpoint — a role pointing at a removed provider fails at call time. `frank-large` -declares `input: [text, image]` (halogen vision tower). There is **no audio -input anywhere in the fleet** (left with `frank-npu`). +declares `input: [text, image]` (its mmproj vision tower). There is **no audio ## Active Services @@ -377,8 +385,12 @@ linked runbooks. - Both hosts allow passwordless sudo for the deploy user. Use plain `sudo` in committed scripts (no `echo pw | sudo -S`); `FRANK_SUDO_PW` covers hosts without it. -- ComfyUI on Dankclaw runs CPU-only (`--cpu`): no ROCm wheels for gfx1151, - and CUDA images are wrong for the hardware. +- ComfyUI on Dankclaw runs with **GPU access** (`/dev/kfd` + `/dev/dri` via + `AddDevice`, `GroupAdd=keep-groups`, plus the udev rule that makes those nodes + world-writable — user-unit spawns carry no supplementary groups). It must NOT + pass `--cpu`: the gfx1151 ROCm wheels exist and the unit's `--cpu` era ended + 2026-09-16, when a stale device-less container was found restarted 9,146 + times (`RuntimeError: No HIP GPUs are available`). - SearXNG outbound traffic must stay on the Gluetun tunnel; do not move its ports off the gluetun service. - `amd_iommu=off` on Dankclaw (bought 13–16 % prefill for frank-large, @@ -396,7 +408,7 @@ Active: | Service | Runbook | |---------|---------| -| frank-large (halogen-flash-server, Dankclaw) | `serverAppConfigs-docs/LLM/halogen-flash-server.md` | +| frank-large (ik_llama.cpp, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-ik-uncensored.md` | | frank-medium (vllm-radiance MXFP4, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` | | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` | | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` | @@ -406,6 +418,7 @@ Retired (rollback targets and history): | Runtime | Host | Window | Runbook | |---------|------|--------|---------| | kingjones STRIX Qwen3.8-Flash-Next ROCmFP4 | Dankclaw | 2026-08-30 → 09-11 | `kingjones-strix-rocmfp4.md` | +| halogen-flash-server (`frank-large`) | Dankclaw | → 2026-09-15 | `halogen-flash-server.md` | | agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` | | eemin Qwen3.8-27B ROCmI4 W4A4 MTP | Dankclaw | 2026-08-18 → 08-25 | `eemin-rocmi4-mtp.md` | | Kairic Edge IU4 | Dankclaw | 2026-08-22 (same-day revert) | `kairic-edge.md` | diff --git a/dankclaw/observability/quadlet/node-exporter.container b/dankclaw/observability/quadlet/node-exporter.container index 7abdc84..c4cb04c 100644 --- a/dankclaw/observability/quadlet/node-exporter.container +++ b/dankclaw/observability/quadlet/node-exporter.container @@ -19,7 +19,9 @@ # the wildcard bind off the table matters: this is the second half of the # pairing that removes the unauthenticated observability surface, and both # consumers are accounted for. Dankserver runs its own copy of this container -# (scraped by the "node-dankserver" job) from its own compose stack. +# (scraped by the "node-dankserver" job) — since 2026-09-16 as a Quadlet unit in +# the same shape (dankserver/observability/quadlet/), because the compose-only +# version it replaced had been dead for a day and a half before anyone noticed. # # MOUNTS. The container reads the host's /proc, /sys and / as /host/* read-only # and is pointed at them with --path.*; it writes nothing anywhere. diff --git a/dankclaw/observability/quadlet/prometheus.container b/dankclaw/observability/quadlet/prometheus.container index 52112f5..b6ee694 100644 --- a/dankclaw/observability/quadlet/prometheus.container +++ b/dankclaw/observability/quadlet/prometheus.container @@ -25,8 +25,15 @@ # namespace and dank owns /opt/prometheus/data. The image's default user # (nobody) maps to a subuid that cannot write it. # -# There are no alerting rules and no Alertmanager anywhere in this stack: the -# scrape config is the whole of its configuration. +# RULES. Five LLM alert rules and one recording rule load from +# /etc/prometheus/rules/ (installed from shared/prometheus/rules/ by the +# deployer). There is still **no Alertmanager**, so nothing is delivered: a +# firing alert is visible in Prometheus and on the Grafana dashboard's ALERTS +# panel and reaches nobody. Delivery is an operator decision, not an oversight; +# the fragility is that an undelivered alert looks identical to a quiet one. +# +# The config volume is also why rules belong in git: editing them in the UI is +# lost on the next deploy, since the deployer overwrites the whole directory. [Unit] Description=Prometheus metrics store (Dankclaw) diff --git a/serverAppConfigs-docs/Infrastructure/Authentik.md b/serverAppConfigs-docs/Infrastructure/Authentik.md index 892cae6..d3fb01a 100644 --- a/serverAppConfigs-docs/Infrastructure/Authentik.md +++ b/serverAppConfigs-docs/Infrastructure/Authentik.md @@ -259,8 +259,9 @@ Two secrets live in the host `.env` (git-ignored; placeholders in - **Logs:** `podman logs authentik-server` / `podman logs authentik-worker`. - **Backups:** `pg_dump` the `authentik-postgres` database (the single source of truth). The `/home/dank/Containers/authentik/data` dir holds media + static files. -- **Upgrades:** bump the image tag in `dankclaw/compose.yaml`, run - `./scripts/deploy.sh dankclaw`. Authentik migrates the schema on startup. +- **Upgrades:** bump the image tag in the three units under + `dankclaw/authentik/quadlet/`, run `./scripts/deploy.sh dankclaw`. Authentik + migrates the schema on startup. ## Outpost claim cache (`proxysession`) — 403 after a group add diff --git a/serverAppConfigs-docs/Infrastructure/Prometheus.md b/serverAppConfigs-docs/Infrastructure/Prometheus.md index f01366e..c58e2ed 100644 --- a/serverAppConfigs-docs/Infrastructure/Prometheus.md +++ b/serverAppConfigs-docs/Infrastructure/Prometheus.md @@ -16,7 +16,7 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) — | Config | `/opt/prometheus/config/prometheus.yml`, installed from `shared/prometheus/prometheus.yml` on every deploy | | Data | `/opt/prometheus/data` (TSDB) | | Retention | `--storage.tsdb.retention.time=30d` | -| Alerting | **none** — no rule files, no Alertmanager anywhere in the stack | +| Alerting | 5 alert rules + 1 recording rule (`shared/prometheus/rules/llm-fleet.yml`). Evaluated, **not delivered**: there is no Alertmanager (see [[Observability]]) | ## Targets @@ -26,12 +26,25 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) — |-----|--------|-----| | `prometheus` | `localhost:9090` | itself | | `node-dankclaw`, `node-dankserver` | `192.168.0.143:9100`, `192.168.0.235:9100` | host metrics from each machine's [[Node-Exporter]] | -| `gitea` | `192.168.0.143:3000` | Gitea's built-in exporter, reached on its published LAN port | -| `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on the `dankclaw_default` network | -| `comfyui` | `comfyui:8188` | UI probe; ComfyUI has no native `/metrics` | -| `services` | `192.168.0.143:8081`, `:3005` | up/down probes for SearXNG and OpenWebUI | +| `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on the `dankclaw_default` network; both serve real `/metrics` | +| `frank-medium`, `frank-large` | `192.168.0.235:7770`, `192.168.0.143:7770` | the LLM fleet — `vllm:` and `llamacpp:` series (LAN addresses, not the guarded vhosts) | -Targets are `host:port`, never URLs. For richer HTTP assertions, add the Prometheus **blackbox** exporter in probe mode rather than scraping HTML. +**Three jobs read down, and they are not outages** (audited 2026-09-16): + +| Job | Target | Why it never reports up | +|-----|--------|-------------------------| +| `comfyui` | `comfyui:8188` | scrapes `/`, which is HTML — "unsupported Content-Type" | +| `services` | `192.168.0.143:8081`, `:3005` | same: up/down intent, implemented as a scrape | +| `gitea` | `192.168.0.143:3000/metrics` | Gitea's exporter is off until `[metrics] ENABLED = true` | + +A scrape cannot express "is this HTTP service answering?" — that is what a +probe is for, and the remedy is the **blackbox exporter** in probe mode +(`/probe?target=…&module=http_2xx`), which also gives the guarded vhosts a real +uptime signal since Prometheus cannot solve an Anubis challenge itself. Until it +is deployed, these three stay red for a structural reason: deleting them would +hide a real service, and fixing them by publishing databases or enabling metrics +would trade a wrong signal for a real exposure. Targets are `host:port`, never +URLs. ## Volumes @@ -54,6 +67,6 @@ Prometheus responds on `/-/healthy` with 200; the unit's health command is the s ## See Also -- [[Grafana]] — the dashboard frontend (no datasource configured yet) +- [[Grafana]] — the dashboard frontend (datasource and LLM dashboard provisioned as code since 2026-09-16) - [[Node-Exporter]] — the host metrics source - [[Observability]] — the whole stack, including what is still unconfigured diff --git a/serverAppConfigs-docs/Infrastructure/authentik-server.md b/serverAppConfigs-docs/Infrastructure/authentik-server.md index 8fa9814..3cabeba 100644 --- a/serverAppConfigs-docs/Infrastructure/authentik-server.md +++ b/serverAppConfigs-docs/Infrastructure/authentik-server.md @@ -5,7 +5,7 @@ This is the service the Caddy `forward_auth` vhosts proxy to (`authentik-server:9000`). It also serves the login page at `https://auth.franksdank.cloud`. -- **Compose service:** `authentik-server` (in `dankclaw/compose.yaml`) +- **Unit:** `dankclaw/authentik/quadlet/authentik-server.container` (`authentik-server.service`) - **Image:** `ghcr.io/goauthentik/server:2026.8.0` (command: `server`) - **Port:** `9000` (published; the outpost + admin + login all live here) - **Data:** `/home/dank/authentik/data` (bind mount, shared with the worker) diff --git a/serverAppConfigs-docs/Infrastructure/authentik-worker.md b/serverAppConfigs-docs/Infrastructure/authentik-worker.md index faef2ea..cdceb3f 100644 --- a/serverAppConfigs-docs/Infrastructure/authentik-worker.md +++ b/serverAppConfigs-docs/Infrastructure/authentik-worker.md @@ -4,7 +4,7 @@ The Authentik worker — background tasks: event processing, provider synchronization, scheduled flows, and certificate renewal. It runs the same image as the server; the `command: worker` selects the role. -- **Compose service:** `authentik-worker` (in `dankclaw/compose.yaml`) +- **Unit:** `dankclaw/authentik/quadlet/authentik-worker.container` (`authentik-worker.service`) - **Image:** `ghcr.io/goauthentik/server:2026.8.0` (command: `worker`) - **Runs as:** `0:0` (rootless: container root maps to the host user that owns the data dir) - **Data:** `/home/dank/authentik/data` (bind mount, shared with the server) diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index 91958a7..6767800 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -85,9 +85,9 @@ covers both workloads). - [[authentik-worker]] — Authentik background worker service - [[Homepage]] — password-protected service index - [[Homepage Design Grilling]] — homepage decision log and open questions -- [[Observability]] — Grafana, Prometheus, Loki, Alloy, node-exporter: the stack, its quadlet deployment, its exposure, and **what is still unconfigured** -- [[Grafana]] — dashboards (no datasource or dashboard configured yet) -- [[Prometheus]] — metrics collection (no alerting rules or Alertmanager) +- [[Observability]] — Grafana, Prometheus, Loki, Alloy, node-exporter: the stack, its quadlet deployment, its exposure, and what is still missing (alert delivery) +- [[Grafana]] — dashboards (datasource and the LLM fleet dashboard provisioned as code since 2026-09-16) +- [[Prometheus]] — metrics collection (5 alert rules + 1 recording rule; evaluated but not delivered — no Alertmanager) - [[Node-Exporter]] — host metrics - [[Loki]] — log storage - [[Alloy]] — log shipping (replaces Promtail) diff --git a/tests/authentik.bats b/tests/authentik.bats index 9681d23..b1ca883 100644 --- a/tests/authentik.bats +++ b/tests/authentik.bats @@ -18,7 +18,8 @@ load helpers -COMPOSE="$REPO_ROOT/dankclaw/compose.yaml" +# (no COMPOSE here: dankclaw/compose.yaml is retired — every container on that +# host is unit-managed, and this seam reads the units.) CADDYFILE="$REPO_ROOT/dankclaw/Caddyfile" ENV_EXAMPLE="$REPO_ROOT/dankclaw/.env.example" diff --git a/tests/caddyfile.bats b/tests/caddyfile.bats index 9040a32..13a7315 100644 --- a/tests/caddyfile.bats +++ b/tests/caddyfile.bats @@ -78,8 +78,11 @@ CADDYFILE="$REPO_ROOT/dankclaw/Caddyfile" @test "compose no longer defines the quadlet-replaced services" { # gitea, its postgres, open-webui, and caddy run as boot-persistent # quadlets on dankclaw (see dankclaw/gitea/quadlet, - # dankclaw/open-webui/quadlet, dankclaw/caddy/quadlet). - ! grep -qE '^\s+(gitea|postgres|openwebui|caddy):\s*$' "$REPO_ROOT/dankclaw/compose.yaml" + # dankclaw/open-webui/quadlet, dankclaw/caddy/quadlet). The compose file + # itself is retired and observability_quadlet.bats asserts that; here the + # regression is a re-added entry, so the check must not pass by absence. + local cf="$REPO_ROOT/dankclaw/compose.yaml" + [ ! -f "$cf" ] || ! grep -qE '^\s+(gitea|postgres|openwebui|caddy):\s*$' "$cf" } @test "the caddy quadlet healthcheck probes the bare-IP catch-all" { @@ -174,9 +177,12 @@ CADDYFILE="$REPO_ROOT/dankclaw/Caddyfile" # --- Observability configs (deployed by deploy.sh step 3) --- -@test "no legacy .local service names remain in shared configs or compose" { +@test "no legacy .local service names remain in deployed config" { + # Scans the surfaces that are actually deployed — shared configs and every + # dankclaw unit — rather than a file that no longer exists. A missing path + # makes grep exit 2 and the negation pass, which is how this went hollow. ! grep -rn 'dankclaw\.local\|dankserver\.local' \ - "$REPO_ROOT/shared" "$REPO_ROOT/dankclaw/compose.yaml" + "$REPO_ROOT/shared" "$REPO_ROOT/dankclaw" } @test "loki compactor has a delete_request_store (required when retention is on)" { diff --git a/tests/comfyui_rocm.bats b/tests/comfyui_rocm.bats index f74cfcd..f68a296 100644 --- a/tests/comfyui_rocm.bats +++ b/tests/comfyui_rocm.bats @@ -32,8 +32,12 @@ load helpers grep -q '^Image=localhost/comfyui-rocm:gfx1151$' "$unit" \ || { echo "unit must run the locally built image"; return 1; } # One owner per container name: a surviving compose service would fight the - # unit for the container, which is how the stale one stayed alive. - ! grep -qE '^\s+comfyui:' "$REPO_ROOT/dankclaw/compose.yaml" + # unit for the container, which is how the stale one stayed alive. The file + # itself is retired (observability_quadlet.bats asserts that), so here the + # regression to catch is a re-added entry — the check must not go hollow + # just because the file is absent. + local cf="$REPO_ROOT/dankclaw/compose.yaml" + [ ! -f "$cf" ] || ! grep -qE '^\s+comfyui:' "$cf" } @test "comfyui Containerfile is based on the gfx1151 torch image" { diff --git a/tests/observability_quadlet.bats b/tests/observability_quadlet.bats index 55ea31d..6654b55 100755 --- a/tests/observability_quadlet.bats +++ b/tests/observability_quadlet.bats @@ -263,14 +263,19 @@ PAIRS grep -q 'build_compose_image alloy' "$REMOTE_DEPLOY" } -@test "the five are no longer compose services" { +@test "no dankclaw compose file competes with the quadlet units" { # Two owners for one container name is how a compose run resurrects a - # stale duplicate next to the quadlet-managed one. - local svc - for svc in "${UNITS[@]}"; do - ! services_in "$REPO_ROOT/dankclaw/compose.yaml" | grep -qx "$svc" \ - || { echo "$svc is still a compose service"; return 1; } - done + # stale duplicate next to the quadlet-managed one. This file was retired on + # 2026-09-16 — every container on the host reports a PODMAN_SYSTEMD_UNIT + # label — and the invariant is that it stays retired: the four assertions + # that used to grep it passed against the missing file, which is how a gate + # goes hollow. + local cf="$REPO_ROOT/dankclaw/compose.yaml" + [ ! -f "$cf" ] || { + echo "$cf is back: every dankclaw container is unit-managed, so a" + echo "compose path can only spawn a duplicate fighting a unit for a name" + return 1 + } } @test "changes under dankclaw/observability/ run this seam" {