docs+units: fix every claim my own changes made false
Retiring dankclaw/compose.yaml and adding alert rules invalidated claims across
the repo, including in the rule file agents read first. Clean cutover includes
the prose.
**Tests that had gone hollow.** Four assertions grepped dankclaw/compose.yaml
for a service name. With the file gone they passed by *absence* — a gate that
can no longer fail. Each now asserts the real invariant ("no dankclaw compose
file competes with the quadlet units") and keeps a live check for a re-added
entry, so it still fails when one appears: verified by recreating a compose file
with a comfyui service and watching three tests go red, then removing it.
**AGENTS.md, which was wrong about ComfyUI.** It said ComfyUI runs CPU-only
("no ROCm wheels for gfx1151, CUDA images wrong"). That was true in the
compose era and is false now: the restored unit runs with /dev/kfd + /dev/dri and
must NOT pass --cpu. An agent trusting the rule file would have re-broken it.
Also corrected: the observability paragraph ("RUNNING BUT UNCONFIGURED, no
datasource, no rules" — all three have been configured since 2026-09-16), the
Active runbook row that still named halogen for frank-large, halogen's absence
from the Retired table, the benchmark policy naming halogen's harness, and the
seam list.
**Unit and page claims.** prometheus.container said "no alerting rules anywhere
in this stack"; node-exporter.container said Dankserver runs from its compose
stack; Prometheus.md documented three jobs as working (they scrape HTML: they
can never report up) and said no dashboard existed; index.md and the three
authentik service pages pointed at the retired file; Authentik.md's upgrade
instruction told you to bump an image tag in it.
One bad edit of my own on the way (a stale line number clobbered a See Also
entry in Prometheus.md) — caught by reading back, fixed, and the vault link
checker now proves the cross-references hold.
This commit is contained in:
Frank
@@ -119,9 +119,14 @@ This rule exists because reboots found gaps twice (2026-08-26: Caddy as a
|
||||
bare compose container never returned; 2026-09-11: five observability
|
||||
services had no units). The observability services now have units under
|
||||
`dankclaw/observability/quadlet/` and their compose definitions were removed —
|
||||
one owner per container name. The stack is RUNNING BUT UNCONFIGURED (no
|
||||
Grafana datasources/dashboards, no Prometheus rules); see
|
||||
`serverAppConfigs-docs/Infrastructure/Observability.md`.
|
||||
one owner per container name. The stack is **configured as of 2026-09-16**: a
|
||||
Grafana datasource and the `llm-fleet` dashboard are provisioned from
|
||||
`shared/grafana/provisioning/`, and `shared/prometheus/rules/llm-fleet.yml`
|
||||
loads 5 alert rules + 1 recording rule. Nothing is *delivered* — there is no
|
||||
Alertmanager, so a firing alert reaches nobody (an operator decision, not an
|
||||
oversight). Five Prometheus targets read down for structural reasons
|
||||
(HTML-scraped "probes", Gitea metrics off); the remedy is a blackbox exporter.
|
||||
See `serverAppConfigs-docs/Infrastructure/Observability.md`.
|
||||
|
||||
## Testing
|
||||
|
||||
@@ -132,8 +137,11 @@ Tests are mandatory before commit (`bats` required on the dev machine):
|
||||
```
|
||||
|
||||
Seams: compose validity, env consistency (no leaked secrets), Caddyfile
|
||||
validity, quadlet units, docs completeness, script syntax, LLM unit
|
||||
integrity, LLM secret hygiene, deployment flow, service-scoped planning.
|
||||
validity, quadlet units, docs completeness **including vault link integrity**,
|
||||
script syntax (shell + python), LLM unit integrity, LLM secret hygiene,
|
||||
deployment flow, service-scoped planning, homepage config/pages/deploy/checker,
|
||||
and `e2e_scaffold` — which fails if a Playwright spec file's number prefix is in
|
||||
no project, because such a spec silently never runs.
|
||||
See `serverAppConfigs-docs/Architecture/Testing Policy.md`.
|
||||
|
||||
## Secret Rules
|
||||
@@ -186,8 +194,9 @@ archive.
|
||||
**STOP the production service BEFORE benchmarking** — these servers serve
|
||||
real users, and a bench beside a live service starves the GPU and corrupts
|
||||
results. Use the runtime's own harness: `llama-bench` (llama.cpp), `vllm
|
||||
bench` (vLLM), halogen's `bench`/`sweep` modes for frank-large. Rolling your
|
||||
own bench requires asking first. Scratch ports are not used on LLM hosts — a
|
||||
bench` (vLLM), llama-bench for ik_llama.cpp (the halogen `bench`/`sweep` modes
|
||||
belong to a retired runtime). Rolling your own bench requires asking first.
|
||||
Scratch ports are not used on LLM hosts — a
|
||||
test process cannot share the single GPU anyway; validate against the service
|
||||
itself.
|
||||
|
||||
@@ -219,8 +228,7 @@ When renaming a service, retiring an endpoint, or moving a model: update the
|
||||
provider block, model `id`, `contextWindow`, `maxTokens`, `input` in
|
||||
`models.yml`, then fix every `modelRoles` entry that named the old endpoint —
|
||||
a role pointing at a removed provider fails at call time. `frank-large`
|
||||
declares `input: [text, image]` (halogen vision tower). There is **no audio
|
||||
input anywhere in the fleet** (left with `frank-npu`).
|
||||
declares `input: [text, image]` (its mmproj vision tower). There is **no audio
|
||||
|
||||
## Active Services
|
||||
|
||||
@@ -377,8 +385,12 @@ linked runbooks.
|
||||
- Both hosts allow passwordless sudo for the deploy user. Use plain `sudo` in
|
||||
committed scripts (no `echo pw | sudo -S`); `FRANK_SUDO_PW` covers hosts
|
||||
without it.
|
||||
- ComfyUI on Dankclaw runs CPU-only (`--cpu`): no ROCm wheels for gfx1151,
|
||||
and CUDA images are wrong for the hardware.
|
||||
- ComfyUI on Dankclaw runs with **GPU access** (`/dev/kfd` + `/dev/dri` via
|
||||
`AddDevice`, `GroupAdd=keep-groups`, plus the udev rule that makes those nodes
|
||||
world-writable — user-unit spawns carry no supplementary groups). It must NOT
|
||||
pass `--cpu`: the gfx1151 ROCm wheels exist and the unit's `--cpu` era ended
|
||||
2026-09-16, when a stale device-less container was found restarted 9,146
|
||||
times (`RuntimeError: No HIP GPUs are available`).
|
||||
- SearXNG outbound traffic must stay on the Gluetun tunnel; do not move its
|
||||
ports off the gluetun service.
|
||||
- `amd_iommu=off` on Dankclaw (bought 13–16 % prefill for frank-large,
|
||||
@@ -396,7 +408,7 @@ Active:
|
||||
|
||||
| Service | Runbook |
|
||||
|---------|---------|
|
||||
| frank-large (halogen-flash-server, Dankclaw) | `serverAppConfigs-docs/LLM/halogen-flash-server.md` |
|
||||
| frank-large (ik_llama.cpp, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-ik-uncensored.md` |
|
||||
| frank-medium (vllm-radiance MXFP4, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` |
|
||||
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
|
||||
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
|
||||
@@ -406,6 +418,7 @@ Retired (rollback targets and history):
|
||||
| Runtime | Host | Window | Runbook |
|
||||
|---------|------|--------|---------|
|
||||
| kingjones STRIX Qwen3.8-Flash-Next ROCmFP4 | Dankclaw | 2026-08-30 → 09-11 | `kingjones-strix-rocmfp4.md` |
|
||||
| halogen-flash-server (`frank-large`) | Dankclaw | → 2026-09-15 | `halogen-flash-server.md` |
|
||||
| agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` |
|
||||
| eemin Qwen3.8-27B ROCmI4 W4A4 MTP | Dankclaw | 2026-08-18 → 08-25 | `eemin-rocmi4-mtp.md` |
|
||||
| Kairic Edge IU4 | Dankclaw | 2026-08-22 (same-day revert) | `kairic-edge.md` |
|
||||
|
||||
@@ -19,7 +19,9 @@
|
||||
# the wildcard bind off the table matters: this is the second half of the
|
||||
# pairing that removes the unauthenticated observability surface, and both
|
||||
# consumers are accounted for. Dankserver runs its own copy of this container
|
||||
# (scraped by the "node-dankserver" job) from its own compose stack.
|
||||
# (scraped by the "node-dankserver" job) — since 2026-09-16 as a Quadlet unit in
|
||||
# the same shape (dankserver/observability/quadlet/), because the compose-only
|
||||
# version it replaced had been dead for a day and a half before anyone noticed.
|
||||
#
|
||||
# MOUNTS. The container reads the host's /proc, /sys and / as /host/* read-only
|
||||
# and is pointed at them with --path.*; it writes nothing anywhere.
|
||||
|
||||
@@ -25,8 +25,15 @@
|
||||
# namespace and dank owns /opt/prometheus/data. The image's default user
|
||||
# (nobody) maps to a subuid that cannot write it.
|
||||
#
|
||||
# There are no alerting rules and no Alertmanager anywhere in this stack: the
|
||||
# scrape config is the whole of its configuration.
|
||||
# RULES. Five LLM alert rules and one recording rule load from
|
||||
# /etc/prometheus/rules/ (installed from shared/prometheus/rules/ by the
|
||||
# deployer). There is still **no Alertmanager**, so nothing is delivered: a
|
||||
# firing alert is visible in Prometheus and on the Grafana dashboard's ALERTS
|
||||
# panel and reaches nobody. Delivery is an operator decision, not an oversight;
|
||||
# the fragility is that an undelivered alert looks identical to a quiet one.
|
||||
#
|
||||
# The config volume is also why rules belong in git: editing them in the UI is
|
||||
# lost on the next deploy, since the deployer overwrites the whole directory.
|
||||
|
||||
[Unit]
|
||||
Description=Prometheus metrics store (Dankclaw)
|
||||
|
||||
@@ -259,8 +259,9 @@ Two secrets live in the host `.env` (git-ignored; placeholders in
|
||||
- **Logs:** `podman logs authentik-server` / `podman logs authentik-worker`.
|
||||
- **Backups:** `pg_dump` the `authentik-postgres` database (the single source of
|
||||
truth). The `/home/dank/Containers/authentik/data` dir holds media + static files.
|
||||
- **Upgrades:** bump the image tag in `dankclaw/compose.yaml`, run
|
||||
`./scripts/deploy.sh dankclaw`. Authentik migrates the schema on startup.
|
||||
- **Upgrades:** bump the image tag in the three units under
|
||||
`dankclaw/authentik/quadlet/`, run `./scripts/deploy.sh dankclaw`. Authentik
|
||||
migrates the schema on startup.
|
||||
|
||||
## Outpost claim cache (`proxysession`) — 403 after a group add
|
||||
|
||||
|
||||
@@ -16,7 +16,7 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
|
||||
| Config | `/opt/prometheus/config/prometheus.yml`, installed from `shared/prometheus/prometheus.yml` on every deploy |
|
||||
| Data | `/opt/prometheus/data` (TSDB) |
|
||||
| Retention | `--storage.tsdb.retention.time=30d` |
|
||||
| Alerting | **none** — no rule files, no Alertmanager anywhere in the stack |
|
||||
| Alerting | 5 alert rules + 1 recording rule (`shared/prometheus/rules/llm-fleet.yml`). Evaluated, **not delivered**: there is no Alertmanager (see [[Observability]]) |
|
||||
|
||||
## Targets
|
||||
|
||||
@@ -26,12 +26,25 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
|
||||
|-----|--------|-----|
|
||||
| `prometheus` | `localhost:9090` | itself |
|
||||
| `node-dankclaw`, `node-dankserver` | `192.168.0.143:9100`, `192.168.0.235:9100` | host metrics from each machine's [[Node-Exporter]] |
|
||||
| `gitea` | `192.168.0.143:3000` | Gitea's built-in exporter, reached on its published LAN port |
|
||||
| `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on the `dankclaw_default` network |
|
||||
| `comfyui` | `comfyui:8188` | UI probe; ComfyUI has no native `/metrics` |
|
||||
| `services` | `192.168.0.143:8081`, `:3005` | up/down probes for SearXNG and OpenWebUI |
|
||||
| `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on the `dankclaw_default` network; both serve real `/metrics` |
|
||||
| `frank-medium`, `frank-large` | `192.168.0.235:7770`, `192.168.0.143:7770` | the LLM fleet — `vllm:` and `llamacpp:` series (LAN addresses, not the guarded vhosts) |
|
||||
|
||||
Targets are `host:port`, never URLs. For richer HTTP assertions, add the Prometheus **blackbox** exporter in probe mode rather than scraping HTML.
|
||||
**Three jobs read down, and they are not outages** (audited 2026-09-16):
|
||||
|
||||
| Job | Target | Why it never reports up |
|
||||
|-----|--------|-------------------------|
|
||||
| `comfyui` | `comfyui:8188` | scrapes `/`, which is HTML — "unsupported Content-Type" |
|
||||
| `services` | `192.168.0.143:8081`, `:3005` | same: up/down intent, implemented as a scrape |
|
||||
| `gitea` | `192.168.0.143:3000/metrics` | Gitea's exporter is off until `[metrics] ENABLED = true` |
|
||||
|
||||
A scrape cannot express "is this HTTP service answering?" — that is what a
|
||||
probe is for, and the remedy is the **blackbox exporter** in probe mode
|
||||
(`/probe?target=…&module=http_2xx`), which also gives the guarded vhosts a real
|
||||
uptime signal since Prometheus cannot solve an Anubis challenge itself. Until it
|
||||
is deployed, these three stay red for a structural reason: deleting them would
|
||||
hide a real service, and fixing them by publishing databases or enabling metrics
|
||||
would trade a wrong signal for a real exposure. Targets are `host:port`, never
|
||||
URLs.
|
||||
|
||||
## Volumes
|
||||
|
||||
@@ -54,6 +67,6 @@ Prometheus responds on `/-/healthy` with 200; the unit's health command is the s
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Grafana]] — the dashboard frontend (no datasource configured yet)
|
||||
- [[Grafana]] — the dashboard frontend (datasource and LLM dashboard provisioned as code since 2026-09-16)
|
||||
- [[Node-Exporter]] — the host metrics source
|
||||
- [[Observability]] — the whole stack, including what is still unconfigured
|
||||
|
||||
@@ -5,7 +5,7 @@ This is the service the Caddy `forward_auth` vhosts proxy to
|
||||
(`authentik-server:9000`). It also serves the login page at
|
||||
`https://auth.franksdank.cloud`.
|
||||
|
||||
- **Compose service:** `authentik-server` (in `dankclaw/compose.yaml`)
|
||||
- **Unit:** `dankclaw/authentik/quadlet/authentik-server.container` (`authentik-server.service`)
|
||||
- **Image:** `ghcr.io/goauthentik/server:2026.8.0` (command: `server`)
|
||||
- **Port:** `9000` (published; the outpost + admin + login all live here)
|
||||
- **Data:** `/home/dank/authentik/data` (bind mount, shared with the worker)
|
||||
|
||||
@@ -4,7 +4,7 @@ The Authentik worker — background tasks: event processing, provider
|
||||
synchronization, scheduled flows, and certificate renewal. It runs the same
|
||||
image as the server; the `command: worker` selects the role.
|
||||
|
||||
- **Compose service:** `authentik-worker` (in `dankclaw/compose.yaml`)
|
||||
- **Unit:** `dankclaw/authentik/quadlet/authentik-worker.container` (`authentik-worker.service`)
|
||||
- **Image:** `ghcr.io/goauthentik/server:2026.8.0` (command: `worker`)
|
||||
- **Runs as:** `0:0` (rootless: container root maps to the host user that owns the data dir)
|
||||
- **Data:** `/home/dank/authentik/data` (bind mount, shared with the server)
|
||||
|
||||
@@ -85,9 +85,9 @@ covers both workloads).
|
||||
- [[authentik-worker]] — Authentik background worker service
|
||||
- [[Homepage]] — password-protected service index
|
||||
- [[Homepage Design Grilling]] — homepage decision log and open questions
|
||||
- [[Observability]] — Grafana, Prometheus, Loki, Alloy, node-exporter: the stack, its quadlet deployment, its exposure, and **what is still unconfigured**
|
||||
- [[Grafana]] — dashboards (no datasource or dashboard configured yet)
|
||||
- [[Prometheus]] — metrics collection (no alerting rules or Alertmanager)
|
||||
- [[Observability]] — Grafana, Prometheus, Loki, Alloy, node-exporter: the stack, its quadlet deployment, its exposure, and what is still missing (alert delivery)
|
||||
- [[Grafana]] — dashboards (datasource and the LLM fleet dashboard provisioned as code since 2026-09-16)
|
||||
- [[Prometheus]] — metrics collection (5 alert rules + 1 recording rule; evaluated but not delivered — no Alertmanager)
|
||||
- [[Node-Exporter]] — host metrics
|
||||
- [[Loki]] — log storage
|
||||
- [[Alloy]] — log shipping (replaces Promtail)
|
||||
|
||||
@@ -18,7 +18,8 @@
|
||||
|
||||
load helpers
|
||||
|
||||
COMPOSE="$REPO_ROOT/dankclaw/compose.yaml"
|
||||
# (no COMPOSE here: dankclaw/compose.yaml is retired — every container on that
|
||||
# host is unit-managed, and this seam reads the units.)
|
||||
CADDYFILE="$REPO_ROOT/dankclaw/Caddyfile"
|
||||
ENV_EXAMPLE="$REPO_ROOT/dankclaw/.env.example"
|
||||
|
||||
|
||||
+10
-4
@@ -78,8 +78,11 @@ CADDYFILE="$REPO_ROOT/dankclaw/Caddyfile"
|
||||
@test "compose no longer defines the quadlet-replaced services" {
|
||||
# gitea, its postgres, open-webui, and caddy run as boot-persistent
|
||||
# quadlets on dankclaw (see dankclaw/gitea/quadlet,
|
||||
# dankclaw/open-webui/quadlet, dankclaw/caddy/quadlet).
|
||||
! grep -qE '^\s+(gitea|postgres|openwebui|caddy):\s*$' "$REPO_ROOT/dankclaw/compose.yaml"
|
||||
# dankclaw/open-webui/quadlet, dankclaw/caddy/quadlet). The compose file
|
||||
# itself is retired and observability_quadlet.bats asserts that; here the
|
||||
# regression is a re-added entry, so the check must not pass by absence.
|
||||
local cf="$REPO_ROOT/dankclaw/compose.yaml"
|
||||
[ ! -f "$cf" ] || ! grep -qE '^\s+(gitea|postgres|openwebui|caddy):\s*$' "$cf"
|
||||
}
|
||||
|
||||
@test "the caddy quadlet healthcheck probes the bare-IP catch-all" {
|
||||
@@ -174,9 +177,12 @@ CADDYFILE="$REPO_ROOT/dankclaw/Caddyfile"
|
||||
|
||||
# --- Observability configs (deployed by deploy.sh step 3) ---
|
||||
|
||||
@test "no legacy .local service names remain in shared configs or compose" {
|
||||
@test "no legacy .local service names remain in deployed config" {
|
||||
# Scans the surfaces that are actually deployed — shared configs and every
|
||||
# dankclaw unit — rather than a file that no longer exists. A missing path
|
||||
# makes grep exit 2 and the negation pass, which is how this went hollow.
|
||||
! grep -rn 'dankclaw\.local\|dankserver\.local' \
|
||||
"$REPO_ROOT/shared" "$REPO_ROOT/dankclaw/compose.yaml"
|
||||
"$REPO_ROOT/shared" "$REPO_ROOT/dankclaw"
|
||||
}
|
||||
|
||||
@test "loki compactor has a delete_request_store (required when retention is on)" {
|
||||
|
||||
@@ -32,8 +32,12 @@ load helpers
|
||||
grep -q '^Image=localhost/comfyui-rocm:gfx1151$' "$unit" \
|
||||
|| { echo "unit must run the locally built image"; return 1; }
|
||||
# One owner per container name: a surviving compose service would fight the
|
||||
# unit for the container, which is how the stale one stayed alive.
|
||||
! grep -qE '^\s+comfyui:' "$REPO_ROOT/dankclaw/compose.yaml"
|
||||
# unit for the container, which is how the stale one stayed alive. The file
|
||||
# itself is retired (observability_quadlet.bats asserts that), so here the
|
||||
# regression to catch is a re-added entry — the check must not go hollow
|
||||
# just because the file is absent.
|
||||
local cf="$REPO_ROOT/dankclaw/compose.yaml"
|
||||
[ ! -f "$cf" ] || ! grep -qE '^\s+comfyui:' "$cf"
|
||||
}
|
||||
|
||||
@test "comfyui Containerfile is based on the gfx1151 torch image" {
|
||||
|
||||
@@ -263,14 +263,19 @@ PAIRS
|
||||
grep -q 'build_compose_image alloy' "$REMOTE_DEPLOY"
|
||||
}
|
||||
|
||||
@test "the five are no longer compose services" {
|
||||
@test "no dankclaw compose file competes with the quadlet units" {
|
||||
# Two owners for one container name is how a compose run resurrects a
|
||||
# stale duplicate next to the quadlet-managed one.
|
||||
local svc
|
||||
for svc in "${UNITS[@]}"; do
|
||||
! services_in "$REPO_ROOT/dankclaw/compose.yaml" | grep -qx "$svc" \
|
||||
|| { echo "$svc is still a compose service"; return 1; }
|
||||
done
|
||||
# stale duplicate next to the quadlet-managed one. This file was retired on
|
||||
# 2026-09-16 — every container on the host reports a PODMAN_SYSTEMD_UNIT
|
||||
# label — and the invariant is that it stays retired: the four assertions
|
||||
# that used to grep it passed against the missing file, which is how a gate
|
||||
# goes hollow.
|
||||
local cf="$REPO_ROOT/dankclaw/compose.yaml"
|
||||
[ ! -f "$cf" ] || {
|
||||
echo "$cf is back: every dankclaw container is unit-managed, so a"
|
||||
echo "compose path can only spawn a duplicate fighting a unit for a name"
|
||||
return 1
|
||||
}
|
||||
}
|
||||
|
||||
@test "changes under dankclaw/observability/ run this seam" {
|
||||
|
||||
Reference in New Issue
Block a user