docs+units: fix every claim my own changes made false

Retiring dankclaw/compose.yaml and adding alert rules invalidated claims across
the repo, including in the rule file agents read first. Clean cutover includes
the prose.

**Tests that had gone hollow.** Four assertions grepped dankclaw/compose.yaml
for a service name. With the file gone they passed by *absence* — a gate that
can no longer fail. Each now asserts the real invariant ("no dankclaw compose
file competes with the quadlet units") and keeps a live check for a re-added
entry, so it still fails when one appears: verified by recreating a compose file
with a comfyui service and watching three tests go red, then removing it.

**AGENTS.md, which was wrong about ComfyUI.** It said ComfyUI runs CPU-only
("no ROCm wheels for gfx1151, CUDA images wrong"). That was true in the
compose era and is false now: the restored unit runs with /dev/kfd + /dev/dri and
must NOT pass --cpu. An agent trusting the rule file would have re-broken it.
Also corrected: the observability paragraph ("RUNNING BUT UNCONFIGURED, no
datasource, no rules" — all three have been configured since 2026-09-16), the
Active runbook row that still named halogen for frank-large, halogen's absence
from the Retired table, the benchmark policy naming halogen's harness, and the
seam list.

**Unit and page claims.** prometheus.container said "no alerting rules anywhere
in this stack"; node-exporter.container said Dankserver runs from its compose
stack; Prometheus.md documented three jobs as working (they scrape HTML: they
can never report up) and said no dashboard existed; index.md and the three
authentik service pages pointed at the retired file; Authentik.md's upgrade
instruction told you to bump an image tag in it.

One bad edit of my own on the way (a stale line number clobbered a See Also
entry in Prometheus.md) — caught by reading back, fixed, and the vault link
checker now proves the cross-references hold.
This commit is contained in:
Frank committed 2026-09-16 07:46:23 -07:00
1 parent 2e33ec36c1
commit f134cc8876
12 files changed
+95 -43

No files matched your search

+25 -12
View File
@@ -119,9 +119,14 @@ This rule exists because reboots found gaps twice (2026-08-26: Caddy as a
bare compose container never returned; 2026-09-11: five observability
services had no units). The observability services now have units under
`dankclaw/observability/quadlet/` and their compose definitions were removed —
one owner per container name. The stack is RUNNING BUT UNCONFIGURED (no
Grafana datasources/dashboards, no Prometheus rules); see
`serverAppConfigs-docs/Infrastructure/Observability.md`.
one owner per container name. The stack is **configured as of 2026-09-16**: a
Grafana datasource and the `llm-fleet` dashboard are provisioned from
`shared/grafana/provisioning/`, and `shared/prometheus/rules/llm-fleet.yml`
loads 5 alert rules + 1 recording rule. Nothing is *delivered* — there is no
Alertmanager, so a firing alert reaches nobody (an operator decision, not an
oversight). Five Prometheus targets read down for structural reasons
(HTML-scraped "probes", Gitea metrics off); the remedy is a blackbox exporter.
See `serverAppConfigs-docs/Infrastructure/Observability.md`.
## Testing
@@ -132,8 +137,11 @@ Tests are mandatory before commit (`bats` required on the dev machine):
```
Seams: compose validity, env consistency (no leaked secrets), Caddyfile
validity, quadlet units, docs completeness, script syntax, LLM unit
integrity, LLM secret hygiene, deployment flow, service-scoped planning.
validity, quadlet units, docs completeness **including vault link integrity**,
script syntax (shell + python), LLM unit integrity, LLM secret hygiene,
deployment flow, service-scoped planning, homepage config/pages/deploy/checker,
and `e2e_scaffold` — which fails if a Playwright spec file's number prefix is in
no project, because such a spec silently never runs.
See `serverAppConfigs-docs/Architecture/Testing Policy.md`.
## Secret Rules
@@ -186,8 +194,9 @@ archive.
**STOP the production service BEFORE benchmarking** — these servers serve
real users, and a bench beside a live service starves the GPU and corrupts
results. Use the runtime's own harness: `llama-bench` (llama.cpp), `vllm
bench` (vLLM), halogen's `bench`/`sweep` modes for frank-large. Rolling your
own bench requires asking first. Scratch ports are not used on LLM hosts — a
bench` (vLLM), llama-bench for ik_llama.cpp (the halogen `bench`/`sweep` modes
belong to a retired runtime). Rolling your own bench requires asking first.
Scratch ports are not used on LLM hosts — a
test process cannot share the single GPU anyway; validate against the service
itself.
@@ -219,8 +228,7 @@ When renaming a service, retiring an endpoint, or moving a model: update the
provider block, model `id`, `contextWindow`, `maxTokens`, `input` in
`models.yml`, then fix every `modelRoles` entry that named the old endpoint —
a role pointing at a removed provider fails at call time. `frank-large`
declares `input: [text, image]` (halogen vision tower). There is **no audio
input anywhere in the fleet** (left with `frank-npu`).
declares `input: [text, image]` (its mmproj vision tower). There is **no audio
## Active Services
@@ -377,8 +385,12 @@ linked runbooks.
- Both hosts allow passwordless sudo for the deploy user. Use plain `sudo` in
committed scripts (no `echo pw | sudo -S`); `FRANK_SUDO_PW` covers hosts
without it.
- ComfyUI on Dankclaw runs CPU-only (`--cpu`): no ROCm wheels for gfx1151,
and CUDA images are wrong for the hardware.
- ComfyUI on Dankclaw runs with **GPU access** (`/dev/kfd` + `/dev/dri` via
`AddDevice`, `GroupAdd=keep-groups`, plus the udev rule that makes those nodes
world-writable — user-unit spawns carry no supplementary groups). It must NOT
pass `--cpu`: the gfx1151 ROCm wheels exist and the unit's `--cpu` era ended
2026-09-16, when a stale device-less container was found restarted 9,146
times (`RuntimeError: No HIP GPUs are available`).
- SearXNG outbound traffic must stay on the Gluetun tunnel; do not move its
ports off the gluetun service.
- `amd_iommu=off` on Dankclaw (bought 13–16 % prefill for frank-large,
@@ -396,7 +408,7 @@ Active:
| Service | Runbook |
|---------|---------|
| frank-large (halogen-flash-server, Dankclaw) | `serverAppConfigs-docs/LLM/halogen-flash-server.md` |
| frank-large (ik_llama.cpp, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-ik-uncensored.md` |
| frank-medium (vllm-radiance MXFP4, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` |
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
@@ -406,6 +418,7 @@ Retired (rollback targets and history):
| Runtime | Host | Window | Runbook |
|---------|------|--------|---------|
| kingjones STRIX Qwen3.8-Flash-Next ROCmFP4 | Dankclaw | 2026-08-30 → 09-11 | `kingjones-strix-rocmfp4.md` |
| halogen-flash-server (`frank-large`) | Dankclaw | → 2026-09-15 | `halogen-flash-server.md` |
| agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` |
| eemin Qwen3.8-27B ROCmI4 W4A4 MTP | Dankclaw | 2026-08-18 → 08-25 | `eemin-rocmi4-mtp.md` |
| Kairic Edge IU4 | Dankclaw | 2026-08-22 (same-day revert) | `kairic-edge.md` |