docs+units: fix every claim my own changes made false
Retiring dankclaw/compose.yaml and adding alert rules invalidated claims across
the repo, including in the rule file agents read first. Clean cutover includes
the prose.
**Tests that had gone hollow.** Four assertions grepped dankclaw/compose.yaml
for a service name. With the file gone they passed by *absence* — a gate that
can no longer fail. Each now asserts the real invariant ("no dankclaw compose
file competes with the quadlet units") and keeps a live check for a re-added
entry, so it still fails when one appears: verified by recreating a compose file
with a comfyui service and watching three tests go red, then removing it.
**AGENTS.md, which was wrong about ComfyUI.** It said ComfyUI runs CPU-only
("no ROCm wheels for gfx1151, CUDA images wrong"). That was true in the
compose era and is false now: the restored unit runs with /dev/kfd + /dev/dri and
must NOT pass --cpu. An agent trusting the rule file would have re-broken it.
Also corrected: the observability paragraph ("RUNNING BUT UNCONFIGURED, no
datasource, no rules" — all three have been configured since 2026-09-16), the
Active runbook row that still named halogen for frank-large, halogen's absence
from the Retired table, the benchmark policy naming halogen's harness, and the
seam list.
**Unit and page claims.** prometheus.container said "no alerting rules anywhere
in this stack"; node-exporter.container said Dankserver runs from its compose
stack; Prometheus.md documented three jobs as working (they scrape HTML: they
can never report up) and said no dashboard existed; index.md and the three
authentik service pages pointed at the retired file; Authentik.md's upgrade
instruction told you to bump an image tag in it.
One bad edit of my own on the way (a stale line number clobbered a See Also
entry in Prometheus.md) — caught by reading back, fixed, and the vault link
checker now proves the cross-references hold.
This commit is contained in:
1 parent
2e33ec36c1
commit
f134cc8876
12 files changed
+95
-43
No files matched your search
@@ -119,9 +119,14 @@ This rule exists because reboots found gaps twice (2026-08-26: Caddy as a
|
||||
bare compose container never returned; 2026-09-11: five observability
|
||||
services had no units). The observability services now have units under
|
||||
`dankclaw/observability/quadlet/` and their compose definitions were removed —
|
||||
one owner per container name. The stack is RUNNING BUT UNCONFIGURED (no
|
||||
Grafana datasources/dashboards, no Prometheus rules); see
|
||||
`serverAppConfigs-docs/Infrastructure/Observability.md`.
|
||||
one owner per container name. The stack is **configured as of 2026-09-16**: a
|
||||
Grafana datasource and the `llm-fleet` dashboard are provisioned from
|
||||
`shared/grafana/provisioning/`, and `shared/prometheus/rules/llm-fleet.yml`
|
||||
loads 5 alert rules + 1 recording rule. Nothing is *delivered* — there is no
|
||||
Alertmanager, so a firing alert reaches nobody (an operator decision, not an
|
||||
oversight). Five Prometheus targets read down for structural reasons
|
||||
(HTML-scraped "probes", Gitea metrics off); the remedy is a blackbox exporter.
|
||||
See `serverAppConfigs-docs/Infrastructure/Observability.md`.
|
||||
|
||||
## Testing
|
||||
|
||||
@@ -132,8 +137,11 @@ Tests are mandatory before commit (`bats` required on the dev machine):
|
||||
```
|
||||
|
||||
Seams: compose validity, env consistency (no leaked secrets), Caddyfile
|
||||
validity, quadlet units, docs completeness, script syntax, LLM unit
|
||||
integrity, LLM secret hygiene, deployment flow, service-scoped planning.
|
||||
validity, quadlet units, docs completeness **including vault link integrity**,
|
||||
script syntax (shell + python), LLM unit integrity, LLM secret hygiene,
|
||||
deployment flow, service-scoped planning, homepage config/pages/deploy/checker,
|
||||
and `e2e_scaffold` — which fails if a Playwright spec file's number prefix is in
|
||||
no project, because such a spec silently never runs.
|
||||
See `serverAppConfigs-docs/Architecture/Testing Policy.md`.
|
||||
|
||||
## Secret Rules
|
||||
@@ -186,8 +194,9 @@ archive.
|
||||
**STOP the production service BEFORE benchmarking** — these servers serve
|
||||
real users, and a bench beside a live service starves the GPU and corrupts
|
||||
results. Use the runtime's own harness: `llama-bench` (llama.cpp), `vllm
|
||||
bench` (vLLM), halogen's `bench`/`sweep` modes for frank-large. Rolling your
|
||||
own bench requires asking first. Scratch ports are not used on LLM hosts — a
|
||||
bench` (vLLM), llama-bench for ik_llama.cpp (the halogen `bench`/`sweep` modes
|
||||
belong to a retired runtime). Rolling your own bench requires asking first.
|
||||
Scratch ports are not used on LLM hosts — a
|
||||
test process cannot share the single GPU anyway; validate against the service
|
||||
itself.
|
||||
|
||||
@@ -219,8 +228,7 @@ When renaming a service, retiring an endpoint, or moving a model: update the
|
||||
provider block, model `id`, `contextWindow`, `maxTokens`, `input` in
|
||||
`models.yml`, then fix every `modelRoles` entry that named the old endpoint —
|
||||
a role pointing at a removed provider fails at call time. `frank-large`
|
||||
declares `input: [text, image]` (halogen vision tower). There is **no audio
|
||||
input anywhere in the fleet** (left with `frank-npu`).
|
||||
declares `input: [text, image]` (its mmproj vision tower). There is **no audio
|
||||
|
||||
## Active Services
|
||||
|
||||
@@ -377,8 +385,12 @@ linked runbooks.
|
||||
- Both hosts allow passwordless sudo for the deploy user. Use plain `sudo` in
|
||||
committed scripts (no `echo pw | sudo -S`); `FRANK_SUDO_PW` covers hosts
|
||||
without it.
|
||||
- ComfyUI on Dankclaw runs CPU-only (`--cpu`): no ROCm wheels for gfx1151,
|
||||
and CUDA images are wrong for the hardware.
|
||||
- ComfyUI on Dankclaw runs with **GPU access** (`/dev/kfd` + `/dev/dri` via
|
||||
`AddDevice`, `GroupAdd=keep-groups`, plus the udev rule that makes those nodes
|
||||
world-writable — user-unit spawns carry no supplementary groups). It must NOT
|
||||
pass `--cpu`: the gfx1151 ROCm wheels exist and the unit's `--cpu` era ended
|
||||
2026-09-16, when a stale device-less container was found restarted 9,146
|
||||
times (`RuntimeError: No HIP GPUs are available`).
|
||||
- SearXNG outbound traffic must stay on the Gluetun tunnel; do not move its
|
||||
ports off the gluetun service.
|
||||
- `amd_iommu=off` on Dankclaw (bought 13–16 % prefill for frank-large,
|
||||
@@ -396,7 +408,7 @@ Active:
|
||||
|
||||
| Service | Runbook |
|
||||
|---------|---------|
|
||||
| frank-large (halogen-flash-server, Dankclaw) | `serverAppConfigs-docs/LLM/halogen-flash-server.md` |
|
||||
| frank-large (ik_llama.cpp, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-ik-uncensored.md` |
|
||||
| frank-medium (vllm-radiance MXFP4, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` |
|
||||
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
|
||||
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
|
||||
@@ -406,6 +418,7 @@ Retired (rollback targets and history):
|
||||
| Runtime | Host | Window | Runbook |
|
||||
|---------|------|--------|---------|
|
||||
| kingjones STRIX Qwen3.8-Flash-Next ROCmFP4 | Dankclaw | 2026-08-30 → 09-11 | `kingjones-strix-rocmfp4.md` |
|
||||
| halogen-flash-server (`frank-large`) | Dankclaw | → 2026-09-15 | `halogen-flash-server.md` |
|
||||
| agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` |
|
||||
| eemin Qwen3.8-27B ROCmI4 W4A4 MTP | Dankclaw | 2026-08-18 → 08-25 | `eemin-rocmi4-mtp.md` |
|
||||
| Kairic Edge IU4 | Dankclaw | 2026-08-22 (same-day revert) | `kairic-edge.md` |
|
||||
|
||||
Reference in new issue
Block a user