Frank
f134cc8876
Retiring dankclaw/compose.yaml and adding alert rules invalidated claims across
the repo, including in the rule file agents read first. Clean cutover includes
the prose.
**Tests that had gone hollow.** Four assertions grepped dankclaw/compose.yaml
for a service name. With the file gone they passed by *absence* — a gate that
can no longer fail. Each now asserts the real invariant ("no dankclaw compose
file competes with the quadlet units") and keeps a live check for a re-added
entry, so it still fails when one appears: verified by recreating a compose file
with a comfyui service and watching three tests go red, then removing it.
**AGENTS.md, which was wrong about ComfyUI.** It said ComfyUI runs CPU-only
("no ROCm wheels for gfx1151, CUDA images wrong"). That was true in the
compose era and is false now: the restored unit runs with /dev/kfd + /dev/dri and
must NOT pass --cpu. An agent trusting the rule file would have re-broken it.
Also corrected: the observability paragraph ("RUNNING BUT UNCONFIGURED, no
datasource, no rules" — all three have been configured since 2026-09-16), the
Active runbook row that still named halogen for frank-large, halogen's absence
from the Retired table, the benchmark policy naming halogen's harness, and the
seam list.
**Unit and page claims.** prometheus.container said "no alerting rules anywhere
in this stack"; node-exporter.container said Dankserver runs from its compose
stack; Prometheus.md documented three jobs as working (they scrape HTML: they
can never report up) and said no dashboard existed; index.md and the three
authentik service pages pointed at the retired file; Authentik.md's upgrade
instruction told you to bump an image tag in it.
One bad edit of my own on the way (a stale line number clobbered a See Also
entry in Prometheus.md) — caught by reading back, fixed, and the vault link
checker now proves the cross-references hold.
101 lines
5.3 KiB
Bash
101 lines
5.3 KiB
Bash
#!/usr/bin/env bats
|
|
# Seam: ComfyUI ROCm GPU configuration on Dankclaw.
|
|
# The gfx1151-enabled PyTorch build is the only verified GPU path on Strix
|
|
# Halo. The compose must:
|
|
# - build from a Containerfile based on the gfx1151 ROCm torch image
|
|
# - pass /dev/kfd and /dev/dri to the container
|
|
# - NOT pass --cpu
|
|
# - grant GPU access via world-writable compute nodes (udev MODE 0666), not
|
|
# named-group membership, because systemd user-unit spawns carry NO
|
|
# supplementary groups at spawn time
|
|
#
|
|
# STATUS (2026-09-16): ComfyUI is a Quadlet unit, not a compose service. The
|
|
# build still comes from Containerfile.comfyui (AMD's gfx1151 torch base) via
|
|
# the deployer's build_compose_image, and the wiring now lives in
|
|
# dankclaw/comfyui/quadlet/comfyui.container: /dev/kfd + /dev/dri, keep-groups
|
|
# only, and the udev rule at dankclaw/udev/99-comfyui-gpu-access.rules (MODE
|
|
# 0666 on kfd + renderD*) for compute-node access, because systemd user-unit
|
|
# spawns carry no supplementary groups. The move happened after the compose
|
|
# container went stale — created before the GPU declarations existed, then
|
|
# restarted 9,146 times without them. See
|
|
# serverAppConfigs-docs/Research/ComfyUI ROCm and Vulkan.md.
|
|
|
|
load helpers
|
|
|
|
@test "comfyui builds from the gfx1151 ROCm Containerfile" {
|
|
# The image is built by the deployer (build_compose_image comfyui, which
|
|
# resolves dankclaw/Containerfile.comfyui) and referenced by the unit.
|
|
grep -q "comfyui)" "$REPO_ROOT/scripts/deploy-remote.sh"
|
|
grep -q 'image=comfyui-rocm:gfx1151' "$REPO_ROOT/scripts/deploy-remote.sh"
|
|
grep -q 'containerfile=dankclaw/Containerfile.comfyui' "$REPO_ROOT/scripts/deploy-remote.sh"
|
|
local unit="$REPO_ROOT/dankclaw/comfyui/quadlet/comfyui.container"
|
|
grep -q '^Image=localhost/comfyui-rocm:gfx1151$' "$unit" \
|
|
|| { echo "unit must run the locally built image"; return 1; }
|
|
# One owner per container name: a surviving compose service would fight the
|
|
# unit for the container, which is how the stale one stayed alive. The file
|
|
# itself is retired (observability_quadlet.bats asserts that), so here the
|
|
# regression to catch is a re-added entry — the check must not go hollow
|
|
# just because the file is absent.
|
|
local cf="$REPO_ROOT/dankclaw/compose.yaml"
|
|
[ ! -f "$cf" ] || ! grep -qE '^\s+comfyui:' "$cf"
|
|
}
|
|
|
|
@test "comfyui Containerfile is based on the gfx1151 torch image" {
|
|
# Resolve the Containerfile from the deployer's declaration — the file the
|
|
# build actually uses — not from the filesystem. A `find … | head -1` here
|
|
# once returned whichever Containerfile the directory listing happened to
|
|
# put first (dankclaw/alloy/Containerfile), so the test failed for sessions
|
|
# while the real ComfyUI Containerfile was correct, and it would equally
|
|
# have passed on a wrong file. Compose no longer names it: ComfyUI is a
|
|
# Quadlet unit and the image is built by deploy_comfyui.
|
|
local cf="$REPO_ROOT/dankclaw/Containerfile.comfyui"
|
|
grep -q 'containerfile=dankclaw/Containerfile.comfyui' "$REPO_ROOT/scripts/deploy-remote.sh" \
|
|
|| { echo "deployer no longer builds that Containerfile"; return 1; }
|
|
[ -f "$cf" ] || { echo "$cf does not exist"; return 1; }
|
|
grep -q "gfx1151" "$cf" || { echo "$cf does not reference gfx1151"; return 1; }
|
|
grep -q "FROM" "$cf" || { echo "$cf has no FROM"; return 1; }
|
|
}
|
|
|
|
@test "comfyui passes /dev/kfd and /dev/dri to the container" {
|
|
# Read the Quadlet unit, not compose: ComfyUI left compose on 2026-09-16
|
|
# after a stale, device-less container was restarted 9,146 times by
|
|
# `restart: unless-stopped` (see comfyui/quadlet/comfyui.container).
|
|
local unit="$REPO_ROOT/dankclaw/comfyui/quadlet/comfyui.container"
|
|
[ -f "$unit" ]
|
|
grep -q '^AddDevice=/dev/kfd$' "$unit" || { echo "no /dev/kfd"; return 1; }
|
|
grep -q '^AddDevice=/dev/dri$' "$unit" || { echo "no /dev/dri"; return 1; }
|
|
}
|
|
|
|
@test "comfyui does not run with --cpu" {
|
|
local unit="$REPO_ROOT/dankclaw/comfyui/quadlet/comfyui.container"
|
|
local exec_line
|
|
exec_line=$(grep -E '^Exec=' "$unit")
|
|
case "$exec_line" in
|
|
*--cpu*) echo "comfyui must not pass --cpu (ROCm GPU build)"; return 1 ;;
|
|
esac
|
|
case "$exec_line" in
|
|
*--listen*) ;;
|
|
*) echo "comfyui must listen on 0.0.0.0"; return 1 ;;
|
|
esac
|
|
}
|
|
|
|
@test "comfyui GPU access uses keep-groups + udev MODE 0666, not named groups" {
|
|
python3 - "$REPO_ROOT/dankclaw/comfyui/quadlet/comfyui.container" "$REPO_ROOT/dankclaw/udev/99-comfyui-gpu-access.rules" <<'PYEOF'
|
|
import sys, re
|
|
unit = open(sys.argv[1]).read()
|
|
groups = re.findall(r"^GroupAdd=(.*)$", unit, re.M)
|
|
# User-unit spawns carry NO supplementary groups, so named group_add entries
|
|
# (video, render) can never be honored at spawn time. The deployed contract is
|
|
# keep-groups plus world-writable compute nodes via the udev rule; a future
|
|
# edit that drops either half must fail here.
|
|
assert groups == ["keep-groups"], \
|
|
f"comfyui GroupAdd must be exactly ['keep-groups']; got: {groups}"
|
|
rule = open(sys.argv[2]).read()
|
|
assert 'KERNEL=="kfd", MODE="0666"' in rule, \
|
|
"udev rule must grant MODE 0666 on kfd with NO SUBSYSTEM guard (misc-class devices have no subsystem to match)"
|
|
assert 'KERNEL=="renderD*", SUBSYSTEM=="drm", MODE="0666"' in rule, \
|
|
"udev rule must grant MODE 0666 on renderD* (SUBSYSTEM drm guard)"
|
|
print("GPU access contract OK")
|
|
PYEOF
|
|
}
|