The first revision guarded the kfd line with SUBSYSTEM=="misc", which never
matches (misc-class devices carry no subsystem for udev matching), so the rule
was read but never fired and /dev/kfd stayed 0660 root:render. Name-only match
applies 0666 on creation and on trigger.
Also fixes test-changed.sh to exit 0 when changed paths map to zero tests
(udev rules are mapped-but-untestable) instead of invoking bats with an empty
selection.
The comfyui container is spawned by a systemd user unit whose spawn context
carries no supplementary groups, so render-group access to /dev/kfd and
renderD128 was structurally unreachable (keep-groups preserved an empty set;
named group_add 'render' fails name lookup inside the userns). World-mode on
the compute nodes is the only spawn-context-independent grant; display cards
keep their modes. Apply: install to /etc/udev/rules.d, udevadm control
--reload-rules, then trigger by subsystem to re-apply modes to live nodes.
Also maps dankclaw/udev/* in the focused-test selector (not bats-testable).
Env file is the source of truth (unit ExecStart runs the serve script which
sources it); serve-script default and the deployer health-gate URL flipped in
lockstep so the post-restart health gate polls the new port. Docs, fleet table,
and both index bullets updated to match.
Prior docs claimed unknown tags silently serve llama3.2:1b. Live probes on the
live endpoint disprove it: /v1/chat/completions answers HTTP 200 with a flat
{"error":"Failed to load <tag> model!"} body and no choices; /v1/completions
answers a nested OpenAI-shaped server_error (code 500) under HTTP 200. Valid
tags route per-request to the named model (response echoes the requested tag);
no restart needed to switch between registry tags. Corrected in the runbook,
the AGENTS.md runtime section, the serve-script header comment, and the
historical FastFlowLM incident note.
The trio ran as bare compose containers and died silently at the Aug 26
reboot — nothing restarts compose containers at boot. Each now has a
boot-persistent unit (WantedBy=default.target; the v1 quadlet generator on
this host cannot enable generated units). The deploy handler retires the
compose-era containers only while the units are not yet active, then
(re)starts each unit.
Units run the image under a localhost/hermes-agent:latest alias tag (the
quadlet create path on this host cannot resolve registry-prefixed refs from
local storage); the deploy handler tags first. The gateway's :8090 publish
is dropped — nothing consumes it; the public vhost now routes to the
dashboard's login-gated UI on 9119.
The first deploy attempt pulled c3ed3a6 on the target but the remote runner
rejected the explicit --service selection because service_is_allowed() keeps
its own per-target allowlist; add the new LLM service to it.
New Dankclaw service frank-default-sightsound: FLM 1.0.1 serving the
Gemma4-E4B-IT-NPU2 pack (vision+audio+reasoning, 131072 ctx) on the XDNA2
NPU at port 52626. Model choice rationale: gemma4-it is the only NPU2 model
family with BOTH image and audio towers; the uncensored candidate qwythos:9b
is text-only, so sight+sound wins over uncensored here.
Deployer wiring: planner maps env/unit/serve files to the new service name;
deploy-remote gains deploy_llm_sightsound (relink + daemon-reload + enable +
restart + health gate on /v1/version — FLM has no /health route). Unlike the
retired frank-default-npu relink-only path, this unit is boot-enabled.
- same qwen3_5_moe arch and native block-fp8 format (e4m3, 128x128
weight_block_size, un-fused per-expert layout) as the official
Qwen FP8 checkpoint: identical FreeToken loader path, pool numbers
unchanged (2048 MoE slots / 453005 KV tokens / 4 requests)
- checkpoint downloaded to /home/dank/models/huihui-qwen36-35b-a3b-abliterated-fp8
(26 shards, ~35G); official checkpoint kept for rollback
- ships froggeric fixed Qwen3.6 chat template, read from checkpoint
- also refresh both .service Descriptions (large still said STRIX_LEAN)
- runtime: charlie build-qwen38 -> kingjones30/ROCmFPX (card recipe
build, LLAMA_BUILD_WEBUI=OFF since npm/HF-bucket provisioning fails;
all 35 of our serving flags verified present in the tree)
- model: agentionai FAST imatrix -> kingjones777 STRIX single GGUF
(on-host copy sha256 == HF LFS oid c6770d74...49fa verified)
- args unchanged from our validated profile (401408x2, b4096/ub512,
t16/tb16, f16 KV, fa on, fit off); mmap stays default per card
(PLE streams via page cache; --no-mmap = silent cgroup OOM)
fa on aborted graph build: build_attn_qsa called ggml_fill on the
F16-converted kq mask (GGML_ASSERT F32 at ggml.c:5371). Patch mirrors the
MLA sparse path in llm_graph_context::build_attn (F32 base mask for the
top-k surgery, F16 cast only at the attention call, gated on
cparams.flash_attn). Vendored at scripts/patches/qwen4exp-qsa-f32-mask.patch
(sha256 f2f4a486...); the build-qwen38 binary on Dankclaw is rebuilt with
it and validated live at CTX_SIZE=32768 (health 200, real completion).
- Main model (v2 joined PLE layout, sha256 1e4160ca…) loads and serves on
build-qwen38 only with --flash-attn off: the QSA indexer path calls
ggml_fill on the F16-cast kq mask and aborts graph build (GGML_ASSERT
F32 at ggml.c:5371). Verified live: model loaded, health 200, real
completion.
- MTP draft head (agentionai FAST, 2.28 GiB) does not load on this binary:
qwen4exp loader requires full per-layer blk.<il>.hc_attn_norm.weight,
head carries only blk.48. Third confirmation (EasiiX Q8_0 + agentionai
FAST vs kingjones STRIX_LEAN previously, now own-matched pair). Spec
decoding runs ngram-mod only; head retained for a future compatible build.
- FLASH_ATTN=off and SPEC_TYPE=ngram-mod wired through env + serve script;
runbook and AGENTS.md updated with the verified constraints.
Model: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF (87.06 GiB,
Q4_0_ROCMFP4_FAST imatrix-calibrated, per-head PLE table inside the GGUF —
fully VRAM-resident). Draft: agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST
(2.28 GiB, matched ROCmFP4-FAST quant, ~66% acceptance at n-max 3).
Vision tower (f16) loaded via mmproj/.
Runtime: charlie12345/ROCmFPX build-qwen38 (HIP, ROCm 7.2.4, gfx1151) —
the only binary with the ROCmFP4 FAST types. Replaces the jcbtc CIRU v1.1
deployment (ciru-ai runtime).
Profile: card sampling table (temp 0.7, top_p 0.8, top_k 20, min_p 0.0,
presence 1.5) mixed with the validated build-qwen38 serving profile:
262144 ctx x 1 slot, b 4096 / ub 4096, t 16 / tb 16, f16 KV, fa on,
cont-batching, prompt cache + 8 GiB cache-ram + idle slots, 16 ctx
checkpoints (no --checkpoint-min-step — unsupported on this binary),
--fit off (auto-fit segfaults on qwen4exp hyper-connection config),
--spec-type draft-mtp,ngram-mod with n-max 3 (adaptive p-split measured
worse per author card), p-min 0.75. Qwen-Sharp chat template kept;
reasoning stays in DeepSeek format.
The 2026-08-28 failure of this draft head on the kingjones STRIX_LEAN
model was a cross-vendor pairing problem: the head carries
blk.48.hc_attn_norm.weight (verified in-GGUF, same shape as the working
CIRU draft). Docs: new runbook agentionai-fast-imatrix, index updates.
Runtime: ciru-ai/Qwen3.8-Flash-CIRU-STRIX-IU4 llama.cpp at tag v1.1, built
with the vendor build-linux-amd.sh flag set plus GGML_CUDA_FA +
GGML_CUDA_FA_ALL_QUANTS (the Q4_1_MOE_FORCE_J path is live in the hipified
mmq kernels). Model: jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4 (Q4_1 IU4 experts,
MTP Q8_0 draft, CIRUPLE1 PLE sidecar) with checksums verified at download.
Vendor production profile verbatim on first deployment: 262144 ctx x 1 slot,
b 2048 / ub 512, t 8 / tb 8, f16 KV, fa on, cont-batching, prompt cache +
8 GiB cache-ram + idle slots, 32 ctx checkpoints (min step 8192), MTP depth
3 with p-split 0.10, PLE pager env knobs (Q41_MOE_FORCE_J=32,
PLE_WORKERS=16, PLE_STRICT_SHA=0, HIPBLASLT=1). Qwen-Sharp chat template
kept deliberately; reasoning stays in DeepSeek format.
The quadlet-dir allowlist silently dropped every change under
dankclaw/authentik/ — the new outpost unit (and its siblings) now route
by exact unit name, mirroring the searxng/gitea pattern. Adds
authentik-outpost to the service inventories; it is a quadlet unit, not
a compose service, so it stays out of the compose set.
In authentik 2026.8.x the proxy-provider URL space (/outpost.goauthentik.io/*)
is served by the Rust worker binary, not the server — verified live: with a
tenant-matching Host header, /if/flow/... answers 200 on the server while
/auth/caddy still 404s. The outpost instance existed in the DB but no process
ever served it, so Caddy's forward_auth subrequests hit a 404 and browsers got
authentik's own Not-Found page instead of the authentication flow.
The new unit runs the same image as server/worker with Exec=worker and flips
into outpost mode via AUTHENTIK_HOST/AUTHENTIK_INSECURE/AUTHENTIK_TOKEN (the
token is read from the DB into the host .env at deploy time; it never enters
the repo). It dials back to core over websocket and serves forward-auth on its
own 9000. Deploy handler restarts only this unit — the rest of the stack is
untouched.
Three compounding deploy-handler bugs found by live deployment:
- podman image exists/pull used a mixed-case ghcr.io ref that podman's
reference parser rejects before any network I/O (repository name must be
lowercase); all refs now lowercase — ghcr resolves case-insensitively.
- the unit runs the image under a localhost/ alias tag: this host's podman
5.7 quadlet path fails container create with 'image not known' for
registry-prefixed refs even when present in storage (verified live).
- the health probe was shell-form and the distroless ko-build image has no
shell; HealthCmd is now a JSON exec-form array, which passes through the
quadlet generator's argv assembly verbatim.
Tests updated to assert the alias-tag reality plus the pull guard.
On this host's podman 5.7 quadlet path, a registry-prefixed Image ref
fails to resolve from local storage at container create time even when
the image is present ('image not known'); the identical argv works from
a shell or transient unit. Verified live: an identical unit with
Image=localhost/techarohq/anubis:v1.13.0 starts immediately. The deploy
handler now tags the alias after the guarded pull; upgrades re-tag
after pulling the new tag.
image inspect on an absent reference can exit non-zero for reasons
unrelated to existence in a non-tty deploy context; image exists is
the boolean primitive and cannot misreport. Absent-image behavior
(pull from ghcr) is unchanged.
podman's reference parser rejects mixed-case repository refs before any
network I/O ('repository name must be uppercase'); upstream's own install
docs pull lowercase (ghcr resolves case-insensitively). The first deploy
of this chain exit-2'd on that parse error — the image was not yet
pulled, so the guarded vhosts 502 until this deploy lands.
The first deploy run dispatched caddy (hot reload, no restart) but exit-2'd
on the inventory guard before deploy_anubis could run: deploy.sh's
valid_service was extended but the remote runner's per-target inventory
was not. Guarded vhosts 502 until this lands.
Bots and AI scrapers hit the SSO-fronted vhosts and eat Authentik auth
work for 403s. Anubis verifies clients first: DENY for known AI scraper
UAs, JS challenge for everyone else; verified traffic resumes the normal
Authentik forward_auth chain on Caddy's internal listener (:8016).
Chain: client -> Caddy (TLS) -> Anubis :8923 (verify) -> Caddy :8016
(auth + routing resumes; Host survives both hops) -> app. Reverse-proxy
through Anubis, not forward_auth: at the pinned tag a CHALLENGE answers
HTTP 200 + HTML, and forward_auth treats 200 as "authenticated" — the
page would pass through unchallenged. Exempt by construction (never
routed through Anubis): plex (plain HTTP legacy), auth.* (IdP login
must be reachable ungated), gitea (git smart-HTTP clients cannot solve
JS challenges), and every app with its own login.
Pinned ghcr.io/TecharoHQ/anubis:v1.13.0 — the tag was source-verified
before pinning: at this tag the policy parser is JSON-only with a
minimal bots[] schema (no imports, no CEL expressions), flags are the
config surface via flagenv (TARGET/POLICY_FNAME map to --target/
--policy-fname), and the image's exec-form HEALTHCHECK is inherited by
the quadlet unit (dodges the v1 generator's HealthCmd space-split quirk).
Rollback nuance: stopping anubis 502s everything behind the chain (the
public listeners proxy through it); a true rollback is reverting the
Caddyfile and redeploying, not just systemctl --user stop anubis.
Probed the live generator on dankclaw: podman 5.x v1 hard-fails unit
generation on dash-prefixed options (--shm-size killed authentik-server's
unit) and rejects Command= entirely; Exec= is the accepted role selector.
Drop --shm-size (uwsgi uses unix sockets under /run, not /dev/shm; default
64m shm is fine) and drop systemctl --user enable — generated units are
transient and cannot be enabled; WantedBy=default.target in each unit file
is what makes them boot-persistent on this generator.
The compose-era authentik trio died at every boot (rootless podman does not
auto-start bare containers). Convert the trio to Quadlet units installed by
deploy-remote.sh into ~/.config/containers/systemd/, enabled + WantedBy=
default.target:
- 3 unit files under dankclaw/authentik/quadlet/ (postgres -> server -> worker
ordering via Wants=/After=; secrets stay in the git-ignored host .env,
expanded by systemd at start time exactly like the gitea units)
- deploy_authentik() handler: install units, daemon-reload, retire stale
compose-era containers only while units are inactive, enable once, then
start in dependency order with podman-health gates between stages
- compose.yaml: authentik service definitions retired to a comment pointing at
the quadlet dir (same pattern as searxng/open-webui/gitea/caddy)
- tests/authentik.bats rewritten from compose assertions to unit-file
contract assertions (images, Exec roles, ordering, loopback-only publish,
pg_isready probe, secret hygiene, dispatch routing)
All in-repo path references (AGENTS.md, README, old scripts, report artifacts,
test code paths) rewritten in the same commit; focused-test map extended for
.gitignore, old/scripts/*, dankclaw/comfyui/*, and report/* per this repo's
own convention ('add one before committing'). Obsidian wikilinks are
title-based and unaffected.
Both the EasiiX Q8_0 and agentionai ROCmFP4-FAST MTP draft heads fail
with 'missing tensor blk.0.hc_attn_norm.weight' against the STRIX_LEAN
Uncensored model. The qwen4exp binary expects this hyper-connection
attn norm tensor which neither draft GGUF carries. Using ngram-mod
only until a compatible MTP head is found.
The EasiiX Q8_0 draft was incompatible with STRIX_LEAN (missing
blk.0.hc_attn_norm.weight). Switched to agentionai/Qwen3.8-Flash-Next-
MTP-ROCmFP4-FAST-GGUF (2.28 GiB, ROCmFP4-FAST quant matched to target).
n_max 3 per the HF card (n_max 4 measured worse). SPEC_TYPE=draft-mtp,ngram-mod.
The script runs ON Dankclaw but was calling ssh() to SSH back to
dankclaw for service management. Nested SSH fails (no key auth for
localhost). Replaced ssh() with local() using subprocess.run(shell=True)
for systemctl and sed commands.
The env file had the old STRIX paths (directory doesn't exist on disk).
Updated to the actual running STRIX_LEAN (Uncensored) variant:
- MODEL_DIR → kingjones-Qwen3.8-Flash-Next-Uncensored-ROCmFP4-STRIX_LEAN
- MODEL_FILE → 3-shard Q4_0 (00001-of-00003)
- MMPROJ_PATH → mmproj-Qwen3.8-Flash-Next-Uncensored-BF16.gguf
- serve-frank-large.sh: add --mmproj $MMPROJ_PATH
Reconciles the uncommitted local changes that were on Dankclaw.
serve-frank-large.sh: only pass --model-draft when SPEC_TYPE contains
'draft-mtp'. ngram-only configs (ngram-mod, ngram-simple, etc.) do not
need a draft model file — loading one wastes VRAM and may error.
bench-spec-decode.py: new benchmark script. Sweeps 4 configs (none,
ngram-mod, draft-mtp, draft-mtp+ngram-mod), generates 16k tokens per
config at temp 0 from a minimal prompt. Uses HTTP API (llama-bench
cannot target --spec-type). Follows benchmarking policy: stop+mask
the production service, bench, unmask+restart, health check.
podman's quadlet generator creates the default.target.wants symlink from
the unit's [Install] section at every user-manager start (is-enabled
reports 'generated' for all healthy quadlets, pihole included). An
explicit 'systemctl --user enable caddy.service' fails with 'transient or
generated' and aborted the deploy after the unit was already installed.
systemd-sysctl applies /etc/sysctl.conf AFTER all of sysctl.d/, so the
stale ip_unprivileged_port_start=80 line there beat every drop-in and kept
the runtime value at 80 after the first deploy pass. deploy_pihole now
comments out an active line in /etc/sysctl.conf (idempotent) and generalizes
the sysctl.d conflict scan to any file setting a value above 53.
Caddy left compose.yaml for a quadlet unit (caddy.service): joins
dankclaw_default for container-name upstreams, Pull=never for the local-only
image, Porkbun credentials via /home/dank/Containers/caddy/env written by
deploy from .env, bare-IP-catch-all health check. deploy_caddy installs,
enables, and starts the unit after retiring the bare container (state is in
host volumes).
Pi-hole: the repo now owns the sysctl drop-in
(dankclaw/pihole/sysctl/99-dnsmasq.conf, ip_unprivileged_port_start=53);
deploy_pihole installs it to /etc/sysctl.d/, stages conflicting later-sorted
drop-ins in the home trash, and applies 'sysctl --system' (the boot code
path). This fixes the 2026-08-26 crash loop (port-53 bind denied, restart
counter 1645) caused by a second drop-in resetting the value to 80.
Tests: new caddy_quadlet.bats seam; caddyfile/deployment_flow assertions
repointed from the removed compose service to the quadlet; pihole.bats pins
the sysctl drop-in and its deployment; test-changed.sh routes dankclaw/caddy/
and the pihole subtree to their seams. Docs: Caddy, Pi-hole, Dankclaw Stack,
work log.
Frank