Verified live 2026-08-31: guarded path serves the Anubis JS challenge at
HTTP 200 to non-browser API clients (status-only consumers silently get
HTML); direct published port 8081 from pass_ip sources returns real JSON.
Decision: no second agent-only instance — same engines and same
rate-sensitive VPN exit IP would see double request volume for zero gain.
Server-side implementation landed on Dankclaw: dedicated 'tunnel' account
(nologin shell, password-locked, key-only auth), sshd drop-in
99-tunnel.conf scoping AllowTcpForwarding to that user with PermitTTY off,
and permitopen="localhost:8787" key-option guidance so a leaked tunnel key
can only forward to the webui port. WebUI stays loopback-only published.
Token exchange chain verified end-to-end in-process on the live provider
(pk 9): authorize -> ORM-minted code -> form-encoded token exchange with
Basic client auth -> userinfo. Key findings documented:
- Application row + grant_types must exist before authorize can complete
- token endpoint consumes form-encoded bodies only (JSON body => no
grant_type parsed -> unsupported_grant_type); client auth is Basic-only
- PropertyMappingEvaluator.evaluate(*args,**kwargs) forwards kwargs to
BaseEvaluator.evaluate(expression_source), so context kwargs become the
expression source and every evaluation raises a swallowed exception;
userinfo therefore answers sub-only claims for every provider in this
fork. Scope granting filter (provider= M2M descriptor) is always empty.
- Grafana identity = sub hash until the evaluator bug is fixed upstream.
Env file is the source of truth (unit ExecStart runs the serve script which
sources it); serve-script default and the deployer health-gate URL flipped in
lockstep so the post-restart health gate polls the new port. Docs, fleet table,
and both index bullets updated to match.
Prior docs claimed unknown tags silently serve llama3.2:1b. Live probes on the
live endpoint disprove it: /v1/chat/completions answers HTTP 200 with a flat
{"error":"Failed to load <tag> model!"} body and no choices; /v1/completions
answers a nested OpenAI-shaped server_error (code 500) under HTTP 200. Valid
tags route per-request to the named model (response echoes the requested tag);
no restart needed to switch between registry tags. Corrected in the runbook,
the AGENTS.md runtime section, the serve-script header comment, and the
historical FastFlowLM incident note.
frank-default-sightsound (FLM v1.0.1, XDNA2 NPU, port 52626) is live on
Dankclaw. Documents the first-deploy crash-loop root cause: iommu=off on the
kernel cmdline breaks the XDNA2 SVA bind (amdxdna_drm_open ret -19), so every
/dev/accel/accel0 open fails with ENODEV; fixed with iommu=pt + reboot.
Whole-registry single-port serving with per-request routing by exact FLM tag;
unknown tags fall back to llama3.2:1b (no 404). Health = GET /v1/version.
Vision + audio towers proven live (image_url color probe; audio control-pair
token-count probe). Updates AGENTS.md fleet table + runtime section, both
index pages, and the change log.
- kingjones-strix-rocmfp4 reactivated as the ACTIVE large runbook:
runtime build recipe (WEBUI=OFF gotcha), deployed serving profile,
fa-on-works-unpatched evidence, MTP heads documented unloadable,
swap procedure + 2026-08-30 validation record
- agentionai-fast-imatrix marked RETIRED (rollback record kept)
- freetoken-frank-default-small: model/lineage/template sections now
Huihui abliterated FP8; new operational quirk: chat API 422 without
explicit model field
- LLM index + AGENTS fleet table/runtime sections current; both
services' history chains fixed (small was still Tiel in AGENTS.md)
- worklog entry added; models.json contextWindow for large corrected
to the real per-slot 131072 (llama clamps n_ctx_train/parallel)
Meta-header routing and forward_auth removed from the described design — both
disproved live. Documents: Host-based provider routing with per-guarded-host
proxy providers (lockstep rule), the null-flows wedge that starves the
embedded outpost config fetch, the Anubis /outpost.goauthentik.io/* transport
exemption, the single-file bind-mount stale-policy inode gotcha, Invalid next
URL as a dead design, and the public-DNS record requirement per guarded host.
fa on aborted graph build: build_attn_qsa called ggml_fill on the
F16-converted kq mask (GGML_ASSERT F32 at ggml.c:5371). Patch mirrors the
MLA sparse path in llm_graph_context::build_attn (F32 base mask for the
top-k surgery, F16 cast only at the attention call, gated on
cparams.flash_attn). Vendored at scripts/patches/qwen4exp-qsa-f32-mask.patch
(sha256 f2f4a486...); the build-qwen38 binary on Dankclaw is rebuilt with
it and validated live at CTX_SIZE=32768 (health 200, real completion).
- Main model (v2 joined PLE layout, sha256 1e4160ca…) loads and serves on
build-qwen38 only with --flash-attn off: the QSA indexer path calls
ggml_fill on the F16-cast kq mask and aborts graph build (GGML_ASSERT
F32 at ggml.c:5371). Verified live: model loaded, health 200, real
completion.
- MTP draft head (agentionai FAST, 2.28 GiB) does not load on this binary:
qwen4exp loader requires full per-layer blk.<il>.hc_attn_norm.weight,
head carries only blk.48. Third confirmation (EasiiX Q8_0 + agentionai
FAST vs kingjones STRIX_LEAN previously, now own-matched pair). Spec
decoding runs ngram-mod only; head retained for a future compatible build.
- FLASH_ATTN=off and SPEC_TYPE=ngram-mod wired through env + serve script;
runbook and AGENTS.md updated with the verified constraints.
Model: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF (87.06 GiB,
Q4_0_ROCMFP4_FAST imatrix-calibrated, per-head PLE table inside the GGUF —
fully VRAM-resident). Draft: agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST
(2.28 GiB, matched ROCmFP4-FAST quant, ~66% acceptance at n-max 3).
Vision tower (f16) loaded via mmproj/.
Runtime: charlie12345/ROCmFPX build-qwen38 (HIP, ROCm 7.2.4, gfx1151) —
the only binary with the ROCmFP4 FAST types. Replaces the jcbtc CIRU v1.1
deployment (ciru-ai runtime).
Profile: card sampling table (temp 0.7, top_p 0.8, top_k 20, min_p 0.0,
presence 1.5) mixed with the validated build-qwen38 serving profile:
262144 ctx x 1 slot, b 4096 / ub 4096, t 16 / tb 16, f16 KV, fa on,
cont-batching, prompt cache + 8 GiB cache-ram + idle slots, 16 ctx
checkpoints (no --checkpoint-min-step — unsupported on this binary),
--fit off (auto-fit segfaults on qwen4exp hyper-connection config),
--spec-type draft-mtp,ngram-mod with n-max 3 (adaptive p-split measured
worse per author card), p-min 0.75. Qwen-Sharp chat template kept;
reasoning stays in DeepSeek format.
The 2026-08-28 failure of this draft head on the kingjones STRIX_LEAN
model was a cross-vendor pairing problem: the head carries
blk.48.hc_attn_norm.weight (verified in-GGUF, same shape as the working
CIRU draft). Docs: new runbook agentionai-fast-imatrix, index updates.
All four arr roots live: /tv, /movies via API; /music, /books created after
v1 validator layers peeled back (quality/metadata profile ids must reference
existing profiles; v1 has no /api/v1/profile endpoint). Auth pitfall: the
literal AuthenticationRequired=Disabled string throws DryIoc.ScopedItemException
in v1 AuthorizationMiddleware on every request — the valid no-auth state is
ABSENCE of the key. Fixed live on four containers, verified boot clean.
Sonarr/Radarr root folders created via API (201). Lidarr/Readarr deferred to
first-run wizard (v1 validators demand quality-profile ids that don't exist
yet). Root cause of the write failures: LSIO PUID=1000 under rootless podman
maps to host uid 101000 — a phantom user owning nothing; PUID=0 maps to the
host user that owns every media dir.
Verified live: indexers->Prowlarr->per-arr QBittorrent download clients with
category routing. Readarr's client parked (qB proxy auth defect in abandoned
upstream). EZTV/1337x parked with reason (CloudFlare-blocked at this egress);
Nyaa/sukebei/RuTor/Byrutor/YTS passed live tests and are enabled. Prowlarr v2
Applications rows now carry real arr baseUrl+apiKeys; before this every sync
attempt threw NullConfig in GetIndexerMappings.
Verified live: AuthenticationMethod=None + AuthenticationRequired=Enabled on
all five arrs; unauthenticated API probes return 401 both locally and through
the vhosts. Also records the as-built divergences (entropy mounts, LA timezone,
readarr tag pin, prowlarr mount fix) that supersede the design compose.
Kestrel rejects the h2c upgrade preface; Caddy surfaced that as a 502 per
request while the backend answered 200 on plain HTTP/1.1 from Caddy's network.
Contrast recorded: Plex PMS needs the TLS transport, Jellyfin needs none.
The page claimed an env-var interface (COPAY_PASS/COPAY_MODE) and a /data
volume convention that this image does not implement: the ac edition consumes
accounts and volumes as argv flags (-a USER:PASS, -v SRC:DST:PERMS), serves
the LUKS array at /w, and answers HTTP 500 with its own error page when zero
volumes are configured. Image ref corrected to copyparty-ac (bare latest tag
does not exist on ghcr).
Live-verified 2026-08-29: plexmediaserver.service is an enabled native
systemd unit on Dankserver (not a quadlet; updates via apt, not deploy.sh),
and tautulli runs as a snap with no auth wall on 0.0.0.0:8181. The hunt list
named both; neither had a page. Index linked per the docs-completeness seam.
The policy must ALLOW Anubis's own endpoints before the challenge-all rule
or the challenge challenges itself: browsers POST-solve via make-challenge
then GET-navigate pass-challenge, and a challenged answer renders as the
browser's own fetch-failure error page. Non-JS clients never reach the
verify step, which is why every curl probe looked green while every human
hit a wall. Also records that v1.13.0's valid action verbs are
ALLOW/CHALLENGE/DENY (ACCEPT crash-looped the unit on first deploy).
The forward_auth target /outpost.goauthentik.io/auth/caddy does not exist
in authentik 2026.8 (verified live: 404 for every header combination; the
URL tree has no outpost auth prefix at all — the outpost worker dials back
to core over ws/outpost/<uuid>/ instead). Unauthenticated subrequests got
404, Caddy treated >=400 as 'not authenticated' and redirected browsers to
the dead path — users saw authentik's own Not-Found page.
Fix: guarded vhosts now proxy the whole request through authentik-server:9000
with X-authentik-meta-outpost/-provider/-app headers (the exact header set
this build's Traefik controller emits, adapted for Caddy). Unauthenticated
clients get the login flow served in place; once a session exists,
authentik proxies on to the provider's internal_host — now
http://caddy:8017, a new post-auth internal listener that routes straight to
the app. The chain terminates; nothing dials back into authentik.
Live DB changes applied alongside (postgres is the source of truth):
provider pk=1 authentication_flow=default-authentication-flow (was NULL —
the actual cause of the 404s) and internal_host=http://caddy:8017 (was
self-referential http://authentik-server:9000).
Replaces the pre-Anubis direct-routing diagram with the live chain:
Caddy TLS -> Anubis :8923 verify -> accepted traffic back to Caddy's
internal listener :8016, where per-vhost routing resumes and Authentik's
embedded outpost gates wrapped apps via forward_auth. Guarded set
(homepage, search, hermes, prometheus, romm, files) and the
exempt-by-construction set (auth, plex, gitea git paths, grafana) are
listed explicitly. Access Policy bullet updated: basic auth was replaced
by the challenge-then-forward_auth layering.
Three compounding deploy-handler bugs found by live deployment:
- podman image exists/pull used a mixed-case ghcr.io ref that podman's
reference parser rejects before any network I/O (repository name must be
lowercase); all refs now lowercase — ghcr resolves case-insensitively.
- the unit runs the image under a localhost/ alias tag: this host's podman
5.7 quadlet path fails container create with 'image not known' for
registry-prefixed refs even when present in storage (verified live).
- the health probe was shell-form and the distroless ko-build image has no
shell; HealthCmd is now a JSON exec-form array, which passes through the
quadlet generator's argv assembly verbatim.
Tests updated to assert the alias-tag reality plus the pull guard.
All in-repo path references (AGENTS.md, README, old scripts, report artifacts,
test code paths) rewritten in the same commit; focused-test map extended for
.gitignore, old/scripts/*, dankclaw/comfyui/*, and report/* per this repo's
own convention ('add one before committing'). Obsidian wikilinks are
title-based and unaffected.
Frank