Mechanical identity rename across the knowledge base, AGENTS.md, README, the
hermes config example, and homepage services.json. No behavior change; served
aliases on the LLM endpoints change with the deploy of commit 69f87c8.
frank-default-large -> frank-large (Dankclaw :7770), frank-default-small ->
frank-small (Dankserver :7770), frank-default-sightsound -> frank-npu
(Dankclaw :7771). Units, env files, serve scripts, deploy-plan mappings,
deploy-remote dispatch (stop/disable old units + relink + enable new), and the
test seams move together. Retired frank-default-npu.* artifacts moved to
scripts/.AGENTTRASH/ (tracked-in-history; ignored on disk).
Served aliases become frank-large / frank-small via --alias; the NPU endpoint
serves the whole FLM registry, so its rename changes no served tags.
Post-change doc reconciliation for the hermes vhosts + DNS deletion batch:
- Hermes-Dashboard.md: access is now the hermesadmin.franksdank.cloud vhost
(direct Caddy proxy to 192.168.0.143:9119, no guard chain — exempt because the
dashboard enforces its own basic-auth login), plus the raw LAN port. The old
text claimed an Authentik-guarded public vhost, which was decommissioned.
- Hermes-WebUI.md: description updated — the webui is now also reachable via the
unguarded hermes.franksdank.cloud vhost (its own password auth is the gate).
- LAN-DNS.md: "add a name in two places" corrected to a two-or-three-step
decision; internal-only names (prometheus/loki/grafana/seerr) deliberately get
no public A record and resolve on the LAN only, and DNS-01 issuance is
unaffected because the challenge is a TXT record in the same zone.
- Seerr.md: DNS row corrected — addn-hosts only since 2026-09-02; NXDOMAIN on WAN.
- hermes-dashboard.container: unit comment no longer claims Authentik termination.
No behavior change in this commit; tests 233/233 green (docs-completeness seam included).
New page Architecture/Container Network Namespaces.md: which local services are
addressable from which Podman network namespace on Dankclaw (rootless podman +
netavark). Verified live 2026-09-02: gateway/webui/CDP run in the host netns and
dial loopback; hermes-dashboard is the only Hermes container on a bridge network
(dankclaw_default, 10.89.1.0/24 gw 10.89.1.1) and can dial published ports only —
unpublished host-netns listeners (signal-cli :8080, webui :8787) are refused from
that namespace even though both bind 0.0.0.0.
Corrected while verifying the above:
- Hermes.md: data root is /home/dank/Containers/hermes/data (not the retired
/home/dank/hermes-data); gateway dials the Signal daemon on loopback; the
daemon itself is a host user unit (signal-cli-daemon.service) bound to
0.0.0.0:8080, so the Signal HTTP API is LAN-reachable.
- Hermes-Dashboard.md: it is a Quadlet unit (Network=dankclaw_default), not a
compose service; data root + /opt/data anonymous volume corrected.
- Hermes-WebUI.md: binds 0.0.0.0:8787 in the host netns (verified: unauthenticated
GET /health from another LAN machine returned 200) — the SSH tunnel is an access
path, not a containment boundary; HealthCmd and data-path corrections.
- Network Architecture.md: no hermes vhost exists in either the tracked or the live
Caddyfile (hermes.franksdank.cloud resolves but routes nowhere); SearXNG exits the
tunnel via the shared named network searxng.network, not service:gluetun.
Docs only — deploy is a no-op for this change.
- tautulli.container: HealthCmd switches to the repo's proven curl pattern
(jellyfin unit) — quadlet's systemd-style value parsing mangles embedded
double quotes; PublishPort=8181:8181 restored (a mis-aimed line edit
replaced it; the PublishPort seam test caught the regression)
- deploy_plex readiness gate: 36x5s -> 120x5s; the 2026-09 cutover showed
the first container boot of a years-old library runs ~10 min while PMS
pre-binds 32400 and empty-replies; 180s failed the gate twice with a
healthy, still-booting service
Evidence from the failed first cutover: podman top shows container root
maps to dank (1000) under UserNS=host rootless (subuid range 100000:65536
cannot express 997), so PUID=997 fell back to root and PMS crash-looped on
write access to its 997:984-owned data root. Fleet model instead: PUID=0/
PGID=0 (container root = dank, same as jellyfin/copyparty data roots) and
deploy_plex chowns /var/lib/plexmediaserver to 1000:1000 inside the
first-cutover block. Zero bytes moved; rollback regains one chown -R
997:984 step, documented in Plex.md. Seam test pins PUID=0/PGID=0 and the
absence of any userns key.
podman 5.7 podman-user-generator on dankserver hard-fails the whole unit on
the unknown key ('unsupported key UsernsMode in group Container'), so the
first cutover attempt masked the native unit and then found no plex.service
at all. Swap both units to UserNS=host (same --userns=host flag), update the
ownership-contract seam test to pin the exact key, and correct both docs.
Pin the seam: the v1 generator rejects unknown Container keys silently at
deploy distance — the test now pins UserNS=, not the misspelled alias.
- runtime rows: podman quadlet under the systemd user generator, pinned
image digests, in-place bind mounts, identity model, appnet aliases
- cutover/rollback section on Plex.md (mask-native-unit lever, decommission
gate); drop the false 'updates are apt-only' and 'snap refresh' gotchas
- update path is now deploy.sh dankserver --service plex|tautulli
The loki/grafana/seerr proxy providers were script-provisioned with no
flows and no Application: null flows made the embedded outpost's Rust
deserializer reject the whole config (all guarded vhosts down after any
outpost reload), and the missing apps bounced post-login to authentik's
user UI instead of the app. Backfilled both live; documented the two
provisioning rules and updated the provider inventory. SSO e2e now
asserts the app landing (pathname leaves /if/, no IdP markers in body)
instead of only the hostname.
- qB 5.x's 204 login breaks Readarr's retired client at runtime; 4.6.7 answers
legacy 200+Ok. Pin recorded in the unit and documented.
- Prowlarr download client created via schema template; all five clients test
200 through the podman gateway.
- PBKDF2 gotcha: qB 5.x conf rewrite left a hash 4.6.7 rejects; fresh rewrite
of the same line restored login.
- E2E: torrent through the Panama tunnel at ~17 MB/s, completed into Finished
Torrents, deleted with data.
Document the Mac *arr/qBittorrent stack (VM placement, gluetun credential
gate, socat LaunchAgents, PBKDF2 qB seed) and log the day: dankserver
recovery + full quadlet cutover of the six media services, the frankbook
skip-propagation/PBKDF2 handler fix, the 7-service frankbook deploy, and
the jellyfin GPU-parity finding.
Phase 1/2 of the podman standardization plan: service pages updated to the
quadlet reality (units, Containers data roots, env-file secret convention,
the v1-generator findings each caught live), Storage Policy documents the
two rootless ownership regimes and the merge-move migration, Deployment
Flow's service-manager table no longer claims dankserver media is bare
inventory-only. Bundle change log entry added.
-v /w::A granted anonymous full rwmda on the LAN (verified: anon PUT -> 201,
PROPFIND -> 207). New perms ::r:ra,dank: browse/download for anonymous
(required by the guarded files.franksdank.cloud chain), admin for dank,
no w/m/d for anyone. Runbook corrected with the --help-accounts semantics.
Adds a digest-pinned headless-shell Chrome DevTools Protocol daemon as a
boot-persistent quadlet unit (chrome-devtools-protocol.container). The
entrypoint override drops the image's socat front so the CDP endpoint binds
loopback-only on 127.0.0.1:9222 in the host network namespace; only the two
host-network consumers (Signal gateway, webui server) can drive it.
hermes-webui flips Network=dankclaw_default -> Network=host so its agent
process shares the daemon's namespace (0.0.0.0:8787 bind is LAN-facing as
before; no exposure change). deploy_hermes installs the new unit and starts
it if inactive without restarting in-flight browser sessions.
Verified live 2026-08-31: guarded path serves the Anubis JS challenge at
HTTP 200 to non-browser API clients (status-only consumers silently get
HTML); direct published port 8081 from pass_ip sources returns real JSON.
Decision: no second agent-only instance — same engines and same
rate-sensitive VPN exit IP would see double request volume for zero gain.
Server-side implementation landed on Dankclaw: dedicated 'tunnel' account
(nologin shell, password-locked, key-only auth), sshd drop-in
99-tunnel.conf scoping AllowTcpForwarding to that user with PermitTTY off,
and permitopen="localhost:8787" key-option guidance so a leaked tunnel key
can only forward to the webui port. WebUI stays loopback-only published.
Token exchange chain verified end-to-end in-process on the live provider
(pk 9): authorize -> ORM-minted code -> form-encoded token exchange with
Basic client auth -> userinfo. Key findings documented:
- Application row + grant_types must exist before authorize can complete
- token endpoint consumes form-encoded bodies only (JSON body => no
grant_type parsed -> unsupported_grant_type); client auth is Basic-only
- PropertyMappingEvaluator.evaluate(*args,**kwargs) forwards kwargs to
BaseEvaluator.evaluate(expression_source), so context kwargs become the
expression source and every evaluation raises a swallowed exception;
userinfo therefore answers sub-only claims for every provider in this
fork. Scope granting filter (provider= M2M descriptor) is always empty.
- Grafana identity = sub hash until the evaluator bug is fixed upstream.
Env file is the source of truth (unit ExecStart runs the serve script which
sources it); serve-script default and the deployer health-gate URL flipped in
lockstep so the post-restart health gate polls the new port. Docs, fleet table,
and both index bullets updated to match.
Prior docs claimed unknown tags silently serve llama3.2:1b. Live probes on the
live endpoint disprove it: /v1/chat/completions answers HTTP 200 with a flat
{"error":"Failed to load <tag> model!"} body and no choices; /v1/completions
answers a nested OpenAI-shaped server_error (code 500) under HTTP 200. Valid
tags route per-request to the named model (response echoes the requested tag);
no restart needed to switch between registry tags. Corrected in the runbook,
the AGENTS.md runtime section, the serve-script header comment, and the
historical FastFlowLM incident note.
frank-default-sightsound (FLM v1.0.1, XDNA2 NPU, port 52626) is live on
Dankclaw. Documents the first-deploy crash-loop root cause: iommu=off on the
kernel cmdline breaks the XDNA2 SVA bind (amdxdna_drm_open ret -19), so every
/dev/accel/accel0 open fails with ENODEV; fixed with iommu=pt + reboot.
Whole-registry single-port serving with per-request routing by exact FLM tag;
unknown tags fall back to llama3.2:1b (no 404). Health = GET /v1/version.
Vision + audio towers proven live (image_url color probe; audio control-pair
token-count probe). Updates AGENTS.md fleet table + runtime section, both
index pages, and the change log.
- kingjones-strix-rocmfp4 reactivated as the ACTIVE large runbook:
runtime build recipe (WEBUI=OFF gotcha), deployed serving profile,
fa-on-works-unpatched evidence, MTP heads documented unloadable,
swap procedure + 2026-08-30 validation record
- agentionai-fast-imatrix marked RETIRED (rollback record kept)
- freetoken-frank-default-small: model/lineage/template sections now
Huihui abliterated FP8; new operational quirk: chat API 422 without
explicit model field
- LLM index + AGENTS fleet table/runtime sections current; both
services' history chains fixed (small was still Tiel in AGENTS.md)
- worklog entry added; models.json contextWindow for large corrected
to the real per-slot 131072 (llama clamps n_ctx_train/parallel)
Meta-header routing and forward_auth removed from the described design — both
disproved live. Documents: Host-based provider routing with per-guarded-host
proxy providers (lockstep rule), the null-flows wedge that starves the
embedded outpost config fetch, the Anubis /outpost.goauthentik.io/* transport
exemption, the single-file bind-mount stale-policy inode gotcha, Invalid next
URL as a dead design, and the public-DNS record requirement per guarded host.
fa on aborted graph build: build_attn_qsa called ggml_fill on the
F16-converted kq mask (GGML_ASSERT F32 at ggml.c:5371). Patch mirrors the
MLA sparse path in llm_graph_context::build_attn (F32 base mask for the
top-k surgery, F16 cast only at the attention call, gated on
cparams.flash_attn). Vendored at scripts/patches/qwen4exp-qsa-f32-mask.patch
(sha256 f2f4a486...); the build-qwen38 binary on Dankclaw is rebuilt with
it and validated live at CTX_SIZE=32768 (health 200, real completion).
- Main model (v2 joined PLE layout, sha256 1e4160ca…) loads and serves on
build-qwen38 only with --flash-attn off: the QSA indexer path calls
ggml_fill on the F16-cast kq mask and aborts graph build (GGML_ASSERT
F32 at ggml.c:5371). Verified live: model loaded, health 200, real
completion.
- MTP draft head (agentionai FAST, 2.28 GiB) does not load on this binary:
qwen4exp loader requires full per-layer blk.<il>.hc_attn_norm.weight,
head carries only blk.48. Third confirmation (EasiiX Q8_0 + agentionai
FAST vs kingjones STRIX_LEAN previously, now own-matched pair). Spec
decoding runs ngram-mod only; head retained for a future compatible build.
- FLASH_ATTN=off and SPEC_TYPE=ngram-mod wired through env + serve script;
runbook and AGENTS.md updated with the verified constraints.
Model: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF (87.06 GiB,
Q4_0_ROCMFP4_FAST imatrix-calibrated, per-head PLE table inside the GGUF —
fully VRAM-resident). Draft: agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST
(2.28 GiB, matched ROCmFP4-FAST quant, ~66% acceptance at n-max 3).
Vision tower (f16) loaded via mmproj/.
Runtime: charlie12345/ROCmFPX build-qwen38 (HIP, ROCm 7.2.4, gfx1151) —
the only binary with the ROCmFP4 FAST types. Replaces the jcbtc CIRU v1.1
deployment (ciru-ai runtime).
Profile: card sampling table (temp 0.7, top_p 0.8, top_k 20, min_p 0.0,
presence 1.5) mixed with the validated build-qwen38 serving profile:
262144 ctx x 1 slot, b 4096 / ub 4096, t 16 / tb 16, f16 KV, fa on,
cont-batching, prompt cache + 8 GiB cache-ram + idle slots, 16 ctx
checkpoints (no --checkpoint-min-step — unsupported on this binary),
--fit off (auto-fit segfaults on qwen4exp hyper-connection config),
--spec-type draft-mtp,ngram-mod with n-max 3 (adaptive p-split measured
worse per author card), p-min 0.75. Qwen-Sharp chat template kept;
reasoning stays in DeepSeek format.
The 2026-08-28 failure of this draft head on the kingjones STRIX_LEAN
model was a cross-vendor pairing problem: the head carries
blk.48.hc_attn_norm.weight (verified in-GGUF, same shape as the working
CIRU draft). Docs: new runbook agentionai-fast-imatrix, index updates.
All four arr roots live: /tv, /movies via API; /music, /books created after
v1 validator layers peeled back (quality/metadata profile ids must reference
existing profiles; v1 has no /api/v1/profile endpoint). Auth pitfall: the
literal AuthenticationRequired=Disabled string throws DryIoc.ScopedItemException
in v1 AuthorizationMiddleware on every request — the valid no-auth state is
ABSENCE of the key. Fixed live on four containers, verified boot clean.
Sonarr/Radarr root folders created via API (201). Lidarr/Readarr deferred to
first-run wizard (v1 validators demand quality-profile ids that don't exist
yet). Root cause of the write failures: LSIO PUID=1000 under rootless podman
maps to host uid 101000 — a phantom user owning nothing; PUID=0 maps to the
host user that owns every media dir.
Verified live: indexers->Prowlarr->per-arr QBittorrent download clients with
category routing. Readarr's client parked (qB proxy auth defect in abandoned
upstream). EZTV/1337x parked with reason (CloudFlare-blocked at this egress);
Nyaa/sukebei/RuTor/Byrutor/YTS passed live tests and are enabled. Prowlarr v2
Applications rows now carry real arr baseUrl+apiKeys; before this every sync
attempt threw NullConfig in GetIndexerMappings.
Verified live: AuthenticationMethod=None + AuthenticationRequired=Enabled on
all five arrs; unauthenticated API probes return 401 both locally and through
the vhosts. Also records the as-built divergences (entropy mounts, LA timezone,
readarr tag pin, prowlarr mount fix) that supersede the design compose.
Kestrel rejects the h2c upgrade preface; Caddy surfaced that as a 502 per
request while the backend answered 200 on plain HTTP/1.1 from Caddy's network.
Contrast recorded: Plex PMS needs the TLS transport, Jellyfin needs none.
The page claimed an env-var interface (COPAY_PASS/COPAY_MODE) and a /data
volume convention that this image does not implement: the ac edition consumes
accounts and volumes as argv flags (-a USER:PASS, -v SRC:DST:PERMS), serves
the LUKS array at /w, and answers HTTP 500 with its own error page when zero
volumes are configured. Image ref corrected to copyparty-ac (bare latest tag
does not exist on ghcr).
Live-verified 2026-08-29: plexmediaserver.service is an enabled native
systemd unit on Dankserver (not a quadlet; updates via apt, not deploy.sh),
and tautulli runs as a snap with no auth wall on 0.0.0.0:8181. The hunt list
named both; neither had a page. Index linked per the docs-completeness seam.
The policy must ALLOW Anubis's own endpoints before the challenge-all rule
or the challenge challenges itself: browsers POST-solve via make-challenge
then GET-navigate pass-challenge, and a challenged answer renders as the
browser's own fetch-failure error page. Non-JS clients never reach the
verify step, which is why every curl probe looked green while every human
hit a wall. Also records that v1.13.0's valid action verbs are
ALLOW/CHALLENGE/DENY (ACCEPT crash-looped the unit on first deploy).
The forward_auth target /outpost.goauthentik.io/auth/caddy does not exist
in authentik 2026.8 (verified live: 404 for every header combination; the
URL tree has no outpost auth prefix at all — the outpost worker dials back
to core over ws/outpost/<uuid>/ instead). Unauthenticated subrequests got
404, Caddy treated >=400 as 'not authenticated' and redirected browsers to
the dead path — users saw authentik's own Not-Found page.
Fix: guarded vhosts now proxy the whole request through authentik-server:9000
with X-authentik-meta-outpost/-provider/-app headers (the exact header set
this build's Traefik controller emits, adapted for Caddy). Unauthenticated
clients get the login flow served in place; once a session exists,
authentik proxies on to the provider's internal_host — now
http://caddy:8017, a new post-auth internal listener that routes straight to
the app. The chain terminates; nothing dials back into authentik.
Live DB changes applied alongside (postgres is the source of truth):
provider pk=1 authentication_flow=default-authentication-flow (was NULL —
the actual cause of the 404s) and internal_host=http://caddy:8017 (was
self-referential http://authentik-server:9000).
Frank