command-center/agent/memories/--home-dank-Projects-serverAppConfigs--/learned.md

• Making a Hermes auxiliary call work against a LOCAL OpenAI-compatible backend on this fleet (solved for title generation on 2026-09-11). Symptom: every session logs "Title generation failed" and falls back to a message-derived title. Root cause chain, in the order worth checking: (1) Hermes hardcodes response_format: json_schema on the title call (agent/title_generator.py _TITLE_RESPONSE_FORMAT), and NO local backend implements constrained decoding — frank-small (FreeToken) and frank-large (halogen) both answer 400 response_format json_object/json_schema is not supported (no constrained decoding); (2) config CANNOT override it, because call_llm merges effective_extra_body = _get_task_extra_body(task); effective_extra_body.update(extra_body or {}) — the caller's dict wins; (3) only temperature and max_tokens have reactive retry paths (_is_unsupported_parameter_error is generic but is only wired to those two); (4) auxiliary.task.enabled: false does NOT avoid the failure cleanly — it sits before apply_instant_title, so it removes titles entirely. Working configuration, verified through the real code path: auxiliary.title_generation with api_mode: codex_responses (a real per-task config key, _resolve_ta[REDACTED] reads it) plus extra_body.reasoning.effort: none. The Responses transport builds resp_kwargs and never forwards response_format (it also omits max_tokens/temperature); the reasoning half is mandatory, not decoration — with thinking on, the 64-token cap is spent on the think and content comes back empty with finish_reason=length. reasoning.effort: none maps to {"effort":"none","summary":"auto"} + include, which FreeToken's /v1/responses honors (4 output tokens, 1.5 s). Note reasoning_effort: none as a shorthand yields {"enabled": False}, which the adapter treats as "omit reasoning" — leaving the server default (thinking ON) and reintroducing the empty-content failure; use the explicit extra_body.reasoning.effort form. Both local runtimes also expose /v1/respon (context: Hermes auto-titling on Dankclaw after frank-npu was retired (2026-09-11): the NPU had been the only non-reasoning endpoint, so the title route fell to frank-small and failed on structured output. Verified with in-container probes calling agent.auxiliary_client.call_llm with title_generator's own constants; final config proven end to end (raw {"title": "Friendly greeting"} - 'Friendly greeting'), c)
• Compose→Quadlet cutover traps on this fleet (learned converting the five Dankclaw observability containers on 2026-09-11). (1) A unit's HealthCmd reaches podman as CMD-SHELL (/bin/sh -c). In a distroless image that silently produces a permanently-unhealthy container: the loki image has no shell, so the busybox command that worked by hand failed as a healthcheck, and podman does NOT accept exec form through --health-cmd (a JSON array is stored as one literal argument). The command has to live in the IMAGE as HEALTHCHECK ... CMD ["binary","arg"], which also means the build must pass --format docker — the default OCI image format has no HEALTHCHECK field and discards it with a warning. (2) Pointing a unit's EnvironmentFile at a repo .env is unsafe: this repo's ROOT .env is a gluetun env file that carries HOME=/root, so systemd sets HOME for the service, podman looks for its config under /root, and the unit dies with stat /root/.config: permission denied. The fleet convention is a per-service file (/home/dank/Containers/svc/env) seeded by the deploy handler with a single-variable awk filter, validated, so a missing secret fails the deploy loudly instead of starting a container whose login cannot complete. (3) For exposure, binding published ports to 127.0.0.1 is sufficient and safe: Caddy reaches containers by name over dankclaw_default and the host-side homepage status timer (/srv/homepage/services.json probes → homepage-status.py) probes loopback, so nothing legitimate needed the 0.0.0.0 publish. The exception is a Network=host exporter that a containerized Prometheus must scrape — it needs the LAN IP (containers cannot reach the host's loopback), so it binds loopback + LAN explicitly and is the only deliberate LAN listener. (4) Convert with matching ContainerName= so Caddy/scrape configs that address containers by name keep working, keep Pull=never for locally built images (loki/alloy), and delete the compose definitions in the same change so only one owner exist (context: Converting prometheus/grafana/loki/alloy/node-exporter from dankclaw/compose.yaml services to rootless Quadlet units (repo commits dab81b9, fe4eeaa, b96359e, 25c9398, 701ab14), including the fixes found by deploying: the grafana HOME=/root failure, the loki CMD-SHELL/distroless healthcheck failure, and the OCI-vs-docker image format that dropped the HEALTHCHECK.)
• Retiring a repo-managed root systemd unit on Dankclaw/Dankserver (the fleet installs them as symlinks: relink_unit does ln -sfn /home/dank/serverAppConfigs/scripts/svc.service /etc/systemd/system/svc.service): systemctl mask svc FAILS with "File '/etc/systemd/system/svc.service' already exists and is a symlink to …" — mask refuses to replace an existing unit file. Correct order, and it also avoids the documented disable-deletes-the-unit-file quirk: (1) systemctl stop, (2) systemctl disable (may or may not delete the file on these hosts — tolerate both), (3) if the file/symlink still exists, mv it into ~/.AGENTTRASH/dated-topic/host-units/ (never rm), (4) mv the /etc/systemd/system/multi-user.target.wants/svc.service enable symlink too or it dangles, (5) systemctl daemon-reload, (6) now systemctl mask svc succeeds and prints "Unit … does not exist, proceeding anyway" then creates the /dev/null symlink. Also: moving the repo file to .AGENTTRASH before the host symlink is cleaned leaves a dangling unit, so do the host side in the same change as the repo deletion. (context: Retiring the frank-npu systemd unit on Dankclaw as part of the 2026-09-11 frank-large swap; deploy-remote.sh's relink_unit installs root units as symlinks into the repo checkout.)
• peonist-ai/halogen-flash-server on Dankclaw (frank-large, deployed 2026-09-11) — traps worth keeping: (1) The podman 5.x quadlet v1 generator accepts every key this image needs (AddDevice, PublishPort, PodmanArgs=--ipc=host --security-opt seccomp=unconfined, Ulimit=memlock=-1:-1, HealthCmd, Pull=never) — verified with a throwaway unit before shipping; the generated ExecStart also carries --replace --rm, so every restart discards the container layer (fine when state is on a bind mount). (2) HALOGEN_DOWNLOAD fires only when the checkpoint is absent and requires a WRITABLE /models; the whole 127 GB repo fetched in 542 s (~240 MB/s) anonymously, no token needed. Start-to-health was ~12 s (engine PONG) when the weights were still in page cache, ~50 s after a cold reboot. (3) The startup ledger is the only truthful memory source: 68.0 GiB weights locked + 7.2 GiB KV pool (262144 positions) + 21.1 GiB working set, and free/MemAvailable overstate available memory by the model size because locked pages count as reclaimable cache — Grafana dashboards will be wrong by ~68 GiB. It also detected the host's 2.0 GiB BIOS iGPU carve-out. (4) Serving contract gotchas: no server-side sampler defaults (omitted temperature = GREEDY), the per-request budget defaults to 8192 and INCLUDES thinking (so a reasoning model + small budget returns empty content with finish_reason=length), the cap is 65536, and response_format is a hard 400 ("no constrained decoding"). (5) That last point matters fleet-wide: FreeToken (frank-small) refuses response_format identically, so Hermes's auto-title call (max_tokens=64 + response_format json_schema) fails against BOTH living local backends and falls back to its message-derived title. Any aux route that needs constrained decoding or a tiny budget needs an endpoint with thinking off and grammar support; the retired FLM NPU was that endpoint. (context: Full frank-large runtime+model swap on Dankclaw 2026-09-11: llama.cpp/kingjones STRIX - halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next w4b, as a rootless Quadlet container on port 7770, plus the amd_iommu=off kernel change that retired the frank-npu endpoint. Repo commit 301914b; runbooks serverAppConfigs-docs/LLM/halogen-flash-server.md and dankclaw-iommu-translation-off.md.)
• halogen-flash-server (github.com/peonist-ai/halogen-flash-server) is an OCI-image-only inference server for gfx1151 Strix Halo, evaluated 2026-09-11 as a replacement for the llama.cpp frank-large unit on Dankclaw. Verified facts: - Image ghcr.io/peonist-ai/halogen-flash-server:0.5.8 (latest CHANGELOG tag). Ports: OpenAI-compatible 8731 (publish this), engine 8730 (NO auth — keep unpublished/loopback only). - deploy/entrypoint.sh modes: all (default; both roles in one container), engine, api (split so front-end restarts don't reload the model), bench [drafters] [max_tokens] [effort] [reps]. Engine binary is flash_serve, CLI --ck FILE (not --checkpoint), --slots/--ctx/--max-tok. - Weights: HF peonist-ai/halogen-qwen3.8-flash-next (public, ungated) — qwen38-flash-next-w4b.hgn 115.55 GiB + ck.overlay.hgn 2.31 GiB quality sidecar (auto-picked up; engine warns if absent) + tokenizer/ (8 files). Optional vision.hgn 0.84 GiB, overlay-speed.hgn (trades o_proj calibration). Container-side download via HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next. - Run requirements: --device /dev/kfd --device /dev/dri --group-add keep-groups --security-opt seccomp=unconfined --ipc=host, memlock unlimited, native Linux only (WSL2//dev/dxg unsupported), gfx1151 only (hard-rejected else). Reference SKU = Ryzen AI Max+ 395 / 128 GB — exactly Dankclaw, so the baked HALOGEN_MATMUL_TUNING_FILE does not warn. - Key flags (env-only, no config file, read once at startup): HALOGEN_CHECKPOINT (default /models/qwen38-flash-next-w4b.hgn), HALOGEN_CK_OVERLAY, HALOGEN_MODEL_ID (default halogen-qwen3.8-flash-next), HALOGEN_DRAFTER_DEFAULT=1 (MTP speculation on by default, byte-identical at temp 0), HALOGEN_PROMPT_CACHE=2 (numeric; 1 = byte-identical resume), HALOGEN_CACHE_ENTRIES=8, HALOGEN_MAX_TOKENS_CAP=65536, HALOGEN_CTX=262144, HALOGEN_KV_POOL_POSITIONS (default 524288 = 35.0 GB; 262144 = 27.8 GB), HALOGEN_KV_SLOTS=4, HALOGEN_KV_POOL_FIT=1, HALOGEN_HOST_RESERVE_GIB=20, HALOGEN_MAX_TOK= (context: Research phase of a grilling session: replacing both the model and runtime of frank-large (Dankclaw) with halogen-flash-server, requested by Frank 2026-09-11. All facts read from the upstream repo (README, docs/FLAGS.md, deploy/entrypoint.sh, docker-compose.yml, CHANGELOG), the HF weights repo API, the ghcr image manifest, and live probes of Dankclaw.)
• On Dankclaw's Hermes stack, hermes-webui ran with the image-baked HERMES_HOME=/opt/data while the gateway used /data; /opt/data is an auto-created anonymous volume at the image's VOLUME point, so every container recreation (restart/redeploy) reset the webui's config home to the image-default template (provider: auto → openrouter.ai with no API key) — this is why model renames "didn't propagate" and chat failed with connect errors. Fix (2026-09-05, commit 184df9a): Environment=HERMES_HOME=/data in the hermes-webui quadlet so webui + gateway share one config/state root. Diagnostic order for "can't access models" on hermes: (1) check which config home each runtime actually reads (podman exec c printenv HERMES_HOME), (2) resolve provider→endpoint with the app's own code (hermes_cli.providers.resolve_user_provider(name, cfg["providers"]) accepts api/url/base_url keys on providers: dict entries), (3) probe the served endpoint with the exact model id. Model ids must match served aliases exactly (frank-large @192.168.0.143:7770, frank-small @192.168.0.235:7770, gemma4-it:e4b @192.168.0.143:7771 — FLM routes only exact registry tags).
• Hermes (Dankclaw) model routing as of 2026-09-05: main default = frank-large (Dankclaw 192.168.0.143:7770/v1), small/aux routes = frank-small (Dankserver 192.168.0.235:7770/v1), vision + title_generation route to the exact FLM registry tag gemma4-it:e4b (192.168.0.143:7771/v1). The pre-rename aliases frank-default-large/-small/-sightsound are dead since the 2026-09-03 rename cutover — never route on them. Live config lives at /home/dank/Containers/hermes/data/config.yaml (owned by uid 109999; needs sudo to read/edit); the repo's dankclaw/hermes/config.yaml.example is a sanitized backup and can lag live, so diff live first before editing either. Signal adapter is currently disabled in live config (platforms.signal.enabled: false).
• On the guarded franksdank.cloud vhosts (Caddy → Anubis :8923 → Authentik outpost), every unauthenticated request — roots and deep paths alike — is answered with HTTP 200 and an HTML challenge page (titleMaking sure you're not a bot!/title), not a 4xx. Verifying guard health by status code alone is meaningless; assert on leakage markers instead (copyparty UI title, homepage id="grid", services.json "category" JSON shape). Non-JS clients (curl/wget/WebDAV) can never solve the PoW challenge, so they are permanently parked at the challenge page — this is the entire security boundary for copyparty, which serves authenticated traffic with anonymous read perms (-v /w::r:ra,dank). Pinned by tests/e2e/specs/15-unauth-fetch.spec.ts (guest project, empty storage state): any guard-route regression that lets app bytes answer fails loudly; any gate implementation passes.
• On Dankclaw's Pi-hole (FTL v6.7, quadlet at /home/dank/Containers/pihole), /etc/pihole/hosts/custom.list (host path /home/dank/Containers/pihole/etc-pihole/hosts/custom.list) is a LIVE-ONLY Custom-DNS source NOT tracked in the repo — it held ~22 A records pointing every *.franksdank.cloud vhost at 192.168.0.143, including names with no public Porkbun record. gravity.db's domainlist table has zero custom-DNS rows on this build, and the pihole CLI's sqlite path (pihole-FTL sqlite3) works but the web API auth was failing at discovery time, so retiring a name from LAN DNS requires editing custom.list directly (sudo sed -i '/name/d') followed by podman exec pihole pihole restartdns reload; deleting only the repo-managed addn-hosts record is not sufficient. Verify with dig +short name @192.168.0.143 → empty answer.
• Pi-hole admin panel at pihole.franksdank.cloud has two auth layers: Caddy basicauth (edge, username dankuser + hash) and Pi-hole internal password (CivetWeb, set via pihole setpassword). The Pi-hole internal password was the actual failure point, not the Caddy hash. Pi-hole resolves to 192.168.0.143 via addn-hosts (all 28 names point here), making it a LAN service behind Caddy's public TLS edge. Options to remove public exposure: (1) remove the Caddy vhost entirely — LAN clients hit Pi-hole quadlet directly on :8083; (2) keep Caddy vhost but remove the Porkbun A record so external DNS won't resolve pihole.franksdank.cloud. Current state: Caddy basicauth hash = $2b$14$cJm.hbvDi1dEh7x0XRMnseaIdvIrRyH.AcTL56jGJf3wd54qvEb9K (dankuser, password pihole2026); Pi-hole internal password = pihole.2026 (set via pihole setpassword, survives restart).
• Pi-hole admin at pihole.franksdank.cloud has two auth layers: (1) Caddy basicauth edge (username dankuser, hash in rendered Caddyfile from CADDY_BASICAUTH_HASH in .env), (2) Pi-hole internal CivetWeb password (stored in cli_pw as bcrypt hash, set via pihole setpassword CLI). The env file WEBPASSWORD is used at container start but the CLI hash is what the running web server validates. When "I can't get into Pi-hole admin" returns 403 even after Caddy basicauth passes, the Pi-hole internal password is the likely culprit. Fix: podman exec pihole pihole setpassword newpw to update the CLI hash, then verify podman exec pihole cat /etc/pihole/cli_pw shows updated value. Verify by curling with basicauth: curl -sk -H 'Authorization: Basic ZGFua3VzZXI6...' https://pihole.franksdank.cloud/admin/ should return 302 (not 401 or 403).
• Pi-hole admin panel at pihole.franksdank.cloud uses Caddy basicauth (not Pi-hole's own CivetWeb login). The hash is shared across services (RomM, Copyparty, Pi-hole) via CADDY_BASICAUTH_HASH in the host .env. The rendered Caddyfile at /home/dank/Containers/caddy/Caddyfile is mounted into the Caddy container — updates are picked up by Caddy's file watcher without restart. Password reset: generate bcrypt hash with python3 + bcrypt (hashpw + gensalt), update the rendered Caddyfile line directly or update .env + deploy. Caddy accepts both 2a and 2b bcrypt variants. (context: Pi-hole admin password reset session: discovered Caddy basicauth (not Anubis), could not find matching password in common candidates, reset to pihole2026, updated rendered Caddyfile directly, synced repo .env files, Caddy file watcher handled hot-reload.)
• On the FrankBook-Pro arr fleet (current lsio images: Sonarr v4/Radarr v5 API v3, Lidarr v2/Readarr API v1), the download-client validation endpoints are NOT usable via API: POST /api/{v}/downloadclient/test/{id} and .../test?forceTest=true both return 405, and POST .../downloadclient/action/getTest exists but 400s without a bespoke body. To prove download-client health after changing a qB host/port, skip the API test route and prove behaviorally instead: podman exec arr curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8080/ (expect 200) plus a real qB login from inside the container (curl --data-urlencode username=admin --data-urlencode password=$QBIT_PW .../api/v2/auth/login, expect 200), then confirm /api/{v}/health shows no DownloadClient-/wbrrelated warnings. Also: arr APIs redact stored client passwords as ******** (sonarr/radarr/lidarr) — PUT the object back with the sentinel untouched to preserve the stored value; readarr (v1) returns the plaintext. When editing via the edit tool, body rows must be prefixed with +, not : — : prefixes get written literally into the file. (context: Capture turn after the arr→qB loopback repoint (2026-09-04): app-side download-client test endpoints were unreachable, so client health was proven behaviorally instead.)
• The *arr fleet (Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) runs ONLY on FrankBook-Pro (192.168.0.112, Mac podman machine) since 2026-09-04 — the dankserver compose copy was removed (config roots in ~/.AGENTTRASH/20260904-dankserver-arr-teardown, 22 MB, awaiting purge approval). Fleet invariant, enforced by tests/frankbook_quadlet.bats: ALL six containers share gluetun's netns (Network=gluetun.container, Wants/After gluetun.service) and publish NO ports; the gluetun unit publishes the union (8080, 6881 tcp+udp, 9696, 8989, 7878, 8686, 8787), so all egress exits ProtonVPN by construction (AllowedIPs=0.0.0.0/0; proven: netns egress 51.179.177.14 vs residential 71.227.227.172). deploy-plan.py couples gluetun → all six joiners (a gluetun recreate kills the whole family). Consequences: (1) the old arr→qB addressing hack 10.88.0.1:8080 (podman-bridge gateway) routes INTO the tunnel from inside the netns — download clients must use 127.0.0.1:8080 (done via arr APIs, verified by real qB login from inside sonarr's container); same rule for any future Prowlarr Application rows (baseUrl 127.0.0.1:arr port). (2) Caddy arr vhosts and homepage probes target 192.168.0.112:port (host-side published ports unaffected by netns changes). (3) Known defects as of 2026-09-04: Prowlarr's Applications table is EMPTY (the four rows from 2026-08-30 were lost — prowlarr→arr sync broken until rebuilt); readarr's qB proxy auth defect needs retest post-move; EZTV/1337x "CloudFlare-blocked" parking is stale (egress now VPN — retest before enabling). (4) E2E lesson: playwright specs 11-admin/12-crawl now derive expected open-link counts from services.json (showLink) instead of hardcoded counts — keep new registry entries consistent. Also: deploy.sh --service X dispatches ONLY X on frankbook (path-coupling applies to changed-path planning); gluetun restart self-heals all netns joiners via Restart=always so a single gluetun deploy converges the family. (context: 2026-09-04 operator decision session: dankserver arr compose copy (booted that morning) removed after Frank chose FrankBook-Pro as the sole arr deployment; netns consolidation + test seam + loopback repoint executed same session. Commits 4c7465d, df4339a.)
• Rootless podman on Dankserver (5.7.0, verified 2026-09-04 via /proc/pid/uid_map across 7 containers): the DEFAULT user namespace maps container uid 0 → host uid 1000 (dank) and container uid N → host 100000+N-1 (e.g. calibre-web PUID=1000 writes as 100999). Consequences: (1) PUID=0 + data roots owned 1000:1000 is the complete fleet recipe — NO UserNS/userns_mode key is needed on quadlets or compose services; the plex quadlet works without one. (2) NEVER add userns_mode: host to a podman-compose service: podman-compose 1.5.0 runs every compose service inside a pod and podman hard-rejects the combo (--userns and --pod cannot be set together), leaving the stack down with no recreate path. (3) Diagnostics: podman exec c id always shows uid=0 inside any mapping — meaningless; ground truth is /proc/container-pid/uid_map, host-side ps -o user= -p pid, or ownership of freshly-written files. (4) *arr boot-crash signature: UnauthorizedAccessException: Access to the path '/config/...' at logger init = data root not writable (e.g. root-owned dir created by sudo mkdir); the app exits before binding its port, so the port probe reads "dead" rather than "crashed" — check podman logs tail for "Press enter to exit" before assuming an outage has an infra cause. (5) NEVER pkill -f rootlessport on these hosts: it kills the port-forward helper of EVERY published container simultaneously (2026-09-03 outage); restart the owning systemd units instead. (context: Dankserver *arr stack restore session, 2026-09-04: stack committed 2026-08-29 never booted (Sentry crash, dead ports read as outage); userns_mode: host fix was a red herring — rejected by podman-compose pods; real fix was chown 1000:1000 of /opt config dirs. Commit e454159 (revert), 8a63624 (docs).)
• Arr stack (6 services) on dankserver (192.168.0.235): radarr:7878, sonarr:8989, lidarr:8686, readarr:8787, prowlarr:9696, qbittorrent:8080. Caddyfile on dankclaw has two-layer pattern: (1) public vhost blocks at top level proxying to 192.168.0.235:PORT, (2) @arr named handlers inside :8017 using "host" matcher for post-auth routing. Homepage services.json adds arr entries with category (user/admin), url (*.franksdank.cloud), probe (192.168.0.235:PORT), method http. Rootlessport issue: if rootlessport process dies, ports are mapped but return 000 — kill -9 rootlessport, restart containers with podman-compose. Fix permission issues with sudo rm -rf config dir. (context: ServerAppConfigs repo: Caddyfile changes added to dankclaw/Caddyfile (public blocks + :8017 named handlers), services.json in shared/homepage/ updated with 6 arr entries. All 233 bats tests pass. Deployed to dankclaw (dankclaw-owned).)
• Arr stack containers on dankserver (Plex/Prometheus/Loki via rootless podman) show 000 from curl despite rootlessport listening on the correct port. Root cause: s6 supervision in linuxserver images runs the HTTP server as a background process, but rootlessport forwards to the container's slirp4netns interface which doesn't reach the backgrounded HTTP server correctly. Symptoms: rootlessport process is running and ss shows port LISTEN, but curl returns 000 (no response, not connection reset). Fix: restart rootlessport (pkill -f rootlessport) then podman-compose down && podman-compose up -d, or use --network host to bypass rootlessport entirely. Verification: ss -tlnp | grep port shows rootlessport, podman exec container cat /proc/net/tcp6 should show the port in LISTEN state (0A). (context: Deep investigation of arr stack containers on dankserver (192.168.0.235) failing to respond via rootlessport despite ports being published correctly. Discovered s6 supervision in linuxserver images causes the HTTP server to run as a background process that rootlessport's port forwarding can't reach.)
• flm-npu-pack-registration managed skill exists; when updating it, merge (read current body via skill:// URL first) rather than clobber — the runbook covers: registry edit (mkstemp+os.replace), manual pack placement with resolve/main/file?download=true (raw URLs return 135-byte LFS pointer stubs), family kernel clone from /opt/fastflowlm/share/flm/xclbins/FamilyPack into ~/.config/flm/xclbins/PackName (sudo cp -a; serve resolves under FLM_XCLBIN_PATH = user dir), verification by real completion POST on :7771 checking choices not just HTTP 200 (unknown tags answer 200 with flat {"error":"Failed to load tag model!"} body), hwctx contention (only one flm serve process can hold the NPU; second process fails every load with DRM_IOCTL_AMDXDNA_CREATE_HWCTX err=-22 = aie2 mgmt msg opcode 0x2 status 0x2000003), boot tag decides residency only (per-request routing by exact registry tag regardless of boot tag; swap cost ~6-11 s on first request after swap), FLM_CTX_LEN=-1 to let each pack load at its own trained context, and bracket-guarded pkill patterns over ssh. (context: Capture turn for the completed NPU pack registration + boot-tag flip work on frank-npu (Dankclaw :7771), 2026-09-03.)
• placeholder
• Checking whether the flm-npu-pack-registration managed skill already covers the full registration runbook (manual pack placement, LFS download=true trap, family kernel clone to FLM_XCLBIN_PATH, hwctx contention rule, boot-tag flip verification) before overwriting it via manage_skill update. (context: manage_skill create failed because the skill already exists; need its current body to merge rather than clobber.)
• FLM (FastFlowLM) v1.0.1 on Dankclaw, registering a custom NPU2 pack tag (verified 2026-09-03 for nuextract3:4b = Atomic-Germ/NuExtract3-4B-NPU2): (1) flm pull tag is BROKEN for custom registry entries — it throws json.exception out_of_range key 'tag:size' not found while building the download list; place pack files manually into ~/.config/flm/models/PackName/ instead (use resolve/main/file?download=true — plain raw URLs return 135-byte LFS pointer stubs, not weights). (2) The qwen3.5-family loader opens vision_weight.q4nx unconditionally even for extraction/text-only packs with vlm:false; a missing vision weight file makes the first request HANG with no error log (silent stall behind NPU lock queue), not fail-fast. Registry files list must include vision_weight.q4nx. (3) NPU hardware-context contention: while one flm serve process holds the NPU, any SECOND serve process fails every model load with DRM_IOCTL_AMDXDNA_CREATE_HWCTX err=-22 (aie2 mgmt msg opcode 0x2 status 0x2000003 = hw resource alloc fail) — even for packs that load fine in the resident process. Only ONE flm serve process may hold the NPU; tag switching works only INSIDE one serve process (swap cost ~6-11 s first request after swap). (4) Kernel packs live at /opt/fastflowlm/share/flm/xclbins/PackName/ (root-owned, sudo cp -a); copy from the same-shape family pack (NuExtract3-4B shares Qwen3.5-4B-NPU2 kernels — config.json shape-matched on hidden_size/layers/head_dim first). Also: pkill -f "pattern" over ssh matches your own ssh wrapper's command line and kills your own session — use bracket-guarded patterns like "[q]wen3.5:4b" or exact PIDs.
• In serverAppConfigs, the docs-completeness test (tests/docs_completeness.bats) scans EVERY .md under serverAppConfigs-docs/ including dot-directories like Dankserver/.AGENTTRASH/, and requires each basename to be index-linked from serverAppConfigs-docs/index.md via a case-sensitive basename grep. So the old trick "move untracked docs to .AGENTTRASH before commit" does NOT satisfy this seam anymore — an unlinked page moved into an in-tree trash dir still fails the test. Correct dedupe of duplicate doc pages (e.g. Prowlarr.md vs prowlarr.md, 2026-09-03): collapse via rename so ONE canonical file carries the current content under the exact basename index.md already links (lowercase prowlarr here); git history retains both versions, zero deletions, test passes. Also: log.md in that tree is newest-first — new dated entries go directly under the # Bundle Change Log header, not appended at file tail.
• Dankclaw Hermes network topology, verified live 2026-09-02: systemd-hermes (gateway), hermes-webui, and chrome-devtools-protocol run with NetworkMode=host, so their local addresses are the host listener table — gateway dials signal-cli at http://127.0.0.1:8080 (SIGNAL_HTTP_URL in dankclaw/.env.example), CDP binds loopback-only 127.0.0.1:9222, webui listens 0.0.0.0:8787. hermes-dashboard alone sits on the compose-era orphan network dankclaw_default (subnet 10.89.1.0/24, gateway 10.89.1.1, zero other connections) and publishes 9119→0.0.0.0:9119; from that netns the host is reachable only at 10.89.1.1 for published ports (3000 Gitea, 8081 SearXNG verified OPEN) while unpublished host listeners (8080 signal-cli, 8787 webui) time out, and host.containers.internal=169.254.1.2 is the pasta alias for the default podman net (10.88.0.0/16, gw 10.88.0.1) so it is NOT routed from dankclaw_default. Fix if the dashboard ever needs the gateway: NetworkMode=host + drop PublishPort=9119, redeploy deploy.sh dankclaw --service hermes-dashboard. Also verified same turn: gitea-postgres (db:5432, net systemd-gitea), searxng-valkey (valkey:6379, net systemd-searxng) and alloy (OTLP 12345) publish NO ports → unreachable from the host netns by construction; Plex PMS 1.43 serves TLS-only on 32400 so an http probe there always reads down (use method tcp); and node-exporter was manually stopped on both hosts (dankserver exit 137 SIGKILL at 2026-09-01 02:18 with OOMKilled=false, dankclaw exit 143 SIGTERM ~6 days prior) leaving Prometheus's node-dankclaw/node-dankserver scrape targets connection refused until restarted — it is a bare compose service with restart=unless-stopped and no quadlet unit, so it will not survive a reboot.
• Rootless podman identity model on dankserver (verified live 2026-09-01, plex/tautulli quadlet migration): rootless podman CANNOT map real host uids outside the user's subuid/subgid range (dank: 100000:65536). UserNS=host rootless maps container root to the invoking user (dank, 1000) — not real root, and PUID=997 silently falls back to root because uid 997 is unmappable. The fleet convention is therefore: PUID=0/PGID=0 (container root = dank), data roots owned 1000:1000 (jellyfin/copyparty precedent); a data root owned by a legacy uid (997:984 for plexmediaserver) needs chown -R 1000:1000 at cutover, and rollback chowns back. Quadlet v1 gotchas proven the hard way: (1) UsernsMode= is COMPOSE syntax — the podman-user-generator hard-fails the WHOLE unit ('unsupported key UsernsMode in group Container') and systemctl --user restart name.service reports 'Unit not found'; the correct key is UserNS=. (2) Quadlet's systemd-style value parsing mangles embedded double quotes in HealthCmd= one-liners like python3 -c "..."; use the repo's proven HealthCmd=curl -fs http://localhost:8181/home pattern instead. (3) Without ContainerName=, the v1 generator names the container systemd-unit (systemd-tautulli). (4) PMS version jump 1.21.1.3795 → 1.43.3.10896 triggers one-time DB migrations/DatabaseFixups on a 32 GB DB that run 20-40+ min; a restart during migration ABORTS and re-runs it — never bounce PMS mid-migration; the 600s /identity deploy gate fails (empty-reply: PMS pre-binds 32400) and leaves deploy-state pending while the unit itself is correct. Also: line-targeted edit churn on the .container files ate three keys (tautulli ContainerName, plex Network, tautulli PublishPort) — after multi-hunk edits on unit files, read the WHOLE file before committing; the quadlet seam tests (PublishPort + enumeration) caught one, post-cutover C5 alias probe caught the other. (context: plex-tautulli podman quadlet migration on dankserver (plan local://plex-tautulli-podman-migration-plan.md), executed 2026-09-01: first cutover failed (UsernsMode hard-fail), second failed (rootless uid mapping crash-loop on 997-owned data root), gate failures during PMS one-time migrations, three keys restored after edit churn. Commits 58aa084, c694053, 39c8135, 6302508.)
• Homepage deploy ownership (serverAppConfigs): homepage is Dankclaw-owned even though its config lives in shared/homepage/ (including the misleading dankserver.services.json). Correct deploy: ./scripts/deploy.sh dankclaw --service homepage — deploy-remote.sh deploy_homepage() copies shared/homepage/services.json, index.html, admin.html, app.js, style.css, fonts to /srv/homepage/ on Dankclaw and restarts the homepage-status timer. --service homepage against dankserver fails ("not deployed on dankserver"); per-host *.services.json files are tracked for reference only and are NOT shipped. Changed files only land if committed+pushed first (deployer pulls revision state). (context: Plex homepage link fix, 2026-09-01: deploy.sh dankserver --service homepage failed ("not deployed on dankserver"); deploy-plan.py's homepage group maps to dankclaw only, and deploy-remote.sh deploy_homepage() refuses non-dankclaw targets. Discovered via dry-run + grep of deploy scripts.)
• Authentik (2026.8, Dankclaw) proxy-provider provisioning contract — every ProxyProvider needs BOTH: (1) all three flows set (authentication/authorization/invalidation). Any null makes the embedded outpost's Rust config deserializer reject the ENTIRE config JSON ("invalid type: null, expected a formatted UUID string") → "Failed to fetch outpost from API, retrying in 3 seconds" forever → ALL guarded vhosts die at the next outpost reload/restart. Silent killer: while the old in-memory config lives, everything works; the outage only surfaces on restart. (2) An attached Application (provider_id set). Without it, login completes but bounces to authentik's own user UI (/if/user/#/library) instead of the app — zero errors logged anywhere. The UI wizard does both automatically; raw API/ORM drives must do both explicitly. Diagnostic: the SSO e2e hostname-only assertion had a blind spot — the outpost serves the login flow ON the guarded vhost, so hostname matching proves nothing; assert pathname left /if/ AND body lacks "Powered by authentik"/"Sign in". Schema quirk: provider MTI is authentik_core_provider (PK=id) ← proxy child keyed by oauth2provider_ptr_id; Application.provider_id → base pk. (context: 2026-09-02 seerr/grafana/loki guard outage on Dankclaw: three script-provisioned proxy providers (created via raw API/ORM with minimal payloads) broke SSO in two silent layers; fixed by backfilling flows from pk1 + creating guard Applications, verified by hardened playwright sweep 7/7.)
• Dankserver (192.168.0.235) hardware, verified via read-only dmidecode/lscpu 2026-09-01: Ryzen 7 7700X (Zen 4/AM5) on ASUS TUF Gaming B650-E WiFi, BIOS 2613 (2024-04). RAM = 4×16 GB G.Skill F5-6000J3038F16G (Trident Z5 Neo, dual-rank, rated DDR5-6000 CL30-38-38-96 @ 1.35V EXPO, Hynix), 64 GB total, no ECC. CRITICAL: board silently defaulted to JEDEC-safe 3600 MT/s @ 1.10V (EXPO never enabled) because 4×2R is AM5's hardest population. Agreed OC ladder (user enters BIOS manually, agent hands-off): Tier 0 = EXPO I 6000 CL30 @1.35V; Tier 1 fallback = manual 5600, FCLK 2000, UCLK 1:1, 30-38-38-96, tRFC 448, DRAM VDD/VDDQ 1.35V, VDDIO 1.30V, SOC 1.20V (≤1.25 daily), MCR disabled while tuning; Tier 2 = tRFC 448→392→360, optionally 5800. Never touch tREFI on this non-ECC 24/7 server (silent corruption risk). Validation: Memtest86+ 4 passes, y-cruncher VST 1h, journalctl -k whea/mce watch (baseline clean as of 2026-09-01). Rationale: FreeToken frank-default-small streams ~35 GB experts from host RAM → memory bandwidth is its prefill bottleneck; 3600→6000 ≈ 53→88 GB/s read [INFERENCE, platform-standard]. (context: Dankserver RAM OC analysis turn, 2026-09-01. Read-only audit; profile delivered in chat for manual BIOS entry. User explicitly forbade edits — no sysfs/module/config changes were made; keep that constraint if resuming this task.)
• FrankBook-Pro qBittorrent version coupling (2026-09-01): qB is PINNED to lscr.io/linuxserver/qbittorrent:4.6.7 because Readarr's EOL qB client (frozen ~2023) rejects qB ≥ 5's 204 empty-body login success as "Authentication failure" — and that bites at RUNTIME, not just Prowlarr-style API creates (an earlier session wrongly assumed runtime cookie auth was unaffected; live evidence disproved it). Never unpin qB to latest while readarr is in the fleet. Second trap: on any qB version flip, qB 5.x's shutdown conf rewrite leaves a WebUI\Password_PBKDF2 @ByteArray line that 4.6.7 rejects — every login returns "invalid credentials" (arrs included); fix = rewrite the same conf line fresh with the same password (PBKDF2-HMAC-SHA512, 100k iterations, 16-byte salt, 64-byte key) while qB is stopped. Facts also documented in serverAppConfigs-docs/Infrastructure/FrankBook-Pro.md (repo is source of truth). (context: FrankBook-Pro qBittorrent pipeline work, 2026-09-01 (supplements same-day cookie/egress lesson))
• qBittorrent cookie-name + egress-verification gotchas (FrankBook-Pro, 2026-09-01): (1) qB 5.x names its WebUI session cookie QBT_SID_port (port-suffixed); qB 4.6.7 uses plain SID — any script hard-coding QBT_SID_8080 breaks on 4.6.7 with sidlen=0 → 403 cascades that LOOK like auth failures but never reach qB as failed logins (no ban accrues; verify via qbittorrent.log "WebAPI login success" lines). Always parse generically: grep -oE '[A-Za-z_]SID[A-Za-z0-9_]=[^;[:space:]]*' | head -1, then pass the whole match to curl -b. (2) Egress checks for gluetun-gated stacks MUST run inside the gluetun container (podman exec gluetun wget https://ipinfo.io/json). Running the same wget one ssh-context out (Mac host via podman machine ssh without the inner podman exec) silently returns the RESIDENTIAL IP — a false leak alarm that reads as a hard-gate violation. Confirm context by requiring the org line (AS9009 M247 for the ProtonVPN Panama config), not just country. (context: FrankBook-Pro gluetun/qBittorrent E2E verification, 2026-09-01)
• Authentik 2026.8 embedded outpost (franksdank.cloud fleet) caches claims per browser session in the authentik_providers_proxy_proxysession table (session_data JSON holds claims incl. groups; observed row TTLs 1–24 h). A session authorized BEFORE a user is added to a group keeps serving the old group list to Caddy header_regexp RBAC gates → persistent 403 despite correct membership, and the 403 generates ZERO authentik events (it never reaches authentik), so logs look silent. Re-login does not help (browser cookie re-maps to the same stale row). Fix (verified live 2026-09-01 for akadmin /admin 403): DELETE rows WHERE session_data-'claims'-'preferred_username' = 'user' in the authentik-postgres container — the django login session is untouched, so the next visit re-authorizes silently and mints fresh claims. Do NOT rotate AUTHENTIK_SECRET_KEY for this (invalidates every outpost session fleet-wide). Diagnosis SQL + silent-reauth verification documented in serverAppConfigs-docs/Infrastructure/Authentik.md (commit ea0bba3). (context: Frank reported persistent 403 at www.franksdank.cloud/admin after being added to homepage-admins; recon found his live proxysession row (expiring 2026-09-02 22:06 UTC) carried ["authentik Admins"] only, minted before the group existed. Silent re-auth proven by deleting qa-e2e-admin's 13 rows and running the e2e admin spec with --no-deps against the stale storage state (2 passed, 2.9 s, no login pr)
• Frank wants periodic "what did we do" HTML change reports rendered in Firefox (asked 2026-09-01; first one at /home/dank/Reports/serverAppConfigs-24h-2026-09-01.html). Preferred standard: Keep a Changelog 1.1.0 categories (Added/Changed/Fixed/Removed/Security, omit empty) inside a post-change-review skeleton (summary → timeline → categories → code samples → verification → incidents → user actions). Reports are deliverables outside the repo at ~/Reports/ (never tracked; keeps tree clean). Frank reads the report to plan his code-review pass, so include a ready-made git show/git diff command list per theme. (context: Built a 38 KB self-contained HTML report for serverAppConfigs covering 39 commits/24h; verified render via headless browser screenshot before handing over; opened via firefox --new-tab file:///....)
• Playwright automation of authentik's Lit flow executor (fleet e2e, tests/e2e/lib/login.ts akLogin): a programmatic page.fill races the stage re-render — the Lit web component re-renders right after mount and wipes the input, so the form submits an EMPTY password ("This field is required") even though fill() succeeded. Hardened pattern: fill, VERIFY the value survived (expect the input's value, short retry), submit with Enter (native form path, no button re-render race), and retry the whole stage up to 3 attempts before failing. Two adjacent musts: every login runs in a FRESH browser context (a reused/half-finished context poisons authentik's pending_user stage and hangs the next login), and the browser needs a real Chrome UA (Anubis denies obvious headless UAs); budget 60s assertions for the first Anubis PoW per fresh context (subsequent guarded vhosts skip challenge thanks to COOKIE_DOMAIN solve-once). Conventions documented in serverAppConfigs-docs/Architecture/E2E Suite.md (commit b945dd0). (context: frank-default e2e suite hardening, 2026-09-01: 10/11 spec flake root-caused to Lit re-render race + Anubis PoW budgets; suite now 17/17 green across setup/standard/admin/guest projects.)
• Fleet VPN truth (verified 2026-09-01): the gluetun VPN_SERVICE_PROVIDER=custom profile on Dankclaw (endpoint 149.102.254.77:51820) points at a ProtonVPN Seattle WireGuard server, NOT a self-run box. Evidence: RDAP via curl -sL https://rdap.org/ip/IP (works when whois is empty) — endpoint IP → CDNEXT-CG-SEA/Datacamp; Dankclaw's tunnel v6 2a07:b944::2:2 → CH-PROTONVPN-20160719; exit IP 149.22.88.19 → CDNEXT-SEA-CG. Full 65535-port scan fingerprint: no SSH anywhere, only silent TLS listeners on 443, 4443–4468, 5995, 7770, 8443 (Proton protocol-stealth port map). Consequences: (1) adding a "peer" is never server admin — it's generating a new WireGuard config in Frank's Proton dashboard; each config = one keypair, Proton has NO BYO pubkey (a locally generated keypair is useless; one was retired to dankclaw ~/.AGENTTRASH/20260901-frankbook-wg-keypair). (2) Two hosts sharing one Proton config make the server bounce the endpoint between them — reusing Dankclaw's config for FrankBook-Pro would kill the Dankclaw tunnel (SearXNG). (3) Mac gate: second dashboard config (e.g. named frankbook) → PrivateKey/Address paste into /home/core/Containers/gluetun/env; server pubkey utqNH4P+C0fA1wTgIUaIfhm1b7ai0iEGvXyiw4KC3Fs=, gluetun ignores the .conf DNS line. General technique: identify mystery infra with rdap.org RDAP + nmap port fingerprint before asking the user to administer it. (context: FrankBook-Pro WG gate investigation, 2026-09-01. Probes: no SSH config aliases on the workstation/Dankclaw, port 22 timeout, full nmap scan, RDAP on 3 IPs. Docs rewritten: serverAppConfigs-docs/Infrastructure/FrankBook-Pro.md "Mac provisioning" section; log.md entry. Repo commits 2d06be8, 3094ffa, f1171b9.)
• When a fleet credential "mystery" arises (auth works/doesn't contradict user memory): (1) assume session-archive images are corrupted — OCR of transcript frames mangles strings while preserving shape (Jellyfin admin password read as "ResFace_3241" instead of ""; the prefix was garbage, the digits were real; the user's own recall had the right prefix but flipped digits). (2) NEVER brute-force a live auth endpoint — pull the stored hash and verify candidates offline: Jellyfin stores it in jellyfin.db (Users table, Password column) as $PBKDF2-SHA512$iterations=210000$salt$hash where salt AND hash are HEX-ENCODED (32 B salt → 64 hex chars, 128-char hash). A parser that assumes base64 silently tests zero candidates — always print back the parsed lengths (saltbytes/hashbytes) and candidate count before concluding "no match". (3) A single matching hash is conclusive; a live POST adds nothing and just logs auth-failure warnings. (context: Jellyfin 401 mystery on Dankserver, 2026-09-01. User's correction of the corrupted prefix + digit flip; offline hash check vs stored DB hash decided it. Files: serverAppConfigs-docs/Dankserver/Jellyfin.md, serverAppConfigs-docs/log.md (both corrected).)
• Rootless podman UID mapping on Dankclaw/Dankserver is CONTINUOUS, not offset: container uid N → host uid 100000 + N - 1 (dank subuid 100000:65536; container 0 → 1000, container 1 → 100000, container 1000 → 100999). Never compute host uid as 100000+N. Before chowning a bind root for a non-root image, read the real mapping from any throwaway container: podman run --rm image cat /proc/self/uid_map ("ID-inside ID-outside range" lines). A wrong chown gives EACCES crash-loops (quadlet auto-restart, container never persists; podman logs empty because --rm). Verified 2026-09-01 with seerr-team/seerr (in-image node uid 1000 → chown 100999 fixed it). (context: Seerr first boot on Dankclaw crash-looped with EACCES mkdir /app/config/logs after chown 101000; /proc/self/uid_map showed "1 100000 65536" ⇒ container 1000 → host 100999. Second off-by-one incident on this fleet (postgres 100069 arithmetic was the first).)
• Two pitfalls from the 2026-09-01 frankbook/dankserver session: (1) zsh (default login shell on the workstation AND dankserver) expands =word at the start of an unquoted word to the command's path — so echo ==FOO== inside an ssh command fails with zsh:1: =FOO== not found. Quote echo arguments (echo "==FOO==") or use printf in any command line destined for zsh. (2) qBittorrent ≥ 4.2 ignores a plaintext WebUI\Password in qBittorrent.conf — it must be @ByteArray(salt:hash) from PBKDF2-HMAC-SHA512 (100000 iterations, 16-byte salt, 64-byte key, both base64). Also when the WebUI is reached through a port forwarder that preserves the LAN Host header (e.g. socat), set HostHeaderValidation=false and ServerDomains=* or qB rejects the request. Reference implementation: VM_SEED heredoc in deploy_frankbook_stack, scripts/deploy-remote.sh. (context: frankbook-pro *arr bring-up + dankserver cutover verification, serverAppConfigs repo)
• FreeToken v0.1.2 on qwen3_5_moe loads ONLY modelopt-style quant configs: block-fp8 (quant_method "fp8" + weight_scale_inv, the Huihui/official-Qwen format) works; every compressed-tensors FP8 checkpoint fails. Two live-verified failure modes (2026-09-01, Dankserver): (1) compressed-tensors per-channel (Ornith-1.5-35B-A3B-FP8) aborts at expert-source resolution — ValueError "Missing MoE expert source layers: {'gate_up': ...}" because expert tensors don't follow mlp.experts.{N}.gate_proj/up_proj/down_proj naming; (2) compressed-tensors block-fp8 128×128 (HauhauCS-Aggressive-fp8) resolves experts but _dequant_fp8_weight multiplies weight × weight_scale directly, so [out/128, in/128] block scales don't broadcast → RuntimeError shape mismatch (tensor a 2048 vs b ...). Both exit cleanly at load (~37 GB fits RAM, no swap-thrash). Before swapping any model onto FreeToken, check config.json quant_method/weight_scale shape; a "healthy after ~6s" from deploy.sh proves nothing (health gate passes while the doomed worker is still starting) — verify via journal Uvicorn startup + a real completion. Evidence: serverAppConfigs-docs/LLM/free[REDACTED].md Quantization Compatibility table. (context: frank-default-small model-swap attempts on Dankserver, 2026-09-01: Ornith FP8 and HauhauCS FP8 both failed to load on FreeToken v0.1.2; reverted to batsclamp Huihui FP8 (commit 4d5bea4), docs commit fe34f04.)
• skill://quadlet-v1-generator-quirks
• Before swapping a model into an LLM service, preflight the CHECKPOINT SIZE (bytes on disk / weight precision) against the host's usable RAM, not just VRAM — the failure is silent and self-sustaining, not a clean crash. Case: thomsonreuters/Thomson-1.0-Small is Qwen3.6-35B-A3B arch but ships BF16 (~66 GB) vs the Huihui FP8 it replaced (~35 GB); Dankserver has 62 GB system RAM. FreeToken streams MoE experts from host RAM, so 66 GB of weights exceed physical RAM. Compounding traps that turned a fit-failure into an outage: (1) 127 GB swap means the kernel PAGES instead of OOM-killing, so sshd is starved and never gets killed — no recovery window opens on its own; (2) systemd Restart=on-failure makes it a crash-reload-rethrash loop; (3) the unit is enabled + WantedBy=multi-user.target, so even a reboot re-thrashes on boot until the bad MODEL_DIR is masked/reverted — a plain power-cycle does NOT fix it. Correct recovery requires a shell to run systemctl mask --now svc FIRST (stops + breaks the reboot trap), then pull the revert. deploy.sh reports "healthy after Ns" as soon as systemd is active, BEFORE the model finishes loading — never treat that as load success; verify with a real completion. Verdict: BF16 35B-A3B cannot run on a 62 GB box; needs an FP8/GGUF quant or a bigger host. RTX 3090 has no FP8 tensor cores so even FP8 runs via upcast but still fits in RAM. (context: 2026-09-01 frank-default-small swap to Thomson-1.0-Small on Dankserver caused a sustained swap-thrash outage; SSH locked out, no OOM relief due to 127GB swap. Repo revert staged (commit 1163844), awaiting physical/console recovery.)
• FLM NPU model swap on Dankclaw: install new model via flm-add HF-repo --family family (downloads to ~/.config/flm/models/Name/), register tag in user model_list.json if missing, swap FLM_MODEL in scripts/frank-default-sightsound.env, then restart. Two common pitfalls: (1) FLM looks for xclbins at $FLM_XCLBIN_PATH/ModelName/ — if family is qwen3.5 but the model name is different, create a symlink from the expected name to the family's xclbin dir (e.g. Qwen3.8-Distilled-Heretic-9B-NPU2 → Qwen3.5-9B-NPU2); (2) flm-add leaves vision_weight as .q4nx.part — must rename to .q4nx before FLM can open it. Service uses FLM_CONFIG_PATH and FLM_XCLBIN_PATH exports for tag resolution; unknown tags return 200 with {"error":"Failed to load tag model!"}.
• FreeToken on Dankclaw builds successfully via the CUDA path (not ROCm). Key details: (1) install nvidia-cuda-toolkit via apt (provides nvcc + libcudart at /usr), (2) set CUDA_HOME=/usr, FREETOKEN_ALLOW_CUDA_MISMATCH=1 (nvcc 12.4 vs torch cu130 is safe), (3) uv installed at /home/dank/.local/bin/uv, (4) source .venv/bin/activate in FreeToken/. The build installs torch 2.11.0+cu130, flashinfer 0.6.18, sgl-kernel 0.4.5. FTW format stores dtype per-tensor in index — FP16, BF16 (default), FP32, FP8 are all first-class. GGUF support is model-specific: only Gemma4GGUFForCausalLM has GGUF in the registry. GGML types: F32/F16/BF16 use native torch views; Q4_0 and Q6_K have dequant functions; Q8_0 is listed in GGML_NAME but has no dequant function (raises NotImplementedError). The GGUF reader uses gguf-py and is_gguf_path checks for .gguf files.
• FreeToken does not need FP8 — the FTW format is dtype-agnostic (stores per-tensor dtype in index). Default dtype is BF16 (torch.bfloat16). FP16, FP32, BF16, FP8, FP4 (DS/NV/MX), and Q4_0 are all supported. convert_checkpoint(dtype=torch.bfloat16) is the default; pass any torch.dtype for non-default.
• FreeToken on Dankclaw: CUDA path requires nvidia-cuda-toolkit (apt package, provides nvcc + libcudart at /usr/bin/nvcc and /usr/lib/x86_64-linux-gnu/libcudart.so). Set CUDA_HOME=/usr and FREETOKEN_ALLOW_CUDA_MISMATCH=1 (nvcc 12.4 vs torch 2.11.0+cu130 CUDA 13.0 kernels — safe override). Build: git clone, uv venv, uv pip install -e ".[accel]". UV installed to /home/dank/.local/bin/uv.
• FreeToken on Dankclaw builds for CUDA (not ROCm) because freetoken.kernel._pinned_tensor links against cudart. Ubuntu CUDA toolkit installs to /usr (not /usr/local/cuda), so CUDA_HOME=/usr is required. nvcc 12.4 + torch cu130 requires FREETOKEN_ALLOW_CUDA_MISMATCH=1. UV installs to /home/dank/.local/bin/uv and must be added to PATH.
• FreeToken can build on Dankclaw for both ROCm and CUDA. ROCm is the natural path: gfx1151 GPU with amdgpu driver loaded, ROCm 6.3 at /opt/rocm/, hipcc 7.2. All build deps present: cmake 4.2.3, git 2.53, python 3.14, gcc 15.2, ninja 1.13.2. Dependency chain for ROCm: torch 2.11 (has rocm6.3 wheels on PyPI), sgl-kernel 0.4.5 (has ROCm wheels), flashinfer-python (has ROCm wheels). CUDA path also works (nvidia driver 575.57, Vulkan libs present) but needs nvcc for JIT-compiled CUDA kernels on first use — install via sudo pacman -S cuda. Build command: pip install -e ".[accel]" (uv optional). Vulkan libs at /usr/lib/x86_64-linux-gnu/ but not a FreeToken build requirement.
• frank-default-large contextWindow in ~/.omp/agent/models.yml is 200000 (per-slot with parallel 2), not 401408 (total CTX_SIZE) or 262144. The serve-frank-large.env has CTX_SIZE=401408 / PARALLEL=2, but the user consistently uses the per-slot value of 200000.
• Hermes CDP browser daemon (deployed 2026-08-31, commit 2072354): chrome-devtools-protocol.container quadlet on Dankclaw runs digest-pinned chromedp/headless-shell with entrypoint override /headless-shell/headless-shell --remote-debugging-address=127.0.0.1 --remote-debugging-port=9222 ... — the override deliberately drops the image's socat front which would bind 0.0.0.0:9222 and expose browser control to the whole LAN. Deployer (deploy_hermes) starts it only if inactive; a restart kills in-flight browser sessions, so restart manually only when intended. hermes-webui was flipped to Network=host so both agent consumers (Signal gateway + webui) share the daemon's host netns and can dial 127.0.0.1:9222; bridge-network clients cannot reach it at all by design. Client wiring lives in live /data/config.yaml under mcp_servers.chrome-devtools = command /usr/local/bin/npx, args ["-y","chrome-devtools-mcp@latest","--browserUrl=http://127.0.0.1:9222"]; npx package cache lives in the container layer, so a recreation (not restart) pays one first-call npm download. /data/config.yaml is owned by the unmapped uid — even world-writable dir doesn't make the file readable by dank; edit via sudo python3 - piped script. Verification gotcha proven live: an MCP stdio client probe must newline-terminate each JSON-RPC message AND keep stdin open (sleep) after writing, because on immediate EOF the server exits before flushing any response and you observe empty output with rc=0; also, \n inside ssh single-quotes → sh -c double-quotes lands as literal backslash-n in printf args (invalid JSON), so pipe a written script via ssh 'bash -s' /tmp/probe.sh instead of nested quoting.
• Hermes containers on dankclaw (systemd-hermes, hermes-webui, hermes-dashboard) run as root with SSH keys at /root/.ssh/id_ed25519. Their pubkeys must be in: (1) dankclaw's /root/.ssh/authorized_keys AND ~dank/.ssh/authorized_keys (hermes SSH to dankclaw as root, but dank's own key is for dank@dankclaw), (2) dankserver's ~dank/.ssh/authorized_keys, (3) FrankBook-Pro's ~dank/.ssh/authorized_keys. The hermes containers SSH to dankclaw as root@192.168.0.143, to dankserver as dank@192.168.0.235, and to FrankBook-Pro as dank@192.168.0.112. Each hermes key is unique per container (not shared). The dank@dankclaw key (dankclaw's own id_ed25519) was also missing from dankserver and FrankBook-Pro and needed adding for host-to-host SSH.
• On FrankBook-Pro (macOS Intel x86_64), gvproxy is not bundled with the standalone podman v5.4.2 binary. Must build it from source using go build -mod=mod -o /usr/local/bin/gvproxy github.com/containers/gvisor-tap-vsock/cmd/gvproxy. The old /usr/local/bin/gvproxy file is only 9 bytes ("Not Found" text from a failed curl download) — must rm -f it before building. After building, it becomes a Mach-O 64-bit x86_64 executable (~19.5 MB). The podman machine on FrankBook-Pro is initialized but also requires vfkit in PATH to start fully (error: exec: "vfkit": executable file not found in $PATH).
• 2026-08-31 state, verified live: (1) Vhost renames must be done in lockstep — the test suite (tests/pihole.bats test 178) fails unless every Caddyfile vhost name has a Pi-hole addn-hosts A record on 192.168.0.143; chat→ai and jellyfin→jelly were renamed this way (Porkbun edit with RELATIVE label + addn-hosts line + Caddyfile vhost in one commit, then deploy.sh). Decommissioned vhosts removed from Caddy that day: comfyui, hermes, radarr, sonarr, lidarr, readarr, prowlarr, qbittorrent; their public DNS was deleted from Porkbun while Pi-hole addn-hosts entries were deliberately KEPT (user's explicit 'remove from Porkbun, not PiHole') so internal consumers still resolve them but hit no vhost — internal consumers of those services must use direct IP/port instead (e.g. Hermes dashboard at 192.168.0.143:9119). (2) hermes-webui (Hermex) auth model: password-only login via POST /api/auth/login {"password":...} returning a signed HttpOnly hermes_session cookie (24h TTL); no username field, no SSH-key concept inside the app — SSH keys authenticate only the transport layer (user tunnel on port 22; Termius-generated keys are non-exportable so the keypair must be generated inside whichever SSH app acts as transport). The webui is published LAN-bound (192.168.0.143:8787-8787) with password gating, not loopback-only anymore; off-network access = SSH tunnel on port 22 then dial http://localhost:8787. (3) The webui unit's [Container] EnvironmentFile is /home/dank/serverAppConfigs/dankclaw/.env (NOT the root .env) — a secret rotation must rewrite BOTH env files or the restarted container serves the old value; verify by podman exec printenv on the live container after restart. Also: podman exec must run UNPRIVILEGED on Dankclaw — sudo/root's namespace does not see rootless containers (exit 125).
• On Dankclaw (rootless podman), sudo podman exec user-container ... fails with exit 125 — root's podman namespace does not see the login user's rootless containers. Run podman exec as the login user (ssh dankclaw 'podman exec ...' with no sudo) when inspecting container env/config (e.g., podman exec hermes-webui printenv KEY). Same rule applies to podman inspect/ps for user-owned containers.
• On Dankclaw (rootless podman), sudo podman exec container ... fails with exit 125 — root's namespace does not see the login user's rootless containers; run podman exec unprivileged as the ssh login user instead. Related durable pattern used when repairing /home/dank/serverAppConfigs/.env on Dankclaw (2026-08-31): a prior fix had appended a raw podman inspect --format "{{json .Config.Env}}" JSON dump into the live .env, corrupting keys (HERMES_WEBUI_PASSWORD=["PATH=...). Repair = back up first (cp -a), then rewrite quoted dump lines to proper KEY=value by stripping the leading quote, and source authoritative secret values programmatically from the live container env via unprivileged podman exec svc printenv KEY rather than transcribing them — never trigger a service restart for a pure .env-file repair. Also note: JSON entries carry a leading "; anchored grep on ^KEY= fails for those lines until de-quoted (repeat of the earlier landmine-fix pitfall — verify with grep -c '^"' == 0 after any dump append).
• Decision (2026-08-31, task: "SearXNG agent-use check under Anubis"): do NOT stand up a dedicated agent-only SearXNG instance on Dankclaw. Verified live: the guarded path (search.franksdank.cloud) answers non-browser API clients with the Anubis JS-challenge page at HTTP 200 — status-code-only clients silently receive HTML where JSON was expected; always check content-type, not just status. The agent path is the direct published port http://192.168.0.143:8081/search?q=...&format=json from pass_ip (trusted) sources only — same single instance already VPN-bound via Network=gluetun.container, unreachable from outside the LAN by construction (no public route to the LAN IP). A duplicate would double upstream request volume from the same rate-sensitive WireGuard exit IP for zero benefit. Rationale recorded in repo docs: serverAppConfigs-docs/Dankclaw/SearXNG.md Gotchas section (commit ba0345a, pushed to Gitea origin) — read that file before ever proposing a second SearXNG stack. General probe heuristic: to test whether API clients can pass an Anubis-fronted route, curl the JSON endpoint with a non-browser UA and compare response content-type against HTTP status; a 200 with text/html body = guard challenge, not a working API path.
• Verified runbook for flipping a Caddy-fronted service onto the Anubis+Authentik guard chain on Dankclaw (executed 2026-08-31 for loki.franksdank.cloud, commit a382a3b): (1) create the outpost ProxyProvider FIRST via ak shell stdin (podman exec -i authentik-server ak shell /tmp/x.py; set p.pk=None before save) — external_host = exact vhost URL with trailing slash, internal_host = http://caddy:8017; (2) flip vhost to reverse_proxy anubis:8923 { header_up X-Real-Ip {remote_host} } and add the post-auth @svc host vhost handle block inside the http://:8017 listener routing to the real upstream, in ONE deploy; (3) verify unauthenticated with curl → Anubis challenge page ("Making sure you're not a bot!" markers), NOT 502. Verification pitfall: do NOT probe the internal post-auth listener as localhost:8017 from the host — it lives inside the caddy container's netns and answers HTTP 000 (connection refused) from host loopback; instead verify the reloaded config byte-exact via podman exec caddy grep -n -A3 handler /etc/caddy/Caddyfile, and confirm outpost reachability with getent hosts caddy from inside authentik-server's netns (resolves 10.89.1.211). Also: service-to-service ingestion must target internal addresses (Alloy→loki:3100 never traverses Caddy); non-browser clients can never solve the JS challenge, so any app whose backend queries a public hostname of a guarded service breaks on flip — audit datasource URLs first (Grafana's grafana.db data_source table had zero rows at flip time). (context: Loki guard-chain flip on Dankclaw; provider pk=7 created before route flip; verified via challenge probe + live-config grep inside caddy container.)
• qa-e2e (pk=6) is a live authentik test account on Dankclaw created 2026-08-31 with NORMAL privileges only: no groups, is_staff=False, is_superuser=False — verified at DB level via ak shell. It passes the full guard chain (Anubis challenge → two-stage default-authentication-flow → outpost callback → app content) and was proven live landing on RomM's app UI as a real authenticated user; it therefore cannot reach any admin-only surface (Grafana/Gitea/hermes dashboards keep their own auth). Credential store for the planned Playwright E2E suite: /home/dank/.config/e2e/qa-e2e.cred, mode 0600, deliberately OUTSIDE the serverAppConfigs repo so it can never be committed or trip deploy.sh's untracked-file guard. The E2E suite design (P4) consumes this file to capture storageState; Gitea-facing tests must use a Gitea token instead — an authentik session buys nothing on the git-smart-HTTP-exempt route. Related bypass map at that time: loki.franksdank.cloud had NO auth at all (LogQL open to anyone who can reach DNS) — biggest remaining exposure; *arr/qbittorrent UIs unguarded but API-key-gated for actions; grafana/pihole gated only by shared edge basicauth.
• Edit-tool auto-prefix quirk (observed 2026-08-31, serverAppConfigs Caddyfile work): when an edit call's output warns "Auto-prefixed bare body row(s)", the tool may write literal prefix characters into the file even though the rendered diff looks clean — a : intended as an output marker landed in the file as a literal :# comment line. In a Caddyfile that parses as a matcher token and fails config validation at deploy time with a misleading error pointing at the next line ("unrecognized directive" on the following site block). Lesson: after any edit whose tool output shows auto-prefixing/marker warnings, byte-verify the exact edited lines (sed -n/cat -A) before commit or deploy — do not trust the rendered diff. Same class of failure can silently ship because validation gates (caddy validate) correctly skip the reload and only log a non-fatal error; verify the reload actually ran, not just that deploy exited 0.
• Rootless-podman GPU access on systemd-user-unit-spawned containers (Dankclaw comfyui, 2026-08-31): containers spawned by systemd user units get NO supplementary groups at spawn time, so group_add: render fails with "unable to find group render" (userns name lookup) AND --group-add keep-groups provably preserves an EMPTY group set — torch then dies with "No HIP GPUs are available" even though /dev/kfd and /dev/dri/renderD* are merely passed through (they show as nobody:nogroup inside = unmapped). The only spawn-context-independent fix is udev MODE 0666 on the compute nodes (kfd, renderD*; leave card* display nodes alone). Two gotchas when applying: (1) udevadm trigger --type=devices does NOT enumerate misc-class devices (they have no subsystem) — use udevadm trigger --action=change --sysname-match=kfd; (2) the rule must be KERNEL=="kfd", MODE="0666" with NO SUBSYSTEM= guard, because a SUBSYSTEM=="misc" match never matches. Verified end to end: torch.cuda.is_available() True, device "Radeon 8060S Graphics", port bound same minute.
• FLM (frank-default-sightsound, port 7771) routes chat completions per request by exact FLM registry tag: each request locks the NPU, loads the requested model if not resident, generates, releases. Measured swap costs (Aug 30 2026): first request to an unloaded tag pays full cold load (~7-90 s); switching back re-loads the previous model (~10 s) — alternating tags = every request pays swap. CRITICAL gotcha: a tag listed in /v1/models is NOT necessarily loadable — load fails with [ERROR] Failed to load model: No such file '/home/dank/.config/flm/xclbins/PackName/name.xclbin' when the kernel-pack dir isn't symlinked under FLM_XCLBIN_PATH (/home/dank/.config/flm/xclbins). Fix = ln -s /opt/fastflowlm/share/flm/xclbins/PackName ~/.config/flm/xclbins/PackName (verified for Qwen3.5-4B-NPU2; other unlisted packs fail the same way). Unknown tags answer HTTP 200 with flat {"error":"Failed to load tag model!"} and no choices — status-code-only clients silently get empty completions; clients must check the error key, not just HTTP status.
• FLM (FastFlowLM) 1.0.1 serving on frank-default-sightsound (Dankclaw, port 52626) answers unknown model tags with HTTP 200 and an error body, not a 404 and not a silent fallback: /v1/chat/completions → flat {"error":"Failed to load tag model!"} with no choices; /v1/completions → nested OpenAI-shaped server_error (code 500) also under HTTP 200. Status-code-only clients therefore see a successful empty response — any client wiring against this endpoint must check for an error key in the body, not just status. Valid tags route per-request to the exact named model (response echoes the requested tag; no restart needed to switch registry tags). This disproved the older llama3.2:1b-fallback behavior documented from 2026-08-16; corrected live-probe evidence is committed in AGENTS.md + serverAppConfigs-docs/LLM/frank-default-sightsound.md at commit 9b9f0f9 — verify against those before relying on this. General lesson: behavior claims written into docs from inference/incident notes get falsified by direct probes; probe the live endpoint (bogus input + control pair) before committing contract claims to docs or client configs. (context: Live probe 2026-08-30 on frank-default-sightsound: bogus tag "totally-bogus:9b" against /v1/chat/completions and /v1/completions; control with valid tag gemma4-it:e4b. Docs + serve-script comment corrected in commit 9b9f0f9, deployed healthy 4 s post-deploy.)
• frank-default-sightsound (FLM v1.0.1, XDNA2 NPU, Dankclaw port 52626) serves the WHOLE local model registry on one port and routes /v1/chat/completions per-request by the EXACT FLM registry tag in the model field; unknown tags do NOT 404 — they silently serve llama3.2:1b instead. Any client wiring must copy tag bytes from live GET /v1/models at wiring time (never hand-type, never use a service alias). Wiring pattern used for pi client models.json on the workstation (2026-08-30): backup file → programmatic python json load/dump adding provider dankclaw-npu with baseUrl http://192.168.0.143:52626/v1, model id = exact live tag gemma4-it:e4b, contextWindow/maxTokens parsed from FLM_CTX_LEN in scripts/frank-default-sightsound.env (not transcribed), input ["text","image","audio"], reasoning true → re-parse to prove valid JSON. Health probe for this service is GET /v1/version (FLM has no /health route). Full runbook: serverAppConfigs-docs/LLM/frank-default-sightsound.md; fleet table + runtime section in AGENTS.md carry the iommu=off-kills-NPU trap (SVA bind needs enabled IOMMU for /dev/accel/accel0).
• Dankclaw hermes stack (fixed 2026-08-30, commit 56763e0): the hermes gateway is a Signal messaging agent with NO HTTP listener on :8090 — the public route hermes.franksdank.cloud → Caddy internal listener :8017 → reverse_proxy 192.168.0.143:9119 (dashboard login-gated admin UI). The gateway connects OUTBOUND to the signal-cli daemon, a HOST process at /opt/signal-cli-0.14.7 listening on host :8080, via SIGNAL_HTTP_URL=http://host.containers.internal:8080 in dankclaw/compose.yaml. Durable quirk: on this rootless-podman/netavark host, containers can only reach the host LAN IP 192.168.0.143 on PUBLISHED (DNAT'd) ports — connections to un-forwarded host ports from container space get instant Connection-refused even though a listener exists; host.containers.internal (netavark gateway alias) is the verified working path for unpublished host ports. Also durable: hermes, hermes-dashboard, and hermes-webui are compose-managed with NO quadlet units — they do not survive reboots (this is how the agent went silent after the Aug 26 reboot); a reboot requires manual podman-compose -f dankclaw/compose.yaml up -d --no-deps svc or deploy.sh dispatch. The hermes gateway drops to an unmapped uid inside its userns: /home/dank/hermes-data must be world-writable (chmod -R a+rwX) or the drop-priv user cannot traverse it and crash-loops on log writes (/data/logs/agent.log PermissionError).
• frank fleet as of 2026-08-30 night: frank-default-large (Dankclaw) = kingjones777 Qwen3.8-Flash-Next ROCmFP4-STRIX 113.47 GiB on the card's pinned kingjones30/ROCmFPX fork at /home/dank/GitPackages/kingjones-ROCmFPX/build (commit 36e9acd) — builds require -DLLAMA_BUILD_WEBUI=OFF here (npm/HF-bucket asset provisioning aborts the link at 98%). --flash-attn on works UNPATCHED on the STRIX model (the QSA F16 ggml_fill assert needing scripts/patches/qwen4exp-qsa-f32-mask.patch is agentionai-FAST-specific); mmap MUST stay on (PLE streams via page cache; --no-mmap = silent cgroup kill); no spec decode; agentionai FAST on build-qwen38 is the rollback (one env commit). frank-default-small (Dankserver) = batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 on the same FreeToken v0.1.2 runtime (native block-fp8, identical loader path + pool triple as official FP8); FreeToken's /v1/chat/completions 422s WITHOUT an explicit "model" field (llama.cpp servers don't). Deploy verification trap: a leftover manually-started llama-server holding :7770 makes both the unit crash-loop on bind AND deploy.sh's /health check pass — verify systemctl show -p MainPID matches the live pgrep, not just health. pi models.json frank-default-large contextWindow corrected to 131072 (llama clamps n_ctx_train/parallel to per-slot; the old 200000 oversold). (context: Repo /home/dank/Projects/serverAppConfigs; commits 3066d81 (large config), aa9c1d0 (small config), 0937cfb (docs). Both swaps validated live end to end.)
• When deploying frank-default-large changes to Dankclaw, the service unit is managed by systemd as frank-default-large.service (NOT a user quadlet unit). The quadlet directory exists but systemd-level unit is the authoritative one. The deploy script targets the systemd service; user-level systemctl --user list-units shows nothing for frank services.
• frank-default-large env file has CTX_SIZE=401408 and PARALLEL=2 which are marked in comments as "untested experiment" vs the validated production config of CTX_SIZE=32768, PARALLEL=1. The env file dead-weight VAR VERBOSITY=4 is never passed to the binary (serve script doesn't use it). These values differ from the proven 32768 ctx × 1 slot deployment that loads clean in ~58s with ~100 GiB in the 119.3 GiB HIP heap.
• qBittorrent 5 (lscr.io image, Dankserver) mints a NEW temporary WebUI password at every boot unless credentials are stored — this silently breaks all arr download clients after any restart, and it rotated mid-session once already. qb5 REMOVED /api/v2/app/setPassword and the lscr image has no WEBUI_AUTH env handling, so persistence must be written into /opt/qbittorrent/config/qBittorrent/qBittorrent.conf with the container stopped, in qb's exact format: WebUI\Username=admin + WebUI\Password_PBKDF2=base64(16-byte-salt) + ":" + base64(PBKDF2-HMAC-SHA512, 100000 iters, 64-byte dk) as a raw string — the classic @ByteArray(salt+hash) wrapper 401s. Verify by surviving a podman restart (login 204). API login from curl also needs Referer: http://127.0.0.1:8080/ (CSRF). The live password lives host-local in /home/dank/.qbpass (0600, owner dank) on Dankserver — never in git. Recipe + gotchas: serverAppConfigs-docs/Dankserver/qbittorrent.md. (context: Installed 2026-08-30; all four arr download clients updated to the permanent password and validated (202) via each app's PUT /downloadclient.)
• Prowlarr→arr Torznab wiring (Dankserver, verified live 2026-08-30, whole grab→import chain proven): (1) Feed route is http://prowlarr:9696/{indexerId}/api?apikey={GLOBAL_KEY}&t=caps|search|... — there is NO /torznab/{id}/feed; any unmatched path hits the Forms-auth catch-all and returns 302 → /login?returnUrl=..., which arrs XML-parse into a fake "auth error" ("'/' is an unexpected token ... Line 54"). Feeds auth with the Prowlarr GLOBAL key from config.xml ApiKey; per-app keys are sync-only. (2) The arrs' torznab joiner is TrimEnd(baseUrl,'/') + apiPath with NO separator — set apiPath="/api" WITH leading slash. (3) Freshly created indexer rows default ALL-OFF — must set enableRss/enableAutomaticSearch/enableInteractiveSearch or the app logs "Searching indexers for [X]. 0 active indexers" (bit Lidarr). (4) Prowlarr v1 Application auto-sync is broken by version skew against Radarr 6.3 (POST /api/v1/applications/test → "Unable to cast object of type 'NzbDrone.Core.Thing...'") — manual torznab rows are the real integration; sync can't duplicate/clobber them. (5) Live search endpoints: Sonarr v5 interactive search = GET /api/v3/release?seriesId=N (POST /search and /release/search are 404/405); Lidarr v1 = GET /api/v1/release?albumId=N. Lidarr POST /artist requires foreignArtistId + qualityProfileId + metadataProfileId (lookup rows carry foreignArtistId, not foreignId). (6) Readarr (abandoned upstream 0.4.x) cannot authenticate against qBittorrent 5 while Radarr/Sonarr/Lidarr pass with identical credentials — books pipeline parked. Full docs: serverAppConfigs-docs/Dankserver/prowlarr.md. (context: Repo /home/dank/Projects/serverAppConfigs; first real grab (Dark Knight 2160p) downloaded and imported proves the chain; 76 Sonarr + 18 Lidarr releases via Nyaa feed id 4.)
• frank-default-large (agentionai Qwen3.8-Flash-Next ROCmFP4 FAST-imatrix-v2, qwen4exp MoE: 12 attention + 36 SSM layers) loaded successfully at CTX_SIZE=401408 / PARALLEL=2 (200,704 tokens per slot) on Dankclaw (119.3 GiB HIP heap) without OOM. Load time: ~74s. Health: 200. The model's architecture explains why this fits: only 12 attention layers use KV caches, 36 SSM layers don't — KV memory grows as O(L_attn × n_heads × head_dim × ctx) per slot, not O(L_total × ctx). This compressed KV footprint allows 200K per slot × 2 slots within the 119.3 GiB heap. models.json contextWindow set to 200000 (real per-session capacity, not the full 262144 n_ctx_train). The n_ctx_seq (200704) n_ctx_train (262144) warning fires but is non-critical — model extrapolates fine.
• frank-default-large (agentionai Qwen3.8-Flash-Next ROCmFP4 FAST-imatrix-v2, qwen4exp MoE: 12 attn + 36 SSM) on Dankclaw loads successfully at CTX_SIZE=401408 / PARALLEL=2 (200,704 per slot) in ~74s without OOM on the 119.3 GiB HIP heap. The model's architecture (only 12 attention layers use KV caches, 36 SSM layers don't) prevents the expected ~160 GiB context buffer. Per-slot KV grows as ~O(L_attn × n_heads × head_dim × ctx), not O(L_total × ctx). The n_ctx_seq (200704) n_ctx_train (262144) warning fires but is not critical — the model extrapolates. Ro cm-smi vram only shows the 4 GiB CPU carve-out; trust rocminfo or the device line for the full SVM pool (116732 MiB). The deployment verified health 200. models.json contextWindow: 200000 reflects the real per-session capacity.
• When --flash-attn on aborts graph build with GGML_ASSERT(a-type == GGML_TYPE_F32) at ggml.c:5371 in the QSA indexer path (build_attn_qsa calls ggml_fill on F16-cast kq_mask): apply the QSA F32-mask patch pattern — mirror the MLA sparse path in llm_graph_context::build_attn by using the F32 base mask (inp-self_kq_mask) for the top-k surgery (ggml_fill/set_rows), then cast to F16 only at the attention call when flash_attn is enabled. The patch has two hunks: 1) change get_kq_mask() to self_kq_mask with explanatory comment about F32 requirement; 2) insert conditional casting before build_attn_mha: "kq_mask_attn = inp-cparams.flash_attn ? ggml_cast(..., GGML_TYPE_F16) : kq_mask_top_k". Validate live: stop service → rebuild fork binary after applying patch → start on port 7999 with --flash-attn on --ctx-size 32768 --fit off → verify model loads (check log for "model loaded"), health 200, real completion → deploy production. Patch file should be vendored in scripts/patches/ with sha256 documented for reproducible rebuilds. (context: frank-default-large (agentionai Qwen3.8-Flash-Next ROCmFP4-FAST imatrix) fa-on restore on Dankclaw, 2026-08-29. Source of truth: scripts/patches/qwen4exp-qsa-f32-mask.patch (sha256 f2f4a486e21cfad80bfa42683281d18e1cba22fa43c5e0e2aa0cfdbe141e925f).)
• frank-default-large (Dankclaw, charlie12345/ROCmFPX build-qwen38) model swap to agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF (87.06 GiB Q4_0_ROCMFP4_FAST, 2026-08-29) — three verified hard constraints not to re-derive: 1. MUST use the v2/ joined-layout GGUF, never the root -ple16 file: build-qwen38's qwen4exp loader (src/models/qwen4exp.cpp:108) asserts GGML_ASSERT(ple_w != nullptr) and the joined file carries the single per_layer_token_embd.weight tensor the loader requires. The -ple16 (per-head split, 93.5 GiB) file only loads in the LaurentZuijdwijk Vulkan fork. Verified via GGUF header strings (ple16 lacks the tensor; v2 has it, 1153 .weight tensors). 2. --flash-attn on ABORTS graph build on this model: the QSA indexer path (build_attn_qsa in src/models/qwen4exp.cpp) calls ggml_fill on the F16-cast kq mask, which asserts GGML_TYPE_F32 (ggml.c:5371). Model loads and serves fine with --flash-attn off (verified live: health 200, real completion). STRIX_LEAN didn't hit this — agentionai's QSA-compressed layers do. 3. CTX_SIZE is capped at 32768 (the author's quick-start): the model is FULLY GPU-resident (PLE n-gram table is inside the GGUF, no streaming from RAM like STRIX_LEAN) leaving ~32 GiB of the 119.3 GiB HIP heap for context. Verified OOM ladder: ctx 262144 → cudaMalloc 103176 MiB failed; 131072 → 51976 MiB failed; 32768 → loads clean in ~58s. HIP heap measured via rocm-smi/rocminfo (125105988 KB ≈ 119.3 GiB); rocm-smi --showmeminfo vram reports only the 4 GiB carve-out, NOT the SVM pool — trust rocminfo heap for budget math. MTP draft head (agentionai's own, 2.28 GiB) is INCOMPATIBLE with build-qwen38 even against its own matched v2 main — THIRD confirmation of missing tensor blk.0.hc_attn_norm.weight (loader requires full per-layer set; every Flash-Next-era head carries only blk.48). Spec decoding runs ngram-mod only. MTP on this arch requires a head with all 48 hc_attn_norm tensors or a fork tolerating partial draft tensors. (context: Deploying agentionai Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2 (87 GiB) to frank-default-large on Dankclaw, 2026-08-29. Three crash-loop causes diagnosed live (ple16 loader assert, fa-on graph assert, ctx OOM ladder), each fixed and verified before the final healthy deploy at commit 101b1f8.)
• Dankclaw authentik 2026.x build does NOT mount the outpost URL space for proxy-provider meta-header routing: probed live from inside caddy's netns, /outpost.goauthentik.io/ping answers 204 with X-authentik-meta-outpost header (outpost object exists), but every app path with full meta headers (outpost UUID + provider id + app slug) returns 404 — the middleware does not intercept in this build regardless of header format (name=caddy and bare caddy both 404). The one auth URL that answers is /if/flow/default-authentication-flow/?next=... (200, serves login page in place; completes instantly with a valid session and bounces to next=). Working transport pattern shipped in the Caddyfile: guarded vhosts on http://:8016 issue redir https://auth.franksdank.cloud/if/flow/default-authentication-flow/?next=https://app-host/ 302 instead of meta-header reverse_proxy — unauthenticated sees login in place, authenticated pays one extra hop per request. If a future Caddy reload reintroduces meta-header routing on :8016, it will 404 again — this is the regression signature. Provider pk=1 lives at id column (not uuid) in authentik_core_provider in this build; app slug is caddy.
• Dankclaw auth chain, state as of 2026-08-30: guarded apps (search/romm/files/etc.) route the whole request through authentik-server:9000 with X-authentik-meta-outpost/-provider/-app headers, but in this 2026.x build that path answers 404 even with full meta headers — the embedded outpost's URL space is mounted at its authentik_host (DB row uuid 7e39655b-58d3-4fe6-a80a-513f31051217, type=proxy, managed goauthentik.io/outposts/embedded, authentik_host=https://auth.franksdank.cloud), which is the auth.* vhost kept EXEMPT from Anubis by design. So post-auth landing = authentik's own 404 page = the user's "can't access searxng after logging in". Verified intact: Anubis→Caddy hop (200 from inside anubis netns), provider pk=1 exists, www callback URIs match external_host (fixed previous turn). PENDING USER DECISION between two shapes: A) point the guarded apps' auth hop at where auth actually lives (route /outpost.goauthentik.io/* + /if/flow/* through the auth.* vhost — one-line Caddyfile change per app, small blast radius, my lean) vs B) flip the outpost to standalone (own container + quadlet unit + AUTHENTIK_TOKEN; heavier). When the user answers A or B, execute that shape; verify before/after by dialing from inside the container netns (podman run --rm --network container:caddy ... wget -S), never from the host.
• Dankclaw edge facts (verified 2026-08-29): the homepage hostname moved to www.franksdank.cloud — Caddyfile now 308-redirects naked/legacy names onto www at the TLS edge, and the authentik proxy provider external_host + OAuth2 stored _redirect_uris were flipped to www as a pair (strict matching rejects any callback not byte-identical to a stored URI; that rejection is what users see as "redirect URI mismatch"). Public DNS: zone franksdank.cloud is served by Porkbun nameservers (salvador/maceio/fortaleza/curitiba.porkbun.com); ingress for all service A records is port-forwarded to WAN IP 71.227.227.172 (not a tunnel), and www.franksdank.cloud had NO public record at all — created www A 71.227.227.172 TTL 300 via Porkbun API v3 that day. A wildcard/already-issued cert means TLS for any new *.franksdank.cloud hostname is instant on first handshake (verified: www answered 200 in 10 ms). Invariant to re-check before declaring any hostname move "working": every vhost carrying traffic must have (a) a public DNS record and (b) a provider external_host + redirect URIs matching that exact host — repo-text tests cannot see either; they live in the live DB and the public zone. Porkbun API v3 shape gotchas: endpoint is dns/retrieve/{domain} (not getAll — wrong path answers with an HTML 404 page, not JSON), and the response puts records under a top-level records key with a cloudflare flag; content is null. Creds live in /home/dank/serverAppConfigs/dankclaw/.env on dankclaw (PORKBUN_API_KEY/PORKBUN_API_SECRET_KEY) — use in place, never print.
• Dankclaw Authentik (2026.x image) operational truths, verified live 2026-08-29: (1) The embedded proxy outpost matches providers by EXACT external_host hostname — a request arriving with Host=www.franksdank.cloud while the provider's external_host is https://franksdank.cloud matches ZERO providers and authentik answers its error page (not 404). Fix applied live: UPDATE authentik_providers_proxy_proxyprovider SET external_host='https://www.franksdank.cloud' WHERE oauth2provider_ptr_id=1 via podman exec authentik-postgres psql -U authentik -d authentik; no restart needed — providers resolve per request. Caddyfile now routes www as the guarded vhost and naked domain + legacy names 308-redirect at the TLS edge (commit 8898740). (2) Live-DB probe recipe: ak shell is interactive-only; ORM imports fail because 2026.x moved models out of Python modules (ProviderEndpoint, ProxyProvider, Token gone from their classic modules); query postgres directly with role/db authentik (POSTGRES_USER=authentik, NOT postgres). Column names differ in this version — pk/id may not exist on user/group tables; use \d or SELECT * instead of guessing columns. (3) Management commands: ak create-install-token does NOT exist in this version (Unknown command); admin seat = membership in the 'authentik Admins' group (is_superuser=t), created via ak create_admin_group user which takes a required positional user arg; live users are akadmin (frank8465@protonmail.com) and admin (admin@franksdank.cloud), both with real pbkdf2 hashes — lost-password escape hatch is ak create_recovery_key. (context: Fixing www.franksdank.cloud serving an authentik error page while the naked domain served the homepage; traced to proxy-provider external_host exact matching, fixed via live DB UPDATE + Caddyfile topology flip deployed through scripts/deploy.sh --service caddy.)
• Dankclaw Authentik (2026.x image) live-ORM probing moved to direct SQL: model imports like from authentik.core.models import ProviderEndpoint / authentik.outposts.models.ProxyProvider now raise ImportError — query the DB directly instead (podman exec authentik-postgres psql -U authentik -d authentik; role/db = authentik, NOT postgres). Introspect before guessing columns: several tables use uuid PKs (groups), so id/pk probes error; SELECT * / \d first. Command surface verified live: NO create-install-token command exists in this version ("Unknown command"); createsuperuser is explicitly deprecated ("should not be used. Instead, use ak create_admin_group"); canonical admin creation = ak create_admin_group user (takes a required positional user). Lost-password escape hatch = ak create_recovery_key. ak swallows piped stdin; works via ssh -tt + podman exec -it which allocates a real PTY. Live state at capture: admin seat held by user akadmin (email frank8465@protonmail.com) in authentik Admins superuser group; second account admin (admin@franksdank.cloud) membership unverified; both carry real pbkdf2 hashes — login failures there are password-memory problems, not missing accounts. Embedded outpost uuid matches Caddyfile meta-outpost header; provider pk=1 exists with internal_host http://caddy:8017. (context: Live status investigation of "Authentik is still not working properly" on Dankclaw — probed DB/commands live instead of trusting docs; several column-name guesses failed before SELECT * / \dt introspection ended the guessing.)
• When Frank asks to "research", "look into", or "hunt" something on the homelab, he wants a verified state REPORT first — not immediate deployment. Acting on findings before his approval draws pushback ("I only said research, don't bring it up immediately"). Correct pattern: probe live state (DBs, logs, APIs, disk), report verdict + evidence + the single unblocking move with cost, then wait for his explicit go before mutating anything. He granted blanket autonomy only when he re-pins a directive verbatim.
• MTP draft heads for Qwen3.8-Flash-Next are INCOMPATIBLE with the STRIX_LEAN Uncensored model on the charlie12345/ROCmFPX qwen4exp binary. Both attempted heads — EasiiX Q8_0 (4.1 GB) and agentionai ROCmFP4-FAST (2.28 GiB) — fail with missing tensor 'blk.0.hc_attn_norm.weight'. The qwen4exp binary expects this hyper-connection attn norm tensor in the draft GGUF, but neither draft head carries it. MTP only ever worked on the 27B model (kingjones-Qwen3.8-27B-STRIX-MTP with mtp-Qwen3.8-27B-Q4_0.gguf), never on any Flash-Next variant. The Flash-Next STRIX (non-LEAN) that ran before the 2026-08-28 session never loaded a draft model at all. Current production config: SPEC_TYPE=ngram-mod only. To get MTP working on STRIX_LEAN, need either a draft head that includes hc_attn_norm tensors or a binary that doesn't require them for draft models. (context: frank-default-large MTP integration attempt on Dankclaw, 2026-08-28. Downloaded two MTP heads from HuggingFace, both failed to load against the STRIX_LEAN Uncensored model. Service crash-looped on each attempt. Fell back to ngram-mod only. Files: scripts/frank-default-large.env, scripts/serve-frank-large.sh.)
• The MTP draft head GGUF mtp-Qwen3.8-Flash-Next-Q8_0.gguf (EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF, 4.1 GB) is INCOMPATIBLE with the STRIX_LEAN Uncensored Flash-Next model on the charlie12345/ROCmFPX binary (build-qwen38). The draft GGUF is missing tensor blk.0.hc_attn_norm.weight (hyper-connection attention norm) that the qwen4exp architecture expects. The service crash-loops: main model loads fine (~59s), then draft model load fails and exits. MTP only ever worked on the older 27B model (Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT + its own mtp-Qwen3.8-27B-Q4_0.gguf head). The Flash-Next STRIX (non-LEAN) that ran before this session never loaded a draft model either — it ran ngram-only. Current production config: SPEC_TYPE=ngram-mod (no draft sidecar). TODO: find a compatible MTP head for STRIX_LEAN or build one from the Flash-Next checkpoint. (context: frank-default-large MTP draft head integration on Dankclaw, 2026-08-28)
• When running Python scripts on a remote host (Dankclaw) that need to manage local services (systemctl, sed), use subprocess.run(cmd, shell=True) for local execution — NOT nested SSH back to the same host. Nested SSH to localhost fails (no key auth for self-loopback). Also: before git pull --ff-only on a remote, verify the branch with git branch --show-current — Dankclaw was on frank-large-strix-model not main, so pulls were no-ops and new files didn't appear. (context: bench-spec-decode.py ran on Dankclaw but used ssh() to SSH back to itself for systemctl/sed. All SSH calls failed with Permission denied. Also: Dankclaw was on frank-large-strix-model branch, not main, so git pull --ff-only was a no-op and the new file didn't appear.)
• When deploying to Dankclaw (or any remote host), verify the remote checkout is on the correct branch (main) before pulling. On 2026-08-28, Dankclaw was on frank-large-strix-model with uncommitted local changes (STRIX_LEAN model paths + mmproj), causing git pull to be a no-op on the wrong branch. The benchmark script wasn't found because it was committed to main but Dankclaw was on a feature branch. Fix: git stash && git checkout main && git pull --ff-only on the remote. (context: frank-default-large MTP benchmark deployment on Dankclaw. The git pull said "Already up to date" because it was on frank-large-strix-model, not main.)
• The docs-completeness test (test 112, tests/docs_completeness.bats) scans ALL .md files under docs/ — including untracked files — and requires every one to be linked from docs/index.md via Title wikilink. Untracked files like docs/Clippings/ will fail this test and block the pre-commit hook. Fix: move untracked docs that aren't meant to be part of the knowledge base to .AGENTTRASH/ before committing. (context: Committing MTP draft head changes to frank-default-large was blocked by pre-existing test 112 failure caused by untracked docs/Clippings/Caddy Anubis.md.)
• To verify GGUF model architecture compatibility without llama-gguf or Python GGUF parsing: strings on the first ~2KB of a GGUF file extracts general.architecture, general.name, general.size_label, and all arch.* metadata keys (block_count, embedding_length, expert_count, etc.). For remote files: curl -sL -r 0-2048 'url' | strings | head -40. The architecture string (e.g. qwen4exp, qwen3moe, llama) is the primary compatibility key — a draft head GGUF must report the same general.architecture as the target model. Cross-reference against the serving binary's --help output for the relevant feature flags (e.g. --spec-type draft-mtp, --model-draft) before declaring compatibility. The Python struct.unpack approach for GGUF header parsing is fragile (fails on custom type codes); strings is more robust for a quick architecture check. (context: Assessed EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF MTP draft head compatibility against frank-default-large (Qwen3.8-Flash-Next qwen4exp) on Dankclaw. Needed to verify architecture match between the draft head GGUF and the target model GGUF without llama-gguf binary available.)
• On these hosts (Dankclaw, Dankserver), systemctl disable service doesn't just remove symlinks — it deletes the unit file itself from /etc/systemd/system/. Use systemctl mask (creates symlink to /dev/null, safe and reversible via unmask) instead of disable during benchmarks. The unit files are in scripts/service.service in the repo. (context: AGENTS.md benchmarking policy section updated. Occurred when running bench for frank-default-large on dankclaw.)
• frank-default-large (Qwen3.8-Flash-Next ROCmFP4 STRIX on dankclaw) decode throughput across threads-batch: TB=8: 3.3 tok/s (76.6s), TB=16: 3.4 tok/s (74.8s), TB=32: 3.4 tok/s (75.0s). Benchmark: ~23.5K token code prompt, 256 generated tokens, HTTP API, fresh server per value. Decode throughput is flat across all tested values; TB=16 is optimal due to prefill advantage (373.5 tok/s vs 354.4 at TB=32). TB=8 is marginally worse than 16 on both prefill and decode. (context: bench-decode-threads-batch.py ran on dankclaw. Model: kingjones-Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf (113.47 GiB). Prompt: /tmp/bench-prompts/prompt_code.txt extended to 8200 words (~23.5K tokens).)
• On dankclaw and dankserver, systemctl disable frank-default-large (and similar systemd service units) doesn't just remove symlinks — it deletes the unit file itself from /etc/systemd/system/. Always use systemctl mask instead (creates symlink to /dev/null, safe and reversible with unmask). When restoring after a disable: copy the unit file from this repo back to /etc/systemd/system/, run systemctl daemon-reload, then enable + start.
• When benchmarking llama.cpp server-side parameters (--threads-batch, --ubatch-size, etc.), use HTTP-based prefill benchmarking: start the server with the target parameter, send a large prompt via /v1/chat/completions with max_tokens=1 (the server crashes on max_tokens=0), measure prompt_tokens / wall_time from the response usage. Do NOT use llama-bench — it benchmarks the bench binary itself, not the server, and cannot target server-side parameters. Script at scripts/bench-threads-batch-http.py demonstrates this pattern.
• frank-default-large (Qwen3.8-Flash-Next ROCmFP4 STRIX on dankclaw) best threads-batch: 16. Measured prefill throughput sweep: tb=16: 373.5 tok/s, tb=32: 354.4, tb=48: 344.1, tb=64: 344.7, tb=80: 344.4, tb=96: 357.1, tb=128: 321.1. Lower thread batching wins for prefill — fewer threads compete for GPU resources. Used HTTP-based benchmarking (llama-bench cannot benchmark --threads-batch, it's a server-side parameter). Used max_tokens=1 instead of 0 (server crashes on 0-token requests). Script: scripts/bench-threads-batch-http.py.
• When editing bash scripts that use :: for default-value expansion (e.g., :: "${VAR:=default}"), the edit tool's parser can misinterpret :: and produce mangled output with extra characters (literal ^ prefix or ::: instead of ::). For targeted fixes to these specific lines, use sed in-place instead of the edit tool. Example: sed -i 's/^:: "/: "/' script.sh. (context: frank-default-large model swap: edited scripts/serve-frank-large.sh default-value lines from :: to single-colon. The edit tool produced ::: with a literal ^ on line 40. Required sed fallback to fix.)