back-merge: main -> develop after PR #5 (status line custom segments)

Conflicts resolved in main's favor: develop's models.yml refactors
(frankdank.cloud consolidation, provider split) predate the October
callsign-rename cutover on main; the two dank-user memory buckets are
superseded by main's learn output.

agent/memories/--home-dank-.omp--/learned.md

@@ -1,4 +1,17 @@
-- Gitea on Dankclaw (192.168.0.143:3000) has the **Projects unit disabled instance-wide** — the Projects API returns 404 even with a correct repo_id; don't re-probe it, kanban boards there are web-UI-only and are Frank's one-time manual task. In the ~/.omp command-center repo (remote = Gitea dank/command-center): v1.0.0 was cut and shipped via full git-flow; release/1.0.0 was deliberately pushed to the remote to preserve the hotfix cut-point, while the merged feature branch docs/kb-rebuild stays local-only by decision. The Gitea access token lives only in .git/config remote URL — never commit it anywhere else. Session end state: repo checked out on develop, clean tree (because the checkout *is* the live deployment, a session must never end on a feature branch). _(context: End of command-center build-out session; verified via git ls-remote and two 404 probes of the Projects API.)_
+- 2026-09-14 the workstation outage: upstream commit 49afdfe ("Noctalia theme migration") committed config.kdl with include "noctalia.kdl" but noctalia.kdl is a per-host GENERATED file, never in the repo; the same pull set ghostty theme = noctalia and alacritty import = themes/noctalia.toml, both also template-generated and absent here. Niri 26.04 does NOT exit on a config error — it logs "error parsing" and boots with DEFAULT config, producing a half-dead desktop (no keybinds/rules/autostart shell, default output placement). noctalia msg templates-apply answers "ok" but is a NO-OP on noctalia 5.0.1 (opt-in schema mismatch). Working fix (offline, no session restart needed): noctalia theme --theme-json ~/.config/noctalia/colors.json --render "builtin template:output" for each: templates/niri/niri.kdl → ~/.config/niri/noctalia.kdl (plain host-local file, NOT a repo symlink), templates/ghostty/ghostty → ~/.config/ghostty/themes/noctalia, templates/alacritty/alacritty.toml → ~/.config/alacritty/themes/noctalia.toml. Tokens resolve fully from the 16-key colors.json (primary_container etc. are derived). Verify: niri validate rc0; fresh login loads full config (journal "loaded config" + custom output placement). Also verified: kill -USR1 was NOT the cause of the session ending that night — journal showed a fresh niri-session login; don't accept blame for user-initiated logouts.
+- On this host's niri build, niri msg do reload-config is rejected ("unrecognized subcommand 'do'") — verified 2026-09-10 during the noctalia unification deploy. The working config-reload path is kill -USR1 $(pidof niri) (SIGUSR1 to the niri process); verified on both fleet hosts (the workstation pid via /proc, blankpad over ssh). Future keybind/config deploys that need live reload must use the SIGUSR1 path, not niri msg do.
+- Noctalia fleet architecture UNIFIED 2026-09-10: both the workstation and blankpad now run bare noctalia (v5 native, package noctalia 5.0.1-1.1) as the desktop shell, launched via niri spawn-at-startup "noctalia" from repo autostart.kdl. The v4 Quickshell-hosted architecture (qs -c noctalia-shell, JSON settings.json) is retired on the workstation; qs host process killed and replaced live by a transient systemd-run unit (noctalia-live) that hands off to autostart at next login — single-instance guard in the binary makes overlap safe. Live config layer on both hosts = TOML only: hand-edit ~/.config/noctalia/config.toml; runtime state ~/.local/state/noctalia/settings.toml loads last and overrides same keys (never hand-edit). Verified mechanics: noctalia msg resolves the running instance via D-Bus session bus, so it works from systemd user-unit context (systemd-run --user --pipe --wait) even though niri's own environ lacks WAYLAND_DISPLAY; spawn-at-startup children do receive a working display connection. Lock keybinds/swayidle unified to noctalia msg session lock; verified verb grammar includes panel-toggle {launcher,session,control-center,...}, volume-up/down/mute, mic-mute, media next/previous/playPause (graceful no-op rc 0 with no active player), brightness-up/down. Wallpapers: wallpaper-next/wallpaper-get confirm rotation draws from the configured [wallpaper] directory on both hosts; current wallpaper stays on the built-in default asset until first rotation after config-reload. Leftover dead packages on the workstation (noctalia-shell 4.7.7, noctalia-qs fork) are inert — removal is optional cleanup, zero dependents post-flip.
+- Divergence explained (as of 2026-09-10): the workstation and blankpad both show noctalia 5.0.1-1.1 but run different live architectures because each host's autostart froze at its adoption-era architecture and package upgrades never rewrote autostart commands. Fossil record from pacman install dates: noctalia-shell 4.7.7-3 installed on the workstation Jul 3 (v4 Quickshell-hosted architecture, JSON config; module files owned at /etc/xdg/quickshell/noctalia-shell/, launched via qs -c noctalia-shell from base niri package autostart.kdl); noctalia-qs 0.0.12-2 (Quickshell fork hosting the shell) installed Aug 24; v5-era noctalia package landed Sep 6 as a parallel package owning ONLY /usr/bin/noctalia + desktop file + README — CLI/client surface, no shell module. blankpad was provisioned fresh Sep 9 entirely from CachyOS repo packaging (noctalia 5.0.1-1.1 + vanilla quickshell 0.3.1-1.1 + cachyos-niri-noctalia 1.4.0-1; no v4-era packages exist in that repo set), so its zz-blankpad override autostart launches bare noctalia (native v5, TOML config). Consequence: settings.json is genuinely the live config on the workstation (v4 code), while blankpad runs true v5 with TOML state at ~/.config/noctalia/config.toml + ~/.local/state/noctalia/settings.toml. Unification candidate when approved: flip both to noctalia as single entrypoint, but verify first that the v5 packaging on the workstation ships the shell module (current -Ql evidence says CLI-only) — a unified flip likely requires installing the full v5 package set on the workstation first. Diagnostic recipe for this class of question: query pacman install dates + package ownership (pacman -Qi/-Ql/-Qo, pgrep -af) to reconstruct which architecture each host actually runs rather than trusting version labels.
+- Noctalia shell runtime split on the fleet (as of 2026-09): both the workstation and blankpad install noctalia 5.0.1-1.1 (-dirty = git build), but launch differently — the workstation hosts it via qs -c noctalia-shell (Quickshell host; reads JSON-era config ~/.config/noctalia/settings.json), blankpad runs the bare v5 binary (reads hand-edit ~/.config/noctalia/config.toml, runtime state written to ~/.local/state/noctalia/settings.toml which overrides declarative values last). Lock chain on both hosts: primary qs -c noctalia-shell ipc call lockScreen lock, fallback noctalia msg session lock|suspend|lock-and-suspend|logout|reboot|shutdown. Quirk: noctalia msg ... from a non-session context (ssh command exec) always answers "noctalia is not running" even when the shell runs — detection needs session-scoped state absent in sshd-spawned shells; DBUS_SESSION_BUS_ADDRESS override does NOT fix it. Correct remote procedure: write config files directly (they hot-reload or apply at next start); never treat the msg failure over ssh as evidence the shell is dead. Wallpaper dir default when unconfigured = XDG Pictures ($HOME/Pictures) via user-dirs.dirs; explicit [wallpaper] directory key overrides it — blankpad was seeded with directory = /home/blank/.local/share/noctalia/wallpapers (repo-tracked symlink target).
+- blank@blankpad (CachyOS, Arch-like) deployment COMPLETED 2026-09-09 (supersedes earlier "pending action awaiting user go-ahead" note): full-replica rsync of /home/dank/.omp/ → /home/blank/.omp/ (.git included; push to Gitea NOT required for this transport) followed by deploy.sh sync on blankpad. SSH was initially blocked there by firewall default-drop (timeout, not refuse) and opened only after the user started sshd locally — expect that blocker pattern again for new fleet hosts. Host facts verified at deploy time: login shell fish; niri runs as niri --session reading the repo-symlinked config.kdl; installed binaries include noctalia CLI (its keybinds override package uses noctalia msg ..., NOT canonical qs -c noctalia-shell ipc call), alacritty, firefox, nautilus, fish, zsh, shelly-notifications; MISSING: qs/quickshell, ghostty, gnome-calculator — canonical keybinds relying on those are silent no-ops if deployed there. ~/.p10k.zsh on blankpad is an inert repo symlink until powerlevel10k-zsh is installed and sourced. zz-blankpad override package (display/keybinds/misc/rules/autostart/.zshrc) is tracked in the repo; deploy.sh's target derivation must strip the nested base-package segment for zz-* packages or overrides land on phantom paths like $HOME/niri/.config/...
+- blankpad is a fleet machine reached only as blank@blankpad.local (mDNS FQDN; bare hostname does not resolve from the workstation; 192.168.0.83). As of 2026-09-09 it runs the command-center deployment at /home/blank/.omp with host-specific divergence preserved as tracked override package dotfiles/zz-blankpad/: its own display.kdl output layout (not the workstation's DP/eDP rig), autostart/keybinds targeting the bare noctalia CLI instead of qs -c noctalia-shell ipc call, misc.kdl without renderD129, rules.kdl without the Noctalia floating-window rule, and blank's own minimal .zshrc. Its login shell is fish. Initial blocker (port 22 drop-not-refuse = sshd down or firewall default-drop) was fixed same day by user locally; if SSH drops again on blankpad, suspect firewall default-drop first, not auth.
+- blankpad (user blank) is fleet member at 192.168.0.83, reachable ONLY via mDNS FQDN blankpad.local — bare hostname does not resolve from the workstation. As of 2026-09-09 its SSH was unreachable from the LAN segment: ping fine (~1 ms, wired) but TCP 22 and alt ports (2222/8022/22222/10022) all TIMEOUT = firewall default-drop or sshd down; drop-not-refuse distinguishes this from sshd-listening-but-auth-failing. Pending action awaiting user go-ahead: deploy command-center to blank@blankpad via full-replica rsync of /home/dank/.omp/ (incl .git so core.hooksPath stays armed pre-push) → deploy.sh --check review → sync; transport deliberately chosen over clone-from-Gitea because main was 6 commits ahead of origin/main and push is reserved as a user-only trigger. Note: origin remote on the workstation embeds the Gitea token in plaintext URL — keep it off other machines' .git/config by using rsync as transport. _(context: User asked to set up the dotfiles/command-center deployment on blank@blankpad with a password credential; SSH entry point unreachable from this segment, so work was staged and handed back with exact unblock steps (systemctl enable --now sshd + firewalld internal-zone ssh service).)_
+- The FLM (FastFlowLM) service on Dankclaw (gemma4-it:e4b, port 7771) only supports PNG and JPEG/JPG image formats. WebP images are rejected with "Unsupported image format, skipping this image." The data URL formats it expects are data:image/png;base64, data:image/jpeg;base64, and data:image/jpg;base64. To use WebP screenshots with the NPU, convert them to PNG first. Verified 2026-08-30: a WebP screenshot (1024x576, 71K) was rejected while the same content as PNG was correctly identified by the vision model. _(context: Dankclaw NPU (FLM, gemma4-it:e4b, port 7771) — image format support investigation)_
+- When configuring dankclaw-npu (FLM NPU endpoint on port 7771) in models.json, the model entry must include a thinkingLevelMap field. The other two providers (dankserver, dankclaw) have it; if absent, schema validation fails. Pattern: {"thinkingLevelMap": {"max": "xhigh"}}. Also verify gemma4-it:e4b model ID is registered and .xclbin kernel symlink exists under /home/dank/.config/flm/xclbins/. _(context: Dankclaw NPU endpoint (dankclaw-npu provider, port 7771) had a schema error because gemma4-it:e4b was missing thinkingLevelMap. Fixed by adding it to both models.json and models.yml. The FLM NPU serves 34 models in its registry; gemma4-it:e4b is the currently wired model.)_
+- Dankclaw NPU serves 34 FLM (FastFlowLM) models on port 7771. Only one is wired in models.json (gemma4-it:e4b) but all 34 are registered and available via per-request model routing without restart. Known behavior: single-lane NPU queues FIFO, per-request model switch costs ~7–11s wall for 4B-class reload. Common pitfall: some registered tags fail if .xclbin kernel pack symlink is missing under /home/dank/.config/flm/xclbins/ (fix: ln -s /opt/fastflowlm/share/flm/xclbins/PackName). Registry probe: curl http://192.168.0.143:7771/v1/models.
+- Frank's cursor size is 36 (up from default 24) on the workstation running niri/Wayland. Applied via three channels: gsettings org.gnome.desktop.interface cursor-size=36 (primary), GTK settings.ini gtk-cursor-theme-size=36 (fallback), XCURSOR_SIZE=36 in ~/.profile (sessions). Cursor theme: capitaine-cursors. _(context: the workstation workstation — Frank requested a bigger mouse pointer; set to 36 across gsettings, GTK config, and environment)_
+- Frank's default browser is Firefox (firefox.desktop), set via xdg-settings on the workstation (niri/Wayland). Previously was Chromium. Changes persist across reboots. _(context: the workstation workstation — Frank changed default from Chromium to Firefox via xdg-settings set default-web-browser firefox.desktop)_
+- the workstation (RTX 4070 Laptop, nvidia-open 610.57.04, kernel 7.2.0-1-cachyos, DynamicPowerManagement=2) suffers the known 610.x NVKMS DIFR-prefetch deadlock: nvidia-modeset/kthread_q livelocks in DifrPrefetchEventDeferredWork→nvWriteGpEntry holding the NVKMS API semaphore; any compositor (niri) atomic commit then wedges in D-state in ApplyModeSetConfig/nvkms_ioctl_from_kapi_try_pmlock. Symptoms: GUI frozen, no VT switch, SSH alive, nvidia-smi still answers (GPU idle P8, no Xid). Trigger: display-idle then flip/modeset. Unrecoverable in place — reboot via systemctl reboot --force --force. Candidate mitigations: NVreg_RegistryDwords="RMDisableDIFR=1" (one upstream report says ineffective) and DynamicPowerManagement=0. Bug class spans 595.71–610.57 upstream. _(context: 2026-08-30 GUI lockup diagnosis on the workstation; user suspected nvidia modesetter, confirmed via hung-task stacks. Pending user decision on writing modprobe mitigation.)_
 - When downloading large model shards (40GB) to dankclaw via SSH: 1. NEVER run rm on partial downloads — always clean up BEFORE starting the first download 2. Ensure only ONE wget process per file — if you see duplicate wget PIDs killing them with kill -9 before restarting, or use fuser to verify exclusivity 3. Use wget -c (continue) to resume partial downloads — SSH connections often drop mid-download on large files (~300s timeout) 4. Download sequentially (not in parallel) when using background task agents that may spawn duplicate processes, or coordinate downloads through a single agent session 5. Verify each file's size against expected values after download completes
 - kingjones777's Qwen3.8-Flash-Next ROCmFP4 builds for Strix Halo (gfx1151) have three tiers: FAST (~88 GiB), STRIX_LEAN (~98.5 GiB, 3-shard), STRIX (~113.5 GiB). All are text-only — no vision tower, no mmproj, no fused MTP draft head. Uses TYPE_100/TYPE_101 ROCmFP4 tensor types (general.file_type=106), requires patched llama.cpp PR #27742 from ROCmFPX fork. STRIX_LEAN carries a 35.76 GiB PLE n-gram table streamed off SSD at runtime; ~63 GiB GTT for non-PLE weights, ~75 GiB at full 262K context. Uncensored variants exist from orcarouter but only for Qwen3.8-27B (not Flash-Next), and in non-ROCmFP4 format. _(context: User requested kingjones777/Qwen3.8-Flash-Next-Uncensored-ROCmFP4-STRIX_LEAN-GGUF for frank-default-large on dankclaw. Search revealed the repo doesn't exist; only the base STRIX_LEAN tier is available, and it's text-only with no MTP.)_
-- Model provider config exists in TWO synchronized copies: ~/.pi/agent/models.json (source of truth) and ~/.omp/agent/models.yml (generated import; header comment states it is imported from the JSON). Any edit to providers, model ids, or display name labels must be applied to BOTH files in the same change or they silently drift. Validate after editing: jq -e '.providers|keys' models.json for the JSON and a YAML parse of the .yml. Label strings are NOT cached in ~/.omp/agent/models.db (model_cache table), so no cache flush is needed after label edits — new labels apply on the next config read.
+- Model provider config is SINGLE-SOURCE: `~/.omp/agent/models.yml` (OMP is the primary harness). The pi-era dual-copy convention (`~/.pi/agent/models.json` as source of truth + generated YAML import) is retired — there is no second file to keep in sync. Validate edits with a YAML parse. Label strings are NOT cached in ~/.omp/agent/models.db (model_cache table), so no cache flush is needed after label edits — new labels apply on the next config read.

agent/memories/--home-dank-Projects-serverAppConfigs--/learned.md

@@ -1,4 +1,79 @@
-- Dankclaw podman 5.7 quadlet gotchas (verified live during the Anubis rollout, 2026-08-28): (1) podman's reference parser rejects mixed-case ghcr.io refs at parse time — always use lowercase ghcr.io/techarohq/anubis:vX.Y.Z; ghcr resolves case-insensitively at pull. (2) The v1 quadlet generator cannot enable units on this host, and registry-prefixed image refs fail container create with "image not known" even when the image is in storage — run containers from a localhost/name:tag alias tag instead (deploy handler pulls canonical ref then podman tags the alias). (3) The generator joins bare HealthCmd into ONE argv element with \x20-escaped spaces → shell-form exec of a missing shell; distroless ko-build images (entrypoint /ko-app/anubis, no /bin/sh) need JSON exec-form health probes: HealthCmd=["/ko-app/anubis","--healthcheck"] — the generator passes a JSON array through verbatim. (4) Guard image pulls with podman image exists canonical-ref first; cold anonymous ghcr token requests can 403 on first fetch — retry once, second attempt resolves locally. (5) Anubis chain shape: Caddy TLS :443 → Anubis verify :8923 → accepted traffic back to Caddy internal listener :8016 → per-vhost routing; guarded set = homepage/search/hermes/prometheus/romm/files; exempt by construction: auth (IdP itself), plex (plain HTTP), gitea git smart-HTTP (git clients can't solve JS challenges), grafana (own login). _(context: Repo /home/dank/Projects/serverAppConfigs; docs already cover the chain in serverAppConfigs-docs/Infrastructure/Anubis.md and Architecture/Network Architecture.md; memory covers the podman/quadlet runtime quirks that bite during deploys on this host specifically.)_
+- Making a Hermes auxiliary call work against a LOCAL OpenAI-compatible backend on this fleet (solved for title generation on 2026-09-11). Symptom: every session logs "Title generation failed" and falls back to a message-derived title. Root cause chain, in the order worth checking: (1) Hermes hardcodes response_format: json_schema on the title call (agent/title_generator.py _TITLE_RESPONSE_FORMAT), and NO local backend implements constrained decoding — frank-small (FreeToken) and frank-large (halogen) both answer 400 response_format json_object/json_schema is not supported (no constrained decoding); (2) config CANNOT override it, because call_llm merges effective_extra_body = _get_task_extra_body(task); effective_extra_body.update(extra_body or {}) — the caller's dict wins; (3) only temperature and max_tokens have reactive retry paths (_is_unsupported_parameter_error is generic but is only wired to those two); (4) auxiliary.task.enabled: false does NOT avoid the failure cleanly — it sits before apply_instant_title, so it removes titles entirely. Working configuration, verified through the real code path: auxiliary.title_generation with api_mode: codex_responses (a real per-task config key, _resolve_ta[REDACTED] reads it) plus extra_body.reasoning.effort: none. The Responses transport builds resp_kwargs and never forwards response_format (it also omits max_tokens/temperature); the reasoning half is mandatory, not decoration — with thinking on, the 64-token cap is spent on the think and content comes back empty with finish_reason=length. reasoning.effort: none maps to {"effort":"none","summary":"auto"} + include, which FreeToken's /v1/responses honors (4 output tokens, 1.5 s). Note reasoning_effort: none as a *shorthand* yields {"enabled": False}, which the adapter treats as "omit reasoning" — leaving the server default (thinking ON) and reintroducing the empty-content failure; use the explicit extra_body.reasoning.effort form. Both local runtimes also expose /v1/respon _(context: Hermes auto-titling on Dankclaw after frank-npu was retired (2026-09-11): the NPU had been the only non-reasoning endpoint, so the title route fell to frank-small and failed on structured output. Verified with in-container probes calling agent.auxiliary_client.call_llm with title_generator's own constants; final config proven end to end (raw {"title": "Friendly greeting"} - 'Friendly greeting'), c)_
+- Compose→Quadlet cutover traps on this fleet (learned converting the five Dankclaw observability containers on 2026-09-11). (1) A unit's HealthCmd reaches podman as **CMD-SHELL** (/bin/sh -c). In a distroless image that silently produces a permanently-unhealthy container: the loki image has no shell, so the busybox command that worked by hand failed as a healthcheck, and podman does NOT accept exec form through --health-cmd (a JSON array is stored as one literal argument). The command has to live in the IMAGE as HEALTHCHECK ... CMD ["binary","arg"], which also means the build must pass --format docker — the default OCI image format has no HEALTHCHECK field and discards it with a warning. (2) Pointing a unit's EnvironmentFile at a repo .env is unsafe: this repo's ROOT .env is a gluetun env file that carries HOME=/root, so systemd sets HOME for the service, podman looks for its config under /root, and the unit dies with stat /root/.config: permission denied. The fleet convention is a per-service file (/home/dank/Containers/svc/env) seeded by the deploy handler with a single-variable awk filter, validated, so a missing secret fails the deploy loudly instead of starting a container whose login cannot complete. (3) For exposure, binding published ports to 127.0.0.1 is sufficient and safe: Caddy reaches containers by name over dankclaw_default and the host-side homepage status timer (/srv/homepage/services.json probes → homepage-status.py) probes loopback, so nothing legitimate needed the 0.0.0.0 publish. The exception is a Network=host exporter that a containerized Prometheus must scrape — it needs the LAN IP (containers cannot reach the host's loopback), so it binds loopback + LAN explicitly and is the only deliberate LAN listener. (4) Convert with matching ContainerName= so Caddy/scrape configs that address containers by name keep working, keep Pull=never for locally built images (loki/alloy), and delete the compose definitions in the same change so only one owner exist _(context: Converting prometheus/grafana/loki/alloy/node-exporter from dankclaw/compose.yaml services to rootless Quadlet units (repo commits dab81b9, fe4eeaa, b96359e, 25c9398, 701ab14), including the fixes found by deploying: the grafana HOME=/root failure, the loki CMD-SHELL/distroless healthcheck failure, and the OCI-vs-docker image format that dropped the HEALTHCHECK.)_
+- Retiring a repo-managed root systemd unit on Dankclaw/Dankserver (the fleet installs them as symlinks: relink_unit does ln -sfn /home/dank/serverAppConfigs/scripts/svc.service /etc/systemd/system/svc.service): systemctl mask svc FAILS with "File '/etc/systemd/system/svc.service' already exists and is a symlink to …" — mask refuses to replace an existing unit file. Correct order, and it also avoids the documented disable-deletes-the-unit-file quirk: (1) systemctl stop, (2) systemctl disable (may or may not delete the file on these hosts — tolerate both), (3) if the file/symlink still exists, mv it into ~/.AGENTTRASH/dated-topic/host-units/ (never rm), (4) mv the /etc/systemd/system/multi-user.target.wants/svc.service enable symlink too or it dangles, (5) systemctl daemon-reload, (6) now systemctl mask svc succeeds and prints "Unit … does not exist, proceeding anyway" then creates the /dev/null symlink. Also: moving the repo file to .AGENTTRASH before the host symlink is cleaned leaves a dangling unit, so do the host side in the same change as the repo deletion. _(context: Retiring the frank-npu systemd unit on Dankclaw as part of the 2026-09-11 frank-large swap; deploy-remote.sh's relink_unit installs root units as symlinks into the repo checkout.)_
+- peonist-ai/halogen-flash-server on Dankclaw (frank-large, deployed 2026-09-11) — traps worth keeping: (1) The podman 5.x quadlet **v1 generator** accepts every key this image needs (AddDevice, PublishPort, PodmanArgs=--ipc=host --security-opt seccomp=unconfined, Ulimit=memlock=-1:-1, HealthCmd, Pull=never) — verified with a throwaway unit before shipping; the generated ExecStart also carries --replace --rm, so every restart discards the container layer (fine when state is on a bind mount). (2) HALOGEN_DOWNLOAD fires only when the checkpoint is absent and requires a WRITABLE /models; the whole 127 GB repo fetched in 542 s (~240 MB/s) anonymously, no token needed. Start-to-health was ~12 s (engine PONG) when the weights were still in page cache, ~50 s after a cold reboot. (3) The startup ledger is the only truthful memory source: 68.0 GiB weights locked + 7.2 GiB KV pool (262144 positions) + 21.1 GiB working set, and free/MemAvailable overstate available memory by the model size because locked pages count as reclaimable cache — Grafana dashboards will be wrong by ~68 GiB. It also detected the host's 2.0 GiB BIOS iGPU carve-out. (4) Serving contract gotchas: no server-side sampler defaults (omitted temperature = GREEDY), the per-request budget defaults to 8192 and INCLUDES thinking (so a reasoning model + small budget returns empty content with finish_reason=length), the cap is 65536, and response_format is a hard 400 ("no constrained decoding"). (5) That last point matters fleet-wide: FreeToken (frank-small) refuses response_format identically, so Hermes's auto-title call (max_tokens=64 + response_format json_schema) fails against BOTH living local backends and falls back to its message-derived title. Any aux route that needs constrained decoding or a tiny budget needs an endpoint with thinking off and grammar support; the retired FLM NPU was that endpoint. _(context: Full frank-large runtime+model swap on Dankclaw 2026-09-11: llama.cpp/kingjones STRIX - halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next w4b, as a rootless Quadlet container on port 7770, plus the amd_iommu=off kernel change that retired the frank-npu endpoint. Repo commit 301914b; runbooks serverAppConfigs-docs/LLM/halogen-flash-server.md and dankclaw-iommu-translation-off.md.)_
+- halogen-flash-server (github.com/peonist-ai/halogen-flash-server) is an OCI-image-only inference server for gfx1151 Strix Halo, evaluated 2026-09-11 as a replacement for the llama.cpp frank-large unit on Dankclaw. Verified facts: - Image ghcr.io/peonist-ai/halogen-flash-server:0.5.8 (latest CHANGELOG tag). Ports: OpenAI-compatible **8731** (publish this), engine **8730** (NO auth — keep unpublished/loopback only). - deploy/entrypoint.sh modes: all (default; both roles in one container), engine, api (split so front-end restarts don't reload the model), bench [drafters] [max_tokens] [effort] [reps]. Engine binary is flash_serve, CLI --ck FILE (not --checkpoint), --slots/--ctx/--max-tok. - Weights: HF peonist-ai/halogen-qwen3.8-flash-next (public, ungated) — qwen38-flash-next-w4b.hgn 115.55 GiB + ck.overlay.hgn 2.31 GiB quality sidecar (auto-picked up; engine warns if absent) + tokenizer/ (8 files). Optional vision.hgn 0.84 GiB, overlay-speed.hgn (trades o_proj calibration). Container-side download via HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next. - Run requirements: --device /dev/kfd --device /dev/dri --group-add keep-groups --security-opt seccomp=unconfined --ipc=host, memlock unlimited, native Linux only (WSL2//dev/dxg unsupported), gfx1151 only (hard-rejected else). Reference SKU = Ryzen AI Max+ 395 / 128 GB — exactly Dankclaw, so the baked HALOGEN_MATMUL_TUNING_FILE does *not* warn. - Key flags (env-only, no config file, read once at startup): HALOGEN_CHECKPOINT (default /models/qwen38-flash-next-w4b.hgn), HALOGEN_CK_OVERLAY, HALOGEN_MODEL_ID (default halogen-qwen3.8-flash-next), HALOGEN_DRAFTER_DEFAULT=1 (MTP speculation on by default, byte-identical at temp 0), HALOGEN_PROMPT_CACHE=2 (numeric; 1 = byte-identical resume), HALOGEN_CACHE_ENTRIES=8, HALOGEN_MAX_TOKENS_CAP=65536, HALOGEN_CTX=262144, HALOGEN_KV_POOL_POSITIONS (default 524288 = 35.0 GB; 262144 = 27.8 GB), HALOGEN_KV_SLOTS=4, HALOGEN_KV_POOL_FIT=1, HALOGEN_HOST_RESERVE_GIB=20, HALOGEN_MAX_TOK= _(context: Research phase of a grilling session: replacing both the model and runtime of frank-large (Dankclaw) with halogen-flash-server, requested by Frank 2026-09-11. All facts read from the upstream repo (README, docs/FLAGS.md, deploy/entrypoint.sh, docker-compose.yml, CHANGELOG), the HF weights repo API, the ghcr image manifest, and live probes of Dankclaw.)_
+- On Dankclaw's Hermes stack, hermes-webui ran with the image-baked HERMES_HOME=/opt/data while the gateway used /data; /opt/data is an auto-created anonymous volume at the image's VOLUME point, so every container recreation (restart/redeploy) reset the webui's config home to the image-default template (provider: auto → openrouter.ai with no API key) — this is why model renames "didn't propagate" and chat failed with connect errors. Fix (2026-09-05, commit 184df9a): Environment=HERMES_HOME=/data in the hermes-webui quadlet so webui + gateway share one config/state root. Diagnostic order for "can't access models" on hermes: (1) check which config home each runtime actually reads (podman exec c printenv HERMES_HOME), (2) resolve provider→endpoint with the app's own code (hermes_cli.providers.resolve_user_provider(name, cfg["providers"]) accepts api/url/base_url keys on providers: dict entries), (3) probe the served endpoint with the exact model id. Model ids must match served aliases exactly (frank-large @192.168.0.143:7770, frank-small @192.168.0.235:7770, gemma4-it:e4b @192.168.0.143:7771 — FLM routes only exact registry tags).
+- Hermes (Dankclaw) model routing as of 2026-09-05: main default = frank-large (Dankclaw 192.168.0.143:7770/v1), small/aux routes = frank-small (Dankserver 192.168.0.235:7770/v1), vision + title_generation route to the exact FLM registry tag gemma4-it:e4b (192.168.0.143:7771/v1). The pre-rename aliases frank-default-large/-small/-sightsound are dead since the 2026-09-03 rename cutover — never route on them. Live config lives at /home/dank/Containers/hermes/data/config.yaml (owned by uid 109999; needs sudo to read/edit); the repo's dankclaw/hermes/config.yaml.example is a sanitized backup and can lag live, so diff live first before editing either. Signal adapter is currently disabled in live config (platforms.signal.enabled: false).
+- On the guarded franksdank.cloud vhosts (Caddy → Anubis :8923 → Authentik outpost), every unauthenticated request — roots and deep paths alike — is answered with HTTP **200** and an HTML challenge page (titleMaking sure you're not a bot!/title), not a 4xx. Verifying guard health by status code alone is meaningless; assert on leakage markers instead (copyparty UI title, homepage id="grid", services.json "category" JSON shape). Non-JS clients (curl/wget/WebDAV) can never solve the PoW challenge, so they are permanently parked at the challenge page — this is the entire security boundary for copyparty, which serves authenticated traffic with anonymous read perms (-v /w::r:ra,dank). Pinned by tests/e2e/specs/15-unauth-fetch.spec.ts (guest project, empty storage state): any guard-route regression that lets app bytes answer fails loudly; any gate implementation passes.
+- On Dankclaw's Pi-hole (FTL v6.7, quadlet at /home/dank/Containers/pihole), /etc/pihole/hosts/custom.list (host path /home/dank/Containers/pihole/etc-pihole/hosts/custom.list) is a LIVE-ONLY Custom-DNS source NOT tracked in the repo — it held ~22 A records pointing every *.franksdank.cloud vhost at 192.168.0.143, including names with no public Porkbun record. gravity.db's domainlist table has zero custom-DNS rows on this build, and the pihole CLI's sqlite path (pihole-FTL sqlite3) works but the web API auth was failing at discovery time, so retiring a name from LAN DNS requires editing custom.list directly (sudo sed -i '/name/d') followed by podman exec pihole pihole restartdns reload; deleting only the repo-managed addn-hosts record is not sufficient. Verify with dig +short name @192.168.0.143 → empty answer.
+- Pi-hole admin panel at pihole.franksdank.cloud has two auth layers: Caddy basicauth (edge, username dankuser + hash) and Pi-hole internal password (CivetWeb, set via pihole setpassword). The Pi-hole internal password was the actual failure point, not the Caddy hash. Pi-hole resolves to 192.168.0.143 via addn-hosts (all 28 names point here), making it a LAN service behind Caddy's public TLS edge. Options to remove public exposure: (1) remove the Caddy vhost entirely — LAN clients hit Pi-hole quadlet directly on :8083; (2) keep Caddy vhost but remove the Porkbun A record so external DNS won't resolve pihole.franksdank.cloud. Current state: Caddy basicauth hash = $2b$14$cJm.hbvDi1dEh7x0XRMnseaIdvIrRyH.AcTL56jGJf3wd54qvEb9K (dankuser, password pihole2026); Pi-hole internal password = pihole.2026 (set via pihole setpassword, survives restart).
+- Pi-hole admin at pihole.franksdank.cloud has two auth layers: (1) Caddy basicauth edge (username dankuser, hash in rendered Caddyfile from CADDY_BASICAUTH_HASH in .env), (2) Pi-hole internal CivetWeb password (stored in cli_pw as bcrypt hash, set via pihole setpassword CLI). The env file WEBPASSWORD is used at container start but the CLI hash is what the running web server validates. When "I can't get into Pi-hole admin" returns 403 even after Caddy basicauth passes, the Pi-hole internal password is the likely culprit. Fix: podman exec pihole pihole setpassword newpw to update the CLI hash, then verify podman exec pihole cat /etc/pihole/cli_pw shows updated value. Verify by curling with basicauth: curl -sk -H 'Authorization: Basic ZGFua3VzZXI6...' https://pihole.franksdank.cloud/admin/ should return 302 (not 401 or 403).
+- Pi-hole admin panel at pihole.franksdank.cloud uses Caddy basicauth (not Pi-hole's own CivetWeb login). The hash is shared across services (RomM, Copyparty, Pi-hole) via CADDY_BASICAUTH_HASH in the host .env. The rendered Caddyfile at /home/dank/Containers/caddy/Caddyfile is mounted into the Caddy container — updates are picked up by Caddy's file watcher without restart. Password reset: generate bcrypt hash with python3 + bcrypt (hashpw + gensalt), update the rendered Caddyfile line directly or update .env + deploy. Caddy accepts both $2a$ and $2b$ bcrypt variants. _(context: Pi-hole admin password reset session: discovered Caddy basicauth (not Anubis), could not find matching password in common candidates, reset to pihole2026, updated rendered Caddyfile directly, synced repo .env files, Caddy file watcher handled hot-reload.)_
+- On the FrankBook-Pro *arr fleet (current lsio images: Sonarr v4/Radarr v5 API v3, Lidarr v2/Readarr API v1), the download-client validation endpoints are NOT usable via API: POST /api/{v}/downloadclient/test/{id} and .../test?forceTest=true both return 405, and POST .../downloadclient/action/getTest exists but 400s without a bespoke body. To prove download-client health after changing a qB host/port, skip the API test route and prove behaviorally instead: podman exec arr curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8080/ (expect 200) plus a real qB login from inside the container (curl --data-urlencode username=admin --data-urlencode password=$QBIT_PW .../api/v2/auth/login, expect 200), then confirm /api/{v}/health shows no DownloadClient-*/wbrrelated warnings. Also: arr APIs redact stored client passwords as ******** (sonarr/radarr/lidarr) — PUT the object back with the sentinel untouched to preserve the stored value; readarr (v1) returns the plaintext. When editing via the edit tool, body rows must be prefixed with +, not : — : prefixes get written literally into the file. _(context: Capture turn after the arr→qB loopback repoint (2026-09-04): app-side download-client test endpoints were unreachable, so client health was proven behaviorally instead.)_
+- The *arr fleet (Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) runs ONLY on FrankBook-Pro (192.168.0.112, Mac podman machine) since 2026-09-04 — the dankserver compose copy was removed (config roots in ~/.AGENTTRASH/20260904-dankserver-arr-teardown, 22 MB, awaiting purge approval). Fleet invariant, enforced by tests/frankbook_quadlet.bats: ALL six containers share gluetun's netns (Network=gluetun.container, Wants/After gluetun.service) and publish NO ports; the gluetun unit publishes the union (8080, 6881 tcp+udp, 9696, 8989, 7878, 8686, 8787), so all egress exits ProtonVPN by construction (AllowedIPs=0.0.0.0/0; proven: netns egress 51.179.177.14 vs residential 71.227.227.172). deploy-plan.py couples gluetun → all six joiners (a gluetun recreate kills the whole family). Consequences: (1) the old arr→qB addressing hack 10.88.0.1:8080 (podman-bridge gateway) routes INTO the tunnel from inside the netns — download clients must use 127.0.0.1:8080 (done via arr APIs, verified by real qB login from inside sonarr's container); same rule for any future Prowlarr Application rows (baseUrl 127.0.0.1:arr port). (2) Caddy arr vhosts and homepage probes target 192.168.0.112:port (host-side published ports unaffected by netns changes). (3) Known defects as of 2026-09-04: Prowlarr's Applications table is EMPTY (the four rows from 2026-08-30 were lost — prowlarr→arr sync broken until rebuilt); readarr's qB proxy auth defect needs retest post-move; EZTV/1337x "CloudFlare-blocked" parking is stale (egress now VPN — retest before enabling). (4) E2E lesson: playwright specs 11-admin/12-crawl now derive expected open-link counts from services.json (showLink) instead of hardcoded counts — keep new registry entries consistent. Also: deploy.sh --service X dispatches ONLY X on frankbook (path-coupling applies to changed-path planning); gluetun restart self-heals all netns joiners via Restart=always so a single gluetun deploy converges the family. _(context: 2026-09-04 operator decision session: dankserver arr compose copy (booted that morning) removed after Frank chose FrankBook-Pro as the sole arr deployment; netns consolidation + test seam + loopback repoint executed same session. Commits 4c7465d, df4339a.)_
+- Rootless podman on Dankserver (5.7.0, verified 2026-09-04 via /proc/pid/uid_map across 7 containers): the DEFAULT user namespace maps container uid 0 → host uid 1000 (dank) and container uid N → host 100000+N-1 (e.g. calibre-web PUID=1000 writes as 100999). Consequences: (1) PUID=0 + data roots owned 1000:1000 is the complete fleet recipe — NO UserNS/userns_mode key is needed on quadlets or compose services; the plex quadlet works without one. (2) NEVER add userns_mode: host to a podman-compose service: podman-compose 1.5.0 runs every compose service inside a pod and podman hard-rejects the combo (--userns and --pod cannot be set together), leaving the stack down with no recreate path. (3) Diagnostics: podman exec c id always shows uid=0 inside any mapping — meaningless; ground truth is /proc/container-pid/uid_map, host-side ps -o user= -p pid, or ownership of freshly-written files. (4) *arr boot-crash signature: UnauthorizedAccessException: Access to the path '/config/...' at logger init = data root not writable (e.g. root-owned dir created by sudo mkdir); the app exits before binding its port, so the port probe reads "dead" rather than "crashed" — check podman logs tail for "Press enter to exit" before assuming an outage has an infra cause. (5) NEVER pkill -f rootlessport on these hosts: it kills the port-forward helper of EVERY published container simultaneously (2026-09-03 outage); restart the owning systemd units instead. _(context: Dankserver *arr stack restore session, 2026-09-04: stack committed 2026-08-29 never booted (Sentry crash, dead ports read as outage); userns_mode: host fix was a red herring — rejected by podman-compose pods; real fix was chown 1000:1000 of /opt config dirs. Commit e454159 (revert), 8a63624 (docs).)_
+- Arr stack (6 services) on dankserver (192.168.0.235): radarr:7878, sonarr:8989, lidarr:8686, readarr:8787, prowlarr:9696, qbittorrent:8080. Caddyfile on dankclaw has two-layer pattern: (1) public vhost blocks at top level proxying to 192.168.0.235:PORT, (2) @arr named handlers inside :8017 using "host" matcher for post-auth routing. Homepage services.json adds arr entries with category (user/admin), url (*.franksdank.cloud), probe (192.168.0.235:PORT), method http. Rootlessport issue: if rootlessport process dies, ports are mapped but return 000 — kill -9 rootlessport, restart containers with podman-compose. Fix permission issues with sudo rm -rf config dir. _(context: ServerAppConfigs repo: Caddyfile changes added to dankclaw/Caddyfile (public blocks + :8017 named handlers), services.json in shared/homepage/ updated with 6 arr entries. All 233 bats tests pass. Deployed to dankclaw (dankclaw-owned).)_
+- Arr stack containers on dankserver (Plex/Prometheus/Loki via rootless podman) show 000 from curl despite rootlessport listening on the correct port. Root cause: s6 supervision in linuxserver images runs the HTTP server as a background process, but rootlessport forwards to the container's slirp4netns interface which doesn't reach the backgrounded HTTP server correctly. Symptoms: rootlessport process is running and ss shows port LISTEN, but curl returns 000 (no response, not connection reset). Fix: restart rootlessport (pkill -f rootlessport) then podman-compose down && podman-compose up -d, or use --network host to bypass rootlessport entirely. Verification: ss -tlnp | grep port shows rootlessport, podman exec container cat /proc/net/tcp6 should show the port in LISTEN state (0A). _(context: Deep investigation of arr stack containers on dankserver (192.168.0.235) failing to respond via rootlessport despite ports being published correctly. Discovered s6 supervision in linuxserver images causes the HTTP server to run as a background process that rootlessport's port forwarding can't reach.)_
+- flm-npu-pack-registration managed skill exists; when updating it, merge (read current body via skill:// URL first) rather than clobber — the runbook covers: registry edit (mkstemp+os.replace), manual pack placement with resolve/main/file?download=true (raw URLs return 135-byte LFS pointer stubs), family kernel clone from /opt/fastflowlm/share/flm/xclbins/FamilyPack into ~/.config/flm/xclbins/PackName (sudo cp -a; serve resolves under FLM_XCLBIN_PATH = user dir), verification by real completion POST on :7771 checking choices not just HTTP 200 (unknown tags answer 200 with flat {"error":"Failed to load tag model!"} body), hwctx contention (only one flm serve process can hold the NPU; second process fails every load with DRM_IOCTL_AMDXDNA_CREATE_HWCTX err=-22 = aie2 mgmt msg opcode 0x2 status 0x2000003), boot tag decides residency only (per-request routing by exact registry tag regardless of boot tag; swap cost ~6-11 s on first request after swap), FLM_CTX_LEN=-1 to let each pack load at its own trained context, and bracket-guarded pkill patterns over ssh. _(context: Capture turn for the completed NPU pack registration + boot-tag flip work on frank-npu (Dankclaw :7771), 2026-09-03.)_
+- placeholder
+- Checking whether the flm-npu-pack-registration managed skill already covers the full registration runbook (manual pack placement, LFS download=true trap, family kernel clone to FLM_XCLBIN_PATH, hwctx contention rule, boot-tag flip verification) before overwriting it via manage_skill update. _(context: manage_skill create failed because the skill already exists; need its current body to merge rather than clobber.)_
+- FLM (FastFlowLM) v1.0.1 on Dankclaw, registering a custom NPU2 pack tag (verified 2026-09-03 for nuextract3:4b = Atomic-Germ/NuExtract3-4B-NPU2): (1) flm pull tag is BROKEN for custom registry entries — it throws json.exception out_of_range key 'tag:size' not found while building the download list; place pack files manually into ~/.config/flm/models/PackName/ instead (use resolve/main/file?download=true — plain raw URLs return 135-byte LFS pointer stubs, not weights). (2) The qwen3.5-family loader opens vision_weight.q4nx unconditionally even for extraction/text-only packs with vlm:false; a missing vision weight file makes the first request HANG with no error log (silent stall behind NPU lock queue), not fail-fast. Registry files list must include vision_weight.q4nx. (3) NPU hardware-context contention: while one flm serve process holds the NPU, any SECOND serve process fails every model load with DRM_IOCTL_AMDXDNA_CREATE_HWCTX err=-22 (aie2 mgmt msg opcode 0x2 status 0x2000003 = hw resource alloc fail) — even for packs that load fine in the resident process. Only ONE flm serve process may hold the NPU; tag switching works only INSIDE one serve process (swap cost ~6-11 s first request after swap). (4) Kernel packs live at /opt/fastflowlm/share/flm/xclbins/PackName/ (root-owned, sudo cp -a); copy from the same-shape family pack (NuExtract3-4B shares Qwen3.5-4B-NPU2 kernels — config.json shape-matched on hidden_size/layers/head_dim first). Also: pkill -f "pattern" over ssh matches your own ssh wrapper's command line and kills your own session — use bracket-guarded patterns like "[q]wen3.5:4b" or exact PIDs.
+- In serverAppConfigs, the docs-completeness test (tests/docs_completeness.bats) scans EVERY .md under serverAppConfigs-docs/ including dot-directories like Dankserver/.AGENTTRASH/, and requires each basename to be index-linked from serverAppConfigs-docs/index.md via a case-sensitive [[basename]] grep. So the old trick "move untracked docs to .AGENTTRASH before commit" does NOT satisfy this seam anymore — an unlinked page moved into an in-tree trash dir still fails the test. Correct dedupe of duplicate doc pages (e.g. Prowlarr.md vs prowlarr.md, 2026-09-03): collapse via rename so ONE canonical file carries the current content under the exact basename index.md already links (lowercase prowlarr here); git history retains both versions, zero deletions, test passes. Also: log.md in that tree is newest-first — new dated entries go directly under the # Bundle Change Log header, not appended at file tail.
+- Dankclaw Hermes network topology, verified live 2026-09-02: systemd-hermes (gateway), hermes-webui, and chrome-devtools-protocol run with NetworkMode=host, so their local addresses are the host listener table — gateway dials signal-cli at http://127.0.0.1:8080 (SIGNAL_HTTP_URL in dankclaw/.env.example), CDP binds loopback-only 127.0.0.1:9222, webui listens 0.0.0.0:8787. hermes-dashboard alone sits on the compose-era orphan network dankclaw_default (subnet 10.89.1.0/24, gateway 10.89.1.1, zero other connections) and publishes 9119→0.0.0.0:9119; from that netns the host is reachable only at 10.89.1.1 for *published* ports (3000 Gitea, 8081 SearXNG verified OPEN) while unpublished host listeners (8080 signal-cli, 8787 webui) time out, and host.containers.internal=169.254.1.2 is the pasta alias for the default podman net (10.88.0.0/16, gw 10.88.0.1) so it is NOT routed from dankclaw_default. Fix if the dashboard ever needs the gateway: NetworkMode=host + drop PublishPort=9119, redeploy deploy.sh dankclaw --service hermes-dashboard. Also verified same turn: gitea-postgres (db:5432, net systemd-gitea), searxng-valkey (valkey:6379, net systemd-searxng) and alloy (OTLP 12345) publish NO ports → unreachable from the host netns by construction; Plex PMS 1.43 serves TLS-only on 32400 so an http probe there always reads down (use method tcp); and node-exporter was manually stopped on both hosts (dankserver exit 137 SIGKILL at 2026-09-01 02:18 with OOMKilled=false, dankclaw exit 143 SIGTERM ~6 days prior) leaving Prometheus's node-dankclaw/node-dankserver scrape targets connection refused until restarted — it is a bare compose service with restart=unless-stopped and no quadlet unit, so it will not survive a reboot.
+- Rootless podman identity model on dankserver (verified live 2026-09-01, plex/tautulli quadlet migration): rootless podman CANNOT map real host uids outside the user's subuid/subgid range (dank: 100000:65536). UserNS=host rootless maps container root to the invoking user (dank, 1000) — not real root, and PUID=997 silently falls back to root because uid 997 is unmappable. The fleet convention is therefore: PUID=0/PGID=0 (container root = dank), data roots owned 1000:1000 (jellyfin/copyparty precedent); a data root owned by a legacy uid (997:984 for plexmediaserver) needs chown -R 1000:1000 at cutover, and rollback chowns back. Quadlet v1 gotchas proven the hard way: (1) UsernsMode= is COMPOSE syntax — the podman-user-generator hard-fails the WHOLE unit ('unsupported key UsernsMode in group Container') and systemctl --user restart name.service reports 'Unit not found'; the correct key is UserNS=. (2) Quadlet's systemd-style value parsing mangles embedded double quotes in HealthCmd= one-liners like python3 -c "..."; use the repo's proven HealthCmd=curl -fs http://localhost:8181/home pattern instead. (3) Without ContainerName=, the v1 generator names the container systemd-unit (systemd-tautulli). (4) PMS version jump 1.21.1.3795 → 1.43.3.10896 triggers one-time DB migrations/DatabaseFixups on a 32 GB DB that run 20-40+ min; a restart during migration ABORTS and re-runs it — never bounce PMS mid-migration; the 600s /identity deploy gate fails (empty-reply: PMS pre-binds 32400) and leaves deploy-state pending while the unit itself is correct. Also: line-targeted edit churn on the .container files ate three keys (tautulli ContainerName, plex Network, tautulli PublishPort) — after multi-hunk edits on unit files, read the WHOLE file before committing; the quadlet seam tests (PublishPort + enumeration) caught one, post-cutover C5 alias probe caught the other. _(context: plex-tautulli podman quadlet migration on dankserver (plan local://plex-tautulli-podman-migration-plan.md), executed 2026-09-01: first cutover failed (UsernsMode hard-fail), second failed (rootless uid mapping crash-loop on 997-owned data root), gate failures during PMS one-time migrations, three keys restored after edit churn. Commits 58aa084, c694053, 39c8135, 6302508.)_
+- Homepage deploy ownership (serverAppConfigs): homepage is Dankclaw-owned even though its config lives in shared/homepage/ (including the misleading dankserver.services.json). Correct deploy: ./scripts/deploy.sh dankclaw --service homepage — deploy-remote.sh deploy_homepage() copies shared/homepage/services.json, index.html, admin.html, app.js, style.css, fonts to /srv/homepage/ on Dankclaw and restarts the homepage-status timer. --service homepage against dankserver fails ("not deployed on dankserver"); per-host *.services.json files are tracked for reference only and are NOT shipped. Changed files only land if committed+pushed first (deployer pulls revision state). _(context: Plex homepage link fix, 2026-09-01: deploy.sh dankserver --service homepage failed ("not deployed on dankserver"); deploy-plan.py's homepage group maps to dankclaw only, and deploy-remote.sh deploy_homepage() refuses non-dankclaw targets. Discovered via dry-run + grep of deploy scripts.)_
+- Authentik (2026.8, Dankclaw) proxy-provider provisioning contract — every ProxyProvider needs BOTH: (1) all three flows set (authentication/authorization/invalidation). Any null makes the embedded outpost's Rust config deserializer reject the ENTIRE config JSON ("invalid type: null, expected a formatted UUID string") → "Failed to fetch outpost from API, retrying in 3 seconds" forever → ALL guarded vhosts die at the next outpost reload/restart. Silent killer: while the old in-memory config lives, everything works; the outage only surfaces on restart. (2) An attached Application (provider_id set). Without it, login completes but bounces to authentik's own user UI (/if/user/#/library) instead of the app — zero errors logged anywhere. The UI wizard does both automatically; raw API/ORM drives must do both explicitly. Diagnostic: the SSO e2e hostname-only assertion had a blind spot — the outpost serves the login flow ON the guarded vhost, so hostname matching proves nothing; assert pathname left /if/ AND body lacks "Powered by authentik"/"Sign in". Schema quirk: provider MTI is authentik_core_provider (PK=id) ← proxy child keyed by oauth2provider_ptr_id; Application.provider_id → base pk. _(context: 2026-09-02 seerr/grafana/loki guard outage on Dankclaw: three script-provisioned proxy providers (created via raw API/ORM with minimal payloads) broke SSO in two silent layers; fixed by backfilling flows from pk1 + creating guard Applications, verified by hardened playwright sweep 7/7.)_
+- Dankserver (192.168.0.235) hardware, verified via read-only dmidecode/lscpu 2026-09-01: Ryzen 7 7700X (Zen 4/AM5) on ASUS TUF Gaming B650-E WiFi, BIOS 2613 (2024-04). RAM = 4×16 GB G.Skill F5-6000J3038F16G (Trident Z5 Neo, dual-rank, rated DDR5-6000 CL30-38-38-96 @ 1.35V EXPO, Hynix), 64 GB total, no ECC. CRITICAL: board silently defaulted to JEDEC-safe 3600 MT/s @ 1.10V (EXPO never enabled) because 4×2R is AM5's hardest population. Agreed OC ladder (user enters BIOS manually, agent hands-off): Tier 0 = EXPO I 6000 CL30 @1.35V; Tier 1 fallback = manual 5600, FCLK 2000, UCLK 1:1, 30-38-38-96, tRFC 448, DRAM VDD/VDDQ 1.35V, VDDIO 1.30V, SOC 1.20V (≤1.25 daily), MCR disabled while tuning; Tier 2 = tRFC 448→392→360, optionally 5800. Never touch tREFI on this non-ECC 24/7 server (silent corruption risk). Validation: Memtest86+ 4 passes, y-cruncher VST 1h, journalctl -k whea/mce watch (baseline clean as of 2026-09-01). Rationale: FreeToken frank-default-small streams ~35 GB experts from host RAM → memory bandwidth is its prefill bottleneck; 3600→6000 ≈ 53→88 GB/s read [INFERENCE, platform-standard]. _(context: Dankserver RAM OC analysis turn, 2026-09-01. Read-only audit; profile delivered in chat for manual BIOS entry. User explicitly forbade edits — no sysfs/module/config changes were made; keep that constraint if resuming this task.)_
+- FrankBook-Pro qBittorrent version coupling (2026-09-01): qB is PINNED to lscr.io/linuxserver/qbittorrent:4.6.7 because Readarr's EOL qB client (frozen ~2023) rejects qB ≥ 5's 204 empty-body login success as "Authentication failure" — and that bites at RUNTIME, not just Prowlarr-style API creates (an earlier session wrongly assumed runtime cookie auth was unaffected; live evidence disproved it). Never unpin qB to latest while readarr is in the fleet. Second trap: on any qB version flip, qB 5.x's shutdown conf rewrite leaves a WebUI\Password_PBKDF2 @ByteArray line that 4.6.7 rejects — every login returns "invalid credentials" (arrs included); fix = rewrite the same conf line fresh with the same password (PBKDF2-HMAC-SHA512, 100k iterations, 16-byte salt, 64-byte key) while qB is stopped. Facts also documented in serverAppConfigs-docs/Infrastructure/FrankBook-Pro.md (repo is source of truth). _(context: FrankBook-Pro qBittorrent pipeline work, 2026-09-01 (supplements same-day cookie/egress lesson))_
+- qBittorrent cookie-name + egress-verification gotchas (FrankBook-Pro, 2026-09-01): (1) qB 5.x names its WebUI session cookie QBT_SID_port (port-suffixed); qB 4.6.7 uses plain SID — any script hard-coding QBT_SID_8080 breaks on 4.6.7 with sidlen=0 → 403 cascades that LOOK like auth failures but never reach qB as failed logins (no ban accrues; verify via qbittorrent.log "WebAPI login success" lines). Always parse generically: grep -oE '[A-Za-z_]*SID[A-Za-z0-9_]*=[^;[:space:]]*' | head -1, then pass the whole match to curl -b. (2) Egress checks for gluetun-gated stacks MUST run inside the gluetun container (podman exec gluetun wget https://ipinfo.io/json). Running the same wget one ssh-context out (Mac host via podman machine ssh without the inner podman exec) silently returns the RESIDENTIAL IP — a false leak alarm that reads as a hard-gate violation. Confirm context by requiring the org line (AS9009 M247 for the ProtonVPN Panama config), not just country. _(context: FrankBook-Pro gluetun/qBittorrent E2E verification, 2026-09-01)_
+- Authentik 2026.8 embedded outpost (franksdank.cloud fleet) caches claims per browser session in the authentik_providers_proxy_proxysession table (session_data JSON holds claims incl. groups; observed row TTLs 1–24 h). A session authorized BEFORE a user is added to a group keeps serving the old group list to Caddy header_regexp RBAC gates → persistent 403 despite correct membership, and the 403 generates ZERO authentik events (it never reaches authentik), so logs look silent. Re-login does not help (browser cookie re-maps to the same stale row). Fix (verified live 2026-09-01 for akadmin /admin 403): DELETE rows WHERE session_data-'claims'-'preferred_username' = 'user' in the authentik-postgres container — the django login session is untouched, so the next visit re-authorizes silently and mints fresh claims. Do NOT rotate AUTHENTIK_SECRET_KEY for this (invalidates every outpost session fleet-wide). Diagnosis SQL + silent-reauth verification documented in serverAppConfigs-docs/Infrastructure/Authentik.md (commit ea0bba3). _(context: Frank reported persistent 403 at www.franksdank.cloud/admin after being added to homepage-admins; recon found his live proxysession row (expiring 2026-09-02 22:06 UTC) carried ["authentik Admins"] only, minted before the group existed. Silent re-auth proven by deleting qa-e2e-admin's 13 rows and running the e2e admin spec with --no-deps against the stale storage state (2 passed, 2.9 s, no login pr)_
+- Frank wants periodic "what did we do" HTML change reports rendered in Firefox (asked 2026-09-01; first one at /home/dank/Reports/serverAppConfigs-24h-2026-09-01.html). Preferred standard: Keep a Changelog 1.1.0 categories (Added/Changed/Fixed/Removed/Security, omit empty) inside a post-change-review skeleton (summary → timeline → categories → code samples → verification → incidents → user actions). Reports are deliverables outside the repo at ~/Reports/ (never tracked; keeps tree clean). Frank reads the report to plan his code-review pass, so include a ready-made git show/git diff command list per theme. _(context: Built a 38 KB self-contained HTML report for serverAppConfigs covering 39 commits/24h; verified render via headless browser screenshot before handing over; opened via firefox --new-tab file:///....)_
+- Playwright automation of authentik's Lit flow executor (fleet e2e, tests/e2e/lib/login.ts akLogin): a programmatic page.fill races the stage re-render — the Lit web component re-renders right after mount and wipes the input, so the form submits an EMPTY password ("This field is required") even though fill() succeeded. Hardened pattern: fill, VERIFY the value survived (expect the input's value, short retry), submit with Enter (native form path, no button re-render race), and retry the whole stage up to 3 attempts before failing. Two adjacent musts: every login runs in a FRESH browser context (a reused/half-finished context poisons authentik's pending_user stage and hangs the next login), and the browser needs a real Chrome UA (Anubis denies obvious headless UAs); budget 60s assertions for the first Anubis PoW per fresh context (subsequent guarded vhosts skip challenge thanks to COOKIE_DOMAIN solve-once). Conventions documented in serverAppConfigs-docs/Architecture/E2E Suite.md (commit b945dd0). _(context: frank-default e2e suite hardening, 2026-09-01: 10/11 spec flake root-caused to Lit re-render race + Anubis PoW budgets; suite now 17/17 green across setup/standard/admin/guest projects.)_
+- Fleet VPN truth (verified 2026-09-01): the gluetun VPN_SERVICE_PROVIDER=custom profile on Dankclaw (endpoint 149.102.254.77:51820) points at a **ProtonVPN Seattle WireGuard server**, NOT a self-run box. Evidence: RDAP via curl -sL https://rdap.org/ip/IP (works when whois is empty) — endpoint IP → CDNEXT-CG-SEA/Datacamp; Dankclaw's tunnel v6 2a07:b944::2:2 → CH-PROTONVPN-20160719; exit IP 149.22.88.19 → CDNEXT-SEA-CG. Full 65535-port scan fingerprint: no SSH anywhere, only silent TLS listeners on 443, 4443–4468, 5995, 7770, 8443 (Proton protocol-stealth port map). Consequences: (1) adding a "peer" is never server admin — it's generating a new WireGuard config in Frank's Proton dashboard; each config = one keypair, Proton has NO BYO pubkey (a locally generated keypair is useless; one was retired to dankclaw ~/.AGENTTRASH/20260901-frankbook-wg-keypair). (2) Two hosts sharing one Proton config make the server bounce the endpoint between them — reusing Dankclaw's config for FrankBook-Pro would kill the Dankclaw tunnel (SearXNG). (3) Mac gate: second dashboard config (e.g. named frankbook) → PrivateKey/Address paste into /home/core/Containers/gluetun/env; server pubkey utqNH4P+C0fA1wTgIUaIfhm1b7ai0iEGvXyiw4KC3Fs=, gluetun ignores the .conf DNS line. General technique: identify mystery infra with rdap.org RDAP + nmap port fingerprint before asking the user to administer it. _(context: FrankBook-Pro WG gate investigation, 2026-09-01. Probes: no SSH config aliases on the workstation/Dankclaw, port 22 timeout, full nmap scan, RDAP on 3 IPs. Docs rewritten: serverAppConfigs-docs/Infrastructure/FrankBook-Pro.md "Mac provisioning" section; log.md entry. Repo commits 2d06be8, 3094ffa, f1171b9.)_
+- When a fleet credential "mystery" arises (auth works/doesn't contradict user memory): (1) assume session-archive images are corrupted — OCR of transcript frames mangles strings while preserving shape (Jellyfin admin password read as "ResFace_3241" instead of "<REDACTED-rotated-2026-09-10>"; the prefix was garbage, the digits were real; the user's own recall had the right prefix but flipped digits). (2) NEVER brute-force a live auth endpoint — pull the stored hash and verify candidates offline: Jellyfin stores it in jellyfin.db (Users table, Password column) as $PBKDF2-SHA512$iterations=210000$salt$hash where salt AND hash are HEX-ENCODED (32 B salt → 64 hex chars, 128-char hash). A parser that assumes base64 silently tests zero candidates — always print back the parsed lengths (saltbytes/hashbytes) and candidate count before concluding "no match". (3) A single matching hash is conclusive; a live POST adds nothing and just logs auth-failure warnings. _(context: Jellyfin 401 mystery on Dankserver, 2026-09-01. User's correction of the corrupted prefix + digit flip; offline hash check vs stored DB hash decided it. Files: serverAppConfigs-docs/Dankserver/Jellyfin.md, serverAppConfigs-docs/log.md (both corrected).)_
+- Rootless podman UID mapping on Dankclaw/Dankserver is CONTINUOUS, not offset: container uid N → host uid 100000 + N - 1 (dank subuid 100000:65536; container 0 → 1000, container 1 → 100000, container 1000 → 100999). Never compute host uid as 100000+N. Before chowning a bind root for a non-root image, read the real mapping from any throwaway container: podman run --rm image cat /proc/self/uid_map ("ID-inside ID-outside range" lines). A wrong chown gives EACCES crash-loops (quadlet auto-restart, container never persists; podman logs empty because --rm). Verified 2026-09-01 with seerr-team/seerr (in-image node uid 1000 → chown 100999 fixed it). _(context: Seerr first boot on Dankclaw crash-looped with EACCES mkdir /app/config/logs after chown 101000; /proc/self/uid_map showed "1 100000 65536" ⇒ container 1000 → host 100999. Second off-by-one incident on this fleet (postgres 100069 arithmetic was the first).)_
+- Two pitfalls from the 2026-09-01 frankbook/dankserver session: (1) zsh (default login shell on the workstation AND dankserver) expands =word at the start of an unquoted word to the command's path — so echo ==FOO== inside an ssh command fails with zsh:1: =FOO== not found. Quote echo arguments (echo "==FOO==") or use printf in any command line destined for zsh. (2) qBittorrent ≥ 4.2 ignores a plaintext WebUI\Password in qBittorrent.conf — it must be @ByteArray(salt:hash) from PBKDF2-HMAC-SHA512 (100000 iterations, 16-byte salt, 64-byte key, both base64). Also when the WebUI is reached through a port forwarder that preserves the LAN Host header (e.g. socat), set HostHeaderValidation=false and ServerDomains=* or qB rejects the request. Reference implementation: VM_SEED heredoc in deploy_frankbook_stack, scripts/deploy-remote.sh. _(context: frankbook-pro *arr bring-up + dankserver cutover verification, serverAppConfigs repo)_
+- FreeToken v0.1.2 on qwen3_5_moe loads ONLY modelopt-style quant configs: block-fp8 (quant_method "fp8" + weight_scale_inv, the Huihui/official-Qwen format) works; every compressed-tensors FP8 checkpoint fails. Two live-verified failure modes (2026-09-01, Dankserver): (1) compressed-tensors per-channel (Ornith-1.5-35B-A3B-FP8) aborts at expert-source resolution — ValueError "Missing MoE expert source layers: {'gate_up': ...}" because expert tensors don't follow mlp.experts.{N}.gate_proj/up_proj/down_proj naming; (2) compressed-tensors block-fp8 128×128 (HauhauCS-Aggressive-fp8) resolves experts but _dequant_fp8_weight multiplies weight × weight_scale directly, so [out/128, in/128] block scales don't broadcast → RuntimeError shape mismatch (tensor a 2048 vs b ...). Both exit cleanly at load (~37 GB fits RAM, no swap-thrash). Before swapping any model onto FreeToken, check config.json quant_method/weight_scale shape; a "healthy after ~6s" from deploy.sh proves nothing (health gate passes while the doomed worker is still starting) — verify via journal Uvicorn startup + a real completion. Evidence: serverAppConfigs-docs/LLM/free[REDACTED].md Quantization Compatibility table. _(context: frank-default-small model-swap attempts on Dankserver, 2026-09-01: Ornith FP8 and HauhauCS FP8 both failed to load on FreeToken v0.1.2; reverted to batsclamp Huihui FP8 (commit 4d5bea4), docs commit fe34f04.)_
+- skill://quadlet-v1-generator-quirks
+- Before swapping a model into an LLM service, preflight the CHECKPOINT SIZE (bytes on disk / weight precision) against the host's usable RAM, not just VRAM — the failure is silent and self-sustaining, not a clean crash. Case: thomsonreuters/Thomson-1.0-Small is Qwen3.6-35B-A3B arch but ships BF16 (~66 GB) vs the Huihui FP8 it replaced (~35 GB); Dankserver has 62 GB system RAM. FreeToken streams MoE experts from host RAM, so 66 GB of weights exceed physical RAM. Compounding traps that turned a fit-failure into an outage: (1) 127 GB swap means the kernel PAGES instead of OOM-killing, so sshd is starved and never gets killed — no recovery window opens on its own; (2) systemd Restart=on-failure makes it a crash-reload-rethrash loop; (3) the unit is enabled + WantedBy=multi-user.target, so even a reboot re-thrashes on boot until the bad MODEL_DIR is masked/reverted — a plain power-cycle does NOT fix it. Correct recovery requires a shell to run systemctl mask --now svc FIRST (stops + breaks the reboot trap), then pull the revert. deploy.sh reports "healthy after Ns" as soon as systemd is active, BEFORE the model finishes loading — never treat that as load success; verify with a real completion. Verdict: BF16 35B-A3B cannot run on a 62 GB box; needs an FP8/GGUF quant or a bigger host. RTX 3090 has no FP8 tensor cores so even FP8 runs via upcast but still fits in RAM. _(context: 2026-09-01 frank-default-small swap to Thomson-1.0-Small on Dankserver caused a sustained swap-thrash outage; SSH locked out, no OOM relief due to 127GB swap. Repo revert staged (commit 1163844), awaiting physical/console recovery.)_
+- FLM NPU model swap on Dankclaw: install new model via flm-add HF-repo --family family (downloads to ~/.config/flm/models/Name/), register tag in user model_list.json if missing, swap FLM_MODEL in scripts/frank-default-sightsound.env, then restart. Two common pitfalls: (1) FLM looks for xclbins at $FLM_XCLBIN_PATH/ModelName/ — if family is qwen3.5 but the model name is different, create a symlink from the expected name to the family's xclbin dir (e.g. Qwen3.8-Distilled-Heretic-9B-NPU2 → Qwen3.5-9B-NPU2); (2) flm-add leaves vision_weight as .q4nx.part — must rename to .q4nx before FLM can open it. Service uses FLM_CONFIG_PATH and FLM_XCLBIN_PATH exports for tag resolution; unknown tags return 200 with {"error":"Failed to load tag model!"}.
+- FreeToken on Dankclaw builds successfully via the CUDA path (not ROCm). Key details: (1) install nvidia-cuda-toolkit via apt (provides nvcc + libcudart at /usr), (2) set CUDA_HOME=/usr, FREETOKEN_ALLOW_CUDA_MISMATCH=1 (nvcc 12.4 vs torch cu130 is safe), (3) uv installed at /home/dank/.local/bin/uv, (4) source .venv/bin/activate in FreeToken/. The build installs torch 2.11.0+cu130, flashinfer 0.6.18, sgl-kernel 0.4.5. FTW format stores dtype per-tensor in index — FP16, BF16 (default), FP32, FP8 are all first-class. GGUF support is model-specific: only Gemma4GGUFForCausalLM has GGUF in the registry. GGML types: F32/F16/BF16 use native torch views; Q4_0 and Q6_K have dequant functions; Q8_0 is listed in GGML_NAME but has no dequant function (raises NotImplementedError). The GGUF reader uses gguf-py and is_gguf_path checks for .gguf files.
+- FreeToken does not need FP8 — the FTW format is dtype-agnostic (stores per-tensor dtype in index). Default dtype is BF16 (torch.bfloat16). FP16, FP32, BF16, FP8, FP4 (DS/NV/MX), and Q4_0 are all supported. convert_checkpoint(dtype=torch.bfloat16) is the default; pass any torch.dtype for non-default.
+- FreeToken on Dankclaw: CUDA path requires nvidia-cuda-toolkit (apt package, provides nvcc + libcudart at /usr/bin/nvcc and /usr/lib/x86_64-linux-gnu/libcudart.so). Set CUDA_HOME=/usr and FREETOKEN_ALLOW_CUDA_MISMATCH=1 (nvcc 12.4 vs torch 2.11.0+cu130 CUDA 13.0 kernels — safe override). Build: git clone, uv venv, uv pip install -e ".[accel]". UV installed to /home/dank/.local/bin/uv.
+- FreeToken on Dankclaw builds for CUDA (not ROCm) because freetoken.kernel._pinned_tensor links against cudart. Ubuntu CUDA toolkit installs to /usr (not /usr/local/cuda), so CUDA_HOME=/usr is required. nvcc 12.4 + torch cu130 requires FREETOKEN_ALLOW_CUDA_MISMATCH=1. UV installs to /home/dank/.local/bin/uv and must be added to PATH.
+- FreeToken can build on Dankclaw for both ROCm and CUDA. ROCm is the natural path: gfx1151 GPU with amdgpu driver loaded, ROCm 6.3 at /opt/rocm/, hipcc 7.2. All build deps present: cmake 4.2.3, git 2.53, python 3.14, gcc 15.2, ninja 1.13.2. Dependency chain for ROCm: torch 2.11 (has rocm6.3 wheels on PyPI), sgl-kernel 0.4.5 (has ROCm wheels), flashinfer-python (has ROCm wheels). CUDA path also works (nvidia driver 575.57, Vulkan libs present) but needs nvcc for JIT-compiled CUDA kernels on first use — install via sudo pacman -S cuda. Build command: pip install -e ".[accel]" (uv optional). Vulkan libs at /usr/lib/x86_64-linux-gnu/ but not a FreeToken build requirement.
+- frank-default-large contextWindow in ~/.omp/agent/models.yml is 200000 (per-slot with parallel 2), not 401408 (total CTX_SIZE) or 262144. The serve-frank-large.env has CTX_SIZE=401408 / PARALLEL=2, but the user consistently uses the per-slot value of 200000.
+- Hermes CDP browser daemon (deployed 2026-08-31, commit 2072354): chrome-devtools-protocol.container quadlet on Dankclaw runs digest-pinned chromedp/headless-shell with entrypoint override /headless-shell/headless-shell --remote-debugging-address=127.0.0.1 --remote-debugging-port=9222 ... — the override deliberately drops the image's socat front which would bind 0.0.0.0:9222 and expose browser control to the whole LAN. Deployer (deploy_hermes) starts it only if inactive; a restart kills in-flight browser sessions, so restart manually only when intended. hermes-webui was flipped to Network=host so both agent consumers (Signal gateway + webui) share the daemon's host netns and can dial 127.0.0.1:9222; bridge-network clients cannot reach it at all by design. Client wiring lives in live /data/config.yaml under mcp_servers.chrome-devtools = command /usr/local/bin/npx, args ["-y","chrome-devtools-mcp@latest","--browserUrl=http://127.0.0.1:9222"]; npx package cache lives in the container layer, so a recreation (not restart) pays one first-call npm download. /data/config.yaml is owned by the unmapped uid — even world-writable dir doesn't make the file readable by dank; edit via sudo python3 - piped script. Verification gotcha proven live: an MCP stdio client probe must newline-terminate each JSON-RPC message AND keep stdin open (sleep) after writing, because on immediate EOF the server exits before flushing any response and you observe empty output with rc=0; also, \n inside ssh single-quotes → sh -c double-quotes lands as literal backslash-n in printf args (invalid JSON), so pipe a written script via ssh 'bash -s' /tmp/probe.sh instead of nested quoting.
+- Hermes containers on dankclaw (systemd-hermes, hermes-webui, hermes-dashboard) run as root with SSH keys at /root/.ssh/id_ed25519. Their pubkeys must be in: (1) dankclaw's /root/.ssh/authorized_keys AND ~dank/.ssh/authorized_keys (hermes SSH to dankclaw as root, but dank's own key is for dank@dankclaw), (2) dankserver's ~dank/.ssh/authorized_keys, (3) FrankBook-Pro's ~dank/.ssh/authorized_keys. The hermes containers SSH to dankclaw as root@192.168.0.143, to dankserver as dank@192.168.0.235, and to FrankBook-Pro as dank@192.168.0.112. Each hermes key is unique per container (not shared). The dank@dankclaw key (dankclaw's own id_ed25519) was also missing from dankserver and FrankBook-Pro and needed adding for host-to-host SSH.
+- On FrankBook-Pro (macOS Intel x86_64), gvproxy is not bundled with the standalone podman v5.4.2 binary. Must build it from source using go build -mod=mod -o /usr/local/bin/gvproxy github.com/containers/gvisor-tap-vsock/cmd/gvproxy. The old /usr/local/bin/gvproxy file is only 9 bytes ("Not Found" text from a failed curl download) — must rm -f it before building. After building, it becomes a Mach-O 64-bit x86_64 executable (~19.5 MB). The podman machine on FrankBook-Pro is initialized but also requires vfkit in PATH to start fully (error: exec: "vfkit": executable file not found in $PATH).
+- 2026-08-31 state, verified live: (1) Vhost renames must be done in lockstep — the test suite (tests/pihole.bats test 178) fails unless every Caddyfile vhost name has a Pi-hole addn-hosts A record on 192.168.0.143; chat→ai and jellyfin→jelly were renamed this way (Porkbun edit with RELATIVE label + addn-hosts line + Caddyfile vhost in one commit, then deploy.sh). Decommissioned vhosts removed from Caddy that day: comfyui, hermes, radarr, sonarr, lidarr, readarr, prowlarr, qbittorrent; their public DNS was deleted from Porkbun while Pi-hole addn-hosts entries were deliberately KEPT (user's explicit 'remove from Porkbun, not PiHole') so internal consumers still resolve them but hit no vhost — internal consumers of those services must use direct IP/port instead (e.g. Hermes dashboard at 192.168.0.143:9119). (2) hermes-webui (Hermex) auth model: password-only login via POST /api/auth/login {"password":...} returning a signed HttpOnly hermes_session cookie (24h TTL); no username field, no SSH-key concept inside the app — SSH keys authenticate only the transport layer (user tunnel on port 22; Termius-generated keys are non-exportable so the keypair must be generated inside whichever SSH app acts as transport). The webui is published LAN-bound (192.168.0.143:8787-8787) with password gating, not loopback-only anymore; off-network access = SSH tunnel on port 22 then dial http://localhost:8787. (3) The webui unit's [Container] EnvironmentFile is /home/dank/serverAppConfigs/dankclaw/.env (NOT the root .env) — a secret rotation must rewrite BOTH env files or the restarted container serves the old value; verify by podman exec printenv on the live container after restart. Also: podman exec must run UNPRIVILEGED on Dankclaw — sudo/root's namespace does not see rootless containers (exit 125).
+- On Dankclaw (rootless podman), sudo podman exec user-container ... fails with exit 125 — root's podman namespace does not see the login user's rootless containers. Run podman exec as the login user (ssh dankclaw 'podman exec ...' with no sudo) when inspecting container env/config (e.g., podman exec hermes-webui printenv KEY). Same rule applies to podman inspect/ps for user-owned containers.
+- On Dankclaw (rootless podman), sudo podman exec container ... fails with exit 125 — root's namespace does not see the login user's rootless containers; run podman exec unprivileged as the ssh login user instead. Related durable pattern used when repairing /home/dank/serverAppConfigs/.env on Dankclaw (2026-08-31): a prior fix had appended a raw podman inspect --format "{{json .Config.Env}}" JSON dump into the live .env, corrupting keys (HERMES_WEBUI_PASSWORD=["PATH=...). Repair = back up first (cp -a), then rewrite quoted dump lines to proper KEY=value by stripping the leading quote, and source authoritative secret values programmatically from the live container env via unprivileged podman exec svc printenv KEY rather than transcribing them — never trigger a service restart for a pure .env-file repair. Also note: JSON entries carry a leading "; anchored grep on ^KEY= fails for those lines until de-quoted (repeat of the earlier landmine-fix pitfall — verify with grep -c '^"' == 0 after any dump append).
+- Decision (2026-08-31, task: "SearXNG agent-use check under Anubis"): do NOT stand up a dedicated agent-only SearXNG instance on Dankclaw. Verified live: the guarded path (search.franksdank.cloud) answers non-browser API clients with the Anubis JS-challenge page at HTTP **200** — status-code-only clients silently receive HTML where JSON was expected; always check content-type, not just status. The agent path is the direct published port http://192.168.0.143:8081/search?q=...&format=json from pass_ip (trusted) sources only — same single instance already VPN-bound via Network=gluetun.container, unreachable from outside the LAN by construction (no public route to the LAN IP). A duplicate would double upstream request volume from the same rate-sensitive WireGuard exit IP for zero benefit. Rationale recorded in repo docs: serverAppConfigs-docs/Dankclaw/SearXNG.md Gotchas section (commit ba0345a, pushed to Gitea origin) — read that file before ever proposing a second SearXNG stack. General probe heuristic: to test whether API clients can pass an Anubis-fronted route, curl the JSON endpoint with a non-browser UA and compare response content-type against HTTP status; a 200 with text/html body = guard challenge, not a working API path.
+- Verified runbook for flipping a Caddy-fronted service onto the Anubis+Authentik guard chain on Dankclaw (executed 2026-08-31 for loki.franksdank.cloud, commit a382a3b): (1) create the outpost ProxyProvider FIRST via ak shell stdin (podman exec -i authentik-server ak shell /tmp/x.py; set p.pk=None before save) — external_host = exact vhost URL with trailing slash, internal_host = http://caddy:8017; (2) flip vhost to reverse_proxy anubis:8923 { header_up X-Real-Ip {remote_host} } and add the post-auth @svc host vhost handle block inside the http://:8017 listener routing to the real upstream, in ONE deploy; (3) verify unauthenticated with curl → Anubis challenge page ("Making sure you're not a bot!" markers), NOT 502. Verification pitfall: do NOT probe the internal post-auth listener as localhost:8017 from the host — it lives inside the caddy container's netns and answers HTTP 000 (connection refused) from host loopback; instead verify the reloaded config byte-exact via podman exec caddy grep -n -A3 handler /etc/caddy/Caddyfile, and confirm outpost reachability with getent hosts caddy from inside authentik-server's netns (resolves 10.89.1.211). Also: service-to-service ingestion must target internal addresses (Alloy→loki:3100 never traverses Caddy); non-browser clients can never solve the JS challenge, so any app whose backend queries a public hostname of a guarded service breaks on flip — audit datasource URLs first (Grafana's grafana.db data_source table had zero rows at flip time). _(context: Loki guard-chain flip on Dankclaw; provider pk=7 created before route flip; verified via challenge probe + live-config grep inside caddy container.)_
+- qa-e2e (pk=6) is a live authentik test account on Dankclaw created 2026-08-31 with NORMAL privileges only: no groups, is_staff=False, is_superuser=False — verified at DB level via ak shell. It passes the full guard chain (Anubis challenge → two-stage default-authentication-flow → outpost callback → app content) and was proven live landing on RomM's app UI as a real authenticated user; it therefore cannot reach any admin-only surface (Grafana/Gitea/hermes dashboards keep their own auth). Credential store for the planned Playwright E2E suite: /home/dank/.config/e2e/qa-e2e.cred, mode 0600, deliberately OUTSIDE the serverAppConfigs repo so it can never be committed or trip deploy.sh's untracked-file guard. The E2E suite design (P4) consumes this file to capture storageState; Gitea-facing tests must use a Gitea token instead — an authentik session buys nothing on the git-smart-HTTP-exempt route. Related bypass map at that time: loki.franksdank.cloud had NO auth at all (LogQL open to anyone who can reach DNS) — biggest remaining exposure; *arr/qbittorrent UIs unguarded but API-key-gated for actions; grafana/pihole gated only by shared edge basicauth.
+- Edit-tool auto-prefix quirk (observed 2026-08-31, serverAppConfigs Caddyfile work): when an edit call's output warns "Auto-prefixed bare body row(s)", the tool may write literal prefix characters into the file even though the rendered diff looks clean — a : intended as an output marker landed in the file as a literal :# comment line. In a Caddyfile that parses as a matcher token and fails config validation at deploy time with a misleading error pointing at the *next* line ("unrecognized directive" on the following site block). Lesson: after any edit whose tool output shows auto-prefixing/marker warnings, byte-verify the exact edited lines (sed -n/cat -A) before commit or deploy — do not trust the rendered diff. Same class of failure can silently ship because validation gates (caddy validate) correctly skip the reload and only log a non-fatal error; verify the reload actually ran, not just that deploy exited 0.
+- Rootless-podman GPU access on systemd-user-unit-spawned containers (Dankclaw comfyui, 2026-08-31): containers spawned by systemd user units get NO supplementary groups at spawn time, so group_add: render fails with "unable to find group render" (userns name lookup) AND --group-add keep-groups provably preserves an EMPTY group set — torch then dies with "No HIP GPUs are available" even though /dev/kfd and /dev/dri/renderD* are merely passed through (they show as nobody:nogroup inside = unmapped). The only spawn-context-independent fix is udev MODE 0666 on the compute nodes (kfd, renderD*; leave card* display nodes alone). Two gotchas when applying: (1) udevadm trigger --type=devices does NOT enumerate misc-class devices (they have no subsystem) — use udevadm trigger --action=change --sysname-match=kfd; (2) the rule must be KERNEL=="kfd", MODE="0666" with NO SUBSYSTEM= guard, because a SUBSYSTEM=="misc" match never matches. Verified end to end: torch.cuda.is_available() True, device "Radeon 8060S Graphics", port bound same minute.
+- FLM (frank-default-sightsound, port 7771) routes chat completions per request by exact FLM registry tag: each request locks the NPU, loads the requested model if not resident, generates, releases. Measured swap costs (Aug 30 2026): first request to an unloaded tag pays full cold load (~7-90 s); switching back re-loads the previous model (~10 s) — alternating tags = every request pays swap. CRITICAL gotcha: a tag listed in /v1/models is NOT necessarily loadable — load fails with [ERROR] Failed to load model: No such file '/home/dank/.config/flm/xclbins/PackName/name.xclbin' when the kernel-pack dir isn't symlinked under FLM_XCLBIN_PATH (/home/dank/.config/flm/xclbins). Fix = ln -s /opt/fastflowlm/share/flm/xclbins/PackName ~/.config/flm/xclbins/PackName (verified for Qwen3.5-4B-NPU2; other unlisted packs fail the same way). Unknown tags answer HTTP 200 with flat {"error":"Failed to load tag model!"} and no choices — status-code-only clients silently get empty completions; clients must check the error key, not just HTTP status.
+- FLM (FastFlowLM) 1.0.1 serving on frank-default-sightsound (Dankclaw, port 52626) answers unknown model tags with HTTP **200** and an error body, not a 404 and not a silent fallback: /v1/chat/completions → flat {"error":"Failed to load tag model!"} with no choices; /v1/completions → nested OpenAI-shaped server_error (code 500) also under HTTP 200. Status-code-only clients therefore see a successful empty response — any client wiring against this endpoint must check for an error key in the body, not just status. Valid tags route per-request to the exact named model (response echoes the requested tag; no restart needed to switch registry tags). This disproved the older llama3.2:1b-fallback behavior documented from 2026-08-16; corrected live-probe evidence is committed in AGENTS.md + serverAppConfigs-docs/LLM/frank-default-sightsound.md at commit 9b9f0f9 — verify against those before relying on this. General lesson: behavior claims written into docs from inference/incident notes get falsified by direct probes; probe the live endpoint (bogus input + control pair) before committing contract claims to docs or client configs. _(context: Live probe 2026-08-30 on frank-default-sightsound: bogus tag "totally-bogus:9b" against /v1/chat/completions and /v1/completions; control with valid tag gemma4-it:e4b. Docs + serve-script comment corrected in commit 9b9f0f9, deployed healthy 4 s post-deploy.)_
+- frank-default-sightsound (FLM v1.0.1, XDNA2 NPU, Dankclaw port 52626) serves the WHOLE local model registry on one port and routes /v1/chat/completions per-request by the EXACT FLM registry tag in the model field; unknown tags do NOT 404 — they silently serve llama3.2:1b instead. Any client wiring must copy tag bytes from live GET /v1/models at wiring time (never hand-type, never use a service alias). Wiring pattern used for pi client models.json on the workstation (2026-08-30): backup file → programmatic python json load/dump adding provider dankclaw-npu with baseUrl http://192.168.0.143:52626/v1, model id = exact live tag gemma4-it:e4b, contextWindow/maxTokens parsed from FLM_CTX_LEN in scripts/frank-default-sightsound.env (not transcribed), input ["text","image","audio"], reasoning true → re-parse to prove valid JSON. Health probe for this service is GET /v1/version (FLM has no /health route). Full runbook: serverAppConfigs-docs/LLM/frank-default-sightsound.md; fleet table + runtime section in AGENTS.md carry the iommu=off-kills-NPU trap (SVA bind needs enabled IOMMU for /dev/accel/accel0).
+- Dankclaw hermes stack (fixed 2026-08-30, commit 56763e0): the hermes gateway is a Signal messaging agent with NO HTTP listener on :8090 — the public route hermes.franksdank.cloud → Caddy internal listener :8017 → reverse_proxy 192.168.0.143:9119 (dashboard login-gated admin UI). The gateway connects OUTBOUND to the signal-cli daemon, a HOST process at /opt/signal-cli-0.14.7 listening on host :8080, via SIGNAL_HTTP_URL=http://host.containers.internal:8080 in dankclaw/compose.yaml. Durable quirk: on this rootless-podman/netavark host, containers can only reach the host LAN IP 192.168.0.143 on PUBLISHED (DNAT'd) ports — connections to un-forwarded host ports from container space get instant Connection-refused even though a listener exists; host.containers.internal (netavark gateway alias) is the verified working path for unpublished host ports. Also durable: hermes, hermes-dashboard, and hermes-webui are compose-managed with NO quadlet units — they do not survive reboots (this is how the agent went silent after the Aug 26 reboot); a reboot requires manual podman-compose -f dankclaw/compose.yaml up -d --no-deps svc or deploy.sh dispatch. The hermes gateway drops to an unmapped uid inside its userns: /home/dank/hermes-data must be world-writable (chmod -R a+rwX) or the drop-priv user cannot traverse it and crash-loops on log writes (/data/logs/agent.log PermissionError).
+- frank fleet as of 2026-08-30 night: frank-default-large (Dankclaw) = kingjones777 Qwen3.8-Flash-Next ROCmFP4-STRIX 113.47 GiB on the card's pinned kingjones30/ROCmFPX fork at /home/dank/GitPackages/kingjones-ROCmFPX/build (commit 36e9acd) — builds require -DLLAMA_BUILD_WEBUI=OFF here (npm/HF-bucket asset provisioning aborts the link at 98%). --flash-attn on works UNPATCHED on the STRIX model (the QSA F16 ggml_fill assert needing scripts/patches/qwen4exp-qsa-f32-mask.patch is agentionai-FAST-specific); mmap MUST stay on (PLE streams via page cache; --no-mmap = silent cgroup kill); no spec decode; agentionai FAST on build-qwen38 is the rollback (one env commit). frank-default-small (Dankserver) = batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 on the same FreeToken v0.1.2 runtime (native block-fp8, identical loader path + pool triple as official FP8); FreeToken's /v1/chat/completions 422s WITHOUT an explicit "model" field (llama.cpp servers don't). Deploy verification trap: a leftover manually-started llama-server holding :7770 makes both the unit crash-loop on bind AND deploy.sh's /health check pass — verify systemctl show -p MainPID matches the live pgrep, not just health. pi models.json frank-default-large contextWindow corrected to 131072 (llama clamps n_ctx_train/parallel to per-slot; the old 200000 oversold). _(context: Repo /home/dank/Projects/serverAppConfigs; commits 3066d81 (large config), aa9c1d0 (small config), 0937cfb (docs). Both swaps validated live end to end.)_
+- When deploying frank-default-large changes to Dankclaw, the service unit is managed by systemd as frank-default-large.service (NOT a user quadlet unit). The quadlet directory exists but systemd-level unit is the authoritative one. The deploy script targets the systemd service; user-level systemctl --user list-units shows nothing for frank services.
+- frank-default-large env file has CTX_SIZE=401408 and PARALLEL=2 which are marked in comments as "untested experiment" vs the validated production config of CTX_SIZE=32768, PARALLEL=1. The env file dead-weight VAR VERBOSITY=4 is never passed to the binary (serve script doesn't use it). These values differ from the proven 32768 ctx × 1 slot deployment that loads clean in ~58s with ~100 GiB in the 119.3 GiB HIP heap.
+- qBittorrent 5 (lscr.io image, Dankserver) mints a NEW temporary WebUI password at every boot unless credentials are stored — this silently breaks all arr download clients after any restart, and it rotated mid-session once already. qb5 REMOVED /api/v2/app/setPassword and the lscr image has no WEBUI_AUTH env handling, so persistence must be written into /opt/qbittorrent/config/qBittorrent/qBittorrent.conf with the container stopped, in qb's exact format: WebUI\Username=admin + WebUI\Password_PBKDF2=base64(16-byte-salt) + ":" + base64(PBKDF2-HMAC-SHA512, 100000 iters, 64-byte dk) as a raw string — the classic @ByteArray(salt+hash) wrapper 401s. Verify by surviving a podman restart (login 204). API login from curl also needs Referer: http://127.0.0.1:8080/ (CSRF). The live password lives host-local in /home/dank/.qbpass (0600, owner dank) on Dankserver — never in git. Recipe + gotchas: serverAppConfigs-docs/Dankserver/qbittorrent.md. _(context: Installed 2026-08-30; all four arr download clients updated to the permanent password and validated (202) via each app's PUT /downloadclient.)_
+- Prowlarr→arr Torznab wiring (Dankserver, verified live 2026-08-30, whole grab→import chain proven): (1) Feed route is http://prowlarr:9696/{indexerId}/api?apikey={GLOBAL_KEY}&t=caps|search|... — there is NO /torznab/{id}/feed; any unmatched path hits the Forms-auth catch-all and returns 302 → /login?returnUrl=..., which arrs XML-parse into a fake "auth error" ("'/' is an unexpected token ... Line 54"). Feeds auth with the Prowlarr GLOBAL key from config.xml ApiKey; per-app keys are sync-only. (2) The arrs' torznab joiner is TrimEnd(baseUrl,'/') + apiPath with NO separator — set apiPath="/api" WITH leading slash. (3) Freshly created indexer rows default ALL-OFF — must set enableRss/enableAutomaticSearch/enableInteractiveSearch or the app logs "Searching indexers for [X]. 0 active indexers" (bit Lidarr). (4) Prowlarr v1 Application auto-sync is broken by version skew against Radarr 6.3 (POST /api/v1/applications/test → "Unable to cast object of type 'NzbDrone.Core.Thing...'") — manual torznab rows are the real integration; sync can't duplicate/clobber them. (5) Live search endpoints: Sonarr v5 interactive search = GET /api/v3/release?seriesId=N (POST /search and /release/search are 404/405); Lidarr v1 = GET /api/v1/release?albumId=N. Lidarr POST /artist requires foreignArtistId + qualityProfileId + metadataProfileId (lookup rows carry foreignArtistId, not foreignId). (6) Readarr (abandoned upstream 0.4.x) cannot authenticate against qBittorrent 5 while Radarr/Sonarr/Lidarr pass with identical credentials — books pipeline parked. Full docs: serverAppConfigs-docs/Dankserver/prowlarr.md. _(context: Repo /home/dank/Projects/serverAppConfigs; first real grab (Dark Knight 2160p) downloaded and imported proves the chain; 76 Sonarr + 18 Lidarr releases via Nyaa feed id 4.)_
+- frank-default-large (agentionai Qwen3.8-Flash-Next ROCmFP4 FAST-imatrix-v2, qwen4exp MoE: 12 attention + 36 SSM layers) loaded successfully at CTX_SIZE=401408 / PARALLEL=2 (200,704 tokens per slot) on Dankclaw (119.3 GiB HIP heap) without OOM. Load time: ~74s. Health: 200. The model's architecture explains why this fits: only 12 attention layers use KV caches, 36 SSM layers don't — KV memory grows as O(L_attn × n_heads × head_dim × ctx) per slot, not O(L_total × ctx). This compressed KV footprint allows 200K per slot × 2 slots within the 119.3 GiB heap. models.json contextWindow set to 200000 (real per-session capacity, not the full 262144 n_ctx_train). The n_ctx_seq (200704) n_ctx_train (262144) warning fires but is non-critical — model extrapolates fine.
+- frank-default-large (agentionai Qwen3.8-Flash-Next ROCmFP4 FAST-imatrix-v2, qwen4exp MoE: 12 attn + 36 SSM) on Dankclaw loads successfully at CTX_SIZE=401408 / PARALLEL=2 (200,704 per slot) in ~74s without OOM on the 119.3 GiB HIP heap. The model's architecture (only 12 attention layers use KV caches, 36 SSM layers don't) prevents the expected ~160 GiB context buffer. Per-slot KV grows as ~O(L_attn × n_heads × head_dim × ctx), not O(L_total × ctx). The n_ctx_seq (200704) n_ctx_train (262144) warning fires but is not critical — the model extrapolates. Ro cm-smi vram only shows the 4 GiB CPU carve-out; trust rocminfo or the device line for the full SVM pool (116732 MiB). The deployment verified health 200. models.json contextWindow: 200000 reflects the real per-session capacity.
+- When --flash-attn on aborts graph build with GGML_ASSERT(a-type == GGML_TYPE_F32) at ggml.c:5371 in the QSA indexer path (build_attn_qsa calls ggml_fill on F16-cast kq_mask): apply the QSA F32-mask patch pattern — mirror the MLA sparse path in llm_graph_context::build_attn by using the F32 base mask (inp-self_kq_mask) for the top-k surgery (ggml_fill/set_rows), then cast to F16 only at the attention call when flash_attn is enabled. The patch has two hunks: 1) change get_kq_mask() to self_kq_mask with explanatory comment about F32 requirement; 2) insert conditional casting before build_attn_mha: "kq_mask_attn = inp-cparams.flash_attn ? ggml_cast(..., GGML_TYPE_F16) : kq_mask_top_k". Validate live: stop service → rebuild fork binary after applying patch → start on port 7999 with --flash-attn on --ctx-size 32768 --fit off → verify model loads (check log for "model loaded"), health 200, real completion → deploy production. Patch file should be vendored in scripts/patches/ with sha256 documented for reproducible rebuilds. _(context: frank-default-large (agentionai Qwen3.8-Flash-Next ROCmFP4-FAST imatrix) fa-on restore on Dankclaw, 2026-08-29. Source of truth: scripts/patches/qwen4exp-qsa-f32-mask.patch (sha256 f2f4a486e21cfad80bfa42683281d18e1cba22fa43c5e0e2aa0cfdbe141e925f).)_
+- frank-default-large (Dankclaw, charlie12345/ROCmFPX build-qwen38) model swap to agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF (87.06 GiB Q4_0_ROCMFP4_FAST, 2026-08-29) — three verified hard constraints not to re-derive: 1. MUST use the v2/ joined-layout GGUF, never the root -ple16 file: build-qwen38's qwen4exp loader (src/models/qwen4exp.cpp:108) asserts GGML_ASSERT(ple_w != nullptr) and the joined file carries the single per_layer_token_embd.weight tensor the loader requires. The -ple16 (per-head split, 93.5 GiB) file only loads in the LaurentZuijdwijk Vulkan fork. Verified via GGUF header strings (ple16 lacks the tensor; v2 has it, 1153 .weight tensors). 2. --flash-attn on ABORTS graph build on this model: the QSA indexer path (build_attn_qsa in src/models/qwen4exp.cpp) calls ggml_fill on the F16-cast kq mask, which asserts GGML_TYPE_F32 (ggml.c:5371). Model loads and serves fine with --flash-attn off (verified live: health 200, real completion). STRIX_LEAN didn't hit this — agentionai's QSA-compressed layers do. 3. CTX_SIZE is capped at 32768 (the author's quick-start): the model is FULLY GPU-resident (PLE n-gram table is inside the GGUF, no streaming from RAM like STRIX_LEAN) leaving ~32 GiB of the 119.3 GiB HIP heap for context. Verified OOM ladder: ctx 262144 → cudaMalloc 103176 MiB failed; 131072 → 51976 MiB failed; 32768 → loads clean in ~58s. HIP heap measured via rocm-smi/rocminfo (125105988 KB ≈ 119.3 GiB); rocm-smi --showmeminfo vram reports only the 4 GiB carve-out, NOT the SVM pool — trust rocminfo heap for budget math. MTP draft head (agentionai's own, 2.28 GiB) is INCOMPATIBLE with build-qwen38 even against its own matched v2 main — THIRD confirmation of missing tensor blk.0.hc_attn_norm.weight (loader requires full per-layer set; every Flash-Next-era head carries only blk.48). Spec decoding runs ngram-mod only. MTP on this arch requires a head with all 48 hc_attn_norm tensors or a fork tolerating partial draft tensors. _(context: Deploying agentionai Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2 (87 GiB) to frank-default-large on Dankclaw, 2026-08-29. Three crash-loop causes diagnosed live (ple16 loader assert, fa-on graph assert, ctx OOM ladder), each fixed and verified before the final healthy deploy at commit 101b1f8.)_
+- Dankclaw authentik 2026.x build does NOT mount the outpost URL space for proxy-provider meta-header routing: probed live from inside caddy's netns, /outpost.goauthentik.io/ping answers 204 with X-authentik-meta-outpost header (outpost object exists), but every app path with full meta headers (outpost UUID + provider id + app slug) returns 404 — the middleware does not intercept in this build regardless of header format (name=caddy and bare caddy both 404). The one auth URL that answers is /if/flow/default-authentication-flow/?next=... (200, serves login page in place; completes instantly with a valid session and bounces to next=). Working transport pattern shipped in the Caddyfile: guarded vhosts on http://:8016 issue redir https://auth.franksdank.cloud/if/flow/default-authentication-flow/?next=https://app-host/ 302 instead of meta-header reverse_proxy — unauthenticated sees login in place, authenticated pays one extra hop per request. If a future Caddy reload reintroduces meta-header routing on :8016, it will 404 again — this is the regression signature. Provider pk=1 lives at id column (not uuid) in authentik_core_provider in this build; app slug is caddy.
+- Dankclaw auth chain, state as of 2026-08-30: guarded apps (search/romm/files/etc.) route the whole request through authentik-server:9000 with X-authentik-meta-outpost/-provider/-app headers, but in this 2026.x build that path answers 404 even with full meta headers — the embedded outpost's URL space is mounted at its authentik_host (DB row uuid 7e39655b-58d3-4fe6-a80a-513f31051217, type=proxy, managed goauthentik.io/outposts/embedded, authentik_host=https://auth.franksdank.cloud), which is the auth.* vhost kept EXEMPT from Anubis by design. So post-auth landing = authentik's own 404 page = the user's "can't access searxng after logging in". Verified intact: Anubis→Caddy hop (200 from inside anubis netns), provider pk=1 exists, www callback URIs match external_host (fixed previous turn). PENDING USER DECISION between two shapes: A) point the guarded apps' auth hop at where auth actually lives (route /outpost.goauthentik.io/* + /if/flow/* through the auth.* vhost — one-line Caddyfile change per app, small blast radius, my lean) vs B) flip the outpost to standalone (own container + quadlet unit + AUTHENTIK_TOKEN; heavier). When the user answers A or B, execute that shape; verify before/after by dialing from inside the container netns (podman run --rm --network container:caddy ... wget -S), never from the host.
+- Dankclaw edge facts (verified 2026-08-29): the homepage hostname moved to www.franksdank.cloud — Caddyfile now 308-redirects naked/legacy names onto www at the TLS edge, and the authentik proxy provider external_host + OAuth2 stored _redirect_uris were flipped to www as a pair (strict matching rejects any callback not byte-identical to a stored URI; that rejection is what users see as "redirect URI mismatch"). Public DNS: zone franksdank.cloud is served by Porkbun nameservers (salvador/maceio/fortaleza/curitiba.porkbun.com); ingress for all service A records is port-forwarded to WAN IP 71.227.227.172 (not a tunnel), and www.franksdank.cloud had NO public record at all — created www A 71.227.227.172 TTL 300 via Porkbun API v3 that day. A wildcard/already-issued cert means TLS for any new *.franksdank.cloud hostname is instant on first handshake (verified: www answered 200 in 10 ms). Invariant to re-check before declaring any hostname move "working": every vhost carrying traffic must have (a) a public DNS record and (b) a provider external_host + redirect URIs matching that exact host — repo-text tests cannot see either; they live in the live DB and the public zone. Porkbun API v3 shape gotchas: endpoint is dns/retrieve/{domain} (not getAll — wrong path answers with an HTML 404 page, not JSON), and the response puts records under a top-level records key with a cloudflare flag; content is null. Creds live in /home/dank/serverAppConfigs/dankclaw/.env on dankclaw (PORKBUN_API_KEY/PORKBUN_API_SECRET_KEY) — use in place, never print.
+- Dankclaw Authentik (2026.x image) operational truths, verified live 2026-08-29: (1) The embedded proxy outpost matches providers by EXACT external_host hostname — a request arriving with Host=www.franksdank.cloud while the provider's external_host is https://franksdank.cloud matches ZERO providers and authentik answers its error page (not 404). Fix applied live: UPDATE authentik_providers_proxy_proxyprovider SET external_host='https://www.franksdank.cloud' WHERE oauth2provider_ptr_id=1 via podman exec authentik-postgres psql -U authentik -d authentik; no restart needed — providers resolve per request. Caddyfile now routes www as the guarded vhost and naked domain + legacy names 308-redirect at the TLS edge (commit 8898740). (2) Live-DB probe recipe: ak shell is interactive-only; ORM imports fail because 2026.x moved models out of Python modules (ProviderEndpoint, ProxyProvider, Token gone from their classic modules); query postgres directly with role/db authentik (POSTGRES_USER=authentik, NOT postgres). Column names differ in this version — pk/id may not exist on user/group tables; use \d or SELECT * instead of guessing columns. (3) Management commands: ak create-install-token does NOT exist in this version (Unknown command); admin seat = membership in the 'authentik Admins' group (is_superuser=t), created via ak create_admin_group user which takes a required positional user arg; live users are akadmin (frank8465@protonmail.com) and admin (admin@franksdank.cloud), both with real pbkdf2 hashes — lost-password escape hatch is ak create_recovery_key. _(context: Fixing www.franksdank.cloud serving an authentik error page while the naked domain served the homepage; traced to proxy-provider external_host exact matching, fixed via live DB UPDATE + Caddyfile topology flip deployed through scripts/deploy.sh --service caddy.)_
+- Dankclaw Authentik (2026.x image) live-ORM probing moved to direct SQL: model imports like from authentik.core.models import ProviderEndpoint / authentik.outposts.models.ProxyProvider now raise ImportError — query the DB directly instead (podman exec authentik-postgres psql -U authentik -d authentik; role/db = authentik, NOT postgres). Introspect before guessing columns: several tables use uuid PKs (groups), so id/pk probes error; SELECT * / \d first. Command surface verified live: NO create-install-token command exists in this version ("Unknown command"); createsuperuser is explicitly deprecated ("should not be used. Instead, use ak create_admin_group"); canonical admin creation = ak create_admin_group user (takes a required positional user). Lost-password escape hatch = ak create_recovery_key. ak swallows piped stdin; works via ssh -tt + podman exec -it which allocates a real PTY. Live state at capture: admin seat held by user akadmin (email frank8465@protonmail.com) in authentik Admins superuser group; second account admin (admin@franksdank.cloud) membership unverified; both carry real pbkdf2 hashes — login failures there are password-memory problems, not missing accounts. Embedded outpost uuid matches Caddyfile meta-outpost header; provider pk=1 exists with internal_host http://caddy:8017. _(context: Live status investigation of "Authentik is still not working properly" on Dankclaw — probed DB/commands live instead of trusting docs; several column-name guesses failed before SELECT * / \dt introspection ended the guessing.)_
+- When Frank asks to "research", "look into", or "hunt" something on the homelab, he wants a verified state REPORT first — not immediate deployment. Acting on findings before his approval draws pushback ("I only said research, don't bring it up immediately"). Correct pattern: probe live state (DBs, logs, APIs, disk), report verdict + evidence + the single unblocking move with cost, then wait for his explicit go before mutating anything. He granted blanket autonomy only when he re-pins a directive verbatim.
 - MTP draft heads for Qwen3.8-Flash-Next are INCOMPATIBLE with the STRIX_LEAN Uncensored model on the charlie12345/ROCmFPX qwen4exp binary. Both attempted heads — EasiiX Q8_0 (4.1 GB) and agentionai ROCmFP4-FAST (2.28 GiB) — fail with missing tensor 'blk.0.hc_attn_norm.weight'. The qwen4exp binary expects this hyper-connection attn norm tensor in the draft GGUF, but neither draft head carries it. MTP only ever worked on the 27B model (kingjones-Qwen3.8-27B-STRIX-MTP with mtp-Qwen3.8-27B-Q4_0.gguf), never on any Flash-Next variant. The Flash-Next STRIX (non-LEAN) that ran before the 2026-08-28 session never loaded a draft model at all. Current production config: SPEC_TYPE=ngram-mod only. To get MTP working on STRIX_LEAN, need either a draft head that includes hc_attn_norm tensors or a binary that doesn't require them for draft models. _(context: frank-default-large MTP integration attempt on Dankclaw, 2026-08-28. Downloaded two MTP heads from HuggingFace, both failed to load against the STRIX_LEAN Uncensored model. Service crash-looped on each attempt. Fell back to ngram-mod only. Files: scripts/frank-default-large.env, scripts/serve-frank-large.sh.)_
 - The MTP draft head GGUF mtp-Qwen3.8-Flash-Next-Q8_0.gguf (EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF, 4.1 GB) is INCOMPATIBLE with the STRIX_LEAN Uncensored Flash-Next model on the charlie12345/ROCmFPX binary (build-qwen38). The draft GGUF is missing tensor blk.0.hc_attn_norm.weight (hyper-connection attention norm) that the qwen4exp architecture expects. The service crash-loops: main model loads fine (~59s), then draft model load fails and exits. MTP only ever worked on the older 27B model (Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT + its own mtp-Qwen3.8-27B-Q4_0.gguf head). The Flash-Next STRIX (non-LEAN) that ran before this session never loaded a draft model either — it ran ngram-only. Current production config: SPEC_TYPE=ngram-mod (no draft sidecar). TODO: find a compatible MTP head for STRIX_LEAN or build one from the Flash-Next checkpoint. _(context: frank-default-large MTP draft head integration on Dankclaw, 2026-08-28)_
 - When running Python scripts on a remote host (Dankclaw) that need to manage local services (systemctl, sed), use subprocess.run(cmd, shell=True) for local execution — NOT nested SSH back to the same host. Nested SSH to localhost fails (no key auth for self-loopback). Also: before git pull --ff-only on a remote, verify the branch with git branch --show-current — Dankclaw was on frank-large-strix-model not main, so pulls were no-ops and new files didn't appear. _(context: bench-spec-decode.py ran on Dankclaw but used ssh() to SSH back to itself for systemctl/sed. All SSH calls failed with Permission denied. Also: Dankclaw was on frank-large-strix-model branch, not main, so git pull --ff-only was a no-op and the new file didn't appear.)_

agent/models.yml

@@ -1,32 +1,232 @@
-# Imported from ~/.pi/agent/models.json (pi agent settings)
+# OMP agent model registry — single source of truth (OMP is the primary
+# harness; there is no second copy to keep in sync).
+# Conversion notes from the pi-era import:
 # - apiKey "EMPTY" -> auth: none (keyless local endpoint)
 # - thinkingLevelMap {max: xhigh} -> compat.reasoningEffortMap
 providers:
-  "dankserver":
+  # Provider NAMES are labels, and every reference to them lives in config.yml
+  # (modelRoles, agentModelOverrides) or in another model's compactionModel —
+  # rename one and a missed reference fails at call time. So each label is
+  # migrated with its model in the same edit. One provider per HOST:PORT, so
+  # Dankclaw's second endpoint (:7771) is a separate block rather than a second
+  # model inside the :7770 one.
+  "llamacpp (dankserver)":
     baseUrl: http://192.168.0.235:7770/v1
     api: openai-completions
     auth: none
     models:
-      - id: frank-default-small
+      - id: frank-archon
-        name: "frank-default-small"
+        # Model id renamed from frank-primary on 2026-10-08 (official
-        contextWindow: 204800
+        # unit-name scheme; the systemd service, scripts and runbooks keep
-        maxTokens: 204800
+        # the frank-primary name — service names are unchanged).
+        # Server-side compaction routes to compaction.remoteEndpoint in
+        # config.yml (192.168.0.143:7771 — frank-zealot's 3090); the
+        # compactionModel candidate keeps the wire model id valid there.
+        compactionModel: llamacpp (dankclaw-3090)/frank-zealot
+        name: "frank-archon"
+        # Dankserver: BOTH Radeon AI PRO R9700s (gfx1201, TP=2 over the
+        # 2026-09-20 board swap's root ports, wht6 wire). Engine:
+        # radiance (StillDeadcode, Apache-2.0) from the digest-pinned
+        # stilldeadcode/radiance image (index digest sha256:4c3f4845..., 287
+        # MiB, kernels prebuilt for gfx1201), plain `podman run` through
+        # scripts/serve-frank-primary.sh, root unit (User=dank, dank's
+        # rootless podman storage). Serving the OFFICIAL
+        # Qwen/Qwen3.8-Flash-Next-FP8 at NATIVE modelopt block-FP8
+        # precision — a rad-convert no-recipe pure copy into
+        # qwen4exp-fp8-native.rad (172.76 GiB measured by plan-only), built by
+        # scripts/build-frank-primary-rad.sh against a staging dir whose
+        # config.json drops quantization_config so the shipped qwen4exp
+        # plugin's (qwen4exp, "") selection key matches; per-weight
+        # encodings are untouched. Experts stream from a 40 GiB/rank
+        # pinned host pool; the 47.7 GiB PLE ngram table is pinned in RAM
+        # (Frank's call); KV fp8; MTP depth 3 (the checkpoint's own mtp.*
+        # head); prefix cache 4 GiB host + 64 GiB disk. Chat template:
+        # froggeric's fixed qwen3.8 v22.5
+        # (scripts/qwen38-froggeric-v22.5.jinja) via
+        # --override-chat-template. Rollback: git tag
+        # frank-primary-freetoken-pr587-20261005 (FreeToken, one card).
+        contextWindow: 200000
+        # maxTokens must survive the harness compaction gap AND the chars/4
+        # token-estimate under-run on code-heavy prompts. The clamp is
+        # max_tokens = min(maxTokens, contextWindow − estimate − 4096); the
+        # estimate under-runs by ~3 % on code. Compaction fires at 85 % of
+        # the window = 170,000 est; worst-case prompt + budget ≈
+        # 170,000 × 1.03 + 16,384 ≈ 191,500 ≤ 200,000 — the engine
+        # refuses a longer request, never truncates.
+        maxTokens: 16384
         reasoning: true
+        # Text + image: the official FP8 checkpoint carries the vision
+        # tower (vision_config depth 27; the first blocks stay bf16 in
+        # modules_to_not_convert) and the plugin declares it
+        # (arch/common/rad_block_vit.h). Video preprocessing configs ship
+        # too; images are the contract that matters here.
         input: [text, image]
         compat:
+          # radiance's --reasoning-format defaults to auto: thinking
+          # comes back in `reasoning_content` (config.cpp). No parser
+          # flags exist or are needed.
+          reasoningContentField: "reasoning_content"
+          # The froggeric template does not gate reasoning_effort the way
+          # Qwen's shipped template did, and radiance special-cases only
+          # none/minimal (core/server/oai.cpp:609). The map stays as the
+          # zero-cost safety net, same as frank-zealot's.
           reasoningEffortMap:
+            minimal: low
+            low: low
+            medium: medium
+            high: xhigh
             max: xhigh
-  "dankclaw":
+          # Qwen recommended sampling. extraBody is merged last and wins.
+          # Every key is inside radiance's known-parameter list
+          # (core/server/oai.cpp:194-205); unknown keys are refused 400.
+          extraBody:
+            temperature: 1.0
+            top_p: 0.95
+            top_k: 20
+            min_p: 0.0
+            presence_penalty: 0.0
+            repetition_penalty: 1.0
+            chat_template_kwargs:
+              enable_thinking: true
+              preserve_thinking: true
+
+  "llamacpp (dankclaw)":
     baseUrl: http://192.168.0.143:7770/v1
     api: openai-completions
     auth: none
     models:
-      - id: frank-default-large
+      - id: frank-dragoon
-        name: "frank-default-large"
+        # Model id renamed from frank-large on 2026-10-08 (official
-        contextWindow: 262144
+        # unit-name scheme; the service/unit/scripts keep the frank-large
-        maxTokens: 262144
+        # name — service names are unchanged).
+        name: "frank-dragoon (dankclaw-gpu)"
+        # Server-side compaction runs on the compaction.remoteEndpoint in
+        # config.yml (192.168.0.143:7771 — frank-zealot's 3090); this steers the
+        # compaction model candidate so the wire model id stays valid
+        # there even from a frank-dragoon session.
+        compactionModel: llamacpp (dankclaw-3090)/frank-zealot
+        # halogen-flash-server 0.16.2 (ghcr.io/peonist-ai/
+        # halogen-flash-server@sha256:0c61bf84, digest-pinned) serving the
+        # Quat3rnion ABLITERATED Swift1.5 finetune of peonist's
+        # Qwen3.8-Flash-Next v2 checkpoint (qwen38-flash-next-v2-swift15-
+        # abliterated.hgn, ~62 GiB resident, + REQUIRED n-gram table +
+        # vision tower) on the Strix Halo iGPU: TWO slots at
+        # HALOGEN_CTX=262144 each with the MTP head ALWAYS ON
+        # (HALOGEN_SPEC_ADAPT=0 — the default adaptive policy halves
+        # decode). Card: 53-57 t/s greedy, ~1,640 t/s prefill at 34k, ~72%
+        # fewer thinking tokens than the censored 27B. ABLITERATED base
+        # (refusals should not return) and VISION ON (OpenAI image_url
+        # parts; >=256 px images or it answers "video"). Runbook:
+        # serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md.
+        # Per-SLOT window the harness may fill — deliberately BELOW the
+        # engine ceiling (HALOGEN_CTX=262144): the clamp below leaves a
+        # 45k gap that absorbs the estimate error.
+        contextWindow: 217088
+        # The clamp is max_tokens = min(maxTokens, contextWindow − est −
+        # 4096); the chars/4 est UNDER-counts the rendered prompt by ~3 %
+        # on code/agent traffic (measured 2026-10-07: est 152,938 ->
+        # server 157,388). Worst-case request = ctx − 4096 + 3 % est
+        # error ≈ 219k, inside the 262,144 engine ceiling with ~43k
+        # margin (robust to a 10 % under-count). Compaction fires at 85 %
+        # of the window = 184,524 est ≈ 190k measured (Frank's 180k
+        # target) and rides on frank-zealot: input ≈ threshold −
+        # keepRecent(20k) ≈ 170k, inside its 200,704 slot.
+        # HALOGEN_MAX_TOKENS_CAP in frank-large.env must stay >= this.
+        maxTokens: 45056
+        reasoning: true
+        # VISION ON as of 2026-10-06: the v2 vision tower loads beside the
+        # checkpoint (HALOGEN_VISION_TOWER; red-PNG probe verified on the
+        # card). image_url content parts only — the legacy top-level
+        # images field is the frank-secondary trap, do not repeat it here.
+        input: [text, image]
+        compat:
+          # flash-server emits thinking as `reasoning_content` — same
+          # field as every engine this endpoint has run, no harness
+          # change needed.
+          reasoningContentField: "reasoning_content"
+          # Same Qwen3.8 template family (it ships inside the tokenizer
+          # bundle): any reasoning_effort outside {xhigh, medium, low} is
+          # rejected, not clamped — the fold stays mandatory.
+          reasoningEffortMap:
+            minimal: low
+            low: low
+            medium: medium
+            high: xhigh
+            max: xhigh
+          # The card's thinking recipe, on the wire. The flash-server
+          # engine HAS a server-side sampler (HALOGEN_TEMPERATURE/TOP_P/
+          # TOP_K = 1.0 / 0.95 / 20 in frank-large.env) — these match it,
+          # so per-request values are consistent, not load-bearing.
+          extraBody:
+            temperature: 1.0
+            top_p: 0.95
+
+  "llamacpp (dankclaw-3090)":
+    baseUrl: http://192.168.0.143:7771/v1
+    api: openai-completions
+    auth: none
+    models:
+      - id: frank-zealot
+        # Model id renamed from frank-secondary on 2026-10-08 (official
+        # unit-name scheme; the service/unit/scripts keep the
+        # frank-secondary name — service names are unchanged).
+        name: "frank-zealot (dankclaw-3090)"
+        # Server-side compaction routes to compaction.remoteEndpoint in
+        # config.yml (192.168.0.143:7771 — this endpoint); the
+        # compactionModel candidate keeps the wire model id valid there.
+        compactionModel: llamacpp (dankclaw-3090)/frank-zealot
+        # Dankclaw's RTX 3090: llama.cpp (master, CUDA sm_86 build) serving
+        # HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive at Q6_K
+        # (7,359,259,008 B) — qwen35 dense hybrid (32 layers: 8 full-attention
+        # + 24 Gated DeltaNet, 4 KV heads x 256 dim, native 262,144), TWO
+        # conversations (--ctx-size 401408 / --parallel 2) at Frank's 200k cap
+        # each. Exact KV: f16 K + f16 V. Vision via the publisher's OWN
+        # BF16 mmproj (921,704,448 B) — the model is natively multimodal.
+        # **MTP IS OFF** because the file carries no nextn/mtp tensors —
+        # SPEC_TYPE=none strips both spec flags. Priced from the MiMo Q6_K
+        # boots at this geometry (21,276/2,849 at ubatch 1024; 21,846/2,279 at
+        # 2048). 2026-10-07 (late) swap from Gemma4-26B-A4B Q4_K_M (1x200,704).
+        # Runbook:
+        # serverAppConfigs-docs/LLM/frank-secondary-llama-qwen35-9b-q6k-2x200704.md.
+        # contextWindow is the PER-SLOT window (Frank's cap): a longer
+        # prompt needs compaction.
+        contextWindow: 200704
+        # Prompt + budget must fit inside the slot window; thinking counts
+        # against the budget. Compaction fires at 85 % of the window =
+        # 170,598 est; worst-case prompt + budget ≈ 170,598 × 1.03 +
+        # 16,384 ≈ 192,100 ≤ 200,704 — a longer prompt is a 400 naming
+        # the limit, never a truncation.
+        maxTokens: 16384
         reasoning: true
         input: [text, image]
         compat:
+          # llama.cpp --reasoning-format deepseek emits thinking as
+          # `reasoning_content` — the same field the MiMo/Ornith eras used;
+          # harness reads unchanged.
+          reasoningContentField: "reasoning_content"
+          # The Qwen3.5 chat template has NO reasoning_effort variable (unlike
+          # the Qwen3.8-template endpoints, where an unmapped level is a hard
+          # Jinja raise). llama.cpp passes chat_template_kwargs through to
+          # Jinja, and Jinja ignores variables a template never reads — so the
+          # map is inert here, kept so role-level overrides resolve identically
+          # across the fleet.
           reasoningEffortMap:
+            minimal: low
+            low: low
+            medium: medium
+            high: xhigh
             max: xhigh
+          # The Qwen3.5 publisher recipe (model card, thinking mode):
+          # temperature 0.6 / top_p 0.95 / top_k 20 / min_p 0. Non-thinking
+          # mode is 0.7/0.8/20/0 — the harness runs thinking mode.
+          # extraBody is merged last and wins over the harness default.
+          extraBody:
+            temperature: 0.6
+            top_p: 0.95
+            top_k: 20
+            min_p: 0.0
+            presence_penalty: 0.0
+            chat_template_kwargs:
+              enable_thinking: true
+            # (no preserve_thinking: that kwarg belongs to the Qwen3.8
+            # templates; nothing in this template reads it)