command-center/agent/memories/--home-dank-Projects-serverAppConfigs--/raw_memories.md

# Raw Memories

## 01a07351-8e05-75bc-9deb-bd9447e1256d
updated_at: 1788678884
ROOT CAUSE: Alias drift (pre-rename IDs persisted) + Config-home drift (HERMES_HOME pointed to /opt/data, an auto-created anonymous volume). Container recreation wipes /opt/data, reverting to image-default template config.
DECISION: Pinned Environment=HERMES_HOME=/data in quadlet unit to unify config and state root under /data. In-place renamed live config IDs to frank-large/frank-small/dankclaw-npu.
WORKFLOWS AND PITFALLS: Podman VOLUME paths get fresh anonymous volumes on recreation; edits lost unless pinned. provider: auto without explicit api: routes to OpenRouter; fails if API key missing. FLM/NPU routes by exact registry tag; HTTP 200 does not guarantee valid model (watch for error: Failed to load tag model!). Named providers in providers: dict resolve via api:/url:/base_url keys; custom:<slug> requires custom_providers[] entry.
VERIFICATION: Use resolve_model_provider() and resolve_user_provider() for routing truth; probe endpoints with exact served IDs; /api/providers lists models with key_source: config_yaml.
ARTIFACTS: /data/config.yaml, api/config.py, providers.py, runtime_provider.py, quadlet units, deploy.sh.

## 01a06ef4-c78b-71df-98e0-e1b930f260a5
updated_at: 1788579402
Category shift: Sonarr moved to 'admin' in shared/homepage/services.json.
Exposure constraint: Sonarr kept local-only; removed 'sonarr.franksdank.cloud' vhost from dankclaw/Caddyfile to prevent external routing.
Resolved pitfall: Prior directive conflated Seerr and Sonarr external reachability; corrected by auditing Caddyfile @sonarr handle block and services.json category mapping.
Paths: shared/homepage/services.json, dankclaw/Caddyfile, frankbook-pro/sonarr/quadlet/sonarr.container.
Workflow: Category update + Caddyfile vhost strip -> commit (b6f5b2b) -> push -> verify.

## 01a06e56-5e70-73f5-83b3-a28b1736f34b
updated_at: 1788561197
- Pi-hole admin password reset to pihole.2026.
- Public DNS for pihole.franksdank.cloud removed from Porkbun (NXDOMAIN externally); LAN resolution persists via addn-hosts (192.168.0.143).
- Caddy vhost retains basicauth edge and TLS termination, reverse proxying to Pi-hole quadlet published on 192.168.0.143:8083.
- Architectural note: Pi-hole serves as LAN DNS authority (dnsmasq/FTL) but sits behind Caddy for edge traffic handling, avoiding direct public exposure while maintaining internal name resolution.

## 01a06b4c-fcea-70e2-a342-a41c5f99d927
updated_at: 1788553425
Consolidated *arr stack to FrankBook-Pro (192.168.0.112), removing old Dankserver (235) instances.
Six containers (QBit + 5 arrs) share a single gluetun network namespace; zero internal PublishPort on arrs, gluetun publishes full port set (8080, 6881, 9696, 8989, 7878, 8686, 8787).
All egress forced through ProtonVPN (AllowedIPs=0.0.0.0/0); verified via podman exec IP echo (51.179.177.14 vs residential 71.227.227.172).
Local DNS: Pi-hole quadlet on dankclaw serves addn-hosts, resolving *arr service names to Caddy (192.168.0.143).
Caddy topology: TLS vhosts -> Anubis :8923 (AI scrapers deny/challenge) -> Caddy internal :8017 -> app containers.
arr→qB client config updated to 127.0.0.1:8080 (resolved 10.88.0.1 gateway routing into tunnel).
Tests: frankbook_quadlet.bats enforces shared netns/zero ports; unit & Playwright E2E suites pass.
Parked defects: 1) Prowlarr Applications table empty (sync dead until rebuilt with loopback baseUrls); 2) readarr qB proxy upstream auth defect (post-move retest); 3) EZTV/1337x CloudFlare-block likely lifted for VPN egress.
Old configs archived in ~/.AGENTTRASH/20260904-dankserver-arr-teardown/. Commits: 4c7465d, df4339a.

## 01a06aa2-9087-753a-8019-50af77fbaddf
updated_at: 1788501666
File: shared/homepage/services.json. Action: Reformatted from compact to multi-line JSON (2-space indent). Committed 7 description updates (SearXNG, OpenWebUI, Gitea, Copyparty, Plex, Jellyfin, Seerr). Workflow: git add/commit, ./scripts/deploy.sh dankclaw --service homepage. Result: Deployment 1967584 successful.

## 01a06894-8947-74fa-ba09-0f104cd6677f
updated_at: 1788495432
DEPLOYMENT & HOSTING:
- frank-npu service deployed to dankclaw via deploy.sh (commit df0c663, atop b563053).
- Unit: serve-frank-npu.service running on port 7771.
- FLM registry path: /home/dank/.config/flm/model_list.json.

MODEL BEHAVIOR & ROUTING:
- Resident model: nuextract3:4b. First probe confirms residency (2.6s latency, correct extraction output).
- Context Length: FLM_CTX_LEN=-1 (no global override). Packs load trained context (nuextract3: 32768, gemma4: 131072).
- Routing: FIFO single-lane NPU. Heretic requests (qwen3.8-distilled-heretic:9b) pay ~6-11s swap cost on tag alternation. Single-tag consumers pay once per switch direction.

NAMING CONVENTIONS & RENAME WORKFLOW:
- Renaming: frank-default-sightsound → frank-npu; frank-default-small → frank-small; frank-default-large → frank-large.
- Scope: Scripts, systemd units, env files, docs.
- Critical Order: Update models.json (~/.pi/agent/models.json) with new aliases BEFORE deploying renames to prevent agent connection loss during LLM runtime restarts.

PLAYWRIGHT TEST SPECIFICATIONS:
- Unauthenticated users must be blocked from direct navigation to Authentik-protected apps (except exempt services like Plex).
- Admin users must retain guaranteed access to admin space and admin apps.

HOST & REPO CONTEXT:
- Repo: /home/dank/Projects/serverAppConfigs (the workstation).
- Live hosts: dankclaw (NPU/LLM), dankserver.
- SSH accessible; deploy.sh handles push and dispatch.

## 01a063a7-4c37-7765-8f58-21f7c860aed7
updated_at: 1788403307
Context/workflow:
- The workstation is the operation host; repo path is /home/dank/Projects/serverAppConfigs.
- Main deployment hosts identified/reconfirmed: Dankserver at 192.168.0.235 and Dankclaw at 192.168.0.143. Both were reachable via passwordless SSH from the workstation.
- Services are primarily defined by per-host Quadlet .container files in the repo and installed as user-systemd units (`systemctl --user ...`) generating `<name>.service`. Some model/LLM services may be system units rather than user units.
- Useful status enumeration commands:
  - `cd /home/dank/Projects/serverAppConfigs && find . -path ./.git -prune -o -name '*.container' -print | sort`
  - For a host: `ssh <host> 'systemctl --user list-units --type=service; podman ps -a --format "{{.Names}} {{.Status}} nets=[{{.Networks}}]"'`

Hermes/Dankclaw diagnostic result:
- Command used:
  `ssh -o ConnectTimeout=10 dankclaw 'echo "== units:"; for u in hermes hermes-webui hermes-dashboard signal-cli-daemon; do printf "%-20s %s\n" "$u" "$(systemctl --user is-active $u.service 2>/dev/null) $(systemctl --user is-enabled $u.service 2>/dev/null)"; done; echo "== containers:"; podman ps -a --format "{{.Names}} {{.Status}} nets=[{{.Networks}}]" | grep -Ei "hermes|signal|chrome"; echo "== listeners:"; ss -lntp 2>/dev/null | grep -wE "8080|8787|9119|9222" | tr -s " "; echo "== webui env (names only):"; podman exec hermes-webui sh -c "env | cut -d= -f1 | grep -Ei \"hermes|signal|base_url\" | sort" 2>&1 | head -20'`
- Output showed:
  - `hermes`: active, generated
  - `hermes-webui`: active, generated
  - `hermes-dashboard`: active, generated
  - `signal-cli-daemon`: inactive, disabled
  - Containers:
    - `chrome-devtools-protocol` up ~2 days, no listed network
    - `systemd-hermes` up ~38h, no listed network
    - `hermes-dashboard` up ~38h, network `dankclaw_default`
    - `hermes-webui` up ~38h, healthy, no listed network
  - Listeners:
    - `0.0.0.0:8787` listening (Hermes WebUI)
    - `127.0.0.1:9222` listening by `headless-shell` (Chrome DevTools Protocol)
    - `*:9119` listening via `rootlessport`
    - No listener on `8080` in this diagnostic, consistent with `signal-cli-daemon` being down.
  - `hermes-webui` env names include `SIGNAL_ACCOUNT`, `SIGNAL_ALLOWED_USERS`, `HERMES_WEBUI_*`, etc.
- Durable diagnosis: if Hermes WebUI says it cannot reach Hermes/Signal, first check `signal-cli-daemon.service`. In this captured state, Hermes-related services/containers were active/healthy, but `signal-cli-daemon` was inactive/disabled and there was no port 8080 listener, so the Signal HTTP transport was likely unavailable even though the WebUI itself responded on 8787.
- Suggested next operational checks/fix path:
  - `ssh dankclaw 'systemctl --user status signal-cli-daemon.service --no-pager; journalctl --user -u signal-cli-daemon.service -n 80 --no-pager'`
  - If appropriate, start/enable it: `ssh dankclaw 'systemctl --user enable --now signal-cli-daemon.service'`
  - Verify: `ssh dankclaw 'systemctl --user is-active signal-cli-daemon.service; ss -lntp | grep -w 8080'`

## 01a06085-8146-72b0-9e52-b486c767a2f6
updated_at: 1788331026
GOAL: Migrate Plex (native plexmediaserver.service) and Tautulli (snap) to podman quadlets on dankserver. Zero-bytes-moved design.

IMAGES:
- Plex: lscr.io/linuxserver/plex:1.43.3.10896-cb3ebc72d-ls322 (superseded initial ls9 pin)
- Tautulli: lscr.io/linuxserver/tautulli:v2.18.1-ls242 (verified ≡ latest digest)

USER & UID MODEL:
- UserNS=host for both. Rootless podman cannot map real uid 997 natively.
- Plex: PUID=0/PGID=0 (container root=dank), data root /var/lib/plexmediaserver chowned to 1000:1000. Rollback chowns back to 997:984.
- Tautulli: PUID=1000/PGID=1000.

VOLUMES:
- Plex: /var/lib/plexmediaserver:/config (RW, in-place), /mnt/entropy:/mnt/entropy:ro
- Tautulli: /home/dank/Containers/tautulli:/config

NETWORK & PORTS:
- App: dankserver_appnet. Plex alias=plex, Tautulli alias=tautulli.
- Plex: 32400:32400. Tautulli: 8181:8181.
- Tautulli config.seed: PMS_Web_Host=plex, PMS_Web_Port=32400, PMS_Secure=0.

CUTOVER & HEALTH:
- Native units stopped & masked (Plex), snap disabled (Tautulli). Instant rollback via unmask/snap enable + chown.
- Plex health gate widened to 600s (32GB DB migrations reset on bounce). No HealthCmd to avoid cold-load flapping.
- Tautulli uses python3 urllib health probe.

PITFALLS & FIXES:
- UsernsMode= (compose syntax) hard-failed v1 generator → corrected to UserNS= or omitted.
- Edit churn ate ContainerName, Network, PublishPort keys → caught by seam test + C5 probe.
- PMS deploy-state record showed pending (cosmetic) → self-heals on next deploy.
- Decommission: apt purge plexmediaserver + snap remove tautulli only on explicit sign-off.

## 01a06035-a0d8-73ad-8a49-279f51c68bf2
updated_at: 1788329168
PLEX_HOMEPAGE: Updated url to http://plex.franksdank.cloud/web/index.html in shared/homepage/dankserver.services.json & services.json. JELLYFIN_MNT: /mnt/entropy:/data:ro (read-only whole volume), health 200, libraries verified working. COPIPARTY_MNT: /mnt/entropy:/w:ro, health 200. JELLYFIN_DB: /home/dank/Containers/jellyfin/config/data/jellyfin.db; table is BaseItems (not TypedBaseItems). DEPLOY: Commit 8895a02, 233/233 tests green. USER_DIRECTIVE: Stop touching Jellyfin/Copyparty after verification.

## 01a05bed-d759-7157-af15-2e178337182a
updated_at: 1788325757
Decisions (Locked):
- D1: Quadlet .container units for all long-running services; compose retired to reference.
- D2: Data root /home/dank/Containers/<svc> on Linux hosts; entropy is mounted volume.
- D3: *arr-stack migrates to FrankBook-Pro; decommissioned on dankserver.
- D4: Keep /Volumes/COMPLETE & /Volumes/COMPLETE/"Finished Torrents" verbatim; rename deferred.
- D5: VPN egress via gluetun container; qbittorrent shares netns (Network=gluetun.container).
- D6: Private LAN-only, direct IP:port, Pi-hole addn-hosts DNS; no Caddy vhost on Mac.
- D7: Scope includes Mac bring-up + cutover.
- D8: Seerr on dankclaw (SQLite, Authentik OIDC SSO), provider created before route flip.

Execution Phases:
- Phase 0: Prereqs (Mac SSH access, WireGuard creds/machine-local env, egress gate check via podman machine ssh).
- Phase 1 (dankserver): Per service (romm, mariadb, redis, jellyfin, calibre-web, copyparty): stop -> mv /opt -> /home/dank/Containers -> chown (uid 1000) -> deploy quadlet. RomM landmine: move DB from anonymous volume to /home/dank/Containers/romm/db.
- Phase 2 (dankclaw): Migrate data roots for gitea, authentik, etc.
- Phase 3/4: Mac stack bring-up & *arr cutover.

Technical Constraints & Patterns:
- Quadlet template: [Container] (Image, ContainerName, EnvironmentFile, Volume=bind-mounts-only), [Service] Restart=always, [Install] WantedBy=default.target.
- Entropy write: Only copyparty -v /mnt/entropy:/w with -a dank:<secret> -v /w::r:ra,dank.
- Deployer: ./scripts/deploy.sh computes changes, pushes, target pulls, dispatches. Handles run_privileged and service health checks.
- Secret hygiene: No secrets in tracked repo files (machine-local envs).
- Gotcha: Verify --userns=host in systemctl --user cat <svc>.service after daemon-reload. Data root mkdir -p handled in deployer.

## 01a05bd4-8dbe-759b-8885-8b86990051ed
updated_at: 1788314412
frank-default-large model/runtime configuration:
- Service alias: frank-default-large, served by scripts/serve-frank-large.sh with tunables in scripts/frank-default-large.env.
- Model: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF, file Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf under /home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX.
- Architecture: Qwen3.8-Flash-Next base / qwen4exp MoE, 512 experts / 10 used, 48 layers = 12 attention + 36 SSM, QSA indexer, hyper-connections, PLE at layer 1.
- Quant/file details: Q4_0_ROCMFP4_STRIX, file_type 105, ~113.47 GiB single GGUF. STRIX top quality tier; PLE table ~50.66 GiB Q8_0, token embeddings + lm head Q6_K.
- Runtime: kingjones30/ROCmFPX fork with qwen4exp-on-rocmfpx patch, built at /home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server using HIP for gfx1151, ROCm 7.2.4. Stock llama.cpp and charlie12345/ROCmFPX alone cannot load this GGUF because ROCmFP4 tensor types and qwen4exp architecture support must be in the same tree.
- Hardware target: Dankclaw / Strix Halo gfx1151 unified memory. Env sets HSA_OVERRIDE_GFX_VERSION=11.5.1, HIP_VISIBLE_DEVICES=0, --device ROCm0.
- Important pitfall: PLE n-gram table must stay out of GPU memory and stream via mmap from OS page cache. Do not add --no-mmap or tensor override/offload (-ot) for the PLE table.
- Important pitfall: --fit must remain off; auto-fit segfaults on qwen4exp hyper-connection config.
- Important pitfall: if graph build aborts on QSA F16 mask with flash attention, port scripts/patches/qwen4exp-qsa-f32-mask.patch into the kingjones ROCmFPX tree and rebuild.
- Speculative decoding currently disabled (SPEC_TYPE=none). Flash-Next MTP heads do not load on this loader and the model card ships no spec flags; draft model defaults exist but should not be enabled casually.
- mmproj: STRIX repo ships no mmproj; serve script conditionally passes /home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/mmproj/mmproj-Qwen3.8-Flash-Next-f16.gguf if readable.
- Serving defaults: host 0.0.0.0 port 7770, ctx 401408, parallel 2 slots, batch 4096, ubatch 512, threads 16, threads-batch 16, f16 KV cache, flash-attn on, split-mode none, gpu-layers 999, ctx-checkpoints 16, cache-ram 8192 MiB, prompt cache and idle-slot cache enabled, metrics enabled.
- Sampling defaults: temp 0.7, top_p 0.8, top_k 20, min_p 0.0, presence_penalty 1.5, reasoning on, reasoning-format deepseek, reasoning-budget -1. Env comments describe this as card sampling table mixed with validated serving profile, non-thinking instruct despite reasoning plumbing being enabled.
- Replaced prior agentionai ROCmFP4-FAST imatrix deployment/charlie build-qwen38 runtime around 2026-08-30.

ASUS TUF GAMING B650-E WIFI BIOS update/backup durable workflow:
- Target latest memory-compat BIOS identified: 3886 beta (2026-07), AGESA 1.2.0.3a Patch A. Current referenced BIOS was 2613.
- Board-name caution: use exact ASUS TUF GAMING B650-E WIFI support page; avoid wrong CAPs for similar boards like B650M-E or B650-PLUS.
- Backup current BIOS before flashing. Safe Linux read-only approach:
  sudo pacman -S flashrom
  sudo flashrom --programmer internal -r bios-2613-backup-1.bin
  sudo flashrom --programmer internal -r bios-2613-backup-2.bin
  sha256sum bios-2613-backup-*.bin; hashes must match or re-read.
- Expected AM5 SPI dump size is about 32 MB. Copy backups to USB and another machine. Never use flashrom -w from Linux for writing; power loss or mismatch can brick board. If flashrom reports multiple chips or refuses, use chip-specific selection only if certain or fall back to UEFI/EZ Flash backup.
- Alternative backup: ASUS EZ Flash 3 often offers save/current-BIOS-to-USB from UEFI. Also download archived 2613 CAP from ASUS as a restore/downgrade image.
- Normal flash path: FAT32 USB with CAP in root, reboot, Del into UEFI, Tool → ASUS EZ Flash 3, select CAP, allow verification, save old BIOS if offered, flash without resets/power cuts.
- Rescue path: USB BIOS FlashBack only if board will not POST. System off but PSU standby on, USB in rear FlashBack-marked port, rename CAP exactly per support page/manual (do not guess), hold FlashBack button ~3s, wait until LED stops blinking.
- Operational note: Dankserver hosts media stack + frank-default-small; BIOS flash is ~15–20 min downtime. First boot after AGESA update can spend 1–5 min on black screen due to DDR5 retraining; wait.
- Post-flash: settings reset to defaults. Re-apply memory profile/manual settings and re-run memtest; new AGESA/training invalidates prior memory stability conclusions. Verify version in BIOS main screen or with sudo dmidecode -s bios-version.

## 01a05a7d-cc94-7589-8926-c24cb5a10490
updated_at: 1788248708
Hermes Agent Persistence: Data survives restarts via host bind mount /home/dank/hermes-data:/data. Requires entrypoint override to hermes gateway run --force -v to prevent default TUI crash-loop on exit. Environment HERMES_DATA_DIR and HERMES_HOME must point to /data. Signal HTTP adapter connects to 127.0.0.1:8080 via network_mode: host. Dashboard runs separately on port 9119.
General Container Patterns: ComfyUI uses keep-groups for rootless podman render group resolution; models/data persist under /home/dank/comfyui-data. SearXNG, Gluetun, and Valkey are boot-persistent Quadlet units sharing network namespaces. qBittorrent reuses SearXNG's netns without its own port.
Migration & Deployment Constraints: *arr stack (Sonarr, Radarr, Lidarr, Readarr, Prowlarr) migrating to FrankBook-Pro Mac. Config dirs archived to /home/dank/archive/<date>-arr-configs.tar.gz before removal; virgin installs skip archive. Podman machine init on Mac requires -v /Volumes/COMPLETE -v /Volumes/INCOMPLETE; later addition requires re-init. VPN egress gate must pass before *arr port publishing. Caddy uses Quadlet units. Verify with systemctl --user is-enabled, podman machine ssh, and health probes. Deploy scripts: deploy.sh, deploy-remote.sh, deploy-plan.py manage service order and target dispatch.

## 01a05a29-d9fd-71ca-86c8-b92c602492c7
updated_at: 1788229381
VOLUME SECURITY: POSIX permissions for read/write user via SFTP/SSH. Create dedicated writer user and readers group. Set volume ownership to writer:readers with chmod 2775 (setgid ensures new files inherit group). Deployment: Host volume or Podman container. Restrict SSH/SFTP access via Match block in sshd_config or ForceCommand internal-sftp.

FREETOKEN BUILD (DANKCLAW): Build-only, no deploy/run. Supports Vulkan or ROCm. Pre-installed: ROCm 6.3 (/opt/rocm), NVIDIA driver 575.57.02, CMake, Python, git. Requirements met for compilation.

GGUF LOADING CONSTRAINTS: Restricted to Gemma4GGUFForCausalLM model class. Supported GGML quantization types: F32(0), F16(1), BF16(30) use native torch views; Q4_0(2) and Q6_K(14) use dequant_q4_0/dequant_q6_k. Q8_0(8) is mapped but lacks dequant function (raises NotImplementedError). Loading pipeline: is_gguf_path -> iter_gguf_tensors -> dequantize -> load into graph.

## 01a04a3f-6596-75d0-8749-8ece9a2b11fd
updated_at: 1788224586
REPO_ROOT: /home/dank/Projects/serverAppConfigs
HOSTS: Dankclaw (192.168.0.143, Caddy host), Dankserver (192.168.0.235)
CADDY_SETUP: Single container on dankclaw fronts all services. DNS via dankclaw Pi-hole quadlet; LAN routed via SURFboard G36 router. TLS via Porkbun DNS-01 (no inbound 80/443 required). Admin interface ON (127.0.0.1:2019) enabling config file watcher hot-reload; turning off caused stale basicauth hash. Basicauth hash rendered from host .env into container Caddyfile.
AUTH: Authentik SSO via Caddy forward_auth. Protects homepage, SearXNG, Hermes, Prometheus, RomM, Copyparty. Apps with native auth (Grafana, Gitea, OpenWebUI, Jellyfin, *arr, Pi-hole) bypass forward_auth.
HERMES_BOT: WebUI network flipped to Network=host. Browser automation via chrome-devtools-protocol quadlet; CDP bound to loopback 127.0.0.1:9222. Deployer patched to start service only if inactive (preserves in-flight browser sessions). Trio + daemon managed via systemctl --user.
CONFIG_QUIRKS: /data/config.yaml mounted but owned by unmapped UID (dank user cannot read directly; requires sudo python for edits). NPM npx cache lives in container layer: first tool call in fresh container pays npm download cost; container restarts do not reset cache, but recreation does.
DEPLOY_WORKFLOW: deploy.sh handles quadlet install, daemon start-if-not-running, config wiring, trio restart. Uses stash-deploy-pop dance. State tracked via STATE[hermes] commit hash. FF-only pull on target.

## 01a059f4-cead-704b-b857-fad89ebde42b
updated_at: 1788224497
Context: Dankserver hosts copyparty for `files.franksdank.cloud` and LAN direct access. Traffic chain is Browser -> Caddy edge -> Anubis :8923 -> Authentik outpost -> post-auth :8017 as UNAUTHENTICATED -> copyparty :3923. LAN direct also reaches copyparty :3923.

Resolved change: in `dankserver/compose.yaml`, copyparty volume argument was changed from `-v /w::A` to `-v /w::r:ra,dank`. This makes `/w` browsable/downloadable but not writable/deletable. The copyparty version observed was v1.20.21. There is no separate dedicated reject-uploads flag; read-only behavior is achieved by omitting `w`, `m`, and `d` permissions. Anonymous must keep `r` because the guarded Authentik chain passes authenticated users to copyparty as unauthenticated.

Deployment workflow: `deploy.sh` correctly refuses to directly modify the compose stack on Dankserver but can pull/record rev. To apply the compose change, run on Dankserver: `podman-compose up -d --force-recreate copyparty`.

Verification results after deploy: anonymous `GET /Movies/` returned 200; anonymous `PUT` returned 401; authenticated `dank` `PUT`/`DELETE` returned 403; recent-uploads DB showed `"ups": []`; `files.franksdank.cloud` returned 200 and the chain remained intact.

Critical pitfall discovered: prior runbook claim that anonymous got nothing was wrong. In copyparty, bare `A` means everyone, so `-v /w::A` had made the entire ~91T array anonymously readable, writable, and deletable on LAN/WebDAV. During probing, one 1-byte test file `/mnt/entropy/w/.readability-probe-delete-me` was created; it was verified by exact path and size and removed, then root listing was rechecked to match original state.

Admin credentials at time of rollout: username `dank`, password `m9FuwPKWotsPz7jUUn`. Stored as `COPAY_PASS=` in `dankserver/.env` on Dankserver (gitignored). Compose interpolates this into container argv, so anyone able to `podman inspect copyparty` on the host can see it. Rotate by editing `.env` and recreating the service. After read-only change, `dank` has read plus admin panel but no writes by design.

Outstanding operational gap: copyparty/dankserver stack was still running as a bare compose container with no Quadlet unit; boot-persistence rule is not satisfied for the whole dankserver stack. This was flagged but not changed.

## 01a05a42-6580-7043-96eb-a73344efd9e6
updated_at: 1788221481
Hermes FLM/Gemma rollout on Dankclaw:
- User request: add Gemma FLM model to Hermes agent.
- Context: Hermes live data is bind-mounted at /home/dank/hermes-data and appears in-container as /data. Repo contains sanitized backup at dankclaw/hermes/config.yaml.example; do not commit live-only secrets such as dashboard password hashes, phone numbers, OAuth credentials, API keys, sessions, memories, or generated state.
- Existing Hermes config shape:
  - model.provider/default selects main model.
  - providers entries use keys like dankclaw-large and dankserver-small with api URL, discover_models: false, and explicit models/context_length.
  - auxiliary entries use compact custom endpoint shape: {provider: custom, model: ..., base_url: ...}.
- Added FLM/NPU provider for Dankclaw to Hermes live config and sanitized repo example:
  - provider name: dankclaw-npu
  - FLM OpenAI-compatible base/API URL: http://192.168.0.143:7771/v1
  - model: gemma4-it:e4b
  - discover_models: false
  - use api_key: not-needed where required by config conventions.
- Routed Hermes auxiliary vision to the Gemma FLM model:
  - auxiliary.vision: {provider: custom, model: gemma4-it:e4b, base_url: http://192.168.0.143:7771/v1}
- Main Hermes model remained frank-default-large via dankclaw-large at http://192.168.0.143:7770/v1.
- Adjusted ~/.omp/agent/models.yml so frank-default-large context length is 200000 (reduced from oversized setting to suit actual Hermes/agent behavior).
- Cleaned stray ~/.pi by moving it to ~/.AGENTTRASH/20260831-pi.
- Restarted Hermes container after live config edit; Hermes became healthy.
- Validation: full repo test suite passed, 204 tests total.
- Durable caution: config.yaml.example is only a sanitized baseline backup of /home/dank/hermes-data/config.yaml, not a full replacement for live state. Preserve live secrets and generated state when editing/syncing.

## 01a059ae-72f3-76d4-8e09-735f8f2e0a5d
updated_at: 1788216058
Hosts and identities:
- FrankBook-Pro is reachable as dank@FrankBook-Pro.local / dank@192.168.0.112. It is macOS on x86_64, not iOS.
- dankclaw is 192.168.0.143.
- dankserver is 192.168.0.235.
- This rollout was run from the workstation/dankclaw context with an existing ed25519 SSH key.

SSH configuration completed and verified:
- Passwordless host-to-host SSH works:
  - dankclaw -> FrankBook-Pro returns user dank.
  - dankclaw -> dankserver returns user dank.
  - dankserver -> dankclaw returns user dank.
  - FrankBook-Pro -> dankclaw returns user dank.
- FrankBook-Pro host keys were already present in known_hosts during setup.

Hermes container SSH completed and verified:
- Containers checked: systemd-hermes, hermes-webui, hermes-dashboard.
- Each container can passwordlessly SSH using /root/.ssh/id_ed25519 with IdentitiesOnly=yes to:
  - root@192.168.0.143 (dankclaw) -> returns root.
  - dank@192.168.0.235 (dankserver) -> returns dank.
  - dank@192.168.0.112 (FrankBook-Pro) -> returns dank.

FrankBook-Pro Podman stack installed and verified:
- podman installed at /usr/local/bin/podman, version 5.4.2, Mach-O x86_64.
- gvproxy installed/built at /usr/local/bin/gvproxy, Mach-O x86_64.
- vfkit installed via Homebrew at /usr/local/bin/vfkit, version v0.6.4, Mach-O x86_64.
- Podman machine uses applehv provider and is running:
  - podman-machine-default* applehv, 4 CPUs, 2GiB RAM, 100GiB disk.
- podman run was verified with an Alpine test and confirmed working.

Operational notes:
- For future verification from dankclaw, use SSH checks to 192.168.0.112/235 and podman exec into Hermes containers to test /root/.ssh/id_ed25519 against the three targets.
- Do not assume FrankBook-Pro is iOS; treat it as macOS with Homebrew and AppleHV/vfkit support.

## 01a0598c-244a-766a-8485-b02f8ab630b5
updated_at: 1788209135
DNS architecture / split-horizon behavior:
- Domain: franksdank.cloud.
- Public DNS provider: Porkbun.
- LAN DNS: Pi-hole on the homelab, fed by addn-hosts / local records.
- Public/WAN Porkbun A records point subdomains to WAN IP 71.227.227.172.
- LAN Pi-hole resolves the same subdomains to Dankclaw LAN IP 192.168.0.143.
- G36 router/DHCP gives LAN clients Pi-hole as DNS, so LAN clients hit Pi-hole local records first. WAN clients resolve via Porkbun/public DNS.

Operational consequence:
- It is safe to remove a regular subdomain A record from Porkbun while keeping the Pi-hole record if the name should be LAN-only.
- LAN clients will continue resolving the name to 192.168.0.143 via Pi-hole.
- WAN clients will get NXDOMAIN/unknown subdomain once the Porkbun record is removed, unless cached elsewhere.
- Removing Porkbun DNS only affects name visibility/resolution; it does not itself close router port forwards or prevent direct access by IP if forwarding remains open.

Caddy/TLS implications:
- Single Caddy on Dankclaw fronts services and routes by Host header; Caddy does not care whether the hostname was resolved by Porkbun or Pi-hole.
- HTTPS certificates are wildcard DNS-01 certs for franksdank.cloud issued/renewed using a custom franksdank/caddy-porkbun image with caddy-dns/porkbun.
- DNS-01 renewal uses Porkbun API credentials to create TXT records under _acme-challenge; deleting ordinary A records does not break existing wildcard cert validity or renewal.
- Porkbun credentials live in dankclaw/.env as PORKBUN_API_KEY and PORKBUN_API_SECRET_KEY and are deployed to /home/dank/Containers/caddy/env for the Caddy quadlet; never commit secrets.

Cautions:
- Do not casually remove public DNS for auth.franksdank.cloud: prior sign-in/debugging showed auth needed a public A record for external login flows.
- Be cautious with gitea.franksdank.cloud because git smart-HTTP/external use may depend on public DNS.
- Keep Porkbun records for any service that must be reachable by external users through the public hostname; remove only for LAN-only/internal services.

Known public/LAN service hostnames previously tracked under franksdank.cloud include: auth, books, romm, files, jellyfin, radarr, sonarr, lidarr, readarr, prowlarr, qbittorrent, search, comfyui, chat, hermes, grafana, prometheus, gitea, pihole, and plex (plex.franksdank.cloud is canonical HTTPS).

## 01a051e2-6ff9-72cc-b980-b6837011e99f
updated_at: 1788126110
Service: frank-default-large on Dankclaw (gfx1151 unified memory).
Model: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF -> file Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf (~113.5 GiB).
Runtime: charlie12345/ROCmFPX llama.cpp (/home/dank/GitPackages/ROCmFPX/build-qwen38). ROCm 7.2.4.
Quant/Format: ROCmFP4 (hand-ported from ciru-ai; upstream llama.cpp incompatible).
Attention: Flash-ATTN ON via QSA indexer. Requires qwen4exp-qsa-f32-mask.patch to fix ggml_fill assert (applied in binary).
MTP/Speculative: Draft head does not load (requires blk.0.hc_attn_norm.weight). Uses ngram-mod speculative decoding.
Context: CTX_SIZE ~400k, Parallel 2.
Config: scripts/frank-default-large.env and scripts/serve-frank-large.sh.
Pitfalls: Avoid bare env vars TEMP, HOST, GPU_MAX_HEAP_SIZE (HIP runtime reads them). llama.cpp respects CLI flags.

## 01a051bf-4728-7028-82be-247e7b5da2bd
updated_at: 1788080182
SERVER: frank-default-large | HOST: Dankclaw (Strix Halo gfx1151, unified memory) | RUNTIME: charlie12345/ROCmFPX llama.cpp (/home/dank/GitPackages/ROCmFPX/build-qwen38/bin/llama-server, ROCm 7.2.4, HIP) | MODEL: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2.gguf (87.06 GiB, Q4_0_ROCMFP4_FAST, qwen4exp MoE: 512 experts/10 used, 48 layers: 12 attn + 36 SSM, QSA indexer, hyper-connections, PLE layer 1, n-gram PLE fully VRAM-resident) | ENV: HIP_VISIBLE_DEVICES=0, HSA_OVERRIDE_GFX_VERSION=11.5.1, LD_LIBRARY_PATH includes ROCm paths | SERVING PROFILE: --parallel 2, --ctx-size 401408, --batch-size 4096, --ubatch-size 4096, --threads 16, --threads-batch 16, --cache-type-k f16, --cache-type-v f16, --ctx-checkpoints 16, --cache-ram 8192, --gpu-layers 999, --split-mode none, --cache-prompt, --cache-idle-slots, --metrics | SAMPLING: --temp 0.7, --top-p 0.8, --top-k 20, --min-p 0.0, --presence-penalty 1.5 | REASONING: --reasoning on, --reasoning-format deepseek, --reasoning-budget -1 | VISION: --mmproj active (f16 mmproj) | SPECULATIVE DECODING: SPEC_TYPE=ngram-mod (MTP head does not load on build-qwen38 due to missing blk.0.hc_attn_norm.weight) | CONSTRAINTS/PITFALLS: --fit off (auto-fit segfaults on qwen4exp hyper-connection config); flash-attn on (known to abort graph build on QSA indexer path); ROCmFP4 quant hand-ported from ciru-ai/ROCmFPX (upstream llama.cpp cannot load this GGUF format). | CONFIG PATHS: /home/dank/Projects/serverAppConfigs/scripts/serve-frank-large.sh, /home/dank/Projects/serverAppConfigs/scripts/frank-default-large.env, /home/dank/Projects/serverAppConfigs/scripts/service-frank-large.service, chat-template.jinja.

## 01a051d8-db9b-76ab-b05f-ac95f79c9e55
updated_at: 1788079840
Tool Constraints: bash tool strictly validates cwd parameter; explicit cwd pointing to non-existent directory causes failure ('Working directory does not exist'). Actual cwd resolved to /home/dank/Projects/serverAppConfigs via read tool. read tool handles relative paths well and returns resultLimit signals. File State: Active file contains git merge conflict markers at lines 25-18 and 64-79.

## 01a05172-a606-7196-bde3-454344369e22
updated_at: 1788076443
Platform: Strix Halo unified memory (~116.7 GiB pool, no separate VRAM). Configuration: frank-default-large updated to n_ctx=200704, n_slots=2 in env/serve scripts. models.json contextWindow set to 200000 for budget calculations. Memory dynamics: Base 32kx1 used ~13 GiB context buffer. 200kx2 scales theoretical buffer ~12.3x, but Qwen3.8-Flash-Next's SSM architecture (12 attention layers out of 48) limits KV cache growth, keeping total load within ~114 GiB HIP heap without OOM. Files: ~/.pi/agent/models.json, /home/dank/serverAppConfigs/scripts/frank-default-large.env, serve-frank-large.sh.

## 01a050b6-e0fb-73a0-889c-e663fedd8c96
updated_at: 1788072313
- **Model Configuration**: Replaced frank-default-large main model with https://huggingface.co/agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF. Configured MTP draft as https://huggingface.co/agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF.
- **MTP Architecture Pitfall**: Previous MTP draft heads failed on STRIX_LEAN due to missing tensor 'blk.0.hc_attn_norm.weight' caused by main/draft mismatch. Using agentionai FAST main with agentionai FAST draft resolves this as a matching pair with compatible general.architecture.
- **Git Workflow Decision**: "Go straight to main" interpreted as skipping branch/worktree ceremony while keeping git as single source of truth. Direct commit and push to main.
- **Deployment Resolution**: Deployment blocked by git state showing deleted file (D) for dankclaw/Caddyfile and presence of .CADDYWIP-backup. Resolved by removing backup and running git checkout HEAD --.
- **Files Modified**: scripts/frank-default-large.env, scripts/serve-frank-large.sh, scripts/patches/qwen4exp-qsa-f32-mask.patch, dankclaw/LLM/agentionai-fast-imatrix.md.
- **Env Tunables**: Updated based on model card settings mixed with existing configuration.

## 01a04fa5-181d-75a9-ac57-3fdf464a36d3
updated_at: 1788060388
RUNTIME: ciru-ai/Qwen3.8-Flash-CIRU-STRIX-IU4 (llamacpp fork, tag v1.1). Binary: /home/dank/GitPackages/ROCmFPX/build-qwen38/bin/llama-server. Build flags: -DGGML_HIP=ON -DGGML_HIP_ROCMI4_W4A4=ON -DCMAKE_HIP_ARCHITECTURES=gfx1151.
MODEL: jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4 (HuggingFace). Replaces kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX. 177B params, Q4_0 quant, PLE table streamed from page cache.
ARCH: Strix Halo (gfx1151), unified memory, SVM-backed layers (--fit off, --gpu-layers 999).
SPEC DECODING: MTP draft head first. Legacy STRIX_LEAN drafts failed (missing blk.0.hc_attn_norm.weight); new runtime/model fixes this. Stack ngram-mod later.
CONTEXT: Start at 128k (CTX_SIZE=131072), promote to 524k+ post-stability.
SLOTS: parallel=1 default. Long prompts can block the single slot, causing timeouts. Monitor n_prompt_[REDACTED] vs n_prompt_tokens. May need parallel adjustment.
TEMPLATE: scripts/chat-template.jinja (DeepSeek reasoning format).
PATHS: frank-default-large.env, frank-default-large.sh. Health check on 192.168.0.143:7770.

## 01a04f40-c5c0-77a4-89c1-f729e282a9e8
updated_at: 1788042611
Plan: Swap frank-default-large model to agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF, use sidecar MTP (not grafted), and stack ngram-mod later. Ignore gitflow, deploy straight to main.

Model & MTP Details: Sidecar MTP confirmed (main GGUF has 0 MTP tensors, block_count=48; head is 2.44GB on Dankclaw). Using FAST main + FAST family head resolves Aug-28 hc_attn_norm mismatch. Draft path relies on qwen4exp-aware binary.

Fork Build: Targeting kingjones30/ROCmFPX fork (previously charlie12345/ROCmFPX) on Dankclaw (/home/dank/GitPackages/ROCmFPX/build-qwen38). Fork tip is 36e9acd40; current live binary built at a19d14e4e (indexer-cache fix).

Decision: Clone fork fresh, pin build to tip main. GATE build on: git merge-base --is-ancestor a19d14e4e <pinned-sha>. If false, stop to avoid silently dropping the indexer fix that prevents crash-loops.

Env Constraints: ROCmFPX llama-server (PR #27742). Do not name bare env vars TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP runtime reads bare vars). --fit off to avoid segfaults on qwen4exp hyper-connection/indexer config.

## 01a046ba-1301-72c1-8d9f-347ba1c28cf7
updated_at: 1787952056
## Homelab Auth Chain: Caddy → Anubis → Authentik Outpost → Apps

### Architecture (dankclaw/dankserver, franksdank.cloud)
- Caddy runs as container on dankclaw (192.168.0.143), fronts all services on both hosts.
- Chain per protected vhost: TLS (ACME dns-01 via Porkbun) → Anubis bot challenge (:8923) → internal :8016 routing → Caddy forward_auth → Authentik outpost (:9000) → app.
- Apps with own auth (Grafana, Gitea, OpenWebUI, Jellyfin, *arr, Pi-hole) are NOT wrapped in forward_auth.
- DNS: flat franksdank.cloud scheme via Pi-hole quadlet on dankclaw; all names resolve to 192.168.0.143.
- TLS: real LE certs via dns-01 (Porkbun module, franksdank/caddy-porkbun image). No inbound ports needed.

### Authentik Outpost Provider Config (verified live from DB)
- mode=proxy, external=https://franksdank.cloud, internal=http://authentik-server:9000
- Authorization flow: implicit-consent
- Outpost resolves provider by **exact Host header match** — this is why naive curl probes 404 (wrong/missing Host) while Caddy works (forwards real Host).

### Critical Pitfall: CLI Verification of Auth Chain
- Unknown-UA clients (curl, bots) get Anubis JS challenge FIRST → respond 200 with HTML challenge page.
- This means `curl -A 'goauthentik.io/outpost/...'` to a protected vhost returns 200 (challenge page), NOT outpost pass-through.
- Caddy's forward_auth subrequest is an internal hop (Caddy→authentik :9000) — NOT affected by Anubis.
- **Conclusion**: You cannot verify end-to-end auth from CLI. Must test in a real browser (solves JS challenge once, gets clearance cookie, then SSO carries across apps).

### Caddy Admin Toggle Pitfall
- `admin off` in Caddyfile disables the config file watcher → rendered Caddyfile changes do NOT hot-reload.
- Symptom: stale basicauth hash sitting on disk unapplied for hours (observed 2026-08-22).
- Fix: keep admin ON (binds to 127.0.0.1:2019 inside container, port not published, safe).

### Unresolved at Thread End: External Login 'Not Found'
- Symptom: externally (phone off wifi), after passing Anubis challenge, user gets Authentik's unstyled 'Not Found' page with dead 'Go home' link + 'Powered by authentik' subtitle.
- Diagnosis direction: outpost 404 = provider host-match failure for external request pattern. External clients may present a Host header that doesn't match the outpost's configured provider hostname (e.g., mobile carrier NAT, different SNI, or outpost endpoint path mismatch for non-LAN IPs).
- Next steps would be: check outpost logs for 404s, verify external DNS resolves to Caddy (not bypassed), confirm outpost endpoint URL matches what Caddy forwards externally.

### Grilling Skill
- Path: /home/dank/.agents/skills/grilling/SKILL.md
- Purpose: stress-test plans before building; one question at a time; look up facts in codebase, put decisions to user.

## 01a04717-a7fd-73ab-b7eb-319c4c1271c1
updated_at: 1787905384
Context: User asked whether HuggingFace EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF should be compatible with their personal llama.cpp runtime fork for frank-default-large's Qwen3.8-Flash-Next model.

Current frank-default-large setup at time of rollout:
- Served by /home/dank/Projects/serverAppConfigs/scripts/serve-frank-large.sh using env /home/dank/Projects/serverAppConfigs/scripts/frank-default-large.env.
- Runtime binary: /home/dank/GitPackages/ROCmFPX/build-qwen38/bin/llama-server from charlie12345/ROCmFPX llama.cpp fork with qwen4exp architecture support / PR #27742 merged, built with HIP/ROCmFP4 for Strix Halo gfx1151.
- Model repo/path: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX / local /home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX, file Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf.
- Model details in serve script: 177B params = 56B MoE + 51.2B PLE n-gram table, 512 experts / 10 used, 48 layers (12 attention + 36 SSM/Mamba), full_attention_interval=4, 2 KV heads x 256 dim, QSA indexer, hyper-connections count=4 low_rank=320, PLE at layer 1. Q4_0_ROCMFP4_STRIX quant, 113.47 GiB single file. PLE table streamed from page cache; GPU resident approx 63-69 GiB; mmap default into Strix Halo unified memory. --fit off because auto-fit segfaults on hyper-connection/indexer config; -ngl 999 puts layers on GPU/SVM.
- Sharp chat template enabled via scripts/chat-template.jinja with --jinja and CHAT_TEMPLATE_KWARGS preserving thinking/reasoning; reasoning format deepseek.

MTP investigation outcome:
- Tested EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF Q8_0 draft head and agentionai ROCmFP4-FAST MTP draft head against the STRIX_LEAN/Uncensored main model and qwen4exp binary.
- Both MTP draft GGUFs failed at load/runtime with missing tensor: blk.0.hc_attn_norm.weight.
- Interpretation: qwen4exp runtime expects hyper-connection attention norm tensors in draft models for this architecture/config, but these published draft GGUFs do not carry them. Therefore they are not compatible with the current frank-default-large runtime/model as-is.
- Durable next step: Need either (a) an MTP draft head GGUF that includes hc_attn_norm tensors matching this qwen4exp/hyper-connection model, or (b) a patched/binary/runtime path that does not require those hyper-connection tensors for draft models.

Operational state after rollout:
- Reverted/left frank-default-large using ngram-mod only, not MTP.
- Health check returned {"status":"ok"} after restart; service healthy on ngram-mod.
- Tests passed: ./tests/run.sh and focused bats tests for LLM units, secrets, script syntax; shellcheck skipped locally because not installed.
- Committed/pushed/pulled serverAppConfigs changes: commit 6d4a96a, message "frank-default-large: ngram-mod only — both MTP draft heads incompatible". Changed scripts/frank-default-large.env and scripts/serve-frank-large.sh. Dankclaw checkout fast-forwarded to commit 6d4a96a.

Pitfall to remember: Do not assume Qwen3.8-Flash-Next MTP heads are compatible merely by model name/quant/Strix Halo labeling. Check tensor inventory for hyper-connection tensors, especially blk.0.hc_attn_norm.weight, before scheduling downtime for swap-in.

## 01a04330-1551-72f6-b57b-8a8f43ffe7e2
updated_at: 1787862517
Context: frank-default-large serves kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF on Dankclaw (Strix Halo gfx1151 unified memory) using charlie12345/ROCmFPX llama.cpp build-qwen38 with qwen4exp support. Model is Q4_0_ROCMFP4_STRIX, ~113.47 GiB GGUF, 56B MoE plus 51.2B PLE n-gram table; PLE table is streamed from page cache and not resident in GPU VRAM. Runtime uses mmap into SVM-backed unified memory, -ngl 999/all layers on GPU, f16 KV, and --fit off because auto-fit segfaults on this model’s qwen4exp hyper-connection/indexer config.

Current/benchmark-relevant env file path: /home/dank/Projects/serverAppConfigs/scripts/frank-default-large.env. Serve script: /home/dank/Projects/serverAppConfigs/scripts/serve-frank-large.sh. Key documented tunables include CTX_SIZE=524288, PARALLEL=2, BATCH_SIZE=4096, UBATCH_SIZE=4096, NUM_THREADS=16, CACHE_TYPE_K=f16, CACHE_TYPE_V=f16, CTX_CHECKPOINTS=16, CACHE_RAM=8192, FIT=off. The prefill tuning rollout settled/documented BATCH_SIZE=4096 and UBATCH_SIZE=4096.

Decode tuning: a thread-batch sweep was run with an 8K prompt and 256 generated tokens. THREAD_BATCH=16 was confirmed as the best decode setting in that sweep. A prior todo requested 512 generated tokens, but that was intentionally not rerun: decode throughput in generated tok/s is independent of generation length for this comparison, so 512 tokens would only double wall-clock time while preserving the same tok/s signal. Use 256 gen tokens for future decode sweeps unless specifically measuring long-run thermal throttling or stability.

Benchmark methodology notes: separate prefill/TTFT benchmarking from decode tok/s benchmarking. For prefill, sweep batch and ubatch while holding other variables fixed and measure prompt processing throughput/TTFT. For decode, use a fixed prompt length (8K was used) and fixed generation length (256 tokens sufficient) while sweeping THREAD_BATCH; evaluate generated tok/s and TTFT. Keep prompts deterministic and repeat enough to avoid cache/page-cache warmup artifacts. Restore production env/service after benchmark and run the repo/service test suite.

Operational pitfalls/constraints: do not define bare env vars TEMP, HOST, or GPU_MAX_HEAP_SIZE in service env files; ROCm/HIP runtime reads those names and prior TEMP incident caused failures. llama.cpp host/port/temp should be passed as CLI flags, not bare env vars. Do not enable --fit for this model unless the qwen4exp auto-fit segfault has been fixed. Avoid interpreting longer generation-length decode runs as better signal unless testing thermal/stability; tok/s is the reusable metric.

## 01a042cc-e1c5-7482-87a2-14deadb82202
updated_at: 1787833610
## frank-default-large: Qwen3.8-Flash-Next tuning on Dankclaw (Strix Halo gfx1151)

### Model & Architecture
- kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF, 177B params (51.2B PLE n-gram + 125.8B MoE/attention)
- 56B MoE: 512 experts (10 used), 48 layers (12 attn + 36 SSM, full_attention_interval=4), 2 KV heads x 256 dim
- QSA indexer: 4 heads, top_k=2048; hyper-connections: count=4, low_rank=320
- PLE layer 1: ngram_size=3, heads_per_ngram=8
- Quant: Q4_0_ROCMFP4_STRIX (file_type=105), 113.47 GiB single GGUF
- PLE table: 50.66 GiB Q8_0, streamed from page cache (NEVER in GPU VRAM)
- GPU resident: ~63-69 GiB on Strix Halo unified memory pool (116 GiB SVM-backed)

### Runtime
- ROCmFPX llama.cpp fork at /home/dank/GitPackages/ROCmFPX/build-qwen38/bin/llama-server
- PR #27742 (qwen4exp architecture) merged
- Build: -DGGML_HIP=ON -DGGML_HIP_ROCMI4_W4A4=ON -DCMAKE_HIP_ARCHITECTURES=gfx1151 -DGGML_CUDA_FA_ALL_QUANTS=ON -DLLAMA_BUILD_WEBUI=OFF -DCMAKE_EXE_LINKER_FLAGS=-no-pie

### Critical Constraints (DO NOT CHANGE)
- FIT=off: auto-fit segfaults on this model's hyper-connection/indexer config
- -ngl 999 places all layers on SVM-backed GPU pool
- NEVER name env vars TEMP, HOST, or GPU_MAX_HEAP_SIZE — ROCm HIP runtime reads bare env vars (2026-08-13 TEMP incident)
- llama.cpp reads --host/--port/--temp as CLI flags only, never env vars
- Text-only: no mmproj, no MTP draft head

### Current Tunings (env file: frank-default-large.env)
- BATCH_SIZE=4096, UBATCH_SIZE=4096
- NUM_THREADS=16, THREAD_BATCH=32
- CTX_SIZE=524288, PARALLEL=2
- CACHE_TYPE_K=f16, CACHE_TYPE_V=f16
- CTX_CHECKPOINTS=16, CACHE_RAM=8192
- HSA_OVERRIDE_GFX_VERSION=11.5.1
- Port 7770 (LISTEN_PORT)

### Benchmark Status (interrupted)
- Goal: tune batch/ubatch for optimal token prefill
- Next planned step: threads-batch sweep
- Benchmark servers targeted ports 7772 and 7773
- Session interrupted during health-check polling loop; servers returned no output via curl on those ports (likely still loading 113 GiB model or not yet started)
- The repeated poll pattern (5+ identical checks, ~5s apart) suggests the assistant was waiting for model load completion before starting benchmarks

### Workflow Pattern
- SSH into dankclaw from controller; check health via `curl -s http://127.0.0.1:PORT/health`
- Model load on unified memory is extremely slow (113 GiB mmap); benchmark scripts must account for long startup
- Use frank-default-small (the serving model for this agent) to orchestrate benchmarks against frank-default-large

## 01a042a4-7e85-7728-802d-c8442896f6e6
updated_at: 1787827055
## frank-default-large model swap (2026-09-02)

**Host:** Dankclaw — AMD Radeon 8060S (Strix Halo), gfx1151 iGPU, 64 GiB unified memory. Runtime: charlie12345/ROCmFPX llama.cpp build-kairic v81. Port 7770.

**Swap:** STRIX_LEAN-GGUF (56B MoE / 35.76B PLE, 3 shards, 98.5 GiB) → Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF (177B total / 51.2B PLE, single file, 113.47 GiB on disk).

**File path:** `/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX/Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf`

**Key architectural fact:** PLE (Projection Layer Embedding) table streams from page cache — never enters GPU VRAM regardless of tier. This is why a 114 GiB file fits in 64 GiB unified memory: only ~63-69 GiB is GPU-resident (same as LEAN variant). The PLE table occupies host RAM/page-cache.

**Performance:** Load time ~55 seconds. Generation speed and prompt throughput within noise of STRIX_LEAN at every context depth. 200/200 tests pass.

**Config files modified:**
- `scripts/frank-default-large.env` — MODEL_DIR, MODEL_FILE, header comments
- `scripts/serve-frank-large.sh` — header comments, default values
- `docs/LLM/kingjones-strix-rocmfp4.md` — new 195-line runbook
- `docs/LLM/index.md` — active model reference updated
- `docs/index.md` — top-level index links new doc

**Git:** Branch `frank-large-strix-model` pushed to origin (Gitea on Dankclaw).

**Prior history on this slot (for context):** CIRU PromptForge v3 → Kairic Edge IU4 (reverted) → kingjones Qwen3.8-27B ROCmFP4 STRIX MTP (Q6 AGENT, standalone MTP draft, Turbo4 V cache, 262K ctx/1 slot) → Q8_0 AGENT experiment (failed, set_rows bug) → STRIX MTP restored → now Flash-Next full 177B.