command-center/agent/memories/--home-dank-Projects-serverAppConfigs--/raw_memories.md

Raw Memories

01a07351-8e05-75bc-9deb-bd9447e1256d

updated_at: 1788678884 ROOT CAUSE: Alias drift (pre-rename IDs persisted) + Config-home drift (HERMES_HOME pointed to /opt/data, an auto-created anonymous volume). Container recreation wipes /opt/data, reverting to image-default template config. DECISION: Pinned Environment=HERMES_HOME=/data in quadlet unit to unify config and state root under /data. In-place renamed live config IDs to frank-large/frank-small/dankclaw-npu. WORKFLOWS AND PITFALLS: Podman VOLUME paths get fresh anonymous volumes on recreation; edits lost unless pinned. provider: auto without explicit api: routes to OpenRouter; fails if API key missing. FLM/NPU routes by exact registry tag; HTTP 200 does not guarantee valid model (watch for error: Failed to load tag model!). Named providers in providers: dict resolve via api:/url:/base_url keys; custom: requires custom_providers[] entry. VERIFICATION: Use resolve_model_provider() and resolve_user_provider() for routing truth; probe endpoints with exact served IDs; /api/providers lists models with key_source: config_yaml. ARTIFACTS: /data/config.yaml, api/config.py, providers.py, runtime_provider.py, quadlet units, deploy.sh.

01a06ef4-c78b-71df-98e0-e1b930f260a5

updated_at: 1788579402 Category shift: Sonarr moved to 'admin' in shared/homepage/services.json. Exposure constraint: Sonarr kept local-only; removed 'sonarr.franksdank.cloud' vhost from dankclaw/Caddyfile to prevent external routing. Resolved pitfall: Prior directive conflated Seerr and Sonarr external reachability; corrected by auditing Caddyfile @sonarr handle block and services.json category mapping. Paths: shared/homepage/services.json, dankclaw/Caddyfile, frankbook-pro/sonarr/quadlet/sonarr.container. Workflow: Category update + Caddyfile vhost strip -> commit (b6f5b2b) -> push -> verify.

01a06e56-5e70-73f5-83b3-a28b1736f34b

updated_at: 1788561197

• Pi-hole admin password reset to pihole.2026.
• Public DNS for pihole.franksdank.cloud removed from Porkbun (NXDOMAIN externally); LAN resolution persists via addn-hosts (192.168.0.143).
• Caddy vhost retains basicauth edge and TLS termination, reverse proxying to Pi-hole quadlet published on 192.168.0.143:8083.
• Architectural note: Pi-hole serves as LAN DNS authority (dnsmasq/FTL) but sits behind Caddy for edge traffic handling, avoiding direct public exposure while maintaining internal name resolution.

01a06b4c-fcea-70e2-a342-a41c5f99d927

updated_at: 1788553425 Consolidated *arr stack to FrankBook-Pro (192.168.0.112), removing old Dankserver (235) instances. Six containers (QBit + 5 arrs) share a single gluetun network namespace; zero internal PublishPort on arrs, gluetun publishes full port set (8080, 6881, 9696, 8989, 7878, 8686, 8787). All egress forced through ProtonVPN (AllowedIPs=0.0.0.0/0); verified via podman exec IP echo (51.179.177.14 vs residential 71.227.227.172). Local DNS: Pi-hole quadlet on dankclaw serves addn-hosts, resolving *arr service names to Caddy (192.168.0.143). Caddy topology: TLS vhosts -> Anubis :8923 (AI scrapers deny/challenge) -> Caddy internal :8017 -> app containers. arr→qB client config updated to 127.0.0.1:8080 (resolved 10.88.0.1 gateway routing into tunnel). Tests: frankbook_quadlet.bats enforces shared netns/zero ports; unit & Playwright E2E suites pass. Parked defects: 1) Prowlarr Applications table empty (sync dead until rebuilt with loopback baseUrls); 2) readarr qB proxy upstream auth defect (post-move retest); 3) EZTV/1337x CloudFlare-block likely lifted for VPN egress. Old configs archived in ~/.AGENTTRASH/20260904-dankserver-arr-teardown/. Commits: 4c7465d, df4339a.

01a06aa2-9087-753a-8019-50af77fbaddf

updated_at: 1788501666 File: shared/homepage/services.json. Action: Reformatted from compact to multi-line JSON (2-space indent). Committed 7 description updates (SearXNG, OpenWebUI, Gitea, Copyparty, Plex, Jellyfin, Seerr). Workflow: git add/commit, ./scripts/deploy.sh dankclaw --service homepage. Result: Deployment 1967584 successful.

01a06894-8947-74fa-ba09-0f104cd6677f

updated_at: 1788495432 DEPLOYMENT & HOSTING:

• frank-npu service deployed to dankclaw via deploy.sh (commit df0c663, atop b563053).
• Unit: serve-frank-npu.service running on port 7771.
• FLM registry path: /home/dank/.config/flm/model_list.json.

MODEL BEHAVIOR & ROUTING:

• Resident model: nuextract3:4b. First probe confirms residency (2.6s latency, correct extraction output).
• Context Length: FLM_CTX_LEN=-1 (no global override). Packs load trained context (nuextract3: 32768, gemma4: 131072).
• Routing: FIFO single-lane NPU. Heretic requests (qwen3.8-distilled-heretic:9b) pay ~6-11s swap cost on tag alternation. Single-tag consumers pay once per switch direction.

NAMING CONVENTIONS & RENAME WORKFLOW:

• Renaming: frank-default-sightsound → frank-npu; frank-default-small → frank-small; frank-default-large → frank-large.
• Scope: Scripts, systemd units, env files, docs.
• Critical Order: Update models.json (~/.pi/agent/models.json) with new aliases BEFORE deploying renames to prevent agent connection loss during LLM runtime restarts.

PLAYWRIGHT TEST SPECIFICATIONS:

• Unauthenticated users must be blocked from direct navigation to Authentik-protected apps (except exempt services like Plex).
• Admin users must retain guaranteed access to admin space and admin apps.

HOST & REPO CONTEXT:

• Repo: /home/dank/Projects/serverAppConfigs (the workstation).
• Live hosts: dankclaw (NPU/LLM), dankserver.
• SSH accessible; deploy.sh handles push and dispatch.

01a063a7-4c37-7765-8f58-21f7c860aed7

updated_at: 1788403307 Context/workflow:

• The workstation is the operation host; repo path is /home/dank/Projects/serverAppConfigs.
• Main deployment hosts identified/reconfirmed: Dankserver at 192.168.0.235 and Dankclaw at 192.168.0.143. Both were reachable via passwordless SSH from the workstation.
• Services are primarily defined by per-host Quadlet .container files in the repo and installed as user-systemd units (systemctl --user ...) generating <name>.service. Some model/LLM services may be system units rather than user units.
• Useful status enumeration commands:
  • cd /home/dank/Projects/serverAppConfigs && find . -path ./.git -prune -o -name '*.container' -print | sort
  • For a host: ssh <host> 'systemctl --user list-units --type=service; podman ps -a --format "{{.Names}} {{.Status}} nets=[{{.Networks}}]"'

Hermes/Dankclaw diagnostic result:

• Command used: ssh -o ConnectTimeout=10 dankclaw 'echo "== units:"; for u in hermes hermes-webui hermes-dashboard signal-cli-daemon; do printf "%-20s %s\n" "$u" "$(systemctl --user is-active $u.service 2>/dev/null) $(systemctl --user is-enabled $u.service 2>/dev/null)"; done; echo "== containers:"; podman ps -a --format "{{.Names}} {{.Status}} nets=[{{.Networks}}]" | grep -Ei "hermes|signal|chrome"; echo "== listeners:"; ss -lntp 2>/dev/null | grep -wE "8080|8787|9119|9222" | tr -s " "; echo "== webui env (names only):"; podman exec hermes-webui sh -c "env | cut -d= -f1 | grep -Ei \"hermes|signal|base_url\" | sort" 2>&1 | head -20'
• Output showed:
  • hermes: active, generated
  • hermes-webui: active, generated
  • hermes-dashboard: active, generated
  • signal-cli-daemon: inactive, disabled
  • Containers:
    • chrome-devtools-protocol up ~2 days, no listed network
    • systemd-hermes up ~38h, no listed network
    • hermes-dashboard up ~38h, network dankclaw_default
    • hermes-webui up ~38h, healthy, no listed network
  • Listeners:
    • 0.0.0.0:8787 listening (Hermes WebUI)
    • 127.0.0.1:9222 listening by headless-shell (Chrome DevTools Protocol)
    • *:9119 listening via rootlessport
    • No listener on 8080 in this diagnostic, consistent with signal-cli-daemon being down.
  • hermes-webui env names include SIGNAL_ACCOUNT, SIGNAL_ALLOWED_USERS, HERMES_WEBUI_*, etc.
• Durable diagnosis: if Hermes WebUI says it cannot reach Hermes/Signal, first check signal-cli-daemon.service. In this captured state, Hermes-related services/containers were active/healthy, but signal-cli-daemon was inactive/disabled and there was no port 8080 listener, so the Signal HTTP transport was likely unavailable even though the WebUI itself responded on 8787.
• Suggested next operational checks/fix path:
  • ssh dankclaw 'systemctl --user status signal-cli-daemon.service --no-pager; journalctl --user -u signal-cli-daemon.service -n 80 --no-pager'
  • If appropriate, start/enable it: ssh dankclaw 'systemctl --user enable --now signal-cli-daemon.service'
  • Verify: ssh dankclaw 'systemctl --user is-active signal-cli-daemon.service; ss -lntp | grep -w 8080'

01a06085-8146-72b0-9e52-b486c767a2f6

updated_at: 1788331026 GOAL: Migrate Plex (native plexmediaserver.service) and Tautulli (snap) to podman quadlets on dankserver. Zero-bytes-moved design.

IMAGES:

• Plex: lscr.io/linuxserver/plex:1.43.3.10896-cb3ebc72d-ls322 (superseded initial ls9 pin)
• Tautulli: lscr.io/linuxserver/tautulli:v2.18.1-ls242 (verified ≡ latest digest)

USER & UID MODEL:

• UserNS=host for both. Rootless podman cannot map real uid 997 natively.
• Plex: PUID=0/PGID=0 (container root=dank), data root /var/lib/plexmediaserver chowned to 1000:1000. Rollback chowns back to 997:984.
• Tautulli: PUID=1000/PGID=1000.

VOLUMES:

• Plex: /var/lib/plexmediaserver:/config (RW, in-place), /mnt/entropy:/mnt/entropy:ro
• Tautulli: /home/dank/Containers/tautulli:/config

NETWORK & PORTS:

• App: dankserver_appnet. Plex alias=plex, Tautulli alias=tautulli.
• Plex: 32400:32400. Tautulli: 8181:8181.
• Tautulli config.seed: PMS_Web_Host=plex, PMS_Web_Port=32400, PMS_Secure=0.

CUTOVER & HEALTH:

• Native units stopped & masked (Plex), snap disabled (Tautulli). Instant rollback via unmask/snap enable + chown.
• Plex health gate widened to 600s (32GB DB migrations reset on bounce). No HealthCmd to avoid cold-load flapping.
• Tautulli uses python3 urllib health probe.

PITFALLS & FIXES:

• UsernsMode= (compose syntax) hard-failed v1 generator → corrected to UserNS= or omitted.
• Edit churn ate ContainerName, Network, PublishPort keys → caught by seam test + C5 probe.
• PMS deploy-state record showed pending (cosmetic) → self-heals on next deploy.
• Decommission: apt purge plexmediaserver + snap remove tautulli only on explicit sign-off.

01a06035-a0d8-73ad-8a49-279f51c68bf2

updated_at: 1788329168 PLEX_HOMEPAGE: Updated url to http://plex.franksdank.cloud/web/index.html in shared/homepage/dankserver.services.json & services.json. JELLYFIN_MNT: /mnt/entropy:/data:ro (read-only whole volume), health 200, libraries verified working. COPIPARTY_MNT: /mnt/entropy:/w:ro, health 200. JELLYFIN_DB: /home/dank/Containers/jellyfin/config/data/jellyfin.db; table is BaseItems (not TypedBaseItems). DEPLOY: Commit 8895a02, 233/233 tests green. USER_DIRECTIVE: Stop touching Jellyfin/Copyparty after verification.

01a05bed-d759-7157-af15-2e178337182a

updated_at: 1788325757 Decisions (Locked):

• D1: Quadlet .container units for all long-running services; compose retired to reference.
• D2: Data root /home/dank/Containers/ on Linux hosts; entropy is mounted volume.
• D3: *arr-stack migrates to FrankBook-Pro; decommissioned on dankserver.
• D4: Keep /Volumes/COMPLETE & /Volumes/COMPLETE/"Finished Torrents" verbatim; rename deferred.
• D5: VPN egress via gluetun container; qbittorrent shares netns (Network=gluetun.container).
• D6: Private LAN-only, direct IP:port, Pi-hole addn-hosts DNS; no Caddy vhost on Mac.
• D7: Scope includes Mac bring-up + cutover.
• D8: Seerr on dankclaw (SQLite, Authentik OIDC SSO), provider created before route flip.

Execution Phases:

• Phase 0: Prereqs (Mac SSH access, WireGuard creds/machine-local env, egress gate check via podman machine ssh).
• Phase 1 (dankserver): Per service (romm, mariadb, redis, jellyfin, calibre-web, copyparty): stop -> mv /opt -> /home/dank/Containers -> chown (uid 1000) -> deploy quadlet. RomM landmine: move DB from anonymous volume to /home/dank/Containers/romm/db.
• Phase 2 (dankclaw): Migrate data roots for gitea, authentik, etc.
• Phase 3/4: Mac stack bring-up & *arr cutover.

Technical Constraints & Patterns:

• Quadlet template: [Container] (Image, ContainerName, EnvironmentFile, Volume=bind-mounts-only), [Service] Restart=always, [Install] WantedBy=default.target.
• Entropy write: Only copyparty -v /mnt/entropy:/w with -a dank: -v /w::r:ra,dank.
• Deployer: ./scripts/deploy.sh computes changes, pushes, target pulls, dispatches. Handles run_privileged and service health checks.
• Secret hygiene: No secrets in tracked repo files (machine-local envs).
• Gotcha: Verify --userns=host in systemctl --user cat .service after daemon-reload. Data root mkdir -p handled in deployer.

01a05bd4-8dbe-759b-8885-8b86990051ed

updated_at: 1788314412 frank-default-large model/runtime configuration:

• Service alias: frank-default-large, served by scripts/serve-frank-large.sh with tunables in scripts/frank-default-large.env.
• Model: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF, file Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf under /home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX.
• Architecture: Qwen3.8-Flash-Next base / qwen4exp MoE, 512 experts / 10 used, 48 layers = 12 attention + 36 SSM, QSA indexer, hyper-connections, PLE at layer 1.
• Quant/file details: Q4_0_ROCMFP4_STRIX, file_type 105, ~113.47 GiB single GGUF. STRIX top quality tier; PLE table ~50.66 GiB Q8_0, token embeddings + lm head Q6_K.
• Runtime: kingjones30/ROCmFPX fork with qwen4exp-on-rocmfpx patch, built at /home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server using HIP for gfx1151, ROCm 7.2.4. Stock llama.cpp and charlie12345/ROCmFPX alone cannot load this GGUF because ROCmFP4 tensor types and qwen4exp architecture support must be in the same tree.
• Hardware target: Dankclaw / Strix Halo gfx1151 unified memory. Env sets HSA_OVERRIDE_GFX_VERSION=11.5.1, HIP_VISIBLE_DEVICES=0, --device ROCm0.
• Important pitfall: PLE n-gram table must stay out of GPU memory and stream via mmap from OS page cache. Do not add --no-mmap or tensor override/offload (-ot) for the PLE table.
• Important pitfall: --fit must remain off; auto-fit segfaults on qwen4exp hyper-connection config.
• Important pitfall: if graph build aborts on QSA F16 mask with flash attention, port scripts/patches/qwen4exp-qsa-f32-mask.patch into the kingjones ROCmFPX tree and rebuild.
• Speculative decoding currently disabled (SPEC_TYPE=none). Flash-Next MTP heads do not load on this loader and the model card ships no spec flags; draft model defaults exist but should not be enabled casually.
• mmproj: STRIX repo ships no mmproj; serve script conditionally passes /home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/mmproj/mmproj-Qwen3.8-Flash-Next-f16.gguf if readable.
• Serving defaults: host 0.0.0.0 port 7770, ctx 401408, parallel 2 slots, batch 4096, ubatch 512, threads 16, threads-batch 16, f16 KV cache, flash-attn on, split-mode none, gpu-layers 999, ctx-checkpoints 16, cache-ram 8192 MiB, prompt cache and idle-slot cache enabled, metrics enabled.
• Sampling defaults: temp 0.7, top_p 0.8, top_k 20, min_p 0.0, presence_penalty 1.5, reasoning on, reasoning-format deepseek, reasoning-budget -1. Env comments describe this as card sampling table mixed with validated serving profile, non-thinking instruct despite reasoning plumbing being enabled.
• Replaced prior agentionai ROCmFP4-FAST imatrix deployment/charlie build-qwen38 runtime around 2026-08-30.

ASUS TUF GAMING B650-E WIFI BIOS update/backup durable workflow:

• Target latest memory-compat BIOS identified: 3886 beta (2026-07), AGESA 1.2.0.3a Patch A. Current referenced BIOS was 2613.
• Board-name caution: use exact ASUS TUF GAMING B650-E WIFI support page; avoid wrong CAPs for similar boards like B650M-E or B650-PLUS.
• Backup current BIOS before flashing. Safe Linux read-only approach: sudo pacman -S flashrom sudo flashrom --programmer internal -r bios-2613-backup-1.bin sudo flashrom --programmer internal -r bios-2613-backup-2.bin sha256sum bios-2613-backup-*.bin; hashes must match or re-read.
• Expected AM5 SPI dump size is about 32 MB. Copy backups to USB and another machine. Never use flashrom -w from Linux for writing; power loss or mismatch can brick board. If flashrom reports multiple chips or refuses, use chip-specific selection only if certain or fall back to UEFI/EZ Flash backup.
• Alternative backup: ASUS EZ Flash 3 often offers save/current-BIOS-to-USB from UEFI. Also download archived 2613 CAP from ASUS as a restore/downgrade image.
• Normal flash path: FAT32 USB with CAP in root, reboot, Del into UEFI, Tool → ASUS EZ Flash 3, select CAP, allow verification, save old BIOS if offered, flash without resets/power cuts.
• Rescue path: USB BIOS FlashBack only if board will not POST. System off but PSU standby on, USB in rear FlashBack-marked port, rename CAP exactly per support page/manual (do not guess), hold FlashBack button ~3s, wait until LED stops blinking.
• Operational note: Dankserver hosts media stack + frank-default-small; BIOS flash is ~15–20 min downtime. First boot after AGESA update can spend 1–5 min on black screen due to DDR5 retraining; wait.
• Post-flash: settings reset to defaults. Re-apply memory profile/manual settings and re-run memtest; new AGESA/training invalidates prior memory stability conclusions. Verify version in BIOS main screen or with sudo dmidecode -s bios-version.

01a05a7d-cc94-7589-8926-c24cb5a10490

updated_at: 1788248708 Hermes Agent Persistence: Data survives restarts via host bind mount /home/dank/hermes-data:/data. Requires entrypoint override to hermes gateway run --force -v to prevent default TUI crash-loop on exit. Environment HERMES_DATA_DIR and HERMES_HOME must point to /data. Signal HTTP adapter connects to 127.0.0.1:8080 via network_mode: host. Dashboard runs separately on port 9119. General Container Patterns: ComfyUI uses keep-groups for rootless podman render group resolution; models/data persist under /home/dank/comfyui-data. SearXNG, Gluetun, and Valkey are boot-persistent Quadlet units sharing network namespaces. qBittorrent reuses SearXNG's netns without its own port. Migration & Deployment Constraints: *arr stack (Sonarr, Radarr, Lidarr, Readarr, Prowlarr) migrating to FrankBook-Pro Mac. Config dirs archived to /home/dank/archive/-arr-configs.tar.gz before removal; virgin installs skip archive. Podman machine init on Mac requires -v /Volumes/COMPLETE -v /Volumes/INCOMPLETE; later addition requires re-init. VPN egress gate must pass before *arr port publishing. Caddy uses Quadlet units. Verify with systemctl --user is-enabled, podman machine ssh, and health probes. Deploy scripts: deploy.sh, deploy-remote.sh, deploy-plan.py manage service order and target dispatch.

01a05a29-d9fd-71ca-86c8-b92c602492c7

updated_at: 1788229381 VOLUME SECURITY: POSIX permissions for read/write user via SFTP/SSH. Create dedicated writer user and readers group. Set volume ownership to writer:readers with chmod 2775 (setgid ensures new files inherit group). Deployment: Host volume or Podman container. Restrict SSH/SFTP access via Match block in sshd_config or ForceCommand internal-sftp.

FREETOKEN BUILD (DANKCLAW): Build-only, no deploy/run. Supports Vulkan or ROCm. Pre-installed: ROCm 6.3 (/opt/rocm), NVIDIA driver 575.57.02, CMake, Python, git. Requirements met for compilation.

GGUF LOADING CONSTRAINTS: Restricted to Gemma4GGUFForCausalLM model class. Supported GGML quantization types: F32(0), F16(1), BF16(30) use native torch views; Q4_0(2) and Q6_K(14) use dequant_q4_0/dequant_q6_k. Q8_0(8) is mapped but lacks dequant function (raises NotImplementedError). Loading pipeline: is_gguf_path -> iter_gguf_tensors -> dequantize -> load into graph.

01a04a3f-6596-75d0-8749-8ece9a2b11fd

updated_at: 1788224586 REPO_ROOT: /home/dank/Projects/serverAppConfigs HOSTS: Dankclaw (192.168.0.143, Caddy host), Dankserver (192.168.0.235) CADDY_SETUP: Single container on dankclaw fronts all services. DNS via dankclaw Pi-hole quadlet; LAN routed via SURFboard G36 router. TLS via Porkbun DNS-01 (no inbound 80/443 required). Admin interface ON (127.0.0.1:2019) enabling config file watcher hot-reload; turning off caused stale basicauth hash. Basicauth hash rendered from host .env into container Caddyfile. AUTH: Authentik SSO via Caddy forward_auth. Protects homepage, SearXNG, Hermes, Prometheus, RomM, Copyparty. Apps with native auth (Grafana, Gitea, OpenWebUI, Jellyfin, *arr, Pi-hole) bypass forward_auth. HERMES_BOT: WebUI network flipped to Network=host. Browser automation via chrome-devtools-protocol quadlet; CDP bound to loopback 127.0.0.1:9222. Deployer patched to start service only if inactive (preserves in-flight browser sessions). Trio + daemon managed via systemctl --user. CONFIG_QUIRKS: /data/config.yaml mounted but owned by unmapped UID (dank user cannot read directly; requires sudo python for edits). NPM npx cache lives in container layer: first tool call in fresh container pays npm download cost; container restarts do not reset cache, but recreation does. DEPLOY_WORKFLOW: deploy.sh handles quadlet install, daemon start-if-not-running, config wiring, trio restart. Uses stash-deploy-pop dance. State tracked via STATE[hermes] commit hash. FF-only pull on target.

01a059f4-cead-704b-b857-fad89ebde42b

updated_at: 1788224497 Context: Dankserver hosts copyparty for files.franksdank.cloud and LAN direct access. Traffic chain is Browser -> Caddy edge -> Anubis :8923 -> Authentik outpost -> post-auth :8017 as UNAUTHENTICATED -> copyparty :3923. LAN direct also reaches copyparty :3923.

Resolved change: in dankserver/compose.yaml, copyparty volume argument was changed from -v /w::A to -v /w::r:ra,dank. This makes /w browsable/downloadable but not writable/deletable. The copyparty version observed was v1.20.21. There is no separate dedicated reject-uploads flag; read-only behavior is achieved by omitting w, m, and d permissions. Anonymous must keep r because the guarded Authentik chain passes authenticated users to copyparty as unauthenticated.

Deployment workflow: deploy.sh correctly refuses to directly modify the compose stack on Dankserver but can pull/record rev. To apply the compose change, run on Dankserver: podman-compose up -d --force-recreate copyparty.

Verification results after deploy: anonymous GET /Movies/ returned 200; anonymous PUT returned 401; authenticated dank PUT/DELETE returned 403; recent-uploads DB showed "ups": []; files.franksdank.cloud returned 200 and the chain remained intact.

Critical pitfall discovered: prior runbook claim that anonymous got nothing was wrong. In copyparty, bare A means everyone, so -v /w::A had made the entire ~91T array anonymously readable, writable, and deletable on LAN/WebDAV. During probing, one 1-byte test file /mnt/entropy/w/.readability-probe-delete-me was created; it was verified by exact path and size and removed, then root listing was rechecked to match original state.

Admin credentials at time of rollout: username dank, password m9FuwPKWotsPz7jUUn. Stored as COPAY_PASS= in dankserver/.env on Dankserver (gitignored). Compose interpolates this into container argv, so anyone able to podman inspect copyparty on the host can see it. Rotate by editing .env and recreating the service. After read-only change, dank has read plus admin panel but no writes by design.

Outstanding operational gap: copyparty/dankserver stack was still running as a bare compose container with no Quadlet unit; boot-persistence rule is not satisfied for the whole dankserver stack. This was flagged but not changed.

01a05a42-6580-7043-96eb-a73344efd9e6

updated_at: 1788221481 Hermes FLM/Gemma rollout on Dankclaw:

• User request: add Gemma FLM model to Hermes agent.
• Context: Hermes live data is bind-mounted at /home/dank/hermes-data and appears in-container as /data. Repo contains sanitized backup at dankclaw/hermes/config.yaml.example; do not commit live-only secrets such as dashboard password hashes, phone numbers, OAuth credentials, API keys, sessions, memories, or generated state.
• Existing Hermes config shape:
  • model.provider/default selects main model.
  • providers entries use keys like dankclaw-large and dankserver-small with api URL, discover_models: false, and explicit models/context_length.
  • auxiliary entries use compact custom endpoint shape: {provider: custom, model: ..., base_url: ...}.
• Added FLM/NPU provider for Dankclaw to Hermes live config and sanitized repo example:
  • provider name: dankclaw-npu
  • FLM OpenAI-compatible base/API URL: http://192.168.0.143:7771/v1
  • model: gemma4-it:e4b
  • discover_models: false
  • use api_key: not-needed where required by config conventions.
• Routed Hermes auxiliary vision to the Gemma FLM model:
  • auxiliary.vision: {provider: custom, model: gemma4-it:e4b, base_url: http://192.168.0.143:7771/v1}
• Main Hermes model remained frank-default-large via dankclaw-large at http://192.168.0.143:7770/v1.
• Adjusted ~/.omp/agent/models.yml so frank-default-large context length is 200000 (reduced from oversized setting to suit actual Hermes/agent behavior).
• Cleaned stray ~/.pi by moving it to ~/.AGENTTRASH/20260831-pi.
• Restarted Hermes container after live config edit; Hermes became healthy.
• Validation: full repo test suite passed, 204 tests total.
• Durable caution: config.yaml.example is only a sanitized baseline backup of /home/dank/hermes-data/config.yaml, not a full replacement for live state. Preserve live secrets and generated state when editing/syncing.

01a059ae-72f3-76d4-8e09-735f8f2e0a5d

updated_at: 1788216058 Hosts and identities:

• FrankBook-Pro is reachable as dank@FrankBook-Pro.local / dank@192.168.0.112. It is macOS on x86_64, not iOS.
• dankclaw is 192.168.0.143.
• dankserver is 192.168.0.235.
• This rollout was run from the workstation/dankclaw context with an existing ed25519 SSH key.

SSH configuration completed and verified:

• Passwordless host-to-host SSH works:
  • dankclaw -> FrankBook-Pro returns user dank.
  • dankclaw -> dankserver returns user dank.
  • dankserver -> dankclaw returns user dank.
  • FrankBook-Pro -> dankclaw returns user dank.
• FrankBook-Pro host keys were already present in known_hosts during setup.

Hermes container SSH completed and verified:

• Containers checked: systemd-hermes, hermes-webui, hermes-dashboard.
• Each container can passwordlessly SSH using /root/.ssh/id_ed25519 with IdentitiesOnly=yes to:
  • root@192.168.0.143 (dankclaw) -> returns root.
  • dank@192.168.0.235 (dankserver) -> returns dank.
  • dank@192.168.0.112 (FrankBook-Pro) -> returns dank.

FrankBook-Pro Podman stack installed and verified:

• podman installed at /usr/local/bin/podman, version 5.4.2, Mach-O x86_64.
• gvproxy installed/built at /usr/local/bin/gvproxy, Mach-O x86_64.
• vfkit installed via Homebrew at /usr/local/bin/vfkit, version v0.6.4, Mach-O x86_64.
• Podman machine uses applehv provider and is running:
  • podman-machine-default* applehv, 4 CPUs, 2GiB RAM, 100GiB disk.
• podman run was verified with an Alpine test and confirmed working.

Operational notes:

• For future verification from dankclaw, use SSH checks to 192.168.0.112/235 and podman exec into Hermes containers to test /root/.ssh/id_ed25519 against the three targets.
• Do not assume FrankBook-Pro is iOS; treat it as macOS with Homebrew and AppleHV/vfkit support.

01a0598c-244a-766a-8485-b02f8ab630b5

updated_at: 1788209135 DNS architecture / split-horizon behavior:

• Domain: franksdank.cloud.
• Public DNS provider: Porkbun.
• LAN DNS: Pi-hole on the homelab, fed by addn-hosts / local records.
• Public/WAN Porkbun A records point subdomains to WAN IP 71.227.227.172.
• LAN Pi-hole resolves the same subdomains to Dankclaw LAN IP 192.168.0.143.
• G36 router/DHCP gives LAN clients Pi-hole as DNS, so LAN clients hit Pi-hole local records first. WAN clients resolve via Porkbun/public DNS.

Operational consequence:

• It is safe to remove a regular subdomain A record from Porkbun while keeping the Pi-hole record if the name should be LAN-only.
• LAN clients will continue resolving the name to 192.168.0.143 via Pi-hole.
• WAN clients will get NXDOMAIN/unknown subdomain once the Porkbun record is removed, unless cached elsewhere.
• Removing Porkbun DNS only affects name visibility/resolution; it does not itself close router port forwards or prevent direct access by IP if forwarding remains open.

Caddy/TLS implications:

• Single Caddy on Dankclaw fronts services and routes by Host header; Caddy does not care whether the hostname was resolved by Porkbun or Pi-hole.
• HTTPS certificates are wildcard DNS-01 certs for franksdank.cloud issued/renewed using a custom franksdank/caddy-porkbun image with caddy-dns/porkbun.
• DNS-01 renewal uses Porkbun API credentials to create TXT records under _acme-challenge; deleting ordinary A records does not break existing wildcard cert validity or renewal.
• Porkbun credentials live in dankclaw/.env as PORKBUN_API_KEY and PORKBUN_API_SECRET_KEY and are deployed to /home/dank/Containers/caddy/env for the Caddy quadlet; never commit secrets.

Cautions:

• Do not casually remove public DNS for auth.franksdank.cloud: prior sign-in/debugging showed auth needed a public A record for external login flows.
• Be cautious with gitea.franksdank.cloud because git smart-HTTP/external use may depend on public DNS.
• Keep Porkbun records for any service that must be reachable by external users through the public hostname; remove only for LAN-only/internal services.

Known public/LAN service hostnames previously tracked under franksdank.cloud include: auth, books, romm, files, jellyfin, radarr, sonarr, lidarr, readarr, prowlarr, qbittorrent, search, comfyui, chat, hermes, grafana, prometheus, gitea, pihole, and plex (plex.franksdank.cloud is canonical HTTPS).

01a051e2-6ff9-72cc-b980-b6837011e99f

updated_at: 1788126110 Service: frank-default-large on Dankclaw (gfx1151 unified memory). Model: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF -> file Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf (~113.5 GiB). Runtime: charlie12345/ROCmFPX llama.cpp (/home/dank/GitPackages/ROCmFPX/build-qwen38). ROCm 7.2.4. Quant/Format: ROCmFP4 (hand-ported from ciru-ai; upstream llama.cpp incompatible). Attention: Flash-ATTN ON via QSA indexer. Requires qwen4exp-qsa-f32-mask.patch to fix ggml_fill assert (applied in binary). MTP/Speculative: Draft head does not load (requires blk.0.hc_attn_norm.weight). Uses ngram-mod speculative decoding. Context: CTX_SIZE ~400k, Parallel 2. Config: scripts/frank-default-large.env and scripts/serve-frank-large.sh. Pitfalls: Avoid bare env vars TEMP, HOST, GPU_MAX_HEAP_SIZE (HIP runtime reads them). llama.cpp respects CLI flags.

01a051bf-4728-7028-82be-247e7b5da2bd

updated_at: 1788080182 SERVER: frank-default-large | HOST: Dankclaw (Strix Halo gfx1151, unified memory) | RUNTIME: charlie12345/ROCmFPX llama.cpp (/home/dank/GitPackages/ROCmFPX/build-qwen38/bin/llama-server, ROCm 7.2.4, HIP) | MODEL: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2.gguf (87.06 GiB, Q4_0_ROCMFP4_FAST, qwen4exp MoE: 512 experts/10 used, 48 layers: 12 attn + 36 SSM, QSA indexer, hyper-connections, PLE layer 1, n-gram PLE fully VRAM-resident) | ENV: HIP_VISIBLE_DEVICES=0, HSA_OVERRIDE_GFX_VERSION=11.5.1, LD_LIBRARY_PATH includes ROCm paths | SERVING PROFILE: --parallel 2, --ctx-size 401408, --batch-size 4096, --ubatch-size 4096, --threads 16, --threads-batch 16, --cache-type-k f16, --cache-type-v f16, --ctx-checkpoints 16, --cache-ram 8192, --gpu-layers 999, --split-mode none, --cache-prompt, --cache-idle-slots, --metrics | SAMPLING: --temp 0.7, --top-p 0.8, --top-k 20, --min-p 0.0, --presence-penalty 1.5 | REASONING: --reasoning on, --reasoning-format deepseek, --reasoning-budget -1 | VISION: --mmproj active (f16 mmproj) | SPECULATIVE DECODING: SPEC_TYPE=ngram-mod (MTP head does not load on build-qwen38 due to missing blk.0.hc_attn_norm.weight) | CONSTRAINTS/PITFALLS: --fit off (auto-fit segfaults on qwen4exp hyper-connection config); flash-attn on (known to abort graph build on QSA indexer path); ROCmFP4 quant hand-ported from ciru-ai/ROCmFPX (upstream llama.cpp cannot load this GGUF format). | CONFIG PATHS: /home/dank/Projects/serverAppConfigs/scripts/serve-frank-large.sh, /home/dank/Projects/serverAppConfigs/scripts/frank-default-large.env, /home/dank/Projects/serverAppConfigs/scripts/service-frank-large.service, chat-template.jinja.

01a051d8-db9b-76ab-b05f-ac95f79c9e55

updated_at: 1788079840 Tool Constraints: bash tool strictly validates cwd parameter; explicit cwd pointing to non-existent directory causes failure ('Working directory does not exist'). Actual cwd resolved to /home/dank/Projects/serverAppConfigs via read tool. read tool handles relative paths well and returns resultLimit signals. File State: Active file contains git merge conflict markers at lines 25-18 and 64-79.

01a05172-a606-7196-bde3-454344369e22

updated_at: 1788076443 Platform: Strix Halo unified memory (~116.7 GiB pool, no separate VRAM). Configuration: frank-default-large updated to n_ctx=200704, n_slots=2 in env/serve scripts. models.json contextWindow set to 200000 for budget calculations. Memory dynamics: Base 32kx1 used ~13 GiB context buffer. 200kx2 scales theoretical buffer ~12.3x, but Qwen3.8-Flash-Next's SSM architecture (12 attention layers out of 48) limits KV cache growth, keeping total load within ~114 GiB HIP heap without OOM. Files: ~/.pi/agent/models.json, /home/dank/serverAppConfigs/scripts/frank-default-large.env, serve-frank-large.sh.

01a050b6-e0fb-73a0-889c-e663fedd8c96

updated_at: 1788072313

• Model Configuration: Replaced frank-default-large main model with https://huggingface.co/agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF. Configured MTP draft as https://huggingface.co/agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF.
• MTP Architecture Pitfall: Previous MTP draft heads failed on STRIX_LEAN due to missing tensor 'blk.0.hc_attn_norm.weight' caused by main/draft mismatch. Using agentionai FAST main with agentionai FAST draft resolves this as a matching pair with compatible general.architecture.
• Git Workflow Decision: "Go straight to main" interpreted as skipping branch/worktree ceremony while keeping git as single source of truth. Direct commit and push to main.
• Deployment Resolution: Deployment blocked by git state showing deleted file (D) for dankclaw/Caddyfile and presence of .CADDYWIP-backup. Resolved by removing backup and running git checkout HEAD --.
• Files Modified: scripts/frank-default-large.env, scripts/serve-frank-large.sh, scripts/patches/qwen4exp-qsa-f32-mask.patch, dankclaw/LLM/agentionai-fast-imatrix.md.
• Env Tunables: Updated based on model card settings mixed with existing configuration.

01a04fa5-181d-75a9-ac57-3fdf464a36d3

updated_at: 1788060388 RUNTIME: ciru-ai/Qwen3.8-Flash-CIRU-STRIX-IU4 (llamacpp fork, tag v1.1). Binary: /home/dank/GitPackages/ROCmFPX/build-qwen38/bin/llama-server. Build flags: -DGGML_HIP=ON -DGGML_HIP_ROCMI4_W4A4=ON -DCMAKE_HIP_ARCHITECTURES=gfx1151. MODEL: jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4 (HuggingFace). Replaces kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX. 177B params, Q4_0 quant, PLE table streamed from page cache. ARCH: Strix Halo (gfx1151), unified memory, SVM-backed layers (--fit off, --gpu-layers 999). SPEC DECODING: MTP draft head first. Legacy STRIX_LEAN drafts failed (missing blk.0.hc_attn_norm.weight); new runtime/model fixes this. Stack ngram-mod later. CONTEXT: Start at 128k (CTX_SIZE=131072), promote to 524k+ post-stability. SLOTS: parallel=1 default. Long prompts can block the single slot, causing timeouts. Monitor n_prompt_[REDACTED] vs n_prompt_tokens. May need parallel adjustment. TEMPLATE: scripts/chat-template.jinja (DeepSeek reasoning format). PATHS: frank-default-large.env, frank-default-large.sh. Health check on 192.168.0.143:7770.

01a04f40-c5c0-77a4-89c1-f729e282a9e8

updated_at: 1788042611 Plan: Swap frank-default-large model to agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF, use sidecar MTP (not grafted), and stack ngram-mod later. Ignore gitflow, deploy straight to main.

Model & MTP Details: Sidecar MTP confirmed (main GGUF has 0 MTP tensors, block_count=48; head is 2.44GB on Dankclaw). Using FAST main + FAST family head resolves Aug-28 hc_attn_norm mismatch. Draft path relies on qwen4exp-aware binary.

Fork Build: Targeting kingjones30/ROCmFPX fork (previously charlie12345/ROCmFPX) on Dankclaw (/home/dank/GitPackages/ROCmFPX/build-qwen38). Fork tip is 36e9acd40; current live binary built at a19d14e4e (indexer-cache fix).

Decision: Clone fork fresh, pin build to tip main. GATE build on: git merge-base --is-ancestor a19d14e4e . If false, stop to avoid silently dropping the indexer fix that prevents crash-loops.

Env Constraints: ROCmFPX llama-server (PR #27742). Do not name bare env vars TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP runtime reads bare vars). --fit off to avoid segfaults on qwen4exp hyper-connection/indexer config.

01a046ba-1301-72c1-8d9f-347ba1c28cf7

updated_at: 1787952056

Homelab Auth Chain: Caddy → Anubis → Authentik Outpost → Apps

Architecture (dankclaw/dankserver, franksdank.cloud)

• Caddy runs as container on dankclaw (192.168.0.143), fronts all services on both hosts.
• Chain per protected vhost: TLS (ACME dns-01 via Porkbun) → Anubis bot challenge (:8923) → internal :8016 routing → Caddy forward_auth → Authentik outpost (:9000) → app.
• Apps with own auth (Grafana, Gitea, OpenWebUI, Jellyfin, *arr, Pi-hole) are NOT wrapped in forward_auth.
• DNS: flat franksdank.cloud scheme via Pi-hole quadlet on dankclaw; all names resolve to 192.168.0.143.
• TLS: real LE certs via dns-01 (Porkbun module, franksdank/caddy-porkbun image). No inbound ports needed.

Authentik Outpost Provider Config (verified live from DB)

• mode=proxy, external=https://franksdank.cloud, internal=http://authentik-server:9000
• Authorization flow: implicit-consent
• Outpost resolves provider by exact Host header match — this is why naive curl probes 404 (wrong/missing Host) while Caddy works (forwards real Host).

Critical Pitfall: CLI Verification of Auth Chain

• Unknown-UA clients (curl, bots) get Anubis JS challenge FIRST → respond 200 with HTML challenge page.
• This means curl -A 'goauthentik.io/outpost/...' to a protected vhost returns 200 (challenge page), NOT outpost pass-through.
• Caddy's forward_auth subrequest is an internal hop (Caddy→authentik :9000) — NOT affected by Anubis.
• Conclusion: You cannot verify end-to-end auth from CLI. Must test in a real browser (solves JS challenge once, gets clearance cookie, then SSO carries across apps).

Caddy Admin Toggle Pitfall

• admin off in Caddyfile disables the config file watcher → rendered Caddyfile changes do NOT hot-reload.
• Symptom: stale basicauth hash sitting on disk unapplied for hours (observed 2026-08-22).
• Fix: keep admin ON (binds to 127.0.0.1:2019 inside container, port not published, safe).

Unresolved at Thread End: External Login 'Not Found'

• Symptom: externally (phone off wifi), after passing Anubis challenge, user gets Authentik's unstyled 'Not Found' page with dead 'Go home' link + 'Powered by authentik' subtitle.
• Diagnosis direction: outpost 404 = provider host-match failure for external request pattern. External clients may present a Host header that doesn't match the outpost's configured provider hostname (e.g., mobile carrier NAT, different SNI, or outpost endpoint path mismatch for non-LAN IPs).
• Next steps would be: check outpost logs for 404s, verify external DNS resolves to Caddy (not bypassed), confirm outpost endpoint URL matches what Caddy forwards externally.

Grilling Skill

• Path: /home/dank/.agents/skills/grilling/SKILL.md
• Purpose: stress-test plans before building; one question at a time; look up facts in codebase, put decisions to user.

01a04717-a7fd-73ab-b7eb-319c4c1271c1

updated_at: 1787905384 Context: User asked whether HuggingFace EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF should be compatible with their personal llama.cpp runtime fork for frank-default-large's Qwen3.8-Flash-Next model.

Current frank-default-large setup at time of rollout:

• Served by /home/dank/Projects/serverAppConfigs/scripts/serve-frank-large.sh using env /home/dank/Projects/serverAppConfigs/scripts/frank-default-large.env.
• Runtime binary: /home/dank/GitPackages/ROCmFPX/build-qwen38/bin/llama-server from charlie12345/ROCmFPX llama.cpp fork with qwen4exp architecture support / PR #27742 merged, built with HIP/ROCmFP4 for Strix Halo gfx1151.
• Model repo/path: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX / local /home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX, file Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf.
• Model details in serve script: 177B params = 56B MoE + 51.2B PLE n-gram table, 512 experts / 10 used, 48 layers (12 attention + 36 SSM/Mamba), full_attention_interval=4, 2 KV heads x 256 dim, QSA indexer, hyper-connections count=4 low_rank=320, PLE at layer 1. Q4_0_ROCMFP4_STRIX quant, 113.47 GiB single file. PLE table streamed from page cache; GPU resident approx 63-69 GiB; mmap default into Strix Halo unified memory. --fit off because auto-fit segfaults on hyper-connection/indexer config; -ngl 999 puts layers on GPU/SVM.
• Sharp chat template enabled via scripts/chat-template.jinja with --jinja and CHAT_TEMPLATE_KWARGS preserving thinking/reasoning; reasoning format deepseek.

MTP investigation outcome:

• Tested EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF Q8_0 draft head and agentionai ROCmFP4-FAST MTP draft head against the STRIX_LEAN/Uncensored main model and qwen4exp binary.
• Both MTP draft GGUFs failed at load/runtime with missing tensor: blk.0.hc_attn_norm.weight.
• Interpretation: qwen4exp runtime expects hyper-connection attention norm tensors in draft models for this architecture/config, but these published draft GGUFs do not carry them. Therefore they are not compatible with the current frank-default-large runtime/model as-is.
• Durable next step: Need either (a) an MTP draft head GGUF that includes hc_attn_norm tensors matching this qwen4exp/hyper-connection model, or (b) a patched/binary/runtime path that does not require those hyper-connection tensors for draft models.

Operational state after rollout:

• Reverted/left frank-default-large using ngram-mod only, not MTP.
• Health check returned {"status":"ok"} after restart; service healthy on ngram-mod.
• Tests passed: ./tests/run.sh and focused bats tests for LLM units, secrets, script syntax; shellcheck skipped locally because not installed.
• Committed/pushed/pulled serverAppConfigs changes: commit 6d4a96a, message "frank-default-large: ngram-mod only — both MTP draft heads incompatible". Changed scripts/frank-default-large.env and scripts/serve-frank-large.sh. Dankclaw checkout fast-forwarded to commit 6d4a96a.

Pitfall to remember: Do not assume Qwen3.8-Flash-Next MTP heads are compatible merely by model name/quant/Strix Halo labeling. Check tensor inventory for hyper-connection tensors, especially blk.0.hc_attn_norm.weight, before scheduling downtime for swap-in.

01a04330-1551-72f6-b57b-8a8f43ffe7e2

updated_at: 1787862517 Context: frank-default-large serves kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF on Dankclaw (Strix Halo gfx1151 unified memory) using charlie12345/ROCmFPX llama.cpp build-qwen38 with qwen4exp support. Model is Q4_0_ROCMFP4_STRIX, ~113.47 GiB GGUF, 56B MoE plus 51.2B PLE n-gram table; PLE table is streamed from page cache and not resident in GPU VRAM. Runtime uses mmap into SVM-backed unified memory, -ngl 999/all layers on GPU, f16 KV, and --fit off because auto-fit segfaults on this model’s qwen4exp hyper-connection/indexer config.

Current/benchmark-relevant env file path: /home/dank/Projects/serverAppConfigs/scripts/frank-default-large.env. Serve script: /home/dank/Projects/serverAppConfigs/scripts/serve-frank-large.sh. Key documented tunables include CTX_SIZE=524288, PARALLEL=2, BATCH_SIZE=4096, UBATCH_SIZE=4096, NUM_THREADS=16, CACHE_TYPE_K=f16, CACHE_TYPE_V=f16, CTX_CHECKPOINTS=16, CACHE_RAM=8192, FIT=off. The prefill tuning rollout settled/documented BATCH_SIZE=4096 and UBATCH_SIZE=4096.

Decode tuning: a thread-batch sweep was run with an 8K prompt and 256 generated tokens. THREAD_BATCH=16 was confirmed as the best decode setting in that sweep. A prior todo requested 512 generated tokens, but that was intentionally not rerun: decode throughput in generated tok/s is independent of generation length for this comparison, so 512 tokens would only double wall-clock time while preserving the same tok/s signal. Use 256 gen tokens for future decode sweeps unless specifically measuring long-run thermal throttling or stability.

Benchmark methodology notes: separate prefill/TTFT benchmarking from decode tok/s benchmarking. For prefill, sweep batch and ubatch while holding other variables fixed and measure prompt processing throughput/TTFT. For decode, use a fixed prompt length (8K was used) and fixed generation length (256 tokens sufficient) while sweeping THREAD_BATCH; evaluate generated tok/s and TTFT. Keep prompts deterministic and repeat enough to avoid cache/page-cache warmup artifacts. Restore production env/service after benchmark and run the repo/service test suite.

Operational pitfalls/constraints: do not define bare env vars TEMP, HOST, or GPU_MAX_HEAP_SIZE in service env files; ROCm/HIP runtime reads those names and prior TEMP incident caused failures. llama.cpp host/port/temp should be passed as CLI flags, not bare env vars. Do not enable --fit for this model unless the qwen4exp auto-fit segfault has been fixed. Avoid interpreting longer generation-length decode runs as better signal unless testing thermal/stability; tok/s is the reusable metric.

01a042cc-e1c5-7482-87a2-14deadb82202

updated_at: 1787833610

frank-default-large: Qwen3.8-Flash-Next tuning on Dankclaw (Strix Halo gfx1151)

Model & Architecture

• kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF, 177B params (51.2B PLE n-gram + 125.8B MoE/attention)
• 56B MoE: 512 experts (10 used), 48 layers (12 attn + 36 SSM, full_attention_interval=4), 2 KV heads x 256 dim
• QSA indexer: 4 heads, top_k=2048; hyper-connections: count=4, low_rank=320
• PLE layer 1: ngram_size=3, heads_per_ngram=8
• Quant: Q4_0_ROCMFP4_STRIX (file_type=105), 113.47 GiB single GGUF
• PLE table: 50.66 GiB Q8_0, streamed from page cache (NEVER in GPU VRAM)
• GPU resident: ~63-69 GiB on Strix Halo unified memory pool (116 GiB SVM-backed)

Runtime

• ROCmFPX llama.cpp fork at /home/dank/GitPackages/ROCmFPX/build-qwen38/bin/llama-server
• PR #27742 (qwen4exp architecture) merged
• Build: -DGGML_HIP=ON -DGGML_HIP_ROCMI4_W4A4=ON -DCMAKE_HIP_ARCHITECTURES=gfx1151 -DGGML_CUDA_FA_ALL_QUANTS=ON -DLLAMA_BUILD_WEBUI=OFF -DCMAKE_EXE_LINKER_FLAGS=-no-pie

Critical Constraints (DO NOT CHANGE)

• FIT=off: auto-fit segfaults on this model's hyper-connection/indexer config
• -ngl 999 places all layers on SVM-backed GPU pool
• NEVER name env vars TEMP, HOST, or GPU_MAX_HEAP_SIZE — ROCm HIP runtime reads bare env vars (2026-08-13 TEMP incident)
• llama.cpp reads --host/--port/--temp as CLI flags only, never env vars
• Text-only: no mmproj, no MTP draft head

Current Tunings (env file: frank-default-large.env)

• BATCH_SIZE=4096, UBATCH_SIZE=4096
• NUM_THREADS=16, THREAD_BATCH=32
• CTX_SIZE=524288, PARALLEL=2
• CACHE_TYPE_K=f16, CACHE_TYPE_V=f16
• CTX_CHECKPOINTS=16, CACHE_RAM=8192
• HSA_OVERRIDE_GFX_VERSION=11.5.1
• Port 7770 (LISTEN_PORT)

Benchmark Status (interrupted)

• Goal: tune batch/ubatch for optimal token prefill
• Next planned step: threads-batch sweep
• Benchmark servers targeted ports 7772 and 7773
• Session interrupted during health-check polling loop; servers returned no output via curl on those ports (likely still loading 113 GiB model or not yet started)
• The repeated poll pattern (5+ identical checks, ~5s apart) suggests the assistant was waiting for model load completion before starting benchmarks

Workflow Pattern

• SSH into dankclaw from controller; check health via curl -s http://127.0.0.1:PORT/health
• Model load on unified memory is extremely slow (113 GiB mmap); benchmark scripts must account for long startup
• Use frank-default-small (the serving model for this agent) to orchestrate benchmarks against frank-default-large

01a042a4-7e85-7728-802d-c8442896f6e6

updated_at: 1787827055

frank-default-large model swap (2026-09-02)

Host: Dankclaw — AMD Radeon 8060S (Strix Halo), gfx1151 iGPU, 64 GiB unified memory. Runtime: charlie12345/ROCmFPX llama.cpp build-kairic v81. Port 7770.

Swap: STRIX_LEAN-GGUF (56B MoE / 35.76B PLE, 3 shards, 98.5 GiB) → Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF (177B total / 51.2B PLE, single file, 113.47 GiB on disk).

File path: /home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX/Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf

Key architectural fact: PLE (Projection Layer Embedding) table streams from page cache — never enters GPU VRAM regardless of tier. This is why a 114 GiB file fits in 64 GiB unified memory: only ~63-69 GiB is GPU-resident (same as LEAN variant). The PLE table occupies host RAM/page-cache.

Performance: Load time ~55 seconds. Generation speed and prompt throughput within noise of STRIX_LEAN at every context depth. 200/200 tests pass.

Config files modified:

• scripts/frank-default-large.env — MODEL_DIR, MODEL_FILE, header comments
• scripts/serve-frank-large.sh — header comments, default values
• docs/LLM/kingjones-strix-rocmfp4.md — new 195-line runbook
• docs/LLM/index.md — active model reference updated
• docs/index.md — top-level index links new doc

Git: Branch frank-large-strix-model pushed to origin (Gitea on Dankclaw).

Prior history on this slot (for context): CIRU PromptForge v3 → Kairic Edge IU4 (reverted) → kingjones Qwen3.8-27B ROCmFP4 STRIX MTP (Q6 AGENT, standalone MTP draft, Turbo4 V cache, 262K ctx/1 slot) → Q8_0 AGENT experiment (failed, set_rows bug) → STRIX MTP restored → now Flash-Next full 177B.