command-center/agent/memories/--home-dank-Projects-serverAppConfigs--/learned.md

• MTP draft heads for Qwen3.8-Flash-Next are INCOMPATIBLE with the STRIX_LEAN Uncensored model on the charlie12345/ROCmFPX qwen4exp binary. Both attempted heads — EasiiX Q8_0 (4.1 GB) and agentionai ROCmFP4-FAST (2.28 GiB) — fail with missing tensor 'blk.0.hc_attn_norm.weight'. The qwen4exp binary expects this hyper-connection attn norm tensor in the draft GGUF, but neither draft head carries it. MTP only ever worked on the 27B model (kingjones-Qwen3.8-27B-STRIX-MTP with mtp-Qwen3.8-27B-Q4_0.gguf), never on any Flash-Next variant. The Flash-Next STRIX (non-LEAN) that ran before the 2026-08-28 session never loaded a draft model at all. Current production config: SPEC_TYPE=ngram-mod only. To get MTP working on STRIX_LEAN, need either a draft head that includes hc_attn_norm tensors or a binary that doesn't require them for draft models. (context: frank-default-large MTP integration attempt on Dankclaw, 2026-08-28. Downloaded two MTP heads from HuggingFace, both failed to load against the STRIX_LEAN Uncensored model. Service crash-looped on each attempt. Fell back to ngram-mod only. Files: scripts/frank-default-large.env, scripts/serve-frank-large.sh.)
• The MTP draft head GGUF mtp-Qwen3.8-Flash-Next-Q8_0.gguf (EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF, 4.1 GB) is INCOMPATIBLE with the STRIX_LEAN Uncensored Flash-Next model on the charlie12345/ROCmFPX binary (build-qwen38). The draft GGUF is missing tensor blk.0.hc_attn_norm.weight (hyper-connection attention norm) that the qwen4exp architecture expects. The service crash-loops: main model loads fine (~59s), then draft model load fails and exits. MTP only ever worked on the older 27B model (Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT + its own mtp-Qwen3.8-27B-Q4_0.gguf head). The Flash-Next STRIX (non-LEAN) that ran before this session never loaded a draft model either — it ran ngram-only. Current production config: SPEC_TYPE=ngram-mod (no draft sidecar). TODO: find a compatible MTP head for STRIX_LEAN or build one from the Flash-Next checkpoint. (context: frank-default-large MTP draft head integration on Dankclaw, 2026-08-28)
• When running Python scripts on a remote host (Dankclaw) that need to manage local services (systemctl, sed), use subprocess.run(cmd, shell=True) for local execution — NOT nested SSH back to the same host. Nested SSH to localhost fails (no key auth for self-loopback). Also: before git pull --ff-only on a remote, verify the branch with git branch --show-current — Dankclaw was on frank-large-strix-model not main, so pulls were no-ops and new files didn't appear. (context: bench-spec-decode.py ran on Dankclaw but used ssh() to SSH back to itself for systemctl/sed. All SSH calls failed with Permission denied. Also: Dankclaw was on frank-large-strix-model branch, not main, so git pull --ff-only was a no-op and the new file didn't appear.)
• When deploying to Dankclaw (or any remote host), verify the remote checkout is on the correct branch (main) before pulling. On 2026-08-28, Dankclaw was on frank-large-strix-model with uncommitted local changes (STRIX_LEAN model paths + mmproj), causing git pull to be a no-op on the wrong branch. The benchmark script wasn't found because it was committed to main but Dankclaw was on a feature branch. Fix: git stash && git checkout main && git pull --ff-only on the remote. (context: frank-default-large MTP benchmark deployment on Dankclaw. The git pull said "Already up to date" because it was on frank-large-strix-model, not main.)
• The docs-completeness test (test 112, tests/docs_completeness.bats) scans ALL .md files under docs/ — including untracked files — and requires every one to be linked from docs/index.md via Title wikilink. Untracked files like docs/Clippings/ will fail this test and block the pre-commit hook. Fix: move untracked docs that aren't meant to be part of the knowledge base to .AGENTTRASH/ before committing. (context: Committing MTP draft head changes to frank-default-large was blocked by pre-existing test 112 failure caused by untracked docs/Clippings/Caddy Anubis.md.)
• To verify GGUF model architecture compatibility without llama-gguf or Python GGUF parsing: strings on the first ~2KB of a GGUF file extracts general.architecture, general.name, general.size_label, and all arch.* metadata keys (block_count, embedding_length, expert_count, etc.). For remote files: curl -sL -r 0-2048 'url' | strings | head -40. The architecture string (e.g. qwen4exp, qwen3moe, llama) is the primary compatibility key — a draft head GGUF must report the same general.architecture as the target model. Cross-reference against the serving binary's --help output for the relevant feature flags (e.g. --spec-type draft-mtp, --model-draft) before declaring compatibility. The Python struct.unpack approach for GGUF header parsing is fragile (fails on custom type codes); strings is more robust for a quick architecture check. (context: Assessed EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF MTP draft head compatibility against frank-default-large (Qwen3.8-Flash-Next qwen4exp) on Dankclaw. Needed to verify architecture match between the draft head GGUF and the target model GGUF without llama-gguf binary available.)
• On these hosts (Dankclaw, Dankserver), systemctl disable service doesn't just remove symlinks — it deletes the unit file itself from /etc/systemd/system/. Use systemctl mask (creates symlink to /dev/null, safe and reversible via unmask) instead of disable during benchmarks. The unit files are in scripts/service.service in the repo. (context: AGENTS.md benchmarking policy section updated. Occurred when running bench for frank-default-large on dankclaw.)
• frank-default-large (Qwen3.8-Flash-Next ROCmFP4 STRIX on dankclaw) decode throughput across threads-batch: TB=8: 3.3 tok/s (76.6s), TB=16: 3.4 tok/s (74.8s), TB=32: 3.4 tok/s (75.0s). Benchmark: ~23.5K token code prompt, 256 generated tokens, HTTP API, fresh server per value. Decode throughput is flat across all tested values; TB=16 is optimal due to prefill advantage (373.5 tok/s vs 354.4 at TB=32). TB=8 is marginally worse than 16 on both prefill and decode. (context: bench-decode-threads-batch.py ran on dankclaw. Model: kingjones-Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf (113.47 GiB). Prompt: /tmp/bench-prompts/prompt_code.txt extended to 8200 words (~23.5K tokens).)
• On dankclaw and dankserver, systemctl disable frank-default-large (and similar systemd service units) doesn't just remove symlinks — it deletes the unit file itself from /etc/systemd/system/. Always use systemctl mask instead (creates symlink to /dev/null, safe and reversible with unmask). When restoring after a disable: copy the unit file from this repo back to /etc/systemd/system/, run systemctl daemon-reload, then enable + start.
• When benchmarking llama.cpp server-side parameters (--threads-batch, --ubatch-size, etc.), use HTTP-based prefill benchmarking: start the server with the target parameter, send a large prompt via /v1/chat/completions with max_tokens=1 (the server crashes on max_tokens=0), measure prompt_tokens / wall_time from the response usage. Do NOT use llama-bench — it benchmarks the bench binary itself, not the server, and cannot target server-side parameters. Script at scripts/bench-threads-batch-http.py demonstrates this pattern.
• frank-default-large (Qwen3.8-Flash-Next ROCmFP4 STRIX on dankclaw) best threads-batch: 16. Measured prefill throughput sweep: tb=16: 373.5 tok/s, tb=32: 354.4, tb=48: 344.1, tb=64: 344.7, tb=80: 344.4, tb=96: 357.1, tb=128: 321.1. Lower thread batching wins for prefill — fewer threads compete for GPU resources. Used HTTP-based benchmarking (llama-bench cannot benchmark --threads-batch, it's a server-side parameter). Used max_tokens=1 instead of 0 (server crashes on 0-token requests). Script: scripts/bench-threads-batch-http.py.
• When editing bash scripts that use :: for default-value expansion (e.g., :: "${VAR:=default}"), the edit tool's parser can misinterpret :: and produce mangled output with extra characters (literal ^ prefix or ::: instead of ::). For targeted fixes to these specific lines, use sed in-place instead of the edit tool. Example: sed -i 's/^:: "/: "/' script.sh. (context: frank-default-large model swap: edited scripts/serve-frank-large.sh default-value lines from :: to single-colon. The edit tool produced ::: with a literal ^ on line 40. Required sed fallback to fix.)