Frank
9bfa0be85d
- 31 new managed skills (anubis/arr/authentik/hermes/freetoken/podman/llm-swap runbooks etc.) now tracked. - Removed superseded skills: omp_harness_config, peon_ping_integration (replaced by user-authored peon-ping-* managed skills), rocmfp4-strix-benchmark-orchestration. - New memory-root skills: niri_compositor_ops, omp_git_workflow, omp_model_config, quadlet-media-migration, homelab-service-triage, hermes-dankclaw-ops, frank-default-large-ops, caddy-auth-dns-playbook.
2.1 KiB
2.1 KiB
frank-default-large operations
Use this when changing or troubleshooting the large model service on Dankclaw.
Current durable model/runtime:
- Host: Dankclaw, Strix Halo gfx1151 unified memory.
- Alias: frank-default-large.
- Endpoint: http://192.168.0.143:7770/v1, health on port 7770.
- Repo files: scripts/serve-frank-large.sh and scripts/frank-default-large.env under /home/dank/Projects/serverAppConfigs.
- Runtime: /home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server.
- Model: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF.
- Local GGUF: /home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX/Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf.
Hard constraints:
- Keep --fit off unless the qwen4exp hyper-connection auto-fit segfault is proven fixed.
- Keep PLE table mmap/page-cache streamed. Do not add --no-mmap and do not force the PLE table into GPU memory.
- Avoid bare env vars TEMP, HOST, and GPU_MAX_HEAP_SIZE. HIP/ROCm can read them unexpectedly.
- Do not enable MTP/speculative decoding casually. Prior MTP heads failed because blk.0.hc_attn_norm.weight was missing. Latest durable state has SPEC_TYPE=none.
- Stock llama.cpp cannot load this model; qwen4exp architecture support and ROCmFP4 tensor support must coexist in the same runtime tree.
- If flash-attn graph build fails on QSA mask, port scripts/patches/qwen4exp-qsa-f32-mask.patch into the active runtime tree and rebuild.
Before edits:
- cd /home/dank/Projects/serverAppConfigs
- git status --short
- Inspect current scripts/frank-default-large.env and scripts/serve-frank-large.sh; do not rely on old memory for exact tunables.
After edits:
- Run repo tests appropriate to the change.
- Deploy/restart using the current repo workflow.
- Verify health: curl -s http://192.168.0.143:7770/health or via localhost on Dankclaw.
- Watch logs during model load; 113+ GiB mmap/model load can take significant time.
Benchmarking notes:
- Separate prefill/TTFT from decode tok/s.
- For decode sweeps, an 8K prompt and 256 generated tokens is sufficient unless testing long-run thermal/stability.
- Prior decode sweep found THREAD_BATCH=16 best, but current repo env always wins.