Frank
9bfa0be85d
- 31 new managed skills (anubis/arr/authentik/hermes/freetoken/podman/llm-swap runbooks etc.) now tracked. - Removed superseded skills: omp_harness_config, peon_ping_integration (replaced by user-authored peon-ping-* managed skills), rocmfp4-strix-benchmark-orchestration. - New memory-root skills: niri_compositor_ops, omp_git_workflow, omp_model_config, quadlet-media-migration, homelab-service-triage, hermes-dankclaw-ops, frank-default-large-ops, caddy-auth-dns-playbook.
37 lines
2.1 KiB
Markdown
37 lines
2.1 KiB
Markdown
# frank-default-large operations
|
|
|
|
Use this when changing or troubleshooting the large model service on Dankclaw.
|
|
|
|
Current durable model/runtime:
|
|
- Host: Dankclaw, Strix Halo gfx1151 unified memory.
|
|
- Alias: frank-default-large.
|
|
- Endpoint: http://192.168.0.143:7770/v1, health on port 7770.
|
|
- Repo files: scripts/serve-frank-large.sh and scripts/frank-default-large.env under /home/dank/Projects/serverAppConfigs.
|
|
- Runtime: /home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server.
|
|
- Model: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF.
|
|
- Local GGUF: /home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX/Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf.
|
|
|
|
Hard constraints:
|
|
- Keep --fit off unless the qwen4exp hyper-connection auto-fit segfault is proven fixed.
|
|
- Keep PLE table mmap/page-cache streamed. Do not add --no-mmap and do not force the PLE table into GPU memory.
|
|
- Avoid bare env vars TEMP, HOST, and GPU_MAX_HEAP_SIZE. HIP/ROCm can read them unexpectedly.
|
|
- Do not enable MTP/speculative decoding casually. Prior MTP heads failed because blk.0.hc_attn_norm.weight was missing. Latest durable state has SPEC_TYPE=none.
|
|
- Stock llama.cpp cannot load this model; qwen4exp architecture support and ROCmFP4 tensor support must coexist in the same runtime tree.
|
|
- If flash-attn graph build fails on QSA mask, port scripts/patches/qwen4exp-qsa-f32-mask.patch into the active runtime tree and rebuild.
|
|
|
|
Before edits:
|
|
1. cd /home/dank/Projects/serverAppConfigs
|
|
2. git status --short
|
|
3. Inspect current scripts/frank-default-large.env and scripts/serve-frank-large.sh; do not rely on old memory for exact tunables.
|
|
|
|
After edits:
|
|
1. Run repo tests appropriate to the change.
|
|
2. Deploy/restart using the current repo workflow.
|
|
3. Verify health: curl -s http://192.168.0.143:7770/health or via localhost on Dankclaw.
|
|
4. Watch logs during model load; 113+ GiB mmap/model load can take significant time.
|
|
|
|
Benchmarking notes:
|
|
- Separate prefill/TTFT from decode tok/s.
|
|
- For decode sweeps, an 8K prompt and 256 generated tokens is sufficient unless testing long-run thermal/stability.
|
|
- Prior decode sweep found THREAD_BATCH=16 best, but current repo env always wins.
|