1.9 KiB
1.9 KiB
Ling-3.0-flash on Dankclaw — vLLM-ROCm Deployment Plan
Decisions (confirmed)
| Decision | Choice |
|---|---|
| Objective | Quality-first; correctness (tool calling) over raw quant speed |
| Runtime | vLLM-ROCm on host ROCm venv (jcbtc validated gfx1151 path) |
| Model | jcbtc/Ling-3.0-Flash-CIRU-int4-Strix-native (INT4 W4A16, ~62 GB) |
| Accuracy | INT4 primary; MXFP4 (olka-fi) as controlled A/B after stability |
| KV cache | Full precision (no FP8 KV) — quality |
| Speculative decoding | MTP (native, lossless) |
| Context | Native 256k; user cap ~200k; 4-6 concurrent sessions (~8 max_num_seqs) |
| Throughput | Maximum (chunked prefill, large max_num_batched_tokens) |
| systemd unit | frank-default-large.service |
| Served alias | frank-default-large |
| Port | 7770 (8080s overused) |
| Migration | Kill llama.cpp, disable llama-server.service, reclaim memory |
| Delivery | Follow AGENTS.md git workflow; scripts/ + docs/ in repo |
Why not alternatives
- FP8: broken/lossy on Strix Halo; no ROCmFP8 Ling model; 124 GB does not fit.
- HIGHER bpw (Q5/Q6/Q8/BF16/FP8): exceed 96 GB usable wall for a 124B MoE.
- ROCmFP4 (raulvidis): defective tool calling; llama.cpp-only concurrency limits.
- Vulkan: llama.cpp backend only, not vLLM; no quality ceiling unlock.
- FP8 KV cache: rejected for quality (full-precision KV affordable because Ling is hybrid linear-attention with small KV footprint).
Steps
- Pre-flight: disk space, ROCm/amdgpu state, memory baseline.
- Bring jcbtc scripts + frank-default-large.service into repo; commit.
- Build ROCm vLLM (install-host-deps.sh + build-vllm-gfx1151.sh).
- Download weights via download-model.sh (~62 GB).
- Validate on staging: MTP engages, tool calling works, quality + throughput.
- Cut over: kill llama.cpp, disable llama-server, enable frank-default-large on 7770.
- Tune max throughput; confirm concurrent-token capacity.
- MXFP4 A/B (post-stable).
- Update AGENTS.md, README, and KB.