dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: add Ling-3.0-flash vLLM-ROCm deployment plan (grilled decisions, port 7770)

Browse Source
This commit is contained in:
Frank
2026-08-10 22:02:28 -07:00
parent 2a7571abdc
commit 91188d4b1b
1 changed files with 40 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ling-3.0-flash-vllm-plan.md
+40
View File
@@ -0,0 +1,40 @@
# Ling-3.0-flash on Dankclaw — vLLM-ROCm Deployment Plan
## Decisions (confirmed)
| Decision | Choice |
|----------|--------|
| Objective | Quality-first; correctness (tool calling) over raw quant speed |
| Runtime | vLLM-ROCm on host ROCm venv (jcbtc validated gfx1151 path) |
| Model | jcbtc/Ling-3.0-Flash-CIRU-int4-Strix-native (INT4 W4A16, ~62 GB) |
| Accuracy | INT4 primary; MXFP4 (olka-fi) as controlled A/B after stability |
| KV cache | Full precision (no FP8 KV) — quality |
| Speculative decoding | MTP (native, lossless) |
| Context | Native 256k; user cap ~200k; 4-6 concurrent sessions (~8 max_num_seqs) |
| Throughput | Maximum (chunked prefill, large max_num_batched_tokens) |
| systemd unit | frank-default-large.service |
| Served alias | frank-default-large |
| Port | 7770 (8080s overused) |
| Migration | Kill llama.cpp, disable llama-server.service, reclaim memory |
| Delivery | Follow AGENTS.md git workflow; scripts/ + docs/ in repo |
## Why not alternatives
- FP8: broken/lossy on Strix Halo; no ROCmFP8 Ling model; 124 GB does not fit.
- HIGHER bpw (Q5/Q6/Q8/BF16/FP8): exceed 96 GB usable wall for a 124B MoE.
- ROCmFP4 (raulvidis): defective tool calling; llama.cpp-only concurrency limits.
- Vulkan: llama.cpp backend only, not vLLM; no quality ceiling unlock.
- FP8 KV cache: rejected for quality (full-precision KV affordable because
Ling is hybrid linear-attention with small KV footprint).
## Steps
1. Pre-flight: disk space, ROCm/amdgpu state, memory baseline.
2. Bring jcbtc scripts + frank-default-large.service into repo; commit.
3. Build ROCm vLLM (install-host-deps.sh + build-vllm-gfx1151.sh).
4. Download weights via download-model.sh (~62 GB).
5. Validate on staging: MTP engages, tool calling works, quality + throughput.
6. Cut over: kill llama.cpp, disable llama-server, enable frank-default-large on 7770.
7. Tune max throughput; confirm concurrent-token capacity.
8. MXFP4 A/B (post-stable).
9. Update AGENTS.md, README, and KB.