From 91188d4b1b1631b232f4e1b51d522237304eb722 Mon Sep 17 00:00:00 2001 From: Frank Date: Mon, 10 Aug 2026 22:02:28 -0700 Subject: [PATCH] docs: add Ling-3.0-flash vLLM-ROCm deployment plan (grilled decisions, port 7770) --- docs/ling-3.0-flash-vllm-plan.md | 40 ++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100644 docs/ling-3.0-flash-vllm-plan.md diff --git a/docs/ling-3.0-flash-vllm-plan.md b/docs/ling-3.0-flash-vllm-plan.md new file mode 100644 index 0000000..9b22a3e --- /dev/null +++ b/docs/ling-3.0-flash-vllm-plan.md @@ -0,0 +1,40 @@ +# Ling-3.0-flash on Dankclaw — vLLM-ROCm Deployment Plan + +## Decisions (confirmed) + +| Decision | Choice | +|----------|--------| +| Objective | Quality-first; correctness (tool calling) over raw quant speed | +| Runtime | vLLM-ROCm on host ROCm venv (jcbtc validated gfx1151 path) | +| Model | jcbtc/Ling-3.0-Flash-CIRU-int4-Strix-native (INT4 W4A16, ~62 GB) | +| Accuracy | INT4 primary; MXFP4 (olka-fi) as controlled A/B after stability | +| KV cache | Full precision (no FP8 KV) — quality | +| Speculative decoding | MTP (native, lossless) | +| Context | Native 256k; user cap ~200k; 4-6 concurrent sessions (~8 max_num_seqs) | +| Throughput | Maximum (chunked prefill, large max_num_batched_tokens) | +| systemd unit | frank-default-large.service | +| Served alias | frank-default-large | +| Port | 7770 (8080s overused) | +| Migration | Kill llama.cpp, disable llama-server.service, reclaim memory | +| Delivery | Follow AGENTS.md git workflow; scripts/ + docs/ in repo | + +## Why not alternatives + +- FP8: broken/lossy on Strix Halo; no ROCmFP8 Ling model; 124 GB does not fit. +- HIGHER bpw (Q5/Q6/Q8/BF16/FP8): exceed 96 GB usable wall for a 124B MoE. +- ROCmFP4 (raulvidis): defective tool calling; llama.cpp-only concurrency limits. +- Vulkan: llama.cpp backend only, not vLLM; no quality ceiling unlock. +- FP8 KV cache: rejected for quality (full-precision KV affordable because + Ling is hybrid linear-attention with small KV footprint). + +## Steps + +1. Pre-flight: disk space, ROCm/amdgpu state, memory baseline. +2. Bring jcbtc scripts + frank-default-large.service into repo; commit. +3. Build ROCm vLLM (install-host-deps.sh + build-vllm-gfx1151.sh). +4. Download weights via download-model.sh (~62 GB). +5. Validate on staging: MTP engages, tool calling works, quality + throughput. +6. Cut over: kill llama.cpp, disable llama-server, enable frank-default-large on 7770. +7. Tune max throughput; confirm concurrent-token capacity. +8. MXFP4 A/B (post-stable). +9. Update AGENTS.md, README, and KB.