docs: boot policy (only PP units enabled), deploy.sh dankserver hazard, stale-claim cleanup
- deployment.md: Boot Policy (2026-08-17) section (disabled units table, re-enable recipes, stale frank-restart-small.sh helper, broken deploy.sh dankserver target); Update Dankserver -> frank-pp-worker; condensed retired 35B history block and 2026-08-13 incident - AGENTS.md: boot-policy note, target map fix (Q6_K, split 0.68), deploy.sh dankserver warning, obsolete quirk entries condensed - README.md: rewrite active services/models tables (PP head + worker) - index.md: topology fix (qwythos:9b, Q6_K split 0.68, NPU disabled) - qwen3.8-spec-matrix.md: superseded configs merged into compact history; stale inline env block removed; bundled-head MTP note - FastFlowLM, PP Deployment: boot-policy + enablement notes - System Specs x2, DFlash doc, 35B FP8 doc, Ling docs: stale active claims fixed, retired banners, never-shipped plan stubbed
This commit is contained in:
1 parent
3f88b8e1ed
commit
968066ff23
13 files changed
+190
-246
No files matched your search
@@ -1,40 +1,10 @@
|
||||
# Ling-3.0-flash on Dankclaw — vLLM-ROCm Deployment Plan
|
||||
# Ling-3.0-flash on Dankclaw — vLLM-ROCm Deployment Plan (never shipped)
|
||||
|
||||
## Decisions (confirmed)
|
||||
This page was the plan for a vLLM-ROCm deployment of
|
||||
`jcbtc/Ling-3.0-Flash-CIRU-int4-Strix-native` (INT4 W4A16, ~62 GB) on
|
||||
Dankclaw. The deployment never shipped. The actual Ling work used the
|
||||
llama.cpp Q6_K fork trial instead: [[ling-3.0-flash-trial]].
|
||||
|
||||
| Decision | Choice |
|
||||
|----------|--------|
|
||||
| Objective | Quality-first; correctness (tool calling) over raw quant speed |
|
||||
| Runtime | vLLM-ROCm on host ROCm venv (jcbtc validated gfx1151 path) |
|
||||
| Model | jcbtc/Ling-3.0-Flash-CIRU-int4-Strix-native (INT4 W4A16, ~62 GB) |
|
||||
| Accuracy | INT4 primary; MXFP4 (olka-fi) as controlled A/B after stability |
|
||||
| KV cache | Full precision (no FP8 KV) — quality |
|
||||
| Speculative decoding | MTP (native, lossless) |
|
||||
| Context | Native 256k; user cap ~200k; 4-6 concurrent sessions (~8 max_num_seqs) |
|
||||
| Throughput | Maximum (chunked prefill, large max_num_batched_tokens) |
|
||||
| systemd unit | frank-default-large.service |
|
||||
| Served alias | frank-default-large |
|
||||
| Port | 7770 (8080s overused) |
|
||||
| Migration | Kill llama.cpp, disable llama-server.service, reclaim memory |
|
||||
| Delivery | Follow AGENTS.md git workflow; scripts/ + docs/ in repo |
|
||||
|
||||
## Why not alternatives
|
||||
|
||||
- FP8: broken/lossy on Strix Halo; no ROCmFP8 Ling model; 124 GB does not fit.
|
||||
- HIGHER bpw (Q5/Q6/Q8/BF16/FP8): exceed 96 GB usable wall for a 124B MoE.
|
||||
- ROCmFP4 (raulvidis): defective tool calling; llama.cpp-only concurrency limits.
|
||||
- Vulkan: llama.cpp backend only, not vLLM; no quality ceiling unlock.
|
||||
- FP8 KV cache: rejected for quality (full-precision KV affordable because
|
||||
Ling is hybrid linear-attention with small KV footprint).
|
||||
|
||||
## Steps
|
||||
|
||||
1. Pre-flight: disk space, ROCm/amdgpu state, memory baseline.
|
||||
2. Bring jcbtc scripts + frank-default-large.service into repo; commit.
|
||||
3. Build ROCm vLLM (install-host-deps.sh + build-vllm-gfx1151.sh).
|
||||
4. Download weights via download-model.sh (~62 GB).
|
||||
5. Validate on staging: MTP engages, tool calling works, quality + throughput.
|
||||
6. Cut over: kill llama.cpp, disable llama-server, enable frank-default-large on 7770.
|
||||
7. Tune max throughput; confirm concurrent-token capacity.
|
||||
8. MXFP4 A/B (post-stable).
|
||||
9. Update AGENTS.md, README, and KB.
|
||||
The trial service (`frank-large-ling`) is retired and disabled under the
|
||||
boot policy (2026-08-17). `frank-default-large` is the Qwen3.8-27B PP
|
||||
head, not the Ling path. See git history for the full decision table.
|
||||
Reference in new issue
Block a user