dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: reflect Ling-3 vLLM (frank-default-large) on Dankclaw

Browse Source 
Update target maps (AGENTS, README, deployment, deploy.sh) from the retired
ROCmFPX llama.cpp stack to the active vLLM-ROCm CIRU INT4 service on port 7770
with MTP speculative decoding. Align Dankserver to the running
vllm-awq-qwen36 (AWQ, port 34401). Refresh KB pages: Dankclaw system specs
(gpu-mem-util 0.95, CIRU venv), quantization choices (INT4 primary, MXFP4
experiment), DFlash/DSpark Dankclaw section, and the KB index topology.
This commit is contained in:
Frank committed 2026-08-10 23:37:27 -07:00
1 parent 52ad985978
commit 903d54c389
9 files changed
+55 -54
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/Quantization by Hardware Architecture.md
+6 -5
View File
@@ -10,7 +10,7 @@ Use this page when you choose a model format for Dankserver or Dankclaw. The sho
- NVIDIA Ada (RTX 40 series): FP8 or AWQ.
- NVIDIA Hopper / Blackwell (H100, RTX 50 series): FP8, NVFP4.
- AMD CDNA (MI300 series): FP8 or MXFP4.
- AMD RDNA 3 / Strix Halo (gfx11): GGUF Q4_K_M or ROCmFP4.
- AMD RDNA 3 / Strix Halo (gfx11): Quark INT4 (vLLM), GGUF Q4_K_M, or ROCmFP4.
- CPU-only: any GGUF quant.
## Quantization Types
@@ -112,13 +112,14 @@ The RTX 3090 has no native FP8 or FP4 tensor cores. Use:
### Dankclaw (Strix Halo gfx1151, RDNA 3.5)
RDNA 3.5 has no native FP8 tensor cores. The APU shares memory. Use GGUF through llama.cpp, or ROCmFPX formats:
RDNA 3.5 has no native FP8 tensor cores. The APU shares memory. The active Ling-3.0-flash service runs vLLM-ROCm with the AMD Quark INT4 (W4A16) format (the jcbtc CIRU Strix-native bundle). INT4 weights keep the 124B model inside the 96 GiB frame buffer with a full-precision KV cache.
| Ranking | Format | Why |
|---------|--------|-----|
| 1 | ROCmFP4 (ROCmFPX fork) | native ROCm kernels; proven with Ling-3.0-flash MTP |
| 2 | GGUF Q4_K_M / Q8_0 | works with upstream llama.cpp |
| 3 | FP8 (vLLM) | software emulation; memory savings only |
| 1 | INT4 W4A16 (Quark/vLLM) | active; fits 96 GiB; native ROCm kernels; concurrency |
| 2 | MXFP4 (Quark) | same-bit, higher accuracy; beta on gfx1151 (A/B experiment) |
| 3 | GGUF Q4_K_M / Q8_0 | works with upstream llama.cpp only |
| 4 | FP8 (vLLM) | software emulation; memory savings only; no compute win |
## See Also