docs: GLM-5.3-Flash readiness revised — tcclaviger fork has native MXFP4/DSFP4
Source reading in the qwen-fp8 container: glm5next is registered with an amd/ backend dir + DSA indexer; r4d.so has native moe_dsfp4/moe_dsfp4_mt1 entries (has_dsfp4()==True), so the old 'MXFP4 = emulation, missing kernels' verdict is dead for this fork. DSFP4 = MXFP4 weights + DeepSeek activation law. Offload recipes exist for mxfp4 and W4A16 (R4dW4a16MoEExperts in offloader/expert.py:114). GLM gate is now the payload format (compressed-tensors MXFP4/W4A16, none public) + a HIP boot probe.
This commit is contained in:
@@ -54,7 +54,7 @@ was right.
|
||||
|---|---|---|---|---|
|
||||
| **DeepSeek-V4.1-Flash** | 552B / 8B prefill, 16B decode | 384 top-6, 40 layers, `moe_intermediate_size` 2304, hidden 5120 | CED 20 enc + 20 dec; CSA2 Full/Reindex/Reuse; **FP4 main KV, 890 B/token**; Engram 196B at layers 1 & 14 | Lucebox `feat/ds41-engine` only (unmerged) |
|
||||
| **MiMo-V2.6-Flash-MOPD** | 309B / 15B | 256 top-8, no shared expert, 48 layers (39 SWA / 9 GA) | hybrid SWA (window 128) + GA; 5-layer MTP | vLLM / SGLang, NVIDIA-first. No AMD path |
|
||||
| **GLM-5.3-Flash** | 320B / 18B | MoE | KDA (linear) + DSA (sparse) hybrid; mHC | llama.cpp (community R9700 guide); TokenSpeed recipe |
|
||||
| **GLM-5.3-Flash** | 320B / 18B | MoE | KDA (linear) + DSA (sparse) hybrid; mHC | **tcclaviger vLLM fork — plausible native path (2026-09-29 source reading: `glm5next` registered with an `amd/` backend dir + DSA indexer + mHC path; r4d.so carries native mxfp4/dsfp4 MoE entries). Gate: a compressed-tensors MXFP4 payload (~165 GiB, none exists publicly) + a HIP boot probe.** llama.cpp GGUF (community R9700 guide) remains the verified-but-slower path |
|
||||
|
||||
## Sizing — the arithmetic that decides it
|
||||
|
||||
@@ -166,6 +166,7 @@ Not advisable.
|
||||
| Runtime | gfx1201 | Architectures | Note |
|
||||
|---|---|---|---|
|
||||
| **Affinity** | yes, built here | **V4-Flash only** — "the geometry is compiled in; there is no architecture abstraction" | 110 tok/s for 284B. Best engine for this box by an order of magnitude, wrong model |
|
||||
| **tcclaviger vLLM fork** | **yes — proven 2026-09-29** | qwen4exp measured 37.4-40.6 tok/s decode (see [[dankserver-qwen38-fp8-expert-offload-tuning]]); **`glm5next` registered** (`Glm5NextForCausalLM`/`MTP`, backend-split `amd/`/`common/`/`nvidia/`, `sparse_indexer.py` for DSA); r4d.so has **native mxfp4 + dsfp4 MoE entries** (`has_dsfp4() == True`); W4A16 MoE class (`R4dW4a16MoEExperts`) wired into the same tiered offload; TunableOp silently skipped under offload (needs `CLAV_TUNABLEOP_WITH_OFFLOAD=1`, and then corrupts numerics — do not use) | the ad hoc qwen-fp8 container's runtime; **the old "MXFP4 = emulation/missing kernels" verdict predates this fork and is dead** — the GLM-5.3-Flash gate is now the payload format + a boot probe, not kernels |
|
||||
| **Lucebox** | yes, image on host | Qwen 3.8, DS4; **V4.1 on the unmerged `feat/ds41-engine` branch** | The only V4.1 path. **Refuses layer split and layer-major prefill for V4.1** |
|
||||
| **radiance-vLLM** (drwolfen) | yes — the active frank-primary | welded to `qwen4_exp`; PLE / QSA / packed-HC live inside that model class | 12–20 person-days per architecture |
|
||||
| **FreeToken-ROCm** | partial | block-FP8 safetensors only (modelopt); known `CpuMoeExecutor` issue under ROCm | cannot load a GGUF |
|
||||
|
||||
@@ -105,6 +105,23 @@
|
||||
band, straddling the 35 target within noise. Final 256k config:
|
||||
`MAXLEN=262144 KV_BYTES=4328521728`, prefetch off. The full-speed
|
||||
alternative stays `MAXLEN=184224` at 3 GiB (37.4-40.6 band).
|
||||
- **GLM-5.3-Flash readiness REVISED by source reading (Frank's question
|
||||
about int4 vs mxfp4):** the tcclaviger fork in the qwen-fp8 container
|
||||
registers `Glm5NextForCausalLM`/`Glm5NextMTP` with a backend-split
|
||||
module (`glm5next/amd|common|nvidia`, `sparse_indexer.py` for DSA);
|
||||
`offloader/expert.py` accepts `R4dDsfp4MoEExperts, R4dW4a16MoEExperts,
|
||||
Fp8HipMoEExperts` — and **`R4dDsfp4MoEExperts` is MXFP4 weights with
|
||||
the DeepSeek activation law** (docstring; SwiGLU `gemm1_clamp_limit`
|
||||
default 10.0), inheriting the mxfp4 repack/GEMM/emulation-fallback.
|
||||
**`r4d_lib.has_dsfp4()` is True** in this build — native `moe_dsfp4` /
|
||||
`moe_dsfp4_mt1` entries exist, so the old "MXFP4 = emulation, missing
|
||||
kernels" GLM verdict is dead for this fork. WNA16 (int4) backend
|
||||
oracle on ROCm reduces to HUMMING/TRITON (Marlin/TRTLLM/XPU are
|
||||
CUDA/NVIDIA/Intel-only), payload must be compressed-tensors W4A16
|
||||
group-wise symmetric. Remaining GLM gate: a compressed-tensors MXFP4
|
||||
(~165 GiB) or W4A16 payload + a glm5next-on-HIP boot probe; community
|
||||
GLM-5.3 quants are GGUF (llama.cpp path only), native FP8 is 328 GB
|
||||
(> 246 GiB RAM). Feasibility doc rows updated.
|
||||
|
||||
## 2026-09-28
|
||||
- **Dankclaw model cleanup** (disk was at 93%, 137 G free). Frank asked to
|
||||
|
||||
Reference in New Issue
Block a user