From f532e03bf4ebb4588777805657dcae1c44cb0f84 Mon Sep 17 00:00:00 2001 From: Frank Date: Tue, 29 Sep 2026 23:28:04 -0700 Subject: [PATCH] =?UTF-8?q?docs:=20GLM-5.3-Flash=20readiness=20revised=20?= =?UTF-8?q?=E2=80=94=20tcclaviger=20fork=20has=20native=20MXFP4/DSFP4?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Source reading in the qwen-fp8 container: glm5next is registered with an amd/ backend dir + DSA indexer; r4d.so has native moe_dsfp4/moe_dsfp4_mt1 entries (has_dsfp4()==True), so the old 'MXFP4 = emulation, missing kernels' verdict is dead for this fork. DSFP4 = MXFP4 weights + DeepSeek activation law. Offload recipes exist for mxfp4 and W4A16 (R4dW4a16MoEExperts in offloader/expert.py:114). GLM gate is now the payload format (compressed-tensors MXFP4/W4A16, none public) + a HIP boot probe. --- .../LLM/dankserver-frontier-moe-feasibility.md | 3 ++- serverAppConfigs-docs/log.md | 17 +++++++++++++++++ 2 files changed, 19 insertions(+), 1 deletion(-) diff --git a/serverAppConfigs-docs/LLM/dankserver-frontier-moe-feasibility.md b/serverAppConfigs-docs/LLM/dankserver-frontier-moe-feasibility.md index f840288..b5df329 100644 --- a/serverAppConfigs-docs/LLM/dankserver-frontier-moe-feasibility.md +++ b/serverAppConfigs-docs/LLM/dankserver-frontier-moe-feasibility.md @@ -54,7 +54,7 @@ was right. |---|---|---|---|---| | **DeepSeek-V4.1-Flash** | 552B / 8B prefill, 16B decode | 384 top-6, 40 layers, `moe_intermediate_size` 2304, hidden 5120 | CED 20 enc + 20 dec; CSA2 Full/Reindex/Reuse; **FP4 main KV, 890 B/token**; Engram 196B at layers 1 & 14 | Lucebox `feat/ds41-engine` only (unmerged) | | **MiMo-V2.6-Flash-MOPD** | 309B / 15B | 256 top-8, no shared expert, 48 layers (39 SWA / 9 GA) | hybrid SWA (window 128) + GA; 5-layer MTP | vLLM / SGLang, NVIDIA-first. No AMD path | -| **GLM-5.3-Flash** | 320B / 18B | MoE | KDA (linear) + DSA (sparse) hybrid; mHC | llama.cpp (community R9700 guide); TokenSpeed recipe | +| **GLM-5.3-Flash** | 320B / 18B | MoE | KDA (linear) + DSA (sparse) hybrid; mHC | **tcclaviger vLLM fork — plausible native path (2026-09-29 source reading: `glm5next` registered with an `amd/` backend dir + DSA indexer + mHC path; r4d.so carries native mxfp4/dsfp4 MoE entries). Gate: a compressed-tensors MXFP4 payload (~165 GiB, none exists publicly) + a HIP boot probe.** llama.cpp GGUF (community R9700 guide) remains the verified-but-slower path | ## Sizing — the arithmetic that decides it @@ -166,6 +166,7 @@ Not advisable. | Runtime | gfx1201 | Architectures | Note | |---|---|---|---| | **Affinity** | yes, built here | **V4-Flash only** — "the geometry is compiled in; there is no architecture abstraction" | 110 tok/s for 284B. Best engine for this box by an order of magnitude, wrong model | +| **tcclaviger vLLM fork** | **yes — proven 2026-09-29** | qwen4exp measured 37.4-40.6 tok/s decode (see [[dankserver-qwen38-fp8-expert-offload-tuning]]); **`glm5next` registered** (`Glm5NextForCausalLM`/`MTP`, backend-split `amd/`/`common/`/`nvidia/`, `sparse_indexer.py` for DSA); r4d.so has **native mxfp4 + dsfp4 MoE entries** (`has_dsfp4() == True`); W4A16 MoE class (`R4dW4a16MoEExperts`) wired into the same tiered offload; TunableOp silently skipped under offload (needs `CLAV_TUNABLEOP_WITH_OFFLOAD=1`, and then corrupts numerics — do not use) | the ad hoc qwen-fp8 container's runtime; **the old "MXFP4 = emulation/missing kernels" verdict predates this fork and is dead** — the GLM-5.3-Flash gate is now the payload format + a boot probe, not kernels | | **Lucebox** | yes, image on host | Qwen 3.8, DS4; **V4.1 on the unmerged `feat/ds41-engine` branch** | The only V4.1 path. **Refuses layer split and layer-major prefill for V4.1** | | **radiance-vLLM** (drwolfen) | yes — the active frank-primary | welded to `qwen4_exp`; PLE / QSA / packed-HC live inside that model class | 12–20 person-days per architecture | | **FreeToken-ROCm** | partial | block-FP8 safetensors only (modelopt); known `CpuMoeExecutor` issue under ROCm | cannot load a GGUF | diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 2a0151b..b0f0836 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -105,6 +105,23 @@ band, straddling the 35 target within noise. Final 256k config: `MAXLEN=262144 KV_BYTES=4328521728`, prefetch off. The full-speed alternative stays `MAXLEN=184224` at 3 GiB (37.4-40.6 band). + - **GLM-5.3-Flash readiness REVISED by source reading (Frank's question + about int4 vs mxfp4):** the tcclaviger fork in the qwen-fp8 container + registers `Glm5NextForCausalLM`/`Glm5NextMTP` with a backend-split + module (`glm5next/amd|common|nvidia`, `sparse_indexer.py` for DSA); + `offloader/expert.py` accepts `R4dDsfp4MoEExperts, R4dW4a16MoEExperts, + Fp8HipMoEExperts` — and **`R4dDsfp4MoEExperts` is MXFP4 weights with + the DeepSeek activation law** (docstring; SwiGLU `gemm1_clamp_limit` + default 10.0), inheriting the mxfp4 repack/GEMM/emulation-fallback. + **`r4d_lib.has_dsfp4()` is True** in this build — native `moe_dsfp4` / + `moe_dsfp4_mt1` entries exist, so the old "MXFP4 = emulation, missing + kernels" GLM verdict is dead for this fork. WNA16 (int4) backend + oracle on ROCm reduces to HUMMING/TRITON (Marlin/TRTLLM/XPU are + CUDA/NVIDIA/Intel-only), payload must be compressed-tensors W4A16 + group-wise symmetric. Remaining GLM gate: a compressed-tensors MXFP4 + (~165 GiB) or W4A16 payload + a glm5next-on-HIP boot probe; community + GLM-5.3 quants are GGUF (llama.cpp path only), native FP8 is 328 GB + (> 246 GiB RAM). Feasibility doc rows updated. ## 2026-09-28 - **Dankclaw model cleanup** (disk was at 93%, 137 G free). Frank asked to