docs(fleet): radiance standalone engine probe - qwen4exp refuses fp8 input, quant happens at convert
rad-convert --plan-only with the deployed orcarouter FP8 payload: 'no architecture plugin claims qwen4exp at quantisation fp8; served at (unquantised)'. Our quantization is reproducible from a bf16/abliterated source by recipe (libr4d reads fp8 experts), not from our file. Image pulled on dankserver, no GPU touched.
This commit is contained in:
@@ -168,11 +168,46 @@ Not advisable.
|
||||
| **Affinity** | yes, built here | **V4-Flash only** — "the geometry is compiled in; there is no architecture abstraction" | 110 tok/s for 284B. Best engine for this box by an order of magnitude, wrong model |
|
||||
| **tcclaviger vLLM fork** | **yes — proven 2026-09-29** | qwen4exp measured 37.4-40.6 tok/s decode (see [[dankserver-qwen38-fp8-expert-offload-tuning]]); **`glm5next` registered** (`Glm5NextForCausalLM`/`MTP`, backend-split `amd/`/`common/`/`nvidia/`, `sparse_indexer.py` for DSA); r4d.so has **native mxfp4 + dsfp4 MoE entries** (`has_dsfp4() == True`); W4A16 MoE class (`R4dW4a16MoEExperts`) wired into the same tiered offload; TunableOp silently skipped under offload (needs `CLAV_TUNABLEOP_WITH_OFFLOAD=1`, and then corrupts numerics — do not use) | the ad hoc qwen-fp8 container's runtime; **the old "MXFP4 = emulation/missing kernels" verdict predates this fork and is dead** — the GLM-5.3-Flash gate is now the payload format + a boot probe, not kernels |
|
||||
| **Lucebox** | yes, image on host | Qwen 3.8, DS4; **V4.1 on the unmerged `feat/ds41-engine` branch** | The only V4.1 path. **Refuses layer split and layer-major prefill for V4.1** |
|
||||
| **radiance-vLLM** (drwolfen) | yes — the active frank-primary | welded to `qwen4_exp`; PLE / QSA / packed-HC live inside that model class | 12–20 person-days per architecture |
|
||||
| **radiance-vLLM** (drwolfen) | yes — served frank-primary 2026-09-27 → 09-30 | welded to `qwen4_exp`; PLE / QSA / packed-HC live inside that model class | 12–20 person-days per architecture |
|
||||
| **FreeToken-ROCm** | partial | block-FP8 safetensors only (modelopt); known `CpuMoeExecutor` issue under ROCm | cannot load a GGUF |
|
||||
| **llama.cpp** | yes | GLM-5.3 confirmed by a community R9700 guide; DS-V4.1 CED unverified | no expert LRU; **8 CPU cores** make CPU expert placement the wrong trade |
|
||||
| **TokenSpeed** | claimed | GLM-5.x incl. a 5.3-Flash recipe | NVIDIA-first, unverified on gfx1201 |
|
||||
| **KTransformers** | no | GLM-5.3 tutorial exists | built for Intel AMX + NVIDIA |
|
||||
| **radiance** (StillDeadcode — the standalone C++/HIP engine, NOT the vLLM fork of the same name) | image `stilldeadcode/radiance` prebuilt for gfx1201 | qwen35 dense/MoE at fp8, qwen4exp quantised-at-convert, llama-shaped | see the probe below: our FP8 payload is REFUSED; tiers, MTP and vision native |
|
||||
|
||||
### radiance (standalone) — 2026-10-05 probe
|
||||
|
||||
`codeberg.org/StillDeadcode/radiance` — Deadcode's from-scratch engine
|
||||
(v1.0.13): same author as the `vllm-radiance` fork we ran in September
|
||||
([[frank-medium-radiance-mxfp4]]), an unrelated codebase — C++/HIP, no
|
||||
Python, kernels/architectures/quantisers as `.so` plugins, flags written
|
||||
for two 32 GB R9700s, native Qwen3.8-Flash-Next with `--placement
|
||||
expert_tiered`, MTP and vision.
|
||||
|
||||
Probe with the DEPLOYED payload (`rad-convert --plan-only`, which stops
|
||||
at declare — no GPU touched, no bytes written):
|
||||
|
||||
```
|
||||
E no architecture plugin claims 'qwen4exp' at quantisation 'fp8'.
|
||||
'qwen4exp' is served at: (unquantised).
|
||||
```
|
||||
|
||||
So the 173 G orcarouter FP8 file cannot enter this engine at all: the
|
||||
qwen4exp plugin takes bf16/f16/f32/GGUF in and every quantisation happens
|
||||
at `rad-convert` time by recipe. Not a lock-in to their published quant —
|
||||
recipes are per-weight-class rules (`rtn`/`gptq`/`awq`/`autoround`/
|
||||
`paroquant`/`ggml`, grids from fp8 128×128 to i4/w4nl/w5nl codebooks,
|
||||
custom `--quant` rules go ahead of the file's), and libr4d lists routed
|
||||
experts-as-fp8 among the encodings it reads — so today's quantization is
|
||||
reproducible **from a bf16 source**, not from our file. Uncensored-family
|
||||
bf16 source candidates: `windowsxp811203/Qwen3.8-Flash-Next-Abliterated-GGUF`
|
||||
(ships the bf16 GGUF) and `mradermacher/Qwen3.8-Flash-Next-Uncensored-GGUF`
|
||||
(i1 quants only). 622 G free fits source + container. A trial is:
|
||||
download → `rad-convert` (CPU-side) → stop+mask frank-primary window to
|
||||
serve. Caveats on record: `--tp-wire wht6` is a lossy int8 all-reduce
|
||||
wire; the recurrent GDN state is f32 by default (144 MiB/sequence) with
|
||||
an opt-in f16 that changes generated text; every pool is budgeted
|
||||
explicitly — no profiling pass, no utilisation fraction.
|
||||
|
||||
The two-R9700 three-owner split (primary + secondary + stream) is **not
|
||||
available on this host**: the retired quadlet recorded that
|
||||
|
||||
Reference in New Issue
Block a user