docs: frontier-MoE feasibility on Dankserver (V4.1 / MiMo / GLM-5.3)
Answers Frank's question: given the 2x R9700 box, which of DeepSeek-V4.1-Flash,
MiMo-V2.6-Flash-MOPD and GLM-5.3-Flash runs best, on what runtime, and can
Affinity / vllm-radiance / FreeToken-AMD be modified to run them fast.
Sizing, measured from the HF API rather than the model cards:
- DS4.1 payload 475.27 GiB, of which 189.13 GiB is Engram (two 94.56 GiB
shards, the layer 1 and 14 lookup tables). Lucebox never maps them, so
they do not evict the expert page cache.
- 240 routed experts x ~11 MB per token at the 37-of-384-per-layer residency
the hot stack can hold = ~2.4 GB across PCIe per token. At 55 GB/s real on
Gen5 x8 that is a ~23 tok/s ceiling, under the 35 tok/s goal before any
attention or dense work.
- The backend refuses layer-major prefill for V4.1, so the only legal
prefill is token-by-token; 1000 t/s is ~20x away.
Affinity (Codeberg StillDeadcode) builds clean here (hipcc 23, gfx1201, zero
errors) and claims 110 tok/s for the 284B V4-Flash, because the 101 GiB
container leaves the cold expert set in a pinned host pool in RAM. V4.1 is
2.5x too big to repeat that. Port scope, from both checkpoints' real
safetensors indices: expert machinery ports unchanged (identical tensor
naming, 15,360 tensors at 40x384), 26/32 per-layer tensors and 34/35 config
keys present. Five gaps - Engram, the disjoint {0,1,2} vs {0,4,128} ratio
vocabulary, CED, the DSpark draft geometry, vision. num_hash_layers is the
only key V4.1 lacks, so model.h:36's default of 3 silently survives and
would route layers 0-2 through a table that does not exist.
Also corrects: Dankserver RAM 60 -> 246 GiB (stale since the 2026-09-20 board
swap) and storage; the top-level index credited V4.1 with V4's geometry
(43 layers / 256 experts vs the actual 40 / 384).
This commit is contained in:
@@ -94,6 +94,13 @@ frank-small, port 7771]
|
||||
(localhost/lucebox-rocm:gfx1201) serving
|
||||
Lucebox/DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB) with streamed experts —
|
||||
the fleet's only non-llama.cpp runtime detour
|
||||
- [[dankserver-frontier-moe-feasibility]] — VERIFIED 2026-09-28: which of
|
||||
DeepSeek-V4.1-Flash (552B), MiMo-V2.6-Flash-MOPD (309B) and GLM-5.3-Flash
|
||||
(320B) can run on the 2x R9700 box, on what runtime, and why V4.1 does not
|
||||
fit (475 GiB payload, 189 GiB of it Engram; decode is PCIe-expert-stream
|
||||
bound, ~23 tok/s ceiling at today's residency). Includes the Affinity
|
||||
V4-Flash-to-V4.1 port assessment — reusable expert tiering, four
|
||||
architecture gaps, one of them a silent-wrongness trap on the Engram key
|
||||
- [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-26 (replaced by
|
||||
[[frank-primary-lucebox-ds4]], then
|
||||
[[frank-primary-llamacpp-qwen38-flash-next-q8]]): vllm-radiance 0.9.3
|
||||
|
||||
Reference in New Issue
Block a user