docs: frontier-MoE feasibility on Dankserver (V4.1 / MiMo / GLM-5.3)

Answers Frank's question: given the 2x R9700 box, which of DeepSeek-V4.1-Flash,
MiMo-V2.6-Flash-MOPD and GLM-5.3-Flash runs best, on what runtime, and can
Affinity / vllm-radiance / FreeToken-AMD be modified to run them fast.

Sizing, measured from the HF API rather than the model cards:
- DS4.1 payload 475.27 GiB, of which 189.13 GiB is Engram (two 94.56 GiB
  shards, the layer 1 and 14 lookup tables). Lucebox never maps them, so
  they do not evict the expert page cache.
- 240 routed experts x ~11 MB per token at the 37-of-384-per-layer residency
  the hot stack can hold = ~2.4 GB across PCIe per token. At 55 GB/s real on
  Gen5 x8 that is a ~23 tok/s ceiling, under the 35 tok/s goal before any
  attention or dense work.
- The backend refuses layer-major prefill for V4.1, so the only legal
  prefill is token-by-token; 1000 t/s is ~20x away.

Affinity (Codeberg StillDeadcode) builds clean here (hipcc 23, gfx1201, zero
errors) and claims 110 tok/s for the 284B V4-Flash, because the 101 GiB
container leaves the cold expert set in a pinned host pool in RAM. V4.1 is
2.5x too big to repeat that. Port scope, from both checkpoints' real
safetensors indices: expert machinery ports unchanged (identical tensor
naming, 15,360 tensors at 40x384), 26/32 per-layer tensors and 34/35 config
keys present. Five gaps - Engram, the disjoint {0,1,2} vs {0,4,128} ratio
vocabulary, CED, the DSpark draft geometry, vision. num_hash_layers is the
only key V4.1 lacks, so model.h:36's default of 3 silently survives and
would route layers 0-2 through a table that does not exist.

Also corrects: Dankserver RAM 60 -> 246 GiB (stale since the 2026-09-20 board
swap) and storage; the top-level index credited V4.1 with V4's geometry
(43 layers / 256 experts vs the actual 40 / 384).
This commit is contained in:
Frank
2026-09-28 14:58:59 -07:00
parent 564a781539
commit d29d104f01
5 changed files with 251 additions and 7 deletions
+12 -3
View File
@@ -146,9 +146,18 @@ The LLM inference fleet on the two hosts. Full index:
computed in host RAM, the placement law, the cutover, and the rollback
- [[frank-primary-lucebox-ds4]] — RETIRED 2026-09-26 (same-day swap;
rollback target): the Lucebox runtime serving
DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB GGUF, 43-layer MoE, 256 routed
experts top-6) on Dankserver's R9700 with the routed experts computed
on CPU from the mmapped page cache — held :7770 for hours
DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB GGUF, **40**-layer MoE, **384**
routed experts top-6, 189 GiB of embedded Engram) on Dankserver's R9700
with the routed experts streamed from the mmapped page cache — held :7770
for hours
- [[dankserver-frontier-moe-feasibility]] — VERIFIED 2026-09-28: can
DeepSeek-V4.1-Flash (552B), MiMo-V2.6-Flash-MOPD (309B) or GLM-5.3-Flash
(320B) run fast on the 2x R9700 box, and can Affinity / vllm-radiance /
FreeToken-AMD be modified for them? The capacity arithmetic (475 GiB
payload, 189 GiB of it Engram, ~2.4 GB of cold expert weights per token
across PCIe = a ~23 tok/s ceiling) plus the Affinity V4-Flash-to-V4.1 port
assessment — reusable expert tiering, four architecture gaps, one a
silent-wrongness trap on the Engram key
- [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-26:
vllm-radiance as ONE tensor-parallel process (TP=2 over the R4D one-shot
allreduce) serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4