docs: frontier-MoE feasibility on Dankserver (V4.1 / MiMo / GLM-5.3)

Answers Frank's question: given the 2x R9700 box, which of DeepSeek-V4.1-Flash,
MiMo-V2.6-Flash-MOPD and GLM-5.3-Flash runs best, on what runtime, and can
Affinity / vllm-radiance / FreeToken-AMD be modified to run them fast.

Sizing, measured from the HF API rather than the model cards:
- DS4.1 payload 475.27 GiB, of which 189.13 GiB is Engram (two 94.56 GiB
  shards, the layer 1 and 14 lookup tables). Lucebox never maps them, so
  they do not evict the expert page cache.
- 240 routed experts x ~11 MB per token at the 37-of-384-per-layer residency
  the hot stack can hold = ~2.4 GB across PCIe per token. At 55 GB/s real on
  Gen5 x8 that is a ~23 tok/s ceiling, under the 35 tok/s goal before any
  attention or dense work.
- The backend refuses layer-major prefill for V4.1, so the only legal
  prefill is token-by-token; 1000 t/s is ~20x away.

Affinity (Codeberg StillDeadcode) builds clean here (hipcc 23, gfx1201, zero
errors) and claims 110 tok/s for the 284B V4-Flash, because the 101 GiB
container leaves the cold expert set in a pinned host pool in RAM. V4.1 is
2.5x too big to repeat that. Port scope, from both checkpoints' real
safetensors indices: expert machinery ports unchanged (identical tensor
naming, 15,360 tensors at 40x384), 26/32 per-layer tensors and 34/35 config
keys present. Five gaps - Engram, the disjoint {0,1,2} vs {0,4,128} ratio
vocabulary, CED, the DSpark draft geometry, vision. num_hash_layers is the
only key V4.1 lacks, so model.h:36's default of 3 silently survives and
would route layers 0-2 through a table that does not exist.

Also corrects: Dankserver RAM 60 -> 246 GiB (stale since the 2026-09-20 board
swap) and storage; the top-level index credited V4.1 with V4's geometry
(43 layers / 256 experts vs the actual 40 / 384).
This commit is contained in:
Frank
2026-09-28 14:58:59 -07:00
parent 564a781539
commit d29d104f01
5 changed files with 251 additions and 7 deletions
+37
View File
@@ -2,6 +2,43 @@
## 2026-09-28
- **Frontier-MoE feasibility study on Dankserver** (research + build, no
production change). Frank asked which of DeepSeek-V4.1-Flash,
MiMo-V2.6-Flash-MOPD and GLM-5.3-Flash would run best on the 2x R9700 box,
and whether Affinity / vllm-radiance / FreeToken-AMD could be modified to run
them fast. Findings, all evidenced:
- **V4.1 does not fit.** 475.27 GiB checkpoint (measured from the HF API),
of which **189.13 GiB is Engram** in two 94.56 GiB shards. A token selects
240 routed experts at ~11 MB; at the 37-of-384-per-layer residency the hot
stack can hold, ~2.4 GB crosses PCIe per token. At 55 GB/s real that is a
**~23 tok/s ceiling** — below the 35 tok/s goal before any attention or
dense work. The 1000 t/s prefill target is ~20x away: the backend refuses
layer-major prefill for V4.1, so the only legal prefill is token-by-token.
- **Affinity** (Codeberg, StillDeadcode) is a *DeepSeek-V4-Flash* engine
claiming 110 tok/s on two R9700s; it **builds clean here** (hipcc 23,
gfx1201, zero errors). The 110 works because the 101 GiB container leaves
the cold expert set in a pinned host pool in RAM. A V4.1 port reuses the
quantiser, container, slab, host pool and placement engine (expert tensor
naming is identical, 15,360 tensors at 40x384) but must write CED, CSA2's
new indexer, the DSpark draft geometry and Engram — and Engram fails
*silently*, because V4.1 has no `num_hash_layers` key so Affinity's
`model.h:36` default of 3 survives. Weeks of work, same capacity wall.
- **Lucebox** is the only engine that runs V4.1 on gfx1201, on the unmerged
`feat/ds41-engine` branch. Its own DS41.md measures 0.8-1.9 tok/s on an
R9700 (best tuned three-tier case 9.1) with **61 GB of RAM**; Dankserver's
246 GiB is the one untested lever, so the 377.5 GiB ROCMFP23 payload is
downloading for a three-arm bench.
- `/serverAppConfigs-docs/LLM/Dankserver System Specs.md` — **RAM corrected
60 GiB -> 246 GiB** (stale since the 2026-09-20 board swap; the Lucebox
runbook had it right) and storage refreshed; `/mnt/entropy` is at 100%
(244 G) and must not be planned for payload.
- `/serverAppConfigs-docs/LLM/dankserver-frontier-moe-feasibility.md` — new
runbook; indexed in `LLM/index.md`.
- **LANDMINE, not changed here:** `/home/dank/models` was cleared by Frank
mid-session to free space. frank-primary is still answering `:7770` from
mmapped-but-unlinked inodes (`/models` is empty inside the running
container) — **the endpoint cannot be restarted until the 175 GiB payload
is re-fetched.** A reboot or a container restart takes frank-primary down.
- Deploy-state reconciliation (Frank's call: no restart) — the autoresearch trio
(`f187165` PLE 1800, `41e9189` prefix caching, `ba2100e` docs) was found already
MERGED to main + pushed AND already live on frank-primary: the running