docs: frontier-MoE feasibility on Dankserver (V4.1 / MiMo / GLM-5.3)
Answers Frank's question: given the 2x R9700 box, which of DeepSeek-V4.1-Flash,
MiMo-V2.6-Flash-MOPD and GLM-5.3-Flash runs best, on what runtime, and can
Affinity / vllm-radiance / FreeToken-AMD be modified to run them fast.
Sizing, measured from the HF API rather than the model cards:
- DS4.1 payload 475.27 GiB, of which 189.13 GiB is Engram (two 94.56 GiB
shards, the layer 1 and 14 lookup tables). Lucebox never maps them, so
they do not evict the expert page cache.
- 240 routed experts x ~11 MB per token at the 37-of-384-per-layer residency
the hot stack can hold = ~2.4 GB across PCIe per token. At 55 GB/s real on
Gen5 x8 that is a ~23 tok/s ceiling, under the 35 tok/s goal before any
attention or dense work.
- The backend refuses layer-major prefill for V4.1, so the only legal
prefill is token-by-token; 1000 t/s is ~20x away.
Affinity (Codeberg StillDeadcode) builds clean here (hipcc 23, gfx1201, zero
errors) and claims 110 tok/s for the 284B V4-Flash, because the 101 GiB
container leaves the cold expert set in a pinned host pool in RAM. V4.1 is
2.5x too big to repeat that. Port scope, from both checkpoints' real
safetensors indices: expert machinery ports unchanged (identical tensor
naming, 15,360 tensors at 40x384), 26/32 per-layer tensors and 34/35 config
keys present. Five gaps - Engram, the disjoint {0,1,2} vs {0,4,128} ratio
vocabulary, CED, the DSpark draft geometry, vision. num_hash_layers is the
only key V4.1 lacks, so model.h:36's default of 3 silently survives and
would route layers 0-2 through a table that does not exist.
Also corrects: Dankserver RAM 60 -> 246 GiB (stale since the 2026-09-20 board
swap) and storage; the top-level index credited V4.1 with V4's geometry
(43 layers / 256 experts vs the actual 40 / 384).
This commit is contained in:
@@ -2,6 +2,43 @@
|
||||
|
||||
|
||||
## 2026-09-28
|
||||
- **Frontier-MoE feasibility study on Dankserver** (research + build, no
|
||||
production change). Frank asked which of DeepSeek-V4.1-Flash,
|
||||
MiMo-V2.6-Flash-MOPD and GLM-5.3-Flash would run best on the 2x R9700 box,
|
||||
and whether Affinity / vllm-radiance / FreeToken-AMD could be modified to run
|
||||
them fast. Findings, all evidenced:
|
||||
- **V4.1 does not fit.** 475.27 GiB checkpoint (measured from the HF API),
|
||||
of which **189.13 GiB is Engram** in two 94.56 GiB shards. A token selects
|
||||
240 routed experts at ~11 MB; at the 37-of-384-per-layer residency the hot
|
||||
stack can hold, ~2.4 GB crosses PCIe per token. At 55 GB/s real that is a
|
||||
**~23 tok/s ceiling** — below the 35 tok/s goal before any attention or
|
||||
dense work. The 1000 t/s prefill target is ~20x away: the backend refuses
|
||||
layer-major prefill for V4.1, so the only legal prefill is token-by-token.
|
||||
- **Affinity** (Codeberg, StillDeadcode) is a *DeepSeek-V4-Flash* engine
|
||||
claiming 110 tok/s on two R9700s; it **builds clean here** (hipcc 23,
|
||||
gfx1201, zero errors). The 110 works because the 101 GiB container leaves
|
||||
the cold expert set in a pinned host pool in RAM. A V4.1 port reuses the
|
||||
quantiser, container, slab, host pool and placement engine (expert tensor
|
||||
naming is identical, 15,360 tensors at 40x384) but must write CED, CSA2's
|
||||
new indexer, the DSpark draft geometry and Engram — and Engram fails
|
||||
*silently*, because V4.1 has no `num_hash_layers` key so Affinity's
|
||||
`model.h:36` default of 3 survives. Weeks of work, same capacity wall.
|
||||
- **Lucebox** is the only engine that runs V4.1 on gfx1201, on the unmerged
|
||||
`feat/ds41-engine` branch. Its own DS41.md measures 0.8-1.9 tok/s on an
|
||||
R9700 (best tuned three-tier case 9.1) with **61 GB of RAM**; Dankserver's
|
||||
246 GiB is the one untested lever, so the 377.5 GiB ROCMFP23 payload is
|
||||
downloading for a three-arm bench.
|
||||
- `/serverAppConfigs-docs/LLM/Dankserver System Specs.md` — **RAM corrected
|
||||
60 GiB -> 246 GiB** (stale since the 2026-09-20 board swap; the Lucebox
|
||||
runbook had it right) and storage refreshed; `/mnt/entropy` is at 100%
|
||||
(244 G) and must not be planned for payload.
|
||||
- `/serverAppConfigs-docs/LLM/dankserver-frontier-moe-feasibility.md` — new
|
||||
runbook; indexed in `LLM/index.md`.
|
||||
- **LANDMINE, not changed here:** `/home/dank/models` was cleared by Frank
|
||||
mid-session to free space. frank-primary is still answering `:7770` from
|
||||
mmapped-but-unlinked inodes (`/models` is empty inside the running
|
||||
container) — **the endpoint cannot be restarted until the 175 GiB payload
|
||||
is re-fetched.** A reboot or a container restart takes frank-primary down.
|
||||
- Deploy-state reconciliation (Frank's call: no restart) — the autoresearch trio
|
||||
(`f187165` PLE 1800, `41e9189` prefix caching, `ba2100e` docs) was found already
|
||||
MERGED to main + pushed AND already live on frank-primary: the running
|
||||
|
||||
Reference in New Issue
Block a user