docs: frontier-MoE feasibility on Dankserver (V4.1 / MiMo / GLM-5.3)

Answers Frank's question: given the 2x R9700 box, which of DeepSeek-V4.1-Flash, MiMo-V2.6-Flash-MOPD and GLM-5.3-Flash runs best, on what runtime, and can Affinity / vllm-radiance / FreeToken-AMD be modified to run them fast. Sizing, measured from the HF API rather than the model cards: - DS4.1 payload 475.27 GiB, of which 189.13 GiB is Engram (two 94.56 GiB shards, the layer 1 and 14 lookup tables). Lucebox never maps them, so they do not evict the expert page cache. - 240 routed experts x ~11 MB per token at the 37-of-384-per-layer residency the hot stack can hold = ~2.4 GB across PCIe per token. At 55 GB/s real on Gen5 x8 that is a ~23 tok/s ceiling, under the 35 tok/s goal before any attention or dense work. - The backend refuses layer-major prefill for V4.1, so the only legal prefill is token-by-token; 1000 t/s is ~20x away. Affinity (Codeberg StillDeadcode) builds clean here (hipcc 23, gfx1201, zero errors) and claims 110 tok/s for the 284B V4-Flash, because the 101 GiB container leaves the cold expert set in a pinned host pool in RAM. V4.1 is 2.5x too big to repeat that. Port scope, from both checkpoints' real safetensors indices: expert machinery ports unchanged (identical tensor naming, 15,360 tensors at 40x384), 26/32 per-layer tensors and 34/35 config keys present. Five gaps - Engram, the disjoint {0,1,2} vs {0,4,128} ratio vocabulary, CED, the DSpark draft geometry, vision. num_hash_layers is the only key V4.1 lacks, so model.h:36's default of 3 silently survives and would route layers 0-2 through a table that does not exist. Also corrects: Dankserver RAM 60 -> 246 GiB (stale since the 2026-09-20 board swap) and storage; the top-level index credited V4.1 with V4's geometry (43 layers / 256 experts vs the actual 40 / 384).
2026-09-28 14:58:59 -07:00
parent 564a781539
commit d29d104f01
5 changed files with 251 additions and 7 deletions
@@ -63,16 +63,16 @@ apt lists remain on disk but no NVIDIA PCI device exists.

 | Property | Value |
 |----------|-------|
-| System RAM | 60 GiB |
-| Swap | 63 GiB |
+| System RAM | **246 GiB** (upgraded for the Lucebox DeepSeek deployment of 2026-09-26; this page said 60 GiB until 2026-09-28 — a stale figure that survived the board swap) |
+| Swap | 63 GiB (`/swapfile`) |
 | /dev/shm | 32 GiB |

 ### Storage

 | Filesystem | Size | Used | Avail | Use% |
 |------------|------|------|-------|------|
-| / (nvme0n1p2) | 915G | ~640G | ~280G | ~70% |
-| /mnt/entropy (dm-0) | 91T | 86T | 910G | 99% |
+| / (nvme0n1p2) | 1.8T | 927G | ~822G | ~54% |
+| /mnt/entropy (dm-0) | 91T | 87T | **244G** | **100%** — full; do not plan model payload here |

 Models live under `/home/dank/models`; the serving build tree under
 `/home/dank/GitPackages/llama.cpp-dflash2` (the older `charlie-ROCmFPX` tree is
@@ -0,0 +1,191 @@
+---
+type: research
+title: Frontier MoE on Dankserver — what fits, on which runtime
+description: VERIFIED 2026-09-28 by measurement and source reading. Compares DeepSeek-V4.1-Flash, MiMo-V2.6-Flash-MOPD and GLM-5.3-Flash against the 2x R9700 (gfx1201) / 246 GiB box, and assesses the Affinity runtime's portability off DeepSeek-V4-Flash. Records the capacity arithmetic that decides the answer before any model is loaded.
+tags:
+  - dankserver
+  - r9700
+  - gfx1201
+  - moe
+  - deepseek-v4.1
+  - deepseek-v4
+  - mimo
+  - glm
+  - affinity
+  - lucebox
+  - feasibility
+timestamp: 2026-09-28
+---
+
+# Frontier MoE on Dankserver — what fits, on which runtime
+
+> **Question this page answers.** Given Dankserver's hardware, which of
+> DeepSeek-V4.1-Flash, MiMo-V2.6-Flash-MOPD and GLM-5.3-Flash runs best, on
+> which runtime, and can Affinity / vllm-radiance / FreeToken-AMD be modified
+> to run them fast?
+>
+> **Answer in one line.** V4.1-Flash does not fit this box at any
+> configuration; the engine that does run it (Lucebox) measures
+> 0.8–9.1 tok/s on R9700 because decode is bound by PCIe expert streaming
+> rather than by the GPUs. The 284B **V4-Flash** on **Affinity** is the only
+> shape that clears 35 tok/s, at 110.
+
+Everything below was measured or read from source on 2026-09-28. Claims
+carried over from the engine authors are marked as theirs.
+
+## Hardware (measured live, 2026-09-28)
+
+| Property | Value |
+|---|---|
+| GPUs | 2x AMD Radeon AI PRO R9700 (Navi 48, **gfx1201**), 32,624 MiB each = **63.7 GiB** total |
+| Interconnect | PCIe 5.0 **x8** each, CPU root ports `00:01.1` / `00:01.3`, ~31.5 GB/s per card |
+| PCIe atomics | `CAP_EXP+28.w = 0x1440` (ReqEn+) on both endpoints — RCCL collectives are legal |
+| CPU | Ryzen 7 7700X — **8 cores / 16 threads** (small; CPU expert compute is expensive) |
+| RAM | **246 GiB** + 63 GiB swap |
+| Disk | 1.8 TB NVMe. `/mnt/entropy` is a separate 91 TB ext4 at **100%** — do not plan payload there |
+
+The knowledge base was wrong about RAM: [[Dankserver System Specs]] said
+60 GiB until 2026-09-28 and has been corrected. The Lucebox runbook's 246 GiB
+was right.
+
+## The three candidates
+
+| Model | Total / active | Routed experts | Attention & KV | Working gfx1201 runtime |
+|---|---|---|---|---|
+| **DeepSeek-V4.1-Flash** | 552B / 8B prefill, 16B decode | 384 top-6, 40 layers, `moe_intermediate_size` 2304, hidden 5120 | CED 20 enc + 20 dec; CSA2 Full/Reindex/Reuse; **FP4 main KV, 890 B/token**; Engram 196B at layers 1 & 14 | Lucebox `feat/ds41-engine` only (unmerged) |
+| **MiMo-V2.6-Flash-MOPD** | 309B / 15B | 256 top-8, no shared expert, 48 layers (39 SWA / 9 GA) | hybrid SWA (window 128) + GA; 5-layer MTP | vLLM / SGLang, NVIDIA-first. No AMD path |
+| **GLM-5.3-Flash** | 320B / 18B | MoE | KDA (linear) + DSA (sparse) hybrid; mHC | llama.cpp (community R9700 guide); TokenSpeed recipe |
+
+## Sizing — the arithmetic that decides it
+
+| Quantity | Value | Note |
+|---|---:|---|
+| DS4.1 official checkpoint | **475.27 GiB** | 48 shards, summed from the HF API |
+| &nbsp;&nbsp;of which Engram | **189.13 GiB** | exactly two 94.56 GiB shards — the layer 1 and 14 lookup tables |
+| DS4.1 routed-expert params | 543.6 B | 40 x 384 x 3 x 5120 x 2304 |
+| &nbsp;&nbsp;shipped as | 269 GiB (fp4) | DeepSeek's own `expert_dtype: fp4` |
+| Lucebox `ROCMFP23.gguf` | **377.51 GiB** | 179 GiB experts (2.5/3.5 bpw) + 189 GiB Engram + ~9.5 GiB dense |
+| V4-Flash official checkpoint | 148.67 GiB | **3.2x smaller than V4.1** |
+| Affinity container for V4-Flash | 101.1 GiB | 92.9 GiB experts @ 2.875 bpw + 6.2 dense + 2.0 embed |
+
+The Engram tables are **never mapped** by Lucebox, so they do not evict the
+expert page cache — that is the one piece of good news in the 377 GiB file.
+
+### The decode wall
+
+A V4.1 token selects `6 x 40 = 240` routed experts at ~11 MB each. At the
+residency the hot stack can hold (37 of 384 per layer, 9.6%), **~2.4 GB of
+expert weights cross PCIe per token**. PCIe 5.0 x8 does ~55 GB/s real:
+
+```
+2.4 GB / 55 GB/s = 44 ms  ->  ~23 tok/s absolute ceiling
+```
+
+That is **below the 35 tok/s target before any attention, Engram or dense work
+is counted.** Raising residency is the only lever, and VRAM is what bounds it:
+63.7 GiB / 11 MB = 5,800 of 15,360 experts = 37.8% maximum, and that assumes
+VRAM holds nothing else.
+
+The engine author measured the same wall on an R9700 with 61 GB of RAM
+(DS41.md, "Measured"): 0.8 tok/s cold / 1.8 tok/s warm at a 6 GB hot budget,
+0.9 / 1.9 at 16 GB, and 9.1 tok/s for the best tuned three-tier case. They
+name the cause: *"90 to 96% of the 240 routed experts a token selects are still
+cold, and decode moves roughly 2.4 GB of expert weights per token through the
+synchronous stream path. That, not the GPU, bounds decode today."*
+
+## Affinity — a DeepSeek-V4-Flash engine, and what a V4.1 port costs
+
+[Affinity](https://codeberg.org/StillDeadcode/affinity) is *"an inference
+engine for DeepSeek-V4-Flash on one or two RDNA4 GPUs"*, C++/HIP, no framework.
+**It builds clean on this host**: CMake 3.24+, `hipcc` 23, `HIP enabled:
+gfx1201`, zero errors, producing `affinity`, `aff-quantize`, `aff-info` and
+`aff-gpucheck`. Its README claims **~110 tok/s on two R9700s** against 8 on
+one. That is 3x Frank's 35 tok/s goal, for the 284B model.
+
+The 110 works because the 101 GiB container leaves the cold expert set in a
+**pinned host pool in RAM** — the SSD tier that Affinity calls "a cliff" is
+never reached. V4.1 would need 182 GiB of experts at the same 2.875 bpw, plus
+189 GiB of Engram, against the same 246 GiB of RAM.
+
+### What a V4.1 port would reuse
+
+The config parser is **fully data-driven** — every dimension is read from
+`config.json`, so nothing is compiled in at the config layer. And the expert
+tensor naming is *identical* between the two checkpoints:
+
+```
+layers.N.ffn.experts.E.w1.weight   (11,008 in V4  = 43 x 256)
+layers.N.ffn.experts.E.w1.weight   (15,360 in V4.1 = 40 x 384)
+```
+
+So the quantiser, the `.aff` container, the expert slab, the host pool and the
+heat/placement engine — **the part that produces the 110 tok/s** — all carry
+over. Hyper-connections are the same family (`hc_mult 4`, Sinkhorn 20).
+
+### What a V4.1 port would have to write
+
+Verified against both checkpoints' real `model.safetensors.index.json` on
+2026-09-28 — not inferred from the config prose.
+
+**Tensor presence.** `aff-quantize` requires 32 per-layer tensors. V4.1
+supplies 26 plus every global (`embed`, `head`, `norm`, `hc_head_*`) and the
+whole expert set. Six are missing:
+
+| Missing in V4.1 | Replaced in V4.1 by |
+|---|---|
+| `ffn.gate.tid2eid` | the Engram tables |
+| `attn.compressor.ape` | nothing — ratio-2 is *pooled, no APE* |
+| `attn.indexer.compressor.{wkv,wgate,ape,norm}.weight` | `attn.indexer.wk` + `attn.indexer.k_norm` |
+
+**Config keys.** Of the 35 Affinity reads, V4.1 supplies **34**. The one it
+lacks is the dangerous one:
+
+| Gap | Detail |
+|---|---|
+| **Engram — fails silently** | `num_hash_layers` (V4: 3) is absent, so `cfg_int` returns its default and `model.h:36`'s `n_hash_layer = 3` survives. It would route layers 0–2 through a table that does not exist: fluent output, wrong numbers, no error. This is the only silent failure; the other four gaps fail loudly. |
+| **Ratio vocabulary is disjoint** | V4 uses `{0, 4, 128}`; V4.1 uses `{0, 1, 2}` — **no overlap in the non-zero case**. Affinity computes `coff = (ratio == 4) ? 2 : 1`, so V4.1's 18 ratio-2 layers get the ratio-1 width; `comp_ape` is requested 38 times for a tensor that does not exist; and the `ratio == 4` indexer branch fires on **0** of the 8 layers that need one (`index_source_layer_ids` = 2, 8, 14, 20, 24, 28, 32, 36) — those 8 would run dense. |
+| **CED** | A 20-layer encoder feeding a 20-layer decoder whose global KV is projected from the final encoder hidden state, through an `aligner`. Affinity's `model.cpp` is a flat layer stack. |
+| **DSpark** | V4.1's draft carries **its own** 128-expert MoE at top-3 (`dspark_n_routed_experts`, `dspark_num_experts_per_tok`), 3 stages rather than 1. Affinity's draft shares the target's slab and routing. |
+| Vision | V4.1 is multimodal; Affinity has no vision path. Text-only is an acceptable loss. |
+
+V4.1 also adds 31 config keys Affinity never reads, including the whole
+`engram_*` block, `kv_source_layer_ids`, `index_source_layer_ids` and the
+candidate block pre-selection.
+
+**Verdict.** The reusable half is the *right* half — the expert quantiser,
+the `.aff` container, the slab, the host pool and the heat/placement engine
+are what produce the 110 tok/s. The rest is 6 tensor bindings, one new ratio
+vocabulary, the CED aligner, a new draft geometry and Engram: weeks of work
+in a third-party codebase, with no cheap way to prove correctness. And it
+meets the same capacity wall regardless, because 182 GiB of experts plus
+189 GiB of Engram against 246 GiB of RAM is exactly the problem Lucebox has.
+Not advisable.
+
+## The other runtimes
+
+| Runtime | gfx1201 | Architectures | Note |
+|---|---|---|---|
+| **Affinity** | yes, built here | **V4-Flash only** — "the geometry is compiled in; there is no architecture abstraction" | 110 tok/s for 284B. Best engine for this box by an order of magnitude, wrong model |
+| **Lucebox** | yes, image on host | Qwen 3.8, DS4; **V4.1 on the unmerged `feat/ds41-engine` branch** | The only V4.1 path. **Refuses layer split and layer-major prefill for V4.1** |
+| **radiance-vLLM** (drwolfen) | yes — the active frank-primary | welded to `qwen4_exp`; PLE / QSA / packed-HC live inside that model class | 12–20 person-days per architecture |
+| **FreeToken-ROCm** | partial | block-FP8 safetensors only (modelopt); known `CpuMoeExecutor` issue under ROCm | cannot load a GGUF |
+| **llama.cpp** | yes | GLM-5.3 confirmed by a community R9700 guide; DS-V4.1 CED unverified | no expert LRU; **8 CPU cores** make CPU expert placement the wrong trade |
+| **TokenSpeed** | claimed | GLM-5.x incl. a 5.3-Flash recipe | NVIDIA-first, unverified on gfx1201 |
+| **KTransformers** | no | GLM-5.3 tutorial exists | built for Intel AMX + NVIDIA |
+
+The two-R9700 three-owner split (primary + secondary + stream) is **not
+available on this host**: the retired quadlet recorded that
+`LUCE_DS4_MOE_TP_INPROC` is qualified for gfx1151+gfx1201 pairs, *not*
+gfx1201+gfx1201. hip:0 takes dense weights and the hot expert stack;
+`LUCE_EXPERT_STREAM_DEVICE=1` gives hip:1 the streamed-expert slot cache.
+
+## Measured on Dankserver
+
+<!-- MEASUREMENT -->
+
+## See Also
+
+- [[frank-primary-lucebox-ds4]] — the retired Lucebox frank-primary runbook
+- [[Dual-R9700-Cooperation]] — the PCIe/atomics facts behind the board swap
+- [[Dankserver System Specs]] — the host (RAM corrected 2026-09-28)
+- [[frank-primary-r9v-radiance-qwen38-q8]] — the ACTIVE frank-primary
@@ -94,6 +94,13 @@ frank-small, port 7771]
  (localhost/lucebox-rocm:gfx1201) serving
  Lucebox/DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB) with streamed experts —
  the fleet's only non-llama.cpp runtime detour
+- [[dankserver-frontier-moe-feasibility]] — VERIFIED 2026-09-28: which of
+  DeepSeek-V4.1-Flash (552B), MiMo-V2.6-Flash-MOPD (309B) and GLM-5.3-Flash
+  (320B) can run on the 2x R9700 box, on what runtime, and why V4.1 does not
+  fit (475 GiB payload, 189 GiB of it Engram; decode is PCIe-expert-stream
+  bound, ~23 tok/s ceiling at today's residency). Includes the Affinity
+  V4-Flash-to-V4.1 port assessment — reusable expert tiering, four
+  architecture gaps, one of them a silent-wrongness trap on the Engram key
 - [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-26 (replaced by
  [[frank-primary-lucebox-ds4]], then
  [[frank-primary-llamacpp-qwen38-flash-next-q8]]): vllm-radiance 0.9.3
@@ -146,9 +146,18 @@ The LLM inference fleet on the two hosts. Full index:
  computed in host RAM, the placement law, the cutover, and the rollback
 - [[frank-primary-lucebox-ds4]] — RETIRED 2026-09-26 (same-day swap;
  rollback target): the Lucebox runtime serving
-  DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB GGUF, 43-layer MoE, 256 routed
-  experts top-6) on Dankserver's R9700 with the routed experts computed
-  on CPU from the mmapped page cache — held :7770 for hours
+  DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB GGUF, **40**-layer MoE, **384**
+  routed experts top-6, 189 GiB of embedded Engram) on Dankserver's R9700
+  with the routed experts streamed from the mmapped page cache — held :7770
+  for hours
+- [[dankserver-frontier-moe-feasibility]] — VERIFIED 2026-09-28: can
+  DeepSeek-V4.1-Flash (552B), MiMo-V2.6-Flash-MOPD (309B) or GLM-5.3-Flash
+  (320B) run fast on the 2x R9700 box, and can Affinity / vllm-radiance /
+  FreeToken-AMD be modified for them? The capacity arithmetic (475 GiB
+  payload, 189 GiB of it Engram, ~2.4 GB of cold expert weights per token
+  across PCIe = a ~23 tok/s ceiling) plus the Affinity V4-Flash-to-V4.1 port
+  assessment — reusable expert tiering, four architecture gaps, one a
+  silent-wrongness trap on the Engram key
 - [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-26:
  vllm-radiance as ONE tensor-parallel process (TP=2 over the R4D one-shot
  allreduce) serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4
@@ -2,6 +2,43 @@


 ## 2026-09-28
+- **Frontier-MoE feasibility study on Dankserver** (research + build, no
+  production change). Frank asked which of DeepSeek-V4.1-Flash,
+  MiMo-V2.6-Flash-MOPD and GLM-5.3-Flash would run best on the 2x R9700 box,
+  and whether Affinity / vllm-radiance / FreeToken-AMD could be modified to run
+  them fast. Findings, all evidenced:
+  - **V4.1 does not fit.** 475.27 GiB checkpoint (measured from the HF API),
+    of which **189.13 GiB is Engram** in two 94.56 GiB shards. A token selects
+    240 routed experts at ~11 MB; at the 37-of-384-per-layer residency the hot
+    stack can hold, ~2.4 GB crosses PCIe per token. At 55 GB/s real that is a
+    **~23 tok/s ceiling** — below the 35 tok/s goal before any attention or
+    dense work. The 1000 t/s prefill target is ~20x away: the backend refuses
+    layer-major prefill for V4.1, so the only legal prefill is token-by-token.
+  - **Affinity** (Codeberg, StillDeadcode) is a *DeepSeek-V4-Flash* engine
+    claiming 110 tok/s on two R9700s; it **builds clean here** (hipcc 23,
+    gfx1201, zero errors). The 110 works because the 101 GiB container leaves
+    the cold expert set in a pinned host pool in RAM. A V4.1 port reuses the
+    quantiser, container, slab, host pool and placement engine (expert tensor
+    naming is identical, 15,360 tensors at 40x384) but must write CED, CSA2's
+    new indexer, the DSpark draft geometry and Engram — and Engram fails
+    *silently*, because V4.1 has no `num_hash_layers` key so Affinity's
+    `model.h:36` default of 3 survives. Weeks of work, same capacity wall.
+  - **Lucebox** is the only engine that runs V4.1 on gfx1201, on the unmerged
+    `feat/ds41-engine` branch. Its own DS41.md measures 0.8-1.9 tok/s on an
+    R9700 (best tuned three-tier case 9.1) with **61 GB of RAM**; Dankserver's
+    246 GiB is the one untested lever, so the 377.5 GiB ROCMFP23 payload is
+    downloading for a three-arm bench.
+ - `/serverAppConfigs-docs/LLM/Dankserver System Specs.md` — **RAM corrected
+   60 GiB -> 246 GiB** (stale since the 2026-09-20 board swap; the Lucebox
+   runbook had it right) and storage refreshed; `/mnt/entropy` is at 100%
+   (244 G) and must not be planned for payload.
+ - `/serverAppConfigs-docs/LLM/dankserver-frontier-moe-feasibility.md` — new
+   runbook; indexed in `LLM/index.md`.
+ - **LANDMINE, not changed here:** `/home/dank/models` was cleared by Frank
+   mid-session to free space. frank-primary is still answering `:7770` from
+   mmapped-but-unlinked inodes (`/models` is empty inside the running
+   container) — **the endpoint cannot be restarted until the 175 GiB payload
+   is re-fetched.** A reboot or a container restart takes frank-primary down.
 - Deploy-state reconciliation (Frank's call: no restart) — the autoresearch trio
  (`f187165` PLE 1800, `41e9189` prefix caching, `ba2100e` docs) was found already
  MERGED to main + pushed AND already live on frank-primary: the running