diff --git a/serverAppConfigs-docs/LLM/Dankserver System Specs.md b/serverAppConfigs-docs/LLM/Dankserver System Specs.md index 4f11986..9573e3c 100644 --- a/serverAppConfigs-docs/LLM/Dankserver System Specs.md +++ b/serverAppConfigs-docs/LLM/Dankserver System Specs.md @@ -63,16 +63,16 @@ apt lists remain on disk but no NVIDIA PCI device exists. | Property | Value | |----------|-------| -| System RAM | 60 GiB | -| Swap | 63 GiB | +| System RAM | **246 GiB** (upgraded for the Lucebox DeepSeek deployment of 2026-09-26; this page said 60 GiB until 2026-09-28 — a stale figure that survived the board swap) | +| Swap | 63 GiB (`/swapfile`) | | /dev/shm | 32 GiB | ### Storage | Filesystem | Size | Used | Avail | Use% | |------------|------|------|-------|------| -| / (nvme0n1p2) | 915G | ~640G | ~280G | ~70% | -| /mnt/entropy (dm-0) | 91T | 86T | 910G | 99% | +| / (nvme0n1p2) | 1.8T | 927G | ~822G | ~54% | +| /mnt/entropy (dm-0) | 91T | 87T | **244G** | **100%** — full; do not plan model payload here | Models live under `/home/dank/models`; the serving build tree under `/home/dank/GitPackages/llama.cpp-dflash2` (the older `charlie-ROCmFPX` tree is diff --git a/serverAppConfigs-docs/LLM/dankserver-frontier-moe-feasibility.md b/serverAppConfigs-docs/LLM/dankserver-frontier-moe-feasibility.md new file mode 100644 index 0000000..f840288 --- /dev/null +++ b/serverAppConfigs-docs/LLM/dankserver-frontier-moe-feasibility.md @@ -0,0 +1,191 @@ +--- +type: research +title: Frontier MoE on Dankserver — what fits, on which runtime +description: VERIFIED 2026-09-28 by measurement and source reading. Compares DeepSeek-V4.1-Flash, MiMo-V2.6-Flash-MOPD and GLM-5.3-Flash against the 2x R9700 (gfx1201) / 246 GiB box, and assesses the Affinity runtime's portability off DeepSeek-V4-Flash. Records the capacity arithmetic that decides the answer before any model is loaded. +tags: + - dankserver + - r9700 + - gfx1201 + - moe + - deepseek-v4.1 + - deepseek-v4 + - mimo + - glm + - affinity + - lucebox + - feasibility +timestamp: 2026-09-28 +--- + +# Frontier MoE on Dankserver — what fits, on which runtime + +> **Question this page answers.** Given Dankserver's hardware, which of +> DeepSeek-V4.1-Flash, MiMo-V2.6-Flash-MOPD and GLM-5.3-Flash runs best, on +> which runtime, and can Affinity / vllm-radiance / FreeToken-AMD be modified +> to run them fast? +> +> **Answer in one line.** V4.1-Flash does not fit this box at any +> configuration; the engine that does run it (Lucebox) measures +> 0.8–9.1 tok/s on R9700 because decode is bound by PCIe expert streaming +> rather than by the GPUs. The 284B **V4-Flash** on **Affinity** is the only +> shape that clears 35 tok/s, at 110. + +Everything below was measured or read from source on 2026-09-28. Claims +carried over from the engine authors are marked as theirs. + +## Hardware (measured live, 2026-09-28) + +| Property | Value | +|---|---| +| GPUs | 2x AMD Radeon AI PRO R9700 (Navi 48, **gfx1201**), 32,624 MiB each = **63.7 GiB** total | +| Interconnect | PCIe 5.0 **x8** each, CPU root ports `00:01.1` / `00:01.3`, ~31.5 GB/s per card | +| PCIe atomics | `CAP_EXP+28.w = 0x1440` (ReqEn+) on both endpoints — RCCL collectives are legal | +| CPU | Ryzen 7 7700X — **8 cores / 16 threads** (small; CPU expert compute is expensive) | +| RAM | **246 GiB** + 63 GiB swap | +| Disk | 1.8 TB NVMe. `/mnt/entropy` is a separate 91 TB ext4 at **100%** — do not plan payload there | + +The knowledge base was wrong about RAM: [[Dankserver System Specs]] said +60 GiB until 2026-09-28 and has been corrected. The Lucebox runbook's 246 GiB +was right. + +## The three candidates + +| Model | Total / active | Routed experts | Attention & KV | Working gfx1201 runtime | +|---|---|---|---|---| +| **DeepSeek-V4.1-Flash** | 552B / 8B prefill, 16B decode | 384 top-6, 40 layers, `moe_intermediate_size` 2304, hidden 5120 | CED 20 enc + 20 dec; CSA2 Full/Reindex/Reuse; **FP4 main KV, 890 B/token**; Engram 196B at layers 1 & 14 | Lucebox `feat/ds41-engine` only (unmerged) | +| **MiMo-V2.6-Flash-MOPD** | 309B / 15B | 256 top-8, no shared expert, 48 layers (39 SWA / 9 GA) | hybrid SWA (window 128) + GA; 5-layer MTP | vLLM / SGLang, NVIDIA-first. No AMD path | +| **GLM-5.3-Flash** | 320B / 18B | MoE | KDA (linear) + DSA (sparse) hybrid; mHC | llama.cpp (community R9700 guide); TokenSpeed recipe | + +## Sizing — the arithmetic that decides it + +| Quantity | Value | Note | +|---|---:|---| +| DS4.1 official checkpoint | **475.27 GiB** | 48 shards, summed from the HF API | +|   of which Engram | **189.13 GiB** | exactly two 94.56 GiB shards — the layer 1 and 14 lookup tables | +| DS4.1 routed-expert params | 543.6 B | 40 x 384 x 3 x 5120 x 2304 | +|   shipped as | 269 GiB (fp4) | DeepSeek's own `expert_dtype: fp4` | +| Lucebox `ROCMFP23.gguf` | **377.51 GiB** | 179 GiB experts (2.5/3.5 bpw) + 189 GiB Engram + ~9.5 GiB dense | +| V4-Flash official checkpoint | 148.67 GiB | **3.2x smaller than V4.1** | +| Affinity container for V4-Flash | 101.1 GiB | 92.9 GiB experts @ 2.875 bpw + 6.2 dense + 2.0 embed | + +The Engram tables are **never mapped** by Lucebox, so they do not evict the +expert page cache — that is the one piece of good news in the 377 GiB file. + +### The decode wall + +A V4.1 token selects `6 x 40 = 240` routed experts at ~11 MB each. At the +residency the hot stack can hold (37 of 384 per layer, 9.6%), **~2.4 GB of +expert weights cross PCIe per token**. PCIe 5.0 x8 does ~55 GB/s real: + +``` +2.4 GB / 55 GB/s = 44 ms -> ~23 tok/s absolute ceiling +``` + +That is **below the 35 tok/s target before any attention, Engram or dense work +is counted.** Raising residency is the only lever, and VRAM is what bounds it: +63.7 GiB / 11 MB = 5,800 of 15,360 experts = 37.8% maximum, and that assumes +VRAM holds nothing else. + +The engine author measured the same wall on an R9700 with 61 GB of RAM +(DS41.md, "Measured"): 0.8 tok/s cold / 1.8 tok/s warm at a 6 GB hot budget, +0.9 / 1.9 at 16 GB, and 9.1 tok/s for the best tuned three-tier case. They +name the cause: *"90 to 96% of the 240 routed experts a token selects are still +cold, and decode moves roughly 2.4 GB of expert weights per token through the +synchronous stream path. That, not the GPU, bounds decode today."* + +## Affinity — a DeepSeek-V4-Flash engine, and what a V4.1 port costs + +[Affinity](https://codeberg.org/StillDeadcode/affinity) is *"an inference +engine for DeepSeek-V4-Flash on one or two RDNA4 GPUs"*, C++/HIP, no framework. +**It builds clean on this host**: CMake 3.24+, `hipcc` 23, `HIP enabled: +gfx1201`, zero errors, producing `affinity`, `aff-quantize`, `aff-info` and +`aff-gpucheck`. Its README claims **~110 tok/s on two R9700s** against 8 on +one. That is 3x Frank's 35 tok/s goal, for the 284B model. + +The 110 works because the 101 GiB container leaves the cold expert set in a +**pinned host pool in RAM** — the SSD tier that Affinity calls "a cliff" is +never reached. V4.1 would need 182 GiB of experts at the same 2.875 bpw, plus +189 GiB of Engram, against the same 246 GiB of RAM. + +### What a V4.1 port would reuse + +The config parser is **fully data-driven** — every dimension is read from +`config.json`, so nothing is compiled in at the config layer. And the expert +tensor naming is *identical* between the two checkpoints: + +``` +layers.N.ffn.experts.E.w1.weight (11,008 in V4 = 43 x 256) +layers.N.ffn.experts.E.w1.weight (15,360 in V4.1 = 40 x 384) +``` + +So the quantiser, the `.aff` container, the expert slab, the host pool and the +heat/placement engine — **the part that produces the 110 tok/s** — all carry +over. Hyper-connections are the same family (`hc_mult 4`, Sinkhorn 20). + +### What a V4.1 port would have to write + +Verified against both checkpoints' real `model.safetensors.index.json` on +2026-09-28 — not inferred from the config prose. + +**Tensor presence.** `aff-quantize` requires 32 per-layer tensors. V4.1 +supplies 26 plus every global (`embed`, `head`, `norm`, `hc_head_*`) and the +whole expert set. Six are missing: + +| Missing in V4.1 | Replaced in V4.1 by | +|---|---| +| `ffn.gate.tid2eid` | the Engram tables | +| `attn.compressor.ape` | nothing — ratio-2 is *pooled, no APE* | +| `attn.indexer.compressor.{wkv,wgate,ape,norm}.weight` | `attn.indexer.wk` + `attn.indexer.k_norm` | + +**Config keys.** Of the 35 Affinity reads, V4.1 supplies **34**. The one it +lacks is the dangerous one: + +| Gap | Detail | +|---|---| +| **Engram — fails silently** | `num_hash_layers` (V4: 3) is absent, so `cfg_int` returns its default and `model.h:36`'s `n_hash_layer = 3` survives. It would route layers 0–2 through a table that does not exist: fluent output, wrong numbers, no error. This is the only silent failure; the other four gaps fail loudly. | +| **Ratio vocabulary is disjoint** | V4 uses `{0, 4, 128}`; V4.1 uses `{0, 1, 2}` — **no overlap in the non-zero case**. Affinity computes `coff = (ratio == 4) ? 2 : 1`, so V4.1's 18 ratio-2 layers get the ratio-1 width; `comp_ape` is requested 38 times for a tensor that does not exist; and the `ratio == 4` indexer branch fires on **0** of the 8 layers that need one (`index_source_layer_ids` = 2, 8, 14, 20, 24, 28, 32, 36) — those 8 would run dense. | +| **CED** | A 20-layer encoder feeding a 20-layer decoder whose global KV is projected from the final encoder hidden state, through an `aligner`. Affinity's `model.cpp` is a flat layer stack. | +| **DSpark** | V4.1's draft carries **its own** 128-expert MoE at top-3 (`dspark_n_routed_experts`, `dspark_num_experts_per_tok`), 3 stages rather than 1. Affinity's draft shares the target's slab and routing. | +| Vision | V4.1 is multimodal; Affinity has no vision path. Text-only is an acceptable loss. | + +V4.1 also adds 31 config keys Affinity never reads, including the whole +`engram_*` block, `kv_source_layer_ids`, `index_source_layer_ids` and the +candidate block pre-selection. + +**Verdict.** The reusable half is the *right* half — the expert quantiser, +the `.aff` container, the slab, the host pool and the heat/placement engine +are what produce the 110 tok/s. The rest is 6 tensor bindings, one new ratio +vocabulary, the CED aligner, a new draft geometry and Engram: weeks of work +in a third-party codebase, with no cheap way to prove correctness. And it +meets the same capacity wall regardless, because 182 GiB of experts plus +189 GiB of Engram against 246 GiB of RAM is exactly the problem Lucebox has. +Not advisable. + +## The other runtimes + +| Runtime | gfx1201 | Architectures | Note | +|---|---|---|---| +| **Affinity** | yes, built here | **V4-Flash only** — "the geometry is compiled in; there is no architecture abstraction" | 110 tok/s for 284B. Best engine for this box by an order of magnitude, wrong model | +| **Lucebox** | yes, image on host | Qwen 3.8, DS4; **V4.1 on the unmerged `feat/ds41-engine` branch** | The only V4.1 path. **Refuses layer split and layer-major prefill for V4.1** | +| **radiance-vLLM** (drwolfen) | yes — the active frank-primary | welded to `qwen4_exp`; PLE / QSA / packed-HC live inside that model class | 12–20 person-days per architecture | +| **FreeToken-ROCm** | partial | block-FP8 safetensors only (modelopt); known `CpuMoeExecutor` issue under ROCm | cannot load a GGUF | +| **llama.cpp** | yes | GLM-5.3 confirmed by a community R9700 guide; DS-V4.1 CED unverified | no expert LRU; **8 CPU cores** make CPU expert placement the wrong trade | +| **TokenSpeed** | claimed | GLM-5.x incl. a 5.3-Flash recipe | NVIDIA-first, unverified on gfx1201 | +| **KTransformers** | no | GLM-5.3 tutorial exists | built for Intel AMX + NVIDIA | + +The two-R9700 three-owner split (primary + secondary + stream) is **not +available on this host**: the retired quadlet recorded that +`LUCE_DS4_MOE_TP_INPROC` is qualified for gfx1151+gfx1201 pairs, *not* +gfx1201+gfx1201. hip:0 takes dense weights and the hot expert stack; +`LUCE_EXPERT_STREAM_DEVICE=1` gives hip:1 the streamed-expert slot cache. + +## Measured on Dankserver + + + +## See Also + +- [[frank-primary-lucebox-ds4]] — the retired Lucebox frank-primary runbook +- [[Dual-R9700-Cooperation]] — the PCIe/atomics facts behind the board swap +- [[Dankserver System Specs]] — the host (RAM corrected 2026-09-28) +- [[frank-primary-r9v-radiance-qwen38-q8]] — the ACTIVE frank-primary diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index c7e4f25..aaf352f 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -94,6 +94,13 @@ frank-small, port 7771] (localhost/lucebox-rocm:gfx1201) serving Lucebox/DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB) with streamed experts — the fleet's only non-llama.cpp runtime detour +- [[dankserver-frontier-moe-feasibility]] — VERIFIED 2026-09-28: which of + DeepSeek-V4.1-Flash (552B), MiMo-V2.6-Flash-MOPD (309B) and GLM-5.3-Flash + (320B) can run on the 2x R9700 box, on what runtime, and why V4.1 does not + fit (475 GiB payload, 189 GiB of it Engram; decode is PCIe-expert-stream + bound, ~23 tok/s ceiling at today's residency). Includes the Affinity + V4-Flash-to-V4.1 port assessment — reusable expert tiering, four + architecture gaps, one of them a silent-wrongness trap on the Engram key - [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-26 (replaced by [[frank-primary-lucebox-ds4]], then [[frank-primary-llamacpp-qwen38-flash-next-q8]]): vllm-radiance 0.9.3 diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index fa9a410..1ea000a 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -146,9 +146,18 @@ The LLM inference fleet on the two hosts. Full index: computed in host RAM, the placement law, the cutover, and the rollback - [[frank-primary-lucebox-ds4]] — RETIRED 2026-09-26 (same-day swap; rollback target): the Lucebox runtime serving - DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB GGUF, 43-layer MoE, 256 routed - experts top-6) on Dankserver's R9700 with the routed experts computed - on CPU from the mmapped page cache — held :7770 for hours + DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB GGUF, **40**-layer MoE, **384** + routed experts top-6, 189 GiB of embedded Engram) on Dankserver's R9700 + with the routed experts streamed from the mmapped page cache — held :7770 + for hours +- [[dankserver-frontier-moe-feasibility]] — VERIFIED 2026-09-28: can + DeepSeek-V4.1-Flash (552B), MiMo-V2.6-Flash-MOPD (309B) or GLM-5.3-Flash + (320B) run fast on the 2x R9700 box, and can Affinity / vllm-radiance / + FreeToken-AMD be modified for them? The capacity arithmetic (475 GiB + payload, 189 GiB of it Engram, ~2.4 GB of cold expert weights per token + across PCIe = a ~23 tok/s ceiling) plus the Affinity V4-Flash-to-V4.1 port + assessment — reusable expert tiering, four architecture gaps, one a + silent-wrongness trap on the Engram key - [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-26: vllm-radiance as ONE tensor-parallel process (TP=2 over the R4D one-shot allreduce) serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4 diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 75fd7e8..993b937 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -2,6 +2,43 @@ ## 2026-09-28 +- **Frontier-MoE feasibility study on Dankserver** (research + build, no + production change). Frank asked which of DeepSeek-V4.1-Flash, + MiMo-V2.6-Flash-MOPD and GLM-5.3-Flash would run best on the 2x R9700 box, + and whether Affinity / vllm-radiance / FreeToken-AMD could be modified to run + them fast. Findings, all evidenced: + - **V4.1 does not fit.** 475.27 GiB checkpoint (measured from the HF API), + of which **189.13 GiB is Engram** in two 94.56 GiB shards. A token selects + 240 routed experts at ~11 MB; at the 37-of-384-per-layer residency the hot + stack can hold, ~2.4 GB crosses PCIe per token. At 55 GB/s real that is a + **~23 tok/s ceiling** — below the 35 tok/s goal before any attention or + dense work. The 1000 t/s prefill target is ~20x away: the backend refuses + layer-major prefill for V4.1, so the only legal prefill is token-by-token. + - **Affinity** (Codeberg, StillDeadcode) is a *DeepSeek-V4-Flash* engine + claiming 110 tok/s on two R9700s; it **builds clean here** (hipcc 23, + gfx1201, zero errors). The 110 works because the 101 GiB container leaves + the cold expert set in a pinned host pool in RAM. A V4.1 port reuses the + quantiser, container, slab, host pool and placement engine (expert tensor + naming is identical, 15,360 tensors at 40x384) but must write CED, CSA2's + new indexer, the DSpark draft geometry and Engram — and Engram fails + *silently*, because V4.1 has no `num_hash_layers` key so Affinity's + `model.h:36` default of 3 survives. Weeks of work, same capacity wall. + - **Lucebox** is the only engine that runs V4.1 on gfx1201, on the unmerged + `feat/ds41-engine` branch. Its own DS41.md measures 0.8-1.9 tok/s on an + R9700 (best tuned three-tier case 9.1) with **61 GB of RAM**; Dankserver's + 246 GiB is the one untested lever, so the 377.5 GiB ROCMFP23 payload is + downloading for a three-arm bench. + - `/serverAppConfigs-docs/LLM/Dankserver System Specs.md` — **RAM corrected + 60 GiB -> 246 GiB** (stale since the 2026-09-20 board swap; the Lucebox + runbook had it right) and storage refreshed; `/mnt/entropy` is at 100% + (244 G) and must not be planned for payload. + - `/serverAppConfigs-docs/LLM/dankserver-frontier-moe-feasibility.md` — new + runbook; indexed in `LLM/index.md`. + - **LANDMINE, not changed here:** `/home/dank/models` was cleared by Frank + mid-session to free space. frank-primary is still answering `:7770` from + mmapped-but-unlinked inodes (`/models` is empty inside the running + container) — **the endpoint cannot be restarted until the 175 GiB payload + is re-fetched.** A reboot or a container restart takes frank-primary down. - Deploy-state reconciliation (Frank's call: no restart) — the autoresearch trio (`f187165` PLE 1800, `41e9189` prefix caching, `ba2100e` docs) was found already MERGED to main + pushed AND already live on frank-primary: the running