Frank
a5fe045b58
Structural repair: - 13 pages were the same document twice (each exactly 2x its HEAD size), a second frontmatter block whose closing fence had been glued onto a body line by a frontmatter rewrite that re-read its own output. Un-doubled by cutting at the second frontmatter and keeping the copy whose body is a superset; the orphan census reports 0 lost tokens against HEAD. - log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first — 66 dated headings merged into 34, titles demoted to bold leads, every content line verified present. The file had been descending at the head and ascending at the tail. Ownership moves: - New `LLM/frank-medium.md` is the locator for the retired Dankserver endpoint: era windows quoted from each era page's own banner, no numbers restated, and the two unresolved date conflicts recorded instead of silently resolved. `frank-archon.md` links to it instead of restating the pre-rename window. - The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone; `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor verdicts and cross-split spec decode, and the four links that pointed at the stubs were repointed at it or at the drafter page. - Authentik's three stub pages fold into `Authentik.md`, with the process detail and the outpost reference split out so the parent stays under the length hint. Verified corrections (host census + live probe, 2026-10-19): - The MXFP4 checkpoint two pages called the "on-disk rollback" is gone: /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on the era page that owns that payload; the swap page now links to it. - frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2 arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run census is kept once (155 requests) instead of twice (~120 and 155). - Fleet shapes re-confirmed against the live endpoints before touching any page: 3x200000 fp8 KV, 2x200704, 2x262144. Index rows are regenerated, not hand-carved. Net -1,849 vault lines.
13 KiB
type, title, description, tags, timestamp
| type | title | description | tags | timestamp | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| reference | Frontier MoE on Dankserver — what fits, on which runtime | The capacity arithmetic that decides whether a frontier mixture-of-experts fits this box before any model is loaded. |
|
2026-09-28 |
Frontier MoE on Dankserver — what fits, on which runtime
Question this page answers. Given Dankserver's hardware, which of DeepSeek-V4.1-Flash, MiMo-V2.6-Flash-MOPD and GLM-5.3-Flash runs best, on which runtime, and can Affinity / vllm-radiance / FreeToken-AMD be modified to run them fast?
Answer in one line. V4.1-Flash does not fit this box at any configuration; the engine that does run it (Lucebox) measures 0.8–9.1 tok/s on R9700 because decode is bound by PCIe expert streaming rather than by the GPUs. The 284B V4-Flash on Affinity is the only shape that clears 35 tok/s, at 110.
Everything below was measured or read from source on 2026-09-28. Claims carried over from the engine authors are marked as theirs.
Hardware (measured live, 2026-09-28)
| Property | Value |
|---|---|
| GPUs | 2x AMD Radeon AI PRO R9700 (Navi 48, gfx1201), 32,624 MiB each = 63.7 GiB total |
| Interconnect | PCIe 5.0 x8 each, CPU root ports 00:01.1 / 00:01.3, ~31.5 GB/s per card |
| PCIe atomics | CAP_EXP+28.w = 0x1440 (ReqEn+) on both endpoints — RCCL collectives are legal |
| CPU | Ryzen 7 7700X — 8 cores / 16 threads (small; CPU expert compute is expensive) |
| RAM | 246 GiB + 63 GiB swap |
| Disk | 1.8 TB NVMe. /mnt/entropy is a separate 91 TB ext4 at 100% — do not plan payload there |
The knowledge base was wrong about RAM: Dankserver System Specs said 60 GiB until 2026-09-28 and has been corrected. The Lucebox runbook's 246 GiB was right.
The three candidates
| Model | Total / active | Routed experts | Attention & KV | Working gfx1201 runtime |
|---|---|---|---|---|
| DeepSeek-V4.1-Flash | 552B / 8B prefill, 16B decode | 384 top-6, 40 layers, moe_intermediate_size 2304, hidden 5120 |
CED 20 enc + 20 dec; CSA2 Full/Reindex/Reuse; FP4 main KV, 890 B/token; Engram 196B at layers 1 & 14 | Lucebox feat/ds41-engine only (unmerged) |
| MiMo-V2.6-Flash-MOPD | 309B / 15B | 256 top-8, no shared expert, 48 layers (39 SWA / 9 GA) | hybrid SWA (window 128) + GA; 5-layer MTP | vLLM / SGLang, NVIDIA-first. No AMD path |
| GLM-5.3-Flash | 320B / 18B | MoE | KDA (linear) + DSA (sparse) hybrid; mHC | tcclaviger vLLM fork — plausible native path (2026-09-29 source reading: glm5next registered with an amd/ backend dir + DSA indexer + mHC path; r4d.so carries native mxfp4/dsfp4 MoE entries). Gate: a compressed-tensors MXFP4 payload (~165 GiB, none exists publicly) + a HIP boot probe. llama.cpp GGUF (community R9700 guide) remains the verified-but-slower path |
Sizing — the arithmetic that decides it
| Quantity | Value | Note |
|---|---|---|
| DS4.1 official checkpoint | 475.27 GiB | 48 shards, summed from the HF API |
| of which Engram | 189.13 GiB | exactly two 94.56 GiB shards — the layer 1 and 14 lookup tables |
| DS4.1 routed-expert params | 543.6 B | 40 x 384 x 3 x 5120 x 2304 |
| shipped as | 269 GiB (fp4) | DeepSeek's own expert_dtype: fp4 |
Lucebox ROCMFP23.gguf |
377.51 GiB | 179 GiB experts (2.5/3.5 bpw) + 189 GiB Engram + ~9.5 GiB dense |
| V4-Flash official checkpoint | 148.67 GiB | 3.2x smaller than V4.1 |
| Affinity container for V4-Flash | 101.1 GiB | 92.9 GiB experts @ 2.875 bpw + 6.2 dense + 2.0 embed |
The Engram tables are never mapped by Lucebox, so they do not evict the expert page cache — that is the one piece of good news in the 377 GiB file.
The decode wall
A V4.1 token selects 6 x 40 = 240 routed experts at ~11 MB each. At the
residency the hot stack can hold (37 of 384 per layer, 9.6%), ~2.4 GB of
expert weights cross PCIe per token. PCIe 5.0 x8 does ~55 GB/s real:
2.4 GB / 55 GB/s = 44 ms -> ~23 tok/s absolute ceiling
That is below the 35 tok/s target before any attention, Engram or dense work is counted. Raising residency is the only lever, and VRAM is what bounds it: 63.7 GiB / 11 MB = 5,800 of 15,360 experts = 37.8% maximum, and that assumes VRAM holds nothing else.
The engine author measured the same wall on an R9700 with 61 GB of RAM (DS41.md, "Measured"): 0.8 tok/s cold / 1.8 tok/s warm at a 6 GB hot budget, 0.9 / 1.9 at 16 GB, and 9.1 tok/s for the best tuned three-tier case. They name the cause: "90 to 96% of the 240 routed experts a token selects are still cold, and decode moves roughly 2.4 GB of expert weights per token through the synchronous stream path. That, not the GPU, bounds decode today."
Affinity — a DeepSeek-V4-Flash engine, and what a V4.1 port costs
Affinity is "an inference
engine for DeepSeek-V4-Flash on one or two RDNA4 GPUs", C++/HIP, no framework.
It builds clean on this host: CMake 3.24+, hipcc 23, HIP enabled: gfx1201, zero errors, producing affinity, aff-quantize, aff-info and
aff-gpucheck. Its README claims ~110 tok/s on two R9700s against 8 on
one. That is 3x Frank's 35 tok/s goal, for the 284B model.
The 110 works because the 101 GiB container leaves the cold expert set in a pinned host pool in RAM — the SSD tier that Affinity calls "a cliff" is never reached. V4.1 would need 182 GiB of experts at the same 2.875 bpw, plus 189 GiB of Engram, against the same 246 GiB of RAM.
What a V4.1 port would reuse
The config parser is fully data-driven — every dimension is read from
config.json, so nothing is compiled in at the config layer. And the expert
tensor naming is identical between the two checkpoints:
layers.N.ffn.experts.E.w1.weight (11,008 in V4 = 43 x 256)
layers.N.ffn.experts.E.w1.weight (15,360 in V4.1 = 40 x 384)
So the quantiser, the .aff container, the expert slab, the host pool and the
heat/placement engine — the part that produces the 110 tok/s — all carry
over. Hyper-connections are the same family (hc_mult 4, Sinkhorn 20).
What a V4.1 port would have to write
Verified against both checkpoints' real model.safetensors.index.json on
2026-09-28 — not inferred from the config prose.
Tensor presence. aff-quantize requires 32 per-layer tensors. V4.1
supplies 26 plus every global (embed, head, norm, hc_head_*) and the
whole expert set. Six are missing:
| Missing in V4.1 | Replaced in V4.1 by |
|---|---|
ffn.gate.tid2eid |
the Engram tables |
attn.compressor.ape |
nothing — ratio-2 is pooled, no APE |
attn.indexer.compressor.{wkv,wgate,ape,norm}.weight |
attn.indexer.wk + attn.indexer.k_norm |
Config keys. Of the 35 Affinity reads, V4.1 supplies 34. The one it lacks is the dangerous one:
| Gap | Detail |
|---|---|
| Engram — fails silently | num_hash_layers (V4: 3) is absent, so cfg_int returns its default and model.h:36's n_hash_layer = 3 survives. It would route layers 0–2 through a table that does not exist: fluent output, wrong numbers, no error. This is the only silent failure; the other four gaps fail loudly. |
| Ratio vocabulary is disjoint | V4 uses {0, 4, 128}; V4.1 uses {0, 1, 2} — no overlap in the non-zero case. Affinity computes coff = (ratio == 4) ? 2 : 1, so V4.1's 18 ratio-2 layers get the ratio-1 width; comp_ape is requested 38 times for a tensor that does not exist; and the ratio == 4 indexer branch fires on 0 of the 8 layers that need one (index_source_layer_ids = 2, 8, 14, 20, 24, 28, 32, 36) — those 8 would run dense. |
| CED | A 20-layer encoder feeding a 20-layer decoder whose global KV is projected from the final encoder hidden state, through an aligner. Affinity's model.cpp is a flat layer stack. |
| DSpark | V4.1's draft carries its own 128-expert MoE at top-3 (dspark_n_routed_experts, dspark_num_experts_per_tok), 3 stages rather than 1. Affinity's draft shares the target's slab and routing. |
| Vision | V4.1 is multimodal; Affinity has no vision path. Text-only is an acceptable loss. |
V4.1 also adds 31 config keys Affinity never reads, including the whole
engram_* block, kv_source_layer_ids, index_source_layer_ids and the
candidate block pre-selection.
Verdict. The reusable half is the right half — the expert quantiser,
the .aff container, the slab, the host pool and the heat/placement engine
are what produce the 110 tok/s. The rest is 6 tensor bindings, one new ratio
vocabulary, the CED aligner, a new draft geometry and Engram: weeks of work
in a third-party codebase, with no cheap way to prove correctness. And it
meets the same capacity wall regardless, because 182 GiB of experts plus
189 GiB of Engram against 246 GiB of RAM is exactly the problem Lucebox has.
Not advisable.
The other runtimes
| Runtime | gfx1201 | Architectures | Note |
|---|---|---|---|
| Affinity | yes, built here | V4-Flash only — "the geometry is compiled in; there is no architecture abstraction" | 110 tok/s for 284B. Best engine for this box by an order of magnitude, wrong model |
| tcclaviger vLLM fork | yes — proven 2026-09-29 | qwen4exp measured 37.4-40.6 tok/s decode (see frank-archon); glm5next registered (Glm5NextForCausalLM/MTP, backend-split amd//common//nvidia/, sparse_indexer.py for DSA); r4d.so has native mxfp4 + dsfp4 MoE entries (has_dsfp4() == True); W4A16 MoE class (R4dW4a16MoEExperts) wired into the same tiered offload; TunableOp silently skipped under offload (needs CLAV_TUNABLEOP_WITH_OFFLOAD=1, and then corrupts numerics — do not use) |
the ad hoc qwen-fp8 container's runtime; the old "MXFP4 = emulation/missing kernels" verdict predates this fork and is dead — the GLM-5.3-Flash gate is now the payload format + a boot probe, not kernels |
| Lucebox | yes, image on host | Qwen 3.8, DS4; V4.1 on the unmerged feat/ds41-engine branch |
The only V4.1 path. Refuses layer split and layer-major prefill for V4.1 |
| radiance-vLLM (drwolfen) | yes — served frank-archon 2026-09-27 → 09-30 | welded to qwen4_exp; PLE / QSA / packed-HC live inside that model class |
12–20 person-days per architecture |
| FreeToken-ROCm | partial | block-FP8 safetensors only (modelopt); known CpuMoeExecutor issue under ROCm |
cannot load a GGUF |
| llama.cpp | yes | GLM-5.3 confirmed by a community R9700 guide; DS-V4.1 CED unverified | no expert LRU; 8 CPU cores make CPU expert placement the wrong trade |
| TokenSpeed | claimed | GLM-5.x incl. a 5.3-Flash recipe | NVIDIA-first, unverified on gfx1201 |
| KTransformers | no | GLM-5.3 tutorial exists | built for Intel AMX + NVIDIA |
| radiance (StillDeadcode — the standalone C++/HIP engine, NOT the vLLM fork of the same name) | image stilldeadcode/radiance prebuilt for gfx1201 |
qwen35 dense/MoE at fp8, qwen4exp quantised-at-convert, llama-shaped | see the probe below: our FP8 payload is REFUSED; tiers, MTP and vision native |
radiance (standalone) — 2026-10-05 probe
codeberg.org/StillDeadcode/radiance — Deadcode's from-scratch engine
(v1.0.13): same author as the vllm-radiance fork we ran in September
(frank-medium-radiance-mxfp4), an unrelated codebase — C++/HIP, no
Python, kernels/architectures/quantisers as .so plugins, flags written
for two 32 GB R9700s, native Qwen3.8-Flash-Next with --placement expert_tiered, MTP and vision.
Probe with the DEPLOYED payload (rad-convert --plan-only, which stops
at declare — no GPU touched, no bytes written):
E no architecture plugin claims 'qwen4exp' at quantisation 'fp8'.
'qwen4exp' is served at: (unquantised).
So the 173 G orcarouter FP8 file cannot enter this engine at all: the
qwen4exp plugin takes bf16/f16/f32/GGUF in and every quantisation happens
at rad-convert time by recipe. Not a lock-in to their published quant —
recipes are per-weight-class rules (rtn/gptq/awq/autoround/
paroquant/ggml, grids from fp8 128×128 to i4/w4nl/w5nl codebooks,
custom --quant rules go ahead of the file's), and libr4d lists routed
experts-as-fp8 among the encodings it reads — so today's quantization is
reproducible from a bf16 source, not from our file. Uncensored-family
bf16 source candidates: windowsxp811203/Qwen3.8-Flash-Next-Abliterated-GGUF
(ships the bf16 GGUF) and mradermacher/Qwen3.8-Flash-Next-Uncensored-GGUF
(i1 quants only). 622 G free fits source + container. A trial is:
download → rad-convert (CPU-side) → stop+mask frank-archon window to
serve. Caveats on record: --tp-wire wht6 is a lossy int8 all-reduce
wire; the recurrent GDN state is f32 by default (144 MiB/sequence) with
an opt-in f16 that changes generated text; every pool is budgeted
explicitly — no profiling pass, no utilisation fraction.
The two-R9700 three-owner split (primary + secondary + stream) is not
available on this host: the retired quadlet recorded that
LUCE_DS4_MOE_TP_INPROC is qualified for gfx1151+gfx1201 pairs, not
gfx1201+gfx1201. hip:0 takes dense weights and the hot expert stack;
LUCE_EXPERT_STREAM_DEVICE=1 gives hip:1 the streamed-expert slot cache.
Measured on Dankserver
See Also
- frank-archon — the retired Lucebox frank-archon runbook
- Dual-R9700-Cooperation — the PCIe/atomics facts behind the board swap
- Dankserver System Specs — the host (RAM corrected 2026-09-28)
- frank-archon — the ACTIVE frank-archon