test(frank-medium): A/B DFlash2 vs MTP on FP8 — DFlash2 wins on decode

DFlash2 (100.0 t/s decode, 287.1 t/s burst-4) beats MTP self-speculation
(85.9 t/s, 179.7 t/s) despite a lower accept length (4.24 vs 5.66): the
small separate drafter steps faster than the target's own MTP layer applied
autoregressively. DFlash2 is the deployed shape; both swap-script paths now
default to dflash/7. Docs updated.
This commit is contained in:
Frank committed 2026-09-20 22:20:53 -07:00
1 parent 1ae73db084
commit f998a2d9f2
6 files changed
+111 -44

No files matched your search

+11 -10
View File
@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
| Target | Service | Runtime | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | `frank-medium` | vllm-radiance 0.9.3 (rootless Quadlet, image `localhost/radiance-vllm:0.9.3`, gfx1201) | orcarouter Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, 30.9 GB / 7 shards) with MTP self-speculation (depth 8, live mean accept length 5.66/8), 2x Radeon AI PRO R9700 in ONE tensor-parallel process (TP=2 over the R4D one-shot allreduce — no RCCL collective; both cards on CPU root ports at Gen5 x8 since the 2026-09-20 ProArt B850-Creator swap), 262144 ctx over up to 8 conversations (auto-sized 461,540-token KV pool = 1.76x the window), fp8 KV, prefix caching, vision | 7770 |
| Dankserver | `frank-medium` | vllm-radiance 0.9.3 (rootless Quadlet, image `localhost/radiance-vllm:0.9.3`, gfx1201) | orcarouter Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, 30.9 GB / 7 shards) with the z-lab DFlash2 external drafter (depth 7, live mean accept length 4.24/7 — A/B tested and beats MTP self-speculation on decode: 100.0 vs 85.9 t/s), 2x Radeon AI PRO R9700 in ONE tensor-parallel process (TP=2 over the R4D one-shot allreduce — no RCCL collective; both cards on CPU root ports at Gen5 x8 since the 2026-09-20 ProArt B850-Creator swap), 262144 ctx over up to 8 conversations (auto-sized 384,238-token KV pool = 1.47x the window), fp8 KV, prefix caching, vision | 7770 |
| Dankclaw | `frank-large` | CIRU v4.4.1 llama.cpp (QSA window hotfix, root systemd unit, gfx1151 HIP build) | jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca on the gfx1151 iGPU (Strix Halo LPDDR), 262144 ctx single stream (1 slot), MTP depth 4, PLE 4096 MiB, text only | 7770 |
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 |
@@ -334,25 +334,26 @@ ubatch 2048 OOMs the draft's pp buffer, and why draft depth 4 + poll
Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`.
Rollback: [[frank-small-qwen38-gsq-rco]].
### frank-medium — radiance TP=2, MTP self-speculation (Dankserver, ACTIVE)
### frank-medium — radiance TP=2, DFlash2 drafter (Dankserver, ACTIVE)
vllm-radiance **0.9.3** (`localhost/radiance-vllm:0.9.3`, rootless **Quadlet**
`dankserver/frank-medium/quadlet/frank-medium.container`) serving
`orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3 block-wise 128×128, dynamic
activation quant, 30.9 GB / 7 shards) with **MTP self-speculation**
(`num_speculative_tokens: 8`, the model's own single-layer MTP head, live mean
accept 5.66/8) in **one tensor-parallel process across both R9700s**
(`--tensor-parallel-size 2`). 262,144 ctx, `--max-num-seqs 8`, chunk 8192, fp8
KV, prefix caching, GPU fill 0.98, **no KV-memory pin** — the pool auto-sized
to 461,540 tokens (1.76× the full window; smaller than the MXFP4 rollback
target's 943,581 because the FP8 weights are bigger). Tools via `qwen3_coder`,
activation quant, 30.9 GB / 7 shards) with the **z-lab DFlash2 external
drafter** (`num_speculative_tokens: 7`, live mean accept 4.24/7) in **one
tensor-parallel process across both R9700s** (`--tensor-parallel-size 2`).
262,144 ctx, `--max-num-seqs 8`, chunk 8192, fp8 KV, prefix caching, GPU fill
0.98, **no KV-memory pin** — the pool auto-sized to 384,238 tokens (1.47× the
full window; the DFlash2 drafter's 2.1 GiB costs the pool vs MTP's 461,540).
DFlash2 was A/B tested against MTP self-speculation on this checkpoint and
wins on decode (100.0 vs 85.9 t/s) — see the runbook. Tools via `qwen3_coder`,
reasoning via `qwen3` (the field is `reasoning`, not llama.cpp's
`reasoning_content`), vision through the Gemma4 merge path.
**The entrypoint is quantization-aware.** `radiance-entrypoint.sh` reads
`quantization_config.quant_method` from the checkpoint's `config.json` and
serves the plain FP8 path (no MXFP4 patches, `--quantization fp8`,
`ROCM_AITER_UNIFIED_ATTN`, MTP self-speculation) or the MXFP4 patch-and-serve
`ROCM_AITER_UNIFIED_ATTN`, the DFlash2 drafter) or the MXFP4 patch-and-serve
path (the DFlash2 external drafter). A checkpoint swap is therefore four quadlet
env vars (`SNAP`, `SPEC_METHOD`, `SPEC`, `KV_MEM`) — the deploy handler reads
the serving shape from the quadlet and gates on the right assets. The MXFP4