test(frank-medium): A/B DFlash2 vs MTP on FP8 — DFlash2 wins on decode
DFlash2 (100.0 t/s decode, 287.1 t/s burst-4) beats MTP self-speculation (85.9 t/s, 179.7 t/s) despite a lower accept length (4.24 vs 5.66): the small separate drafter steps faster than the target's own MTP layer applied autoregressively. DFlash2 is the deployed shape; both swap-script paths now default to dflash/7. Docs updated.
This commit is contained in:
Frank
@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
|
||||
| Target | Service | Runtime | Model | Port |
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-medium` | vllm-radiance 0.9.3 (rootless Quadlet, image `localhost/radiance-vllm:0.9.3`, gfx1201) | orcarouter Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, 30.9 GB / 7 shards) with MTP self-speculation (depth 8, live mean accept length 5.66/8), 2x Radeon AI PRO R9700 in ONE tensor-parallel process (TP=2 over the R4D one-shot allreduce — no RCCL collective; both cards on CPU root ports at Gen5 x8 since the 2026-09-20 ProArt B850-Creator swap), 262144 ctx over up to 8 conversations (auto-sized 461,540-token KV pool = 1.76x the window), fp8 KV, prefix caching, vision | 7770 |
|
||||
| Dankserver | `frank-medium` | vllm-radiance 0.9.3 (rootless Quadlet, image `localhost/radiance-vllm:0.9.3`, gfx1201) | orcarouter Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, 30.9 GB / 7 shards) with the z-lab DFlash2 external drafter (depth 7, live mean accept length 4.24/7 — A/B tested and beats MTP self-speculation on decode: 100.0 vs 85.9 t/s), 2x Radeon AI PRO R9700 in ONE tensor-parallel process (TP=2 over the R4D one-shot allreduce — no RCCL collective; both cards on CPU root ports at Gen5 x8 since the 2026-09-20 ProArt B850-Creator swap), 262144 ctx over up to 8 conversations (auto-sized 384,238-token KV pool = 1.47x the window), fp8 KV, prefix caching, vision | 7770 |
|
||||
| Dankclaw | `frank-large` | CIRU v4.4.1 llama.cpp (QSA window hotfix, root systemd unit, gfx1151 HIP build) | jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca on the gfx1151 iGPU (Strix Halo LPDDR), 262144 ctx single stream (1 slot), MTP depth 4, PLE 4096 MiB, text only | 7770 |
|
||||
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 |
|
||||
|
||||
@@ -334,25 +334,26 @@ ubatch 2048 OOMs the draft's pp buffer, and why draft depth 4 + poll
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`.
|
||||
Rollback: [[frank-small-qwen38-gsq-rco]].
|
||||
|
||||
### frank-medium — radiance TP=2, MTP self-speculation (Dankserver, ACTIVE)
|
||||
### frank-medium — radiance TP=2, DFlash2 drafter (Dankserver, ACTIVE)
|
||||
|
||||
vllm-radiance **0.9.3** (`localhost/radiance-vllm:0.9.3`, rootless **Quadlet**
|
||||
`dankserver/frank-medium/quadlet/frank-medium.container`) serving
|
||||
`orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3 block-wise 128×128, dynamic
|
||||
activation quant, 30.9 GB / 7 shards) with **MTP self-speculation**
|
||||
(`num_speculative_tokens: 8`, the model's own single-layer MTP head, live mean
|
||||
accept 5.66/8) in **one tensor-parallel process across both R9700s**
|
||||
(`--tensor-parallel-size 2`). 262,144 ctx, `--max-num-seqs 8`, chunk 8192, fp8
|
||||
KV, prefix caching, GPU fill 0.98, **no KV-memory pin** — the pool auto-sized
|
||||
to 461,540 tokens (1.76× the full window; smaller than the MXFP4 rollback
|
||||
target's 943,581 because the FP8 weights are bigger). Tools via `qwen3_coder`,
|
||||
activation quant, 30.9 GB / 7 shards) with the **z-lab DFlash2 external
|
||||
drafter** (`num_speculative_tokens: 7`, live mean accept 4.24/7) in **one
|
||||
tensor-parallel process across both R9700s** (`--tensor-parallel-size 2`).
|
||||
262,144 ctx, `--max-num-seqs 8`, chunk 8192, fp8 KV, prefix caching, GPU fill
|
||||
0.98, **no KV-memory pin** — the pool auto-sized to 384,238 tokens (1.47× the
|
||||
full window; the DFlash2 drafter's 2.1 GiB costs the pool vs MTP's 461,540).
|
||||
DFlash2 was A/B tested against MTP self-speculation on this checkpoint and
|
||||
wins on decode (100.0 vs 85.9 t/s) — see the runbook. Tools via `qwen3_coder`,
|
||||
reasoning via `qwen3` (the field is `reasoning`, not llama.cpp's
|
||||
`reasoning_content`), vision through the Gemma4 merge path.
|
||||
|
||||
**The entrypoint is quantization-aware.** `radiance-entrypoint.sh` reads
|
||||
`quantization_config.quant_method` from the checkpoint's `config.json` and
|
||||
serves the plain FP8 path (no MXFP4 patches, `--quantization fp8`,
|
||||
`ROCM_AITER_UNIFIED_ATTN`, MTP self-speculation) or the MXFP4 patch-and-serve
|
||||
`ROCM_AITER_UNIFIED_ATTN`, the DFlash2 drafter) or the MXFP4 patch-and-serve
|
||||
path (the DFlash2 external drafter). A checkpoint swap is therefore four quadlet
|
||||
env vars (`SNAP`, `SPEC_METHOD`, `SPEC`, `KV_MEM`) — the deploy handler reads
|
||||
the serving shape from the quadlet and gates on the right assets. The MXFP4
|
||||
|
||||
@@ -110,10 +110,11 @@ Environment=MAXLEN=262144
|
||||
# 400 here. Vision traffic routes to frank-small on dankclaw (harness
|
||||
# modelRoles vision -> frank-small).
|
||||
Environment=GPU_UTIL=0.98
|
||||
# Speculative decoding: A/B test — DFlash2 external drafter at depth 7 (the
|
||||
# documented DFlash2 depth), to compare against the MTP self-speculation
|
||||
# baseline (depth 8, mean accept 5.66/8). The DFlash2 drafter is a separate
|
||||
# quantization-agnostic checkpoint at /models/Qwen3.8-27B-DFlash2-FP8.
|
||||
# Speculative decoding: DFlash2 external drafter at depth 7. A/B tested
|
||||
# against MTP self-speculation (depth 8) on this FP8 checkpoint: DFlash2
|
||||
# wins on decode (100.0 vs 85.9 t/s geomean, 287.1 vs 179.7 t/s burst-4)
|
||||
# despite a lower accept length (4.24 vs 5.66) — the small separate drafter
|
||||
# steps faster than the target's own MTP layer applied autoregressively.
|
||||
Environment=SPEC_METHOD=dflash
|
||||
Environment=SPEC=7
|
||||
# KV cache: 0 = profile the shape (the documented plain-FP8 path's default;
|
||||
|
||||
@@ -91,10 +91,10 @@ quant="$(ssh "$HOST" "python3 -c 'import json,sys;print(json.load(open(sys.argv[
|
||||
echo "==> Detected quantization: ${quant:-<none>}"
|
||||
|
||||
case "$quant" in
|
||||
fp8) spec_method=mtp; default_spec=8 ;;
|
||||
fp8) spec_method=dflash; default_spec=7 ;;
|
||||
mxfp4|quark|"") spec_method=dflash; default_spec=7 ;;
|
||||
*) spec_method=dflash; default_spec=7
|
||||
echo "WARNING: unknown quant '$quant'; defaulting to the MXFP4 path" >&2 ;;
|
||||
echo "WARNING: unknown quant '$quant'; defaulting to the DFlash2 path" >&2 ;;
|
||||
esac
|
||||
[[ -n "$spec" ]] || spec="$default_spec"
|
||||
|
||||
|
||||
@@ -84,15 +84,19 @@ The entrypoint (`radiance-entrypoint.sh`) reads `quant_method` and branches:
|
||||
|
||||
| `quant_method` | Path | Attention | Speculation | Drafter |
|
||||
|---|---|---|---|---|
|
||||
| `fp8` | plain FP8 | `ROCM_AITER_UNIFIED_ATTN` | MTP self-speculation (`--quantization fp8`) | the model's own MTP head (`mtp.layers.0`) |
|
||||
| `fp8` | plain FP8 | `ROCM_AITER_UNIFIED_ATTN` | **DFlash2 external drafter** (A/B winner) | `/models/Qwen3.8-27B-DFlash2-FP8` |
|
||||
| `mxfp4` / `quark` / empty | MXFP4 | `R4D` (if `R4D_ATTN=1`) | DFlash2 external drafter | `/models/Qwen3.8-27B-DFlash2-FP8` |
|
||||
|
||||
The quadlet `SPEC_METHOD`/`SPEC` the swap script sets must match the path:
|
||||
`fp8 → mtp/8`, `mxfp4|quark → dflash/7`. The FP8 model carries its own
|
||||
single-layer MTP head (22 tensors), so MTP self-speculation needs no external
|
||||
drafter and no `fp8_mtp.py` rewrite. The MXFP4 path applies the
|
||||
`radiance-vllm-mxfp4` patches at every container start and uses the DFlash2
|
||||
drafter.
|
||||
The quadlet `SPEC_METHOD`/`SPEC` the swap script sets: **both paths use
|
||||
`dflash/7`** (the DFlash2 drafter is quantization-agnostic — it proposes
|
||||
tokens, the target verifies, so it works with any Qwen3.8-27B target). The
|
||||
FP8 model also carries its own single-layer MTP head (22 tensors), so MTP
|
||||
self-speculation (`mtp/8`) is available as an alternative — but A/B tested on
|
||||
this checkpoint, **DFlash2 wins on decode** (100.0 vs 85.9 t/s geomean, 287.1
|
||||
vs 179.7 t/s burst-4) despite a lower accept length (4.24 vs 5.66), because
|
||||
the small separate drafter steps faster than the target's own MTP layer
|
||||
applied autoregressively. The MXFP4 path additionally applies the
|
||||
`radiance-vllm-mxfp4` patches at every container start.
|
||||
|
||||
**R4D allreduce is independent of the weight quantization.** The FP8 path
|
||||
uses `ROCM_AITER_UNIFIED_ATTN` (the documented plain-FP8 default); R4D can be
|
||||
@@ -159,15 +163,23 @@ safetensors) with `SPEC_METHOD=dflash`, `SPEC=7`.
|
||||
- **Model**: `just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` (MXFP4,
|
||||
19.37 GiB) → `orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3 block-wise
|
||||
128×128, dynamic activation quant, 30.9 GB / 7 shards).
|
||||
- **Serving path**: DFlash2 external drafter → MTP self-speculation
|
||||
(`SPEC_METHOD=mtp`, `SPEC=8`); `R4D` → `ROCM_AITER_UNIFIED_ATTN`.
|
||||
- **MTP**: mean acceptance length 5.66/8, 66.2% draft acceptance,
|
||||
per-position rates 0.89 → 0.41.
|
||||
- **KV pool**: auto-sized to 461,540 tokens (1.76× full-window) — smaller than
|
||||
MXFP4's 943,581 because FP8 weights are bigger (30.9 vs 19.4 GiB).
|
||||
- **Battery** (vs MXFP4 baseline): geomean decode 85.9 vs 92.4 t/s (−7%),
|
||||
mean TTFT 0.06 vs 0.22 s (3.7× faster), geomean prefill 1097 vs ~1490 t/s
|
||||
(−27%), burst-4 aggregate 179.7 vs 248.9 t/s (−28%). The trade-off is the
|
||||
expected one: higher precision (8-bit vs 4-bit weights), decode within 7%,
|
||||
TTFT much faster, prefill/burst slower (the `ROCM_AITER_UNIFIED_ATTN` path
|
||||
vs R4D).
|
||||
- **Serving path**: DFlash2 external drafter → DFlash2 external drafter
|
||||
(kept; `R4D` → `ROCM_AITER_UNIFIED_ATTN`). After the swap, MTP
|
||||
self-speculation (`mtp/8`) was A/B tested against the DFlash2 drafter
|
||||
(`dflash/7`) on the FP8 target.
|
||||
- **DFlash2 vs MTP A/B** (the deployed shape is DFlash2):
|
||||
- DFlash2: mean accept 4.24/7, geomean decode **100.0 t/s**, TTFT 0.19 s,
|
||||
prefill 923.0 t/s, burst-4 **287.1 t/s**, KV pool 384,238 (1.47×).
|
||||
- MTP: mean accept 5.66/8, geomean decode 85.9 t/s, TTFT 0.06 s, prefill
|
||||
1096.9 t/s, burst-4 179.7 t/s, KV pool 461,540 (1.76×).
|
||||
- **DFlash2 wins on decode (+16 %) and burst-4 (+60 %)**; MTP wins on TTFT
|
||||
and prefill. Accept length is a proxy, not the metric — the small separate
|
||||
drafter steps faster than the target's own MTP layer applied
|
||||
autoregressively, and the faster step more than makes up the lower accept
|
||||
length. FP8 + DFlash2 (100.0 t/s) also beats the original MXFP4 + DFlash2
|
||||
(92.4 t/s) — the higher-precision FP8 target verifies more of the drafter's
|
||||
proposals.
|
||||
- **KV pool**: auto-sized to 384,238 tokens (1.47× full-window) — the
|
||||
DFlash2 drafter's 2.1 GiB costs the pool vs MTP's 461,540 (1.76×); both
|
||||
smaller than MXFP4's 943,581 (3.59×) because FP8 weights are bigger (30.9
|
||||
vs 19.4 GiB).
|
||||
|
||||
@@ -54,7 +54,7 @@ From `dankserver/frank-medium/quadlet/frank-medium.container`:
|
||||
| Setting | Value |
|
||||
|---|---|
|
||||
| Model | `orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3 block-wise 128×128, 30.9 GB / 7 shards) |
|
||||
| Speculative | `--speculative-config {"method":"mtp","num_speculative_tokens":8}` (the model's own single-layer MTP head; mean acceptance 5.66/8, 66.2% draft acceptance) |
|
||||
| Speculative | `--speculative-config {"method":"dflash","model":"/models/Qwen3.8-27B-DFlash2-FP8","num_speculative_tokens":7}` (the z-lab DFlash2 external drafter; mean acceptance 4.24/7. A/B tested against MTP self-speculation — DFlash2 wins on decode, see below) |
|
||||
| Parallelism | `--tensor-parallel-size 2` (`ROCR_VISIBLE_DEVICES=0,1`) |
|
||||
| Context | `--max-model-len 262144` |
|
||||
| Concurrency | `--max-num-seqs 8`, chunk 8192 |
|
||||
@@ -67,17 +67,43 @@ From `dankserver/frank-medium/quadlet/frank-medium.container`:
|
||||
| Chat template | `qwen-fixed-v22.3.jinja` (patch dir bind-mounted, `MM_MERGE_ENGINE=AXROW_SEPARATE` for Gemma4 vision) |
|
||||
|
||||
**The KV pool is auto-sized, not pinned.** Production ships no `KV_MEM` pin
|
||||
(`KV_MEM=0`): the pool comes up at whatever VRAM the weights leave. The FP8
|
||||
checkpoint is bigger than the MXFP4 one (30.9 vs 19.4 GiB of weights), so it
|
||||
leaves less for the pool: the 2026-09-20 FP8 start auto-sized to **461,540
|
||||
tokens = 1.76× the full window** (`GPU KV cache size: 461,540 tokens`,
|
||||
`Maximum concurrency for 262,144 tokens per request: 1.76x`). The MXFP4
|
||||
rollback target auto-sized to 943,581 tokens (3.59×) on the same cards.
|
||||
Per-token cost in steady state is 19,673 B — fp8 KV across the model's 16
|
||||
full-attention layers plus the MTP drafter's. After the first healthy start,
|
||||
(`KV_MEM=0`): the pool comes up at whatever VRAM the weights + drafter leave.
|
||||
The FP8 checkpoint is bigger than the MXFP4 one (30.9 vs 19.4 GiB of
|
||||
weights), and the DFlash2 drafter is a separate 2.1 GiB model, so the pool
|
||||
is the smallest of the three shapes: the 2026-09-20 DFlash2 start auto-sized
|
||||
to **384,238 tokens = 1.47× the full window** (`GPU KV cache size: 384,238
|
||||
tokens`, `Maximum concurrency for 262,144 tokens per request: 1.47x`). The
|
||||
MTP shape (no external drafter) auto-sized to 461,540 (1.76×); the MXFP4
|
||||
rollback target to 943,581 (3.59×). After the first healthy start,
|
||||
recalibrate with `calibrate-kv.sh` to use the extra VRAM on the 34.2 GiB
|
||||
cards.
|
||||
|
||||
## DFlash2 vs MTP — A/B on the FP8 checkpoint (2026-09-20)
|
||||
|
||||
The FP8 checkpoint carries its own single-layer MTP head, so it can self-
|
||||
speculate (`SPEC_METHOD=mtp`) or use the z-lab DFlash2 external drafter
|
||||
(`SPEC_METHOD=dflash`, a separate quantization-agnostic 2.1 GiB
|
||||
`DFlash2DraftModel`). Both were deployed and measured on the same FP8 target
|
||||
(`scripts/bench-conversation.py`, 10 real prompts, `max_tokens=512`, greedy,
|
||||
salted, `--burst 4`):
|
||||
|
||||
| Arm | Mean accept | Decode (geomean) | TTFT (mean) | Prefill (geomean) | Burst-4 aggregate | KV pool |
|
||||
|---|---|---|---|---|---|---|
|
||||
| **DFlash2** (depth 7) | 4.24/7 | **100.0 t/s** | 0.19 s | 923.0 t/s | **287.1 t/s** | 384,238 (1.47×) |
|
||||
| MTP self-spec (depth 8) | 5.66/8 | 85.9 t/s | 0.06 s | 1096.9 t/s | 179.7 t/s | 461,540 (1.76×) |
|
||||
|
||||
**DFlash2 wins on decode (+16 %) and burst-4 (+60 %); MTP wins on TTFT and
|
||||
prefill.** The counterintuitive part: DFlash2 accepts *fewer* tokens per step
|
||||
(4.24 vs 5.66) yet decodes *faster*. Accept length is a proxy, not the metric
|
||||
— throughput is `(accept + 1) / (draft time + verify time)`. The DFlash2
|
||||
drafter is a small separate model that steps fast; the MTP head is the
|
||||
target's own layer applied autoregressively 8×, so its draft is slower. The
|
||||
faster step more than makes up the lower accept length. FP8 + DFlash2 (100.0
|
||||
t/s) also beats the original MXFP4 + DFlash2 (92.4 t/s) — the higher-precision
|
||||
FP8 target verifies more of the drafter's proposals. The drafter costs 2.1 GiB
|
||||
of VRAM (the KV pool is 1.47× vs MTP's 1.76×), an acceptable trade for the
|
||||
decode win. **DFlash2 is the deployed shape.**
|
||||
|
||||
## Prefix caching — measured granularity (2026-09-20)
|
||||
|
||||
`--enable-prefix-caching` is on, and radiance says so at boot: *"Prefix caching
|
||||
|
||||
@@ -11,6 +11,33 @@ timestamp: 2026-09-14
|
||||
|
||||
# Bundle Change Log
|
||||
|
||||
## 2026-09-20 — DFlash2 vs MTP A/B on the FP8 checkpoint: DFlash2 wins on decode
|
||||
|
||||
- **The test.** The FP8 checkpoint carries its own single-layer MTP head, so
|
||||
it can self-speculate (`SPEC_METHOD=mtp`) or use the z-lab DFlash2 external
|
||||
drafter (`SPEC_METHOD=dflash`). Both were deployed and measured on the same
|
||||
FP8 target (`scripts/bench-conversation.py`, 10 real prompts, `max_tokens=512`,
|
||||
greedy, salted, `--burst 4`):
|
||||
- **DFlash2** (depth 7): mean accept 4.24/7, geomean decode **100.0 t/s**,
|
||||
TTFT 0.19 s, prefill 923.0 t/s, burst-4 **287.1 t/s**, KV pool 384,238 (1.47×).
|
||||
- **MTP** (depth 8): mean accept 5.66/8, geomean decode 85.9 t/s, TTFT 0.06 s,
|
||||
prefill 1096.9 t/s, burst-4 179.7 t/s, KV pool 461,540 (1.76×).
|
||||
- **Verdict: DFlash2 wins on decode (+16 %) and burst-4 (+60 %); MTP wins on
|
||||
TTFT and prefill.** The counterintuitive part: DFlash2 accepts *fewer*
|
||||
tokens per step (4.24 vs 5.66) yet decodes *faster*. Accept length is a
|
||||
proxy, not the metric — throughput is `(accept + 1) / (draft time + verify
|
||||
time)`. The DFlash2 drafter is a small separate model that steps fast; the
|
||||
MTP head is the target's own layer applied autoregressively 8×, so its draft
|
||||
is slower. The faster step more than makes up the lower accept length.
|
||||
- **FP8 + DFlash2 (100.0 t/s) beats the original MXFP4 + DFlash2 (92.4 t/s)**
|
||||
— the higher-precision FP8 target verifies more of the drafter's proposals.
|
||||
The drafter costs 2.1 GiB of VRAM (KV pool 1.47× vs MTP's 1.76×), an
|
||||
acceptable trade for the decode win.
|
||||
- **DFlash2 is the deployed shape.** The quadlet comment, the radiance
|
||||
runbook (deployed-flags table, KV pool, a new A/B section), AGENTS.md
|
||||
(topology row + frank-medium section), and the model-swap runbook + swap
|
||||
script (both paths now default to `dflash/7`) all updated.
|
||||
|
||||
## 2026-09-20 — frank-medium checkpoint swapped MXFP4 → FP8; the swap is now one command
|
||||
|
||||
- **The swap.** `just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` (MXFP4,
|
||||
|
||||
Reference in New Issue
Block a user