diff --git a/AGENTS.md b/AGENTS.md index 6170106..749f835 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| -| Dankserver | `frank-medium` | vllm-radiance 0.9.3 (rootless Quadlet, image `localhost/radiance-vllm:0.9.3`, gfx1201) | orcarouter Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, 30.9 GB / 7 shards) with MTP self-speculation (depth 8, live mean accept length 5.66/8), 2x Radeon AI PRO R9700 in ONE tensor-parallel process (TP=2 over the R4D one-shot allreduce — no RCCL collective; both cards on CPU root ports at Gen5 x8 since the 2026-09-20 ProArt B850-Creator swap), 262144 ctx over up to 8 conversations (auto-sized 461,540-token KV pool = 1.76x the window), fp8 KV, prefix caching, vision | 7770 | +| Dankserver | `frank-medium` | vllm-radiance 0.9.3 (rootless Quadlet, image `localhost/radiance-vllm:0.9.3`, gfx1201) | orcarouter Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, 30.9 GB / 7 shards) with the z-lab DFlash2 external drafter (depth 7, live mean accept length 4.24/7 — A/B tested and beats MTP self-speculation on decode: 100.0 vs 85.9 t/s), 2x Radeon AI PRO R9700 in ONE tensor-parallel process (TP=2 over the R4D one-shot allreduce — no RCCL collective; both cards on CPU root ports at Gen5 x8 since the 2026-09-20 ProArt B850-Creator swap), 262144 ctx over up to 8 conversations (auto-sized 384,238-token KV pool = 1.47x the window), fp8 KV, prefix caching, vision | 7770 | | Dankclaw | `frank-large` | CIRU v4.4.1 llama.cpp (QSA window hotfix, root systemd unit, gfx1151 HIP build) | jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca on the gfx1151 iGPU (Strix Halo LPDDR), 262144 ctx single stream (1 slot), MTP depth 4, PLE 4096 MiB, text only | 7770 | | Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 | @@ -334,25 +334,26 @@ ubatch 2048 OOMs the draft's pp buffer, and why draft depth 4 + poll Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`. Rollback: [[frank-small-qwen38-gsq-rco]]. -### frank-medium — radiance TP=2, MTP self-speculation (Dankserver, ACTIVE) +### frank-medium — radiance TP=2, DFlash2 drafter (Dankserver, ACTIVE) vllm-radiance **0.9.3** (`localhost/radiance-vllm:0.9.3`, rootless **Quadlet** `dankserver/frank-medium/quadlet/frank-medium.container`) serving `orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3 block-wise 128×128, dynamic -activation quant, 30.9 GB / 7 shards) with **MTP self-speculation** -(`num_speculative_tokens: 8`, the model's own single-layer MTP head, live mean -accept 5.66/8) in **one tensor-parallel process across both R9700s** -(`--tensor-parallel-size 2`). 262,144 ctx, `--max-num-seqs 8`, chunk 8192, fp8 -KV, prefix caching, GPU fill 0.98, **no KV-memory pin** — the pool auto-sized -to 461,540 tokens (1.76× the full window; smaller than the MXFP4 rollback -target's 943,581 because the FP8 weights are bigger). Tools via `qwen3_coder`, +activation quant, 30.9 GB / 7 shards) with the **z-lab DFlash2 external +drafter** (`num_speculative_tokens: 7`, live mean accept 4.24/7) in **one +tensor-parallel process across both R9700s** (`--tensor-parallel-size 2`). +262,144 ctx, `--max-num-seqs 8`, chunk 8192, fp8 KV, prefix caching, GPU fill +0.98, **no KV-memory pin** — the pool auto-sized to 384,238 tokens (1.47× the +full window; the DFlash2 drafter's 2.1 GiB costs the pool vs MTP's 461,540). +DFlash2 was A/B tested against MTP self-speculation on this checkpoint and +wins on decode (100.0 vs 85.9 t/s) — see the runbook. Tools via `qwen3_coder`, reasoning via `qwen3` (the field is `reasoning`, not llama.cpp's `reasoning_content`), vision through the Gemma4 merge path. **The entrypoint is quantization-aware.** `radiance-entrypoint.sh` reads `quantization_config.quant_method` from the checkpoint's `config.json` and serves the plain FP8 path (no MXFP4 patches, `--quantization fp8`, -`ROCM_AITER_UNIFIED_ATTN`, MTP self-speculation) or the MXFP4 patch-and-serve +`ROCM_AITER_UNIFIED_ATTN`, the DFlash2 drafter) or the MXFP4 patch-and-serve path (the DFlash2 external drafter). A checkpoint swap is therefore four quadlet env vars (`SNAP`, `SPEC_METHOD`, `SPEC`, `KV_MEM`) — the deploy handler reads the serving shape from the quadlet and gates on the right assets. The MXFP4 diff --git a/dankserver/frank-medium/quadlet/frank-medium.container b/dankserver/frank-medium/quadlet/frank-medium.container index a74af63..4643536 100644 --- a/dankserver/frank-medium/quadlet/frank-medium.container +++ b/dankserver/frank-medium/quadlet/frank-medium.container @@ -110,10 +110,11 @@ Environment=MAXLEN=262144 # 400 here. Vision traffic routes to frank-small on dankclaw (harness # modelRoles vision -> frank-small). Environment=GPU_UTIL=0.98 -# Speculative decoding: A/B test — DFlash2 external drafter at depth 7 (the -# documented DFlash2 depth), to compare against the MTP self-speculation -# baseline (depth 8, mean accept 5.66/8). The DFlash2 drafter is a separate -# quantization-agnostic checkpoint at /models/Qwen3.8-27B-DFlash2-FP8. +# Speculative decoding: DFlash2 external drafter at depth 7. A/B tested +# against MTP self-speculation (depth 8) on this FP8 checkpoint: DFlash2 +# wins on decode (100.0 vs 85.9 t/s geomean, 287.1 vs 179.7 t/s burst-4) +# despite a lower accept length (4.24 vs 5.66) — the small separate drafter +# steps faster than the target's own MTP layer applied autoregressively. Environment=SPEC_METHOD=dflash Environment=SPEC=7 # KV cache: 0 = profile the shape (the documented plain-FP8 path's default; diff --git a/scripts/swap-frank-medium.sh b/scripts/swap-frank-medium.sh index e6a887a..61eb040 100755 --- a/scripts/swap-frank-medium.sh +++ b/scripts/swap-frank-medium.sh @@ -91,10 +91,10 @@ quant="$(ssh "$HOST" "python3 -c 'import json,sys;print(json.load(open(sys.argv[ echo "==> Detected quantization: ${quant:-}" case "$quant" in - fp8) spec_method=mtp; default_spec=8 ;; + fp8) spec_method=dflash; default_spec=7 ;; mxfp4|quark|"") spec_method=dflash; default_spec=7 ;; *) spec_method=dflash; default_spec=7 - echo "WARNING: unknown quant '$quant'; defaulting to the MXFP4 path" >&2 ;; + echo "WARNING: unknown quant '$quant'; defaulting to the DFlash2 path" >&2 ;; esac [[ -n "$spec" ]] || spec="$default_spec" diff --git a/serverAppConfigs-docs/LLM/frank-medium-model-swap.md b/serverAppConfigs-docs/LLM/frank-medium-model-swap.md index 2122912..e9572b4 100644 --- a/serverAppConfigs-docs/LLM/frank-medium-model-swap.md +++ b/serverAppConfigs-docs/LLM/frank-medium-model-swap.md @@ -84,15 +84,19 @@ The entrypoint (`radiance-entrypoint.sh`) reads `quant_method` and branches: | `quant_method` | Path | Attention | Speculation | Drafter | |---|---|---|---|---| -| `fp8` | plain FP8 | `ROCM_AITER_UNIFIED_ATTN` | MTP self-speculation (`--quantization fp8`) | the model's own MTP head (`mtp.layers.0`) | +| `fp8` | plain FP8 | `ROCM_AITER_UNIFIED_ATTN` | **DFlash2 external drafter** (A/B winner) | `/models/Qwen3.8-27B-DFlash2-FP8` | | `mxfp4` / `quark` / empty | MXFP4 | `R4D` (if `R4D_ATTN=1`) | DFlash2 external drafter | `/models/Qwen3.8-27B-DFlash2-FP8` | -The quadlet `SPEC_METHOD`/`SPEC` the swap script sets must match the path: -`fp8 → mtp/8`, `mxfp4|quark → dflash/7`. The FP8 model carries its own -single-layer MTP head (22 tensors), so MTP self-speculation needs no external -drafter and no `fp8_mtp.py` rewrite. The MXFP4 path applies the -`radiance-vllm-mxfp4` patches at every container start and uses the DFlash2 -drafter. +The quadlet `SPEC_METHOD`/`SPEC` the swap script sets: **both paths use +`dflash/7`** (the DFlash2 drafter is quantization-agnostic — it proposes +tokens, the target verifies, so it works with any Qwen3.8-27B target). The +FP8 model also carries its own single-layer MTP head (22 tensors), so MTP +self-speculation (`mtp/8`) is available as an alternative — but A/B tested on +this checkpoint, **DFlash2 wins on decode** (100.0 vs 85.9 t/s geomean, 287.1 +vs 179.7 t/s burst-4) despite a lower accept length (4.24 vs 5.66), because +the small separate drafter steps faster than the target's own MTP layer +applied autoregressively. The MXFP4 path additionally applies the +`radiance-vllm-mxfp4` patches at every container start. **R4D allreduce is independent of the weight quantization.** The FP8 path uses `ROCM_AITER_UNIFIED_ATTN` (the documented plain-FP8 default); R4D can be @@ -159,15 +163,23 @@ safetensors) with `SPEC_METHOD=dflash`, `SPEC=7`. - **Model**: `just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` (MXFP4, 19.37 GiB) → `orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3 block-wise 128×128, dynamic activation quant, 30.9 GB / 7 shards). -- **Serving path**: DFlash2 external drafter → MTP self-speculation - (`SPEC_METHOD=mtp`, `SPEC=8`); `R4D` → `ROCM_AITER_UNIFIED_ATTN`. -- **MTP**: mean acceptance length 5.66/8, 66.2% draft acceptance, - per-position rates 0.89 → 0.41. -- **KV pool**: auto-sized to 461,540 tokens (1.76× full-window) — smaller than - MXFP4's 943,581 because FP8 weights are bigger (30.9 vs 19.4 GiB). -- **Battery** (vs MXFP4 baseline): geomean decode 85.9 vs 92.4 t/s (−7%), - mean TTFT 0.06 vs 0.22 s (3.7× faster), geomean prefill 1097 vs ~1490 t/s - (−27%), burst-4 aggregate 179.7 vs 248.9 t/s (−28%). The trade-off is the - expected one: higher precision (8-bit vs 4-bit weights), decode within 7%, - TTFT much faster, prefill/burst slower (the `ROCM_AITER_UNIFIED_ATTN` path - vs R4D). +- **Serving path**: DFlash2 external drafter → DFlash2 external drafter + (kept; `R4D` → `ROCM_AITER_UNIFIED_ATTN`). After the swap, MTP + self-speculation (`mtp/8`) was A/B tested against the DFlash2 drafter + (`dflash/7`) on the FP8 target. +- **DFlash2 vs MTP A/B** (the deployed shape is DFlash2): + - DFlash2: mean accept 4.24/7, geomean decode **100.0 t/s**, TTFT 0.19 s, + prefill 923.0 t/s, burst-4 **287.1 t/s**, KV pool 384,238 (1.47×). + - MTP: mean accept 5.66/8, geomean decode 85.9 t/s, TTFT 0.06 s, prefill + 1096.9 t/s, burst-4 179.7 t/s, KV pool 461,540 (1.76×). + - **DFlash2 wins on decode (+16 %) and burst-4 (+60 %)**; MTP wins on TTFT + and prefill. Accept length is a proxy, not the metric — the small separate + drafter steps faster than the target's own MTP layer applied + autoregressively, and the faster step more than makes up the lower accept + length. FP8 + DFlash2 (100.0 t/s) also beats the original MXFP4 + DFlash2 + (92.4 t/s) — the higher-precision FP8 target verifies more of the drafter's + proposals. +- **KV pool**: auto-sized to 384,238 tokens (1.47× full-window) — the + DFlash2 drafter's 2.1 GiB costs the pool vs MTP's 461,540 (1.76×); both + smaller than MXFP4's 943,581 (3.59×) because FP8 weights are bigger (30.9 + vs 19.4 GiB). diff --git a/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md b/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md index e514d83..25abbaf 100644 --- a/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md +++ b/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md @@ -54,7 +54,7 @@ From `dankserver/frank-medium/quadlet/frank-medium.container`: | Setting | Value | |---|---| | Model | `orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3 block-wise 128×128, 30.9 GB / 7 shards) | -| Speculative | `--speculative-config {"method":"mtp","num_speculative_tokens":8}` (the model's own single-layer MTP head; mean acceptance 5.66/8, 66.2% draft acceptance) | +| Speculative | `--speculative-config {"method":"dflash","model":"/models/Qwen3.8-27B-DFlash2-FP8","num_speculative_tokens":7}` (the z-lab DFlash2 external drafter; mean acceptance 4.24/7. A/B tested against MTP self-speculation — DFlash2 wins on decode, see below) | | Parallelism | `--tensor-parallel-size 2` (`ROCR_VISIBLE_DEVICES=0,1`) | | Context | `--max-model-len 262144` | | Concurrency | `--max-num-seqs 8`, chunk 8192 | @@ -67,17 +67,43 @@ From `dankserver/frank-medium/quadlet/frank-medium.container`: | Chat template | `qwen-fixed-v22.3.jinja` (patch dir bind-mounted, `MM_MERGE_ENGINE=AXROW_SEPARATE` for Gemma4 vision) | **The KV pool is auto-sized, not pinned.** Production ships no `KV_MEM` pin -(`KV_MEM=0`): the pool comes up at whatever VRAM the weights leave. The FP8 -checkpoint is bigger than the MXFP4 one (30.9 vs 19.4 GiB of weights), so it -leaves less for the pool: the 2026-09-20 FP8 start auto-sized to **461,540 -tokens = 1.76× the full window** (`GPU KV cache size: 461,540 tokens`, -`Maximum concurrency for 262,144 tokens per request: 1.76x`). The MXFP4 -rollback target auto-sized to 943,581 tokens (3.59×) on the same cards. -Per-token cost in steady state is 19,673 B — fp8 KV across the model's 16 -full-attention layers plus the MTP drafter's. After the first healthy start, +(`KV_MEM=0`): the pool comes up at whatever VRAM the weights + drafter leave. +The FP8 checkpoint is bigger than the MXFP4 one (30.9 vs 19.4 GiB of +weights), and the DFlash2 drafter is a separate 2.1 GiB model, so the pool +is the smallest of the three shapes: the 2026-09-20 DFlash2 start auto-sized +to **384,238 tokens = 1.47× the full window** (`GPU KV cache size: 384,238 +tokens`, `Maximum concurrency for 262,144 tokens per request: 1.47x`). The +MTP shape (no external drafter) auto-sized to 461,540 (1.76×); the MXFP4 +rollback target to 943,581 (3.59×). After the first healthy start, recalibrate with `calibrate-kv.sh` to use the extra VRAM on the 34.2 GiB cards. +## DFlash2 vs MTP — A/B on the FP8 checkpoint (2026-09-20) + +The FP8 checkpoint carries its own single-layer MTP head, so it can self- +speculate (`SPEC_METHOD=mtp`) or use the z-lab DFlash2 external drafter +(`SPEC_METHOD=dflash`, a separate quantization-agnostic 2.1 GiB +`DFlash2DraftModel`). Both were deployed and measured on the same FP8 target +(`scripts/bench-conversation.py`, 10 real prompts, `max_tokens=512`, greedy, +salted, `--burst 4`): + +| Arm | Mean accept | Decode (geomean) | TTFT (mean) | Prefill (geomean) | Burst-4 aggregate | KV pool | +|---|---|---|---|---|---|---| +| **DFlash2** (depth 7) | 4.24/7 | **100.0 t/s** | 0.19 s | 923.0 t/s | **287.1 t/s** | 384,238 (1.47×) | +| MTP self-spec (depth 8) | 5.66/8 | 85.9 t/s | 0.06 s | 1096.9 t/s | 179.7 t/s | 461,540 (1.76×) | + +**DFlash2 wins on decode (+16 %) and burst-4 (+60 %); MTP wins on TTFT and +prefill.** The counterintuitive part: DFlash2 accepts *fewer* tokens per step +(4.24 vs 5.66) yet decodes *faster*. Accept length is a proxy, not the metric +— throughput is `(accept + 1) / (draft time + verify time)`. The DFlash2 +drafter is a small separate model that steps fast; the MTP head is the +target's own layer applied autoregressively 8×, so its draft is slower. The +faster step more than makes up the lower accept length. FP8 + DFlash2 (100.0 +t/s) also beats the original MXFP4 + DFlash2 (92.4 t/s) — the higher-precision +FP8 target verifies more of the drafter's proposals. The drafter costs 2.1 GiB +of VRAM (the KV pool is 1.47× vs MTP's 1.76×), an acceptable trade for the +decode win. **DFlash2 is the deployed shape.** + ## Prefix caching — measured granularity (2026-09-20) `--enable-prefix-caching` is on, and radiance says so at boot: *"Prefix caching diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 83b6311..1cb2d90 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -11,6 +11,33 @@ timestamp: 2026-09-14 # Bundle Change Log +## 2026-09-20 — DFlash2 vs MTP A/B on the FP8 checkpoint: DFlash2 wins on decode + +- **The test.** The FP8 checkpoint carries its own single-layer MTP head, so + it can self-speculate (`SPEC_METHOD=mtp`) or use the z-lab DFlash2 external + drafter (`SPEC_METHOD=dflash`). Both were deployed and measured on the same + FP8 target (`scripts/bench-conversation.py`, 10 real prompts, `max_tokens=512`, + greedy, salted, `--burst 4`): + - **DFlash2** (depth 7): mean accept 4.24/7, geomean decode **100.0 t/s**, + TTFT 0.19 s, prefill 923.0 t/s, burst-4 **287.1 t/s**, KV pool 384,238 (1.47×). + - **MTP** (depth 8): mean accept 5.66/8, geomean decode 85.9 t/s, TTFT 0.06 s, + prefill 1096.9 t/s, burst-4 179.7 t/s, KV pool 461,540 (1.76×). +- **Verdict: DFlash2 wins on decode (+16 %) and burst-4 (+60 %); MTP wins on + TTFT and prefill.** The counterintuitive part: DFlash2 accepts *fewer* + tokens per step (4.24 vs 5.66) yet decodes *faster*. Accept length is a + proxy, not the metric — throughput is `(accept + 1) / (draft time + verify + time)`. The DFlash2 drafter is a small separate model that steps fast; the + MTP head is the target's own layer applied autoregressively 8×, so its draft + is slower. The faster step more than makes up the lower accept length. +- **FP8 + DFlash2 (100.0 t/s) beats the original MXFP4 + DFlash2 (92.4 t/s)** + — the higher-precision FP8 target verifies more of the drafter's proposals. + The drafter costs 2.1 GiB of VRAM (KV pool 1.47× vs MTP's 1.76×), an + acceptable trade for the decode win. +- **DFlash2 is the deployed shape.** The quadlet comment, the radiance + runbook (deployed-flags table, KV pool, a new A/B section), AGENTS.md + (topology row + frank-medium section), and the model-swap runbook + swap + script (both paths now default to `dflash/7`) all updated. + ## 2026-09-20 — frank-medium checkpoint swapped MXFP4 → FP8; the swap is now one command - **The swap.** `just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` (MXFP4,