From d8dbd4f5bc2644d55b79af08460c43a8c05546e8 Mon Sep 17 00:00:00 2001 From: Frank Date: Mon, 21 Sep 2026 17:08:33 -0700 Subject: [PATCH] docs(frank-medium): record the MXFP4 + clean-KV cut; correct vision claim Knowledge-base follow-up to a45f396. The runbook, both index one-liners, and the AGENTS.md topology row all still declared FP8 active; flipped to just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq with unquantized bf16 KV (pool 486,464 tokens, full 262,144 window), and recorded the causal nuance: fp8 KV was on both arms, so the FP8-vs-MXFP4 gap was weight precision, not KV - clean KV is a quality hypothesis for W4A8, not a throughput one. No verdict yet. The quadlet's "TEXT ONLY ... 400 here" comment was wrong: vision works on the MXFP4 checkpoint (Qwen3_5ForConditionalGeneration carries the vision tower through the Gemma4 merge path, unquantized) - verified by naming a solid red 48x48 PNG "Red". Comment corrected; harness models.yml input stays [text,image]. Docs-only; no re-deploy (the endpoint already serves vision). --- AGENTS.md | 2 +- .../quadlet/frank-medium.container | 10 ++-- .../LLM/frank-medium-radiance-mxfp4.md | 49 +++++++++++-------- serverAppConfigs-docs/LLM/index.md | 25 +++++----- serverAppConfigs-docs/index.md | 14 +++--- serverAppConfigs-docs/log.md | 22 +++++++++ 6 files changed, 76 insertions(+), 46 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 286911c..bb0d504 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| -| Dankserver | `frank-medium` | vllm-radiance 0.9.3 (rootless Quadlet, image `localhost/radiance-vllm:0.9.3`, gfx1201) | orcarouter Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, 30.9 GB / 7 shards) with the z-lab DFlash2 external drafter (depth 7, live mean accept length 4.24/7 — A/B tested and beats MTP self-speculation on decode: 100.0 vs 85.9 t/s), 2x Radeon AI PRO R9700 in ONE tensor-parallel process (TP=2 over the R4D one-shot allreduce — no RCCL collective; both cards on CPU root ports at Gen5 x8 since the 2026-09-20 ProArt B850-Creator swap), 262144 ctx over up to 8 conversations (auto-sized 384,238-token KV pool = 1.47x the window), fp8 KV, prefix caching, vision | 7770 | +| Dankserver | `frank-medium` | vllm-radiance 0.9.3 (rootless Quadlet, image `localhost/radiance-vllm:0.9.3`, gfx1201) | just1moremodel Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4/quark W4A8, 19.37 GiB) with the z-lab DFlash2 external drafter (depth 7), 2x Radeon AI PRO R9700 in ONE tensor-parallel process (TP=2 over the R4D one-shot allreduce — no RCCL collective; both cards on CPU root ports at Gen5 x8 since the 2026-09-20 ProArt B850-Creator swap), 262144 ctx over up to 8 conversations, UNQUANTIZED bf16 KV (`KV_DTYPE=auto`, cut 2026-09-21 from the FP8/fp8-KV shape to test whether fp8 KV had been degrading W4A8 output quality; auto-sized 486,464-token pool = 1.86x the window), prefix caching, vision | 7770 | | Dankclaw | `frank-large` | unsloth llama.cpp b11030 prebuilt (ROCm gfx1151, root systemd unit) | unsloth Qwen3.8-Flash-Next UD-IQ4_XS on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 200000 ctx single stream (1 slot), MTP depth 2 via the explicit `-md` shared-Q8_0 head, text only | 7770 | | Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 | diff --git a/dankserver/frank-medium/quadlet/frank-medium.container b/dankserver/frank-medium/quadlet/frank-medium.container index 35ee1c9..d8913c5 100644 --- a/dankserver/frank-medium/quadlet/frank-medium.container +++ b/dankserver/frank-medium/quadlet/frank-medium.container @@ -110,10 +110,12 @@ Environment=MAXLEN=262144 # excludes the others until it finishes (vLLM preempts rather than refuses). # The llama deployment it replaced advertised 327,680 across 2 slots. # -# TEXT ONLY. The MXFP4 checkpoint carries language_model_only=true and the -# launcher loads no vision projector, so an input_audio/image request is a -# 400 here. Vision traffic routes to frank-small on dankclaw (harness -# modelRoles vision -> frank-small). +# VISION WORKS (verified 2026-09-21: a solid red 48x48 PNG is named "Red"). +# config.json sets language_model_only=true and there is no separate mmproj, +# but the Qwen3_5ForConditionalGeneration architecture carries the vision tower +# through the Gemma4 merge path and it is not MXFP4-quantized, so vLLM serves +# image input on either checkpoint. harness models.yml keeps frank-medium +# input: [text, image]. Environment=GPU_UTIL=0.98 # Speculative decoding: DFlash2 external drafter at depth 7. A/B tested # against MTP self-speculation (depth 8) on this FP8 checkpoint: DFlash2 diff --git a/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md b/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md index 8fc64e8..63a4fbf 100644 --- a/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md +++ b/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md @@ -1,7 +1,7 @@ --- type: configuration title: frank-medium vllm-radiance MXFP4 Runbook -description: ACTIVE frank-medium runbook since 2026-09-20 — vllm-radiance serving orcarouter/Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, MTP self-speculation) as ONE tensor-parallel process (TP=2) on Dankserver's two AMD Radeon AI PRO R9700 (gfx1201). The quantization-aware entrypoint serves the plain FP8 path; the MXFP4 checkpoint (just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq + the z-lab DFlash2-FP8 drafter) remains on disk as the rollback target. Model-swap process: [[frank-medium-model-swap]]. The sections below describe the 2026-09-16 two-replica era. +description: ACTIVE frank-medium runbook — vllm-radiance serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4/quark W4A8, DFlash2 drafter, unquantized bf16 KV) as ONE tensor-parallel process (TP=2) on Dankserver's two AMD Radeon AI PRO R9700 (gfx1201). Returned to MXFP4 on 2026-09-21 with KV_DTYPE=auto (clean KV) to test whether the endpoint's historical fp8 KV had been degrading W4A8 attention quality; the FP8 checkpoint (orcarouter/Qwen3.8-27B-Uncensored-FP8, the 2026-09-20 to 09-21 shape) is the on-disk rollback. Model-swap process: [[frank-medium-model-swap]]. The sections below describe the 2026-09-16 two-replica era. tags: - frank-medium - dankserver @@ -28,15 +28,21 @@ It replaced the mainline-llama.cpp layer-split deployment model had earlier run here as two single-GPU replicas, which is the shape most of this page describes. -**Checkpoint swapped to FP8 on 2026-09-20.** The endpoint now serves -`orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3 block-wise, 30.9 GB / 7 -shards) with MTP self-speculation — the quantization-aware entrypoint reads -`quant_method` from `config.json` and serves the plain FP8 path (no MXFP4 -patches, `--quantization fp8`, `ROCM_AITER_UNIFIED_ATTN`). The MXFP4 -checkpoint above remains on disk as the rollback target. The swap process, -the toolchain, and the measured trade-off are in [[frank-medium-model-swap]]; -the "Deployed flags" table below still shows the MXFP4 shape (the rollback -target). +**Serving shape since 2026-09-21: MXFP4 with unquantized (bf16) KV.** This +endpoint briefly served `orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3, +`--quantization fp8`, DFlash2 drafter) from 2026-09-20 to 2026-09-21; it has +since been cut back to the MXFP4 checkpoint above, now with +`--kv-cache-dtype auto` (unquantized bf16 KV) instead of the endpoint's +historical `fp8`. Hypothesis: fp8 KV sat on *both* the FP8 and MXFP4 arms (the +flag was on the common exec path), so it could not explain the FP8-vs-MXFP4 +*throughput* gap — but MXFP4's weights are already W4A8, and fp8 KV on top +compounds attention-path precision loss, which shows up as degraded *output +quality*, not t/s. **Measured on deploy**: `kv_cache_dtype=auto`, no fp8 +banner, pool **486,464 tokens** (`Maximum concurrency for 262,144 tokens per +request: 1`) — exactly half MXFP4's 943,581-token fp8 pool, still clearing the +full 262,144 window so `MAXLEN` stays native. No quality verdict yet; this is +the setup. Rollback: flip `SNAP` + `SPEC` back to the FP8 path with +`KV_DTYPE=fp8` (that shape auto-sized to ~384k and held the full window). **Deployed shape since 2026-09-20: one rootless Quadlet, `--tensor-parallel-size 2`.** The ProArt B850-Creator WiFi NEO board swap put both R9700s on CPU GPP root @@ -53,12 +59,12 @@ From `dankserver/frank-medium/quadlet/frank-medium.container`: | Setting | Value | |---|---| -| Model | `orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3 block-wise 128×128, 30.9 GB / 7 shards) | +| Model | `just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` (MXFP4 / quark, W4A8, 19.37 GiB single shard) | | Speculative | `--speculative-config {"method":"dflash","model":"/models/Qwen3.8-27B-DFlash2-FP8","num_speculative_tokens":7}` (the z-lab DFlash2 external drafter; mean acceptance 4.24/7. A/B tested against MTP self-speculation — DFlash2 wins on decode, see below) | | Parallelism | `--tensor-parallel-size 2` (`ROCR_VISIBLE_DEVICES=0,1`) | | Context | `--max-model-len 262144` | | Concurrency | `--max-num-seqs 8`, chunk 8192 | -| KV | fp8 (`--kv-cache-dtype fp8`), **auto-sized — no `KV_CACHE_MEMORY_MB`** | +| KV | auto = **unquantized bf16** (`--kv-cache-dtype auto`), **auto-sized — pool 486,464 tokens**; the endpoint's historical fp8 KV halved precision on top of MXFP4's W4A8 (rollback: `KV_DTYPE=fp8` -> 943,581 tokens) | | Prompt cache | `--enable-prefix-caching` | | GPU fill | `--gpu-memory-utilization 0.98` | | Parsers | `--enable-auto-tool-choice --tool-call-parser qwen3_coder`, `--reasoning-parser qwen3` | @@ -68,15 +74,16 @@ From `dankserver/frank-medium/quadlet/frank-medium.container`: **The KV pool is auto-sized, not pinned.** Production ships no `KV_MEM` pin (`KV_MEM=0`): the pool comes up at whatever VRAM the weights + drafter leave. -The FP8 checkpoint is bigger than the MXFP4 one (30.9 vs 19.4 GiB of -weights), and the DFlash2 drafter is a separate 2.1 GiB model, so the pool -is the smallest of the three shapes: the 2026-09-20 DFlash2 start auto-sized -to **384,238 tokens = 1.47× the full window** (`GPU KV cache size: 384,238 -tokens`, `Maximum concurrency for 262,144 tokens per request: 1.47x`). The -MTP shape (no external drafter) auto-sized to 461,540 (1.76×); the MXFP4 -rollback target to 943,581 (3.59×). After the first healthy start, -recalibrate with `calibrate-kv.sh` to use the extra VRAM on the 34.2 GiB -cards. +MXFP4's weights are smaller than FP8's (19.4 vs 30.9 GiB), which frees VRAM for +the pool — so even with unquantized bf16 KV the full window fits. The 2026-09-21 +clean-KV start auto-sized to **486,464 tokens** (`GPU KV cache size: 486,464 +tokens`, `Maximum concurrency for 262,144 tokens per request: 1`) — 1.86x the +full window, and exactly half MXFP4's fp8-KV pool of 943,581 (the 2x f16-vs-fp8 +KV ratio). For reference the FP8 checkpoint (now the rollback) auto-sized to +384,238 with fp8 KV (1.47x) and would only reach ~192,000 unquantized — which +is why FP8 and MXFP4 differ on the window, and why MXFP4 is the shape that +affords clean KV at the native 262,144. After the first healthy start, +recalibrate with `calibrate-kv.sh`. ## DFlash2 vs MTP — A/B on the FP8 checkpoint (2026-09-20) diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index 14af88f..adf6dc1 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -2,8 +2,8 @@ Documentation for the LLM inference fleet on the two hosts (2026-09-21): **vllm-radiance on Dankserver** serving frank-medium as ONE tensor-parallel -process across both Radeon AI PRO R9700 cards (orcarouter -Qwen3.8-27B-Uncensored-FP8 + the z-lab DFlash2 drafter, TP=2 over the +process across both Radeon AI PRO R9700 cards (just1moremodel +Qwen3.8-27B-Uncensored-MXFP4 + the z-lab DFlash2 drafter, bf16 KV, TP=2 over the R4D one-shot allreduce), and TWO endpoints on Dankclaw — the unsloth llama.cpp prebuilt (frank-large, Qwen3.8-Flash-Next UD-IQ4_XS on the gfx1151 iGPU, port 7770) and mainline llama.cpp (frank-small, HauhauCS Gemma4-12B QAT @@ -23,7 +23,7 @@ graph LR gfx1201, 32 GiB each] B2[vllm-radiance 0.9.3, TP=2 frank-medium, port 7770] - C[Qwen3.8-27B FP8 + DFlash2 drafter + C[Qwen3.8-27B MXFP4 + DFlash2 drafter ONE process on both cards, 262144 ctx, up to 8 seqs] A --> B2 --> C end @@ -57,16 +57,15 @@ frank-small, port 7771] ## Active Deployment -- [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook since - 2026-09-20: vllm-radiance (stilldeadcode/vllm-radiance:0.9.3) serving - orcarouter/Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, 30.9 GB / 7 - shards) as ONE tensor-parallel process (TP=2) behind :7770, with MTP - self-speculation (the model's own single-layer MTP head). The quantization- - aware entrypoint serves the plain FP8 path; the MXFP4 checkpoint - (just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq + the z-lab DFlash2-FP8 - drafter) remains on disk as the rollback target. The three-arm battery that - chose TP=2, the auto-sized KV pool, and verified capabilities (tools, - reasoning, vision) +- [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook: + vllm-radiance (stilldeadcode/vllm-radiance:0.9.3) serving + just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4/quark W4A8) with the + z-lab DFlash2-FP8 drafter as ONE tensor-parallel process (TP=2) behind + :7770, now on unquantized bf16 KV (`KV_DTYPE=auto`, cut 2026-09-21 to test + whether the endpoint's historical fp8 KV had been degrading W4A8 quality). + The FP8 checkpoint (orcarouter/Qwen3.8-27B-Uncensored-FP8, the 09-20→09-21 + shape) is the on-disk rollback. The three-arm battery that chose TP=2, the + auto-sized KV pool, and verified capabilities (tools, reasoning) - [[frank-medium-model-swap]] — how to swap the checkpoint behind frank-medium: the streamlining toolchain (hf-manifest.py, hf-fetch.sh, swap-frank-medium.sh), the quantization-aware entrypoint contract, the diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index 63717e4..7200a53 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -115,13 +115,13 @@ The LLM inference fleet on the two hosts. Full index: - [[deployment]] — unified deploy and update process for both servers - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change -- [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook since - 2026-09-20: vllm-radiance as ONE tensor-parallel process (TP=2 over the R4D - one-shot allreduce) serving orcarouter/Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 - block-wise, MTP self-speculation) on Dankserver's 2x Radeon AI PRO R9700 - behind the :7770 endpoint — the quantization-aware entrypoint, the - three-arm battery that chose TP=2, verified capabilities, and the record of - the 2026-09-16 two-replica era +- [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook: + vllm-radiance as ONE tensor-parallel process (TP=2 over the R4D one-shot + allreduce) serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4 + W4A8, DFlash2 drafter, unquantized bf16 KV since 2026-09-21) on Dankserver's + 2x Radeon AI PRO R9700 behind the :7770 endpoint — the quantization-aware + entrypoint, the three-arm battery that chose TP=2, verified capabilities, + and the record of the 2026-09-16 two-replica era - [[frank-medium-model-swap]] — how to swap the checkpoint behind frank-medium: the streamlining toolchain (hf-manifest.py, hf-fetch.sh, swap-frank-medium.sh), the quantization-aware entrypoint contract, the diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 51f7a33..0999727 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -11,6 +11,28 @@ timestamp: 2026-09-14 # Bundle Change Log +## 2026-09-21 — frank-medium cut back to MXFP4 with unquantized bf16 KV + +- **The change.** frank-medium (dankserver :7770) served the FP8 checkpoint + (`orcarouter/Qwen3.8-27B-Uncensored-FP8`) with fp8 KV from 2026-09-20 to + 09-21; it is now cut back to the MXFP4 checkpoint + (`just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq`, quark/W4A8) with + `--kv-cache-dtype auto` — unquantized bf16 KV instead of the endpoint's + historical fp8. `KV_DTYPE` is now a quadlet tunable (entrypoint default fp8); + `MAXLEN` stays at the native 262,144. Commit a45f396. +- **Why.** The operator's read that fp8 KV was hurting MXFP4. The nuance: fp8 + KV sat on BOTH the FP8 and MXFP4 arms (hardcoded on the common exec line in + every commit since the MXFP4 era), so it could not explain the FP8-vs-MXFP4 + *throughput* gap — that was weight precision. But MXFP4 is W4A8, and fp8 KV + on top compounds attention-path precision loss; clean KV tests whether that + was degrading *output quality*. No verdict yet — this is the setup. +- **Measured on deploy.** `kv_cache_dtype=auto`, no fp8 banner; pool auto-sized + to **486,464 tokens** (`Maximum concurrency for 262,144 tokens: 1`) — exactly + half MXFP4's 943,581-token fp8 pool (the 2x f16/fp8 ratio), and MXFP4's freed + weights (19.4 vs 30.9 GiB) keep the full 262,144 window fitting. Vision + re-verified on MXFP4 (a solid red 48x48 PNG named "Red"); the quadlet's old + "TEXT ONLY" comment was wrong and is corrected. e2e 32/32. + ## 2026-09-21 — Calibre Content Server deployed (dankserver :8084) - **The change.** The Calibre Content Server (`calibre-server`) is now a