dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(frank-medium): record the MXFP4 + clean-KV cut; correct vision claim

Browse Source 
Knowledge-base follow-up to a45f396. The runbook, both index one-liners, and
the AGENTS.md topology row all still declared FP8 active; flipped to
just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq with unquantized bf16 KV (pool
486,464 tokens, full 262,144 window), and recorded the causal nuance: fp8 KV
was on both arms, so the FP8-vs-MXFP4 gap was weight precision, not KV - clean
KV is a quality hypothesis for W4A8, not a throughput one. No verdict yet.

The quadlet's "TEXT ONLY ... 400 here" comment was wrong: vision works on the
MXFP4 checkpoint (Qwen3_5ForConditionalGeneration carries the vision tower
through the Gemma4 merge path, unquantized) - verified by naming a solid red
48x48 PNG "Red". Comment corrected; harness models.yml input stays [text,image].
Docs-only; no re-deploy (the endpoint already serves vision).
This commit is contained in:
Frank
2026-09-21 17:08:33 -07:00
parent a45f396c39
commit d8dbd4f5bc
6 changed files with 76 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md
+28 -21
View File
@@ -1,7 +1,7 @@
---
type: configuration
title: frank-medium vllm-radiance MXFP4 Runbook
description: ACTIVE frank-medium runbook since 2026-09-20 — vllm-radiance serving orcarouter/Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, MTP self-speculation) as ONE tensor-parallel process (TP=2) on Dankserver's two AMD Radeon AI PRO R9700 (gfx1201). The quantization-aware entrypoint serves the plain FP8 path; the MXFP4 checkpoint (just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq + the z-lab DFlash2-FP8 drafter) remains on disk as the rollback target. Model-swap process: [[frank-medium-model-swap]]. The sections below describe the 2026-09-16 two-replica era.
description: ACTIVE frank-medium runbook — vllm-radiance serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4/quark W4A8, DFlash2 drafter, unquantized bf16 KV) as ONE tensor-parallel process (TP=2) on Dankserver's two AMD Radeon AI PRO R9700 (gfx1201). Returned to MXFP4 on 2026-09-21 with KV_DTYPE=auto (clean KV) to test whether the endpoint's historical fp8 KV had been degrading W4A8 attention quality; the FP8 checkpoint (orcarouter/Qwen3.8-27B-Uncensored-FP8, the 2026-09-20 to 09-21 shape) is the on-disk rollback. Model-swap process: [[frank-medium-model-swap]]. The sections below describe the 2026-09-16 two-replica era.
tags:
- frank-medium
- dankserver
@@ -28,15 +28,21 @@ It replaced the mainline-llama.cpp layer-split deployment
model had earlier run here as two single-GPU replicas, which is the shape most
of this page describes.
**Checkpoint swapped to FP8 on 2026-09-20.** The endpoint now serves
`orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3 block-wise, 30.9 GB / 7
shards) with MTP self-speculation — the quantization-aware entrypoint reads
`quant_method` from `config.json` and serves the plain FP8 path (no MXFP4
patches, `--quantization fp8`, `ROCM_AITER_UNIFIED_ATTN`). The MXFP4
checkpoint above remains on disk as the rollback target. The swap process,
the toolchain, and the measured trade-off are in [[frank-medium-model-swap]];
the "Deployed flags" table below still shows the MXFP4 shape (the rollback
target).
**Serving shape since 2026-09-21: MXFP4 with unquantized (bf16) KV.** This
endpoint briefly served `orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3,
`--quantization fp8`, DFlash2 drafter) from 2026-09-20 to 2026-09-21; it has
since been cut back to the MXFP4 checkpoint above, now with
`--kv-cache-dtype auto` (unquantized bf16 KV) instead of the endpoint's
historical `fp8`. Hypothesis: fp8 KV sat on *both* the FP8 and MXFP4 arms (the
flag was on the common exec path), so it could not explain the FP8-vs-MXFP4
*throughput* gap — but MXFP4's weights are already W4A8, and fp8 KV on top
compounds attention-path precision loss, which shows up as degraded *output
quality*, not t/s. **Measured on deploy**: `kv_cache_dtype=auto`, no fp8
banner, pool **486,464 tokens** (`Maximum concurrency for 262,144 tokens per
request: 1`) — exactly half MXFP4's 943,581-token fp8 pool, still clearing the
full 262,144 window so `MAXLEN` stays native. No quality verdict yet; this is
the setup. Rollback: flip `SNAP` + `SPEC` back to the FP8 path with
`KV_DTYPE=fp8` (that shape auto-sized to ~384k and held the full window).
**Deployed shape since 2026-09-20: one rootless Quadlet, `--tensor-parallel-size 2`.**
The ProArt B850-Creator WiFi NEO board swap put both R9700s on CPU GPP root
@@ -53,12 +59,12 @@ From `dankserver/frank-medium/quadlet/frank-medium.container`:
| Setting | Value |
|---|---|
| Model | `orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3 block-wise 128×128, 30.9 GB / 7 shards) |
| Model | `just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` (MXFP4 / quark, W4A8, 19.37 GiB single shard) |
| Speculative | `--speculative-config {"method":"dflash","model":"/models/Qwen3.8-27B-DFlash2-FP8","num_speculative_tokens":7}` (the z-lab DFlash2 external drafter; mean acceptance 4.24/7. A/B tested against MTP self-speculation — DFlash2 wins on decode, see below) |
| Parallelism | `--tensor-parallel-size 2` (`ROCR_VISIBLE_DEVICES=0,1`) |
| Context | `--max-model-len 262144` |
| Concurrency | `--max-num-seqs 8`, chunk 8192 |
| KV | fp8 (`--kv-cache-dtype fp8`), **auto-sized — no `KV_CACHE_MEMORY_MB`** |
| KV | auto = **unquantized bf16** (`--kv-cache-dtype auto`), **auto-sized — pool 486,464 tokens**; the endpoint's historical fp8 KV halved precision on top of MXFP4's W4A8 (rollback: `KV_DTYPE=fp8` -> 943,581 tokens) |
| Prompt cache | `--enable-prefix-caching` |
| GPU fill | `--gpu-memory-utilization 0.98` |
| Parsers | `--enable-auto-tool-choice --tool-call-parser qwen3_coder`, `--reasoning-parser qwen3` |
@@ -68,15 +74,16 @@ From `dankserver/frank-medium/quadlet/frank-medium.container`:
**The KV pool is auto-sized, not pinned.** Production ships no `KV_MEM` pin
(`KV_MEM=0`): the pool comes up at whatever VRAM the weights + drafter leave.
The FP8 checkpoint is bigger than the MXFP4 one (30.9 vs 19.4 GiB of
weights), and the DFlash2 drafter is a separate 2.1 GiB model, so the pool
is the smallest of the three shapes: the 2026-09-20 DFlash2 start auto-sized
to **384,238 tokens = 1.47× the full window** (`GPU KV cache size: 384,238
tokens`, `Maximum concurrency for 262,144 tokens per request: 1.47x`). The
MTP shape (no external drafter) auto-sized to 461,540 (1.76×); the MXFP4
rollback target to 943,581 (3.59×). After the first healthy start,
recalibrate with `calibrate-kv.sh` to use the extra VRAM on the 34.2 GiB
cards.
MXFP4's weights are smaller than FP8's (19.4 vs 30.9 GiB), which frees VRAM for
the pool — so even with unquantized bf16 KV the full window fits. The 2026-09-21
clean-KV start auto-sized to **486,464 tokens** (`GPU KV cache size: 486,464
tokens`, `Maximum concurrency for 262,144 tokens per request: 1`) — 1.86x the
full window, and exactly half MXFP4's fp8-KV pool of 943,581 (the 2x f16-vs-fp8
KV ratio). For reference the FP8 checkpoint (now the rollback) auto-sized to
384,238 with fp8 KV (1.47x) and would only reach ~192,000 unquantized — which
is why FP8 and MXFP4 differ on the window, and why MXFP4 is the shape that
affords clean KV at the native 262,144. After the first healthy start,
recalibrate with `calibrate-kv.sh`.
## DFlash2 vs MTP — A/B on the FP8 checkpoint (2026-09-20)
serverAppConfigs-docs/LLM/index.md
+12 -13
View File
@@ -2,8 +2,8 @@
Documentation for the LLM inference fleet on the two hosts (2026-09-21):
**vllm-radiance on Dankserver** serving frank-medium as ONE tensor-parallel
process across both Radeon AI PRO R9700 cards (orcarouter
Qwen3.8-27B-Uncensored-FP8 + the z-lab DFlash2 drafter, TP=2 over the
process across both Radeon AI PRO R9700 cards (just1moremodel
Qwen3.8-27B-Uncensored-MXFP4 + the z-lab DFlash2 drafter, bf16 KV, TP=2 over the
R4D one-shot allreduce), and TWO endpoints on Dankclaw — the unsloth
llama.cpp prebuilt (frank-large, Qwen3.8-Flash-Next UD-IQ4_XS on the gfx1151
iGPU, port 7770) and mainline llama.cpp (frank-small, HauhauCS Gemma4-12B QAT
@@ -23,7 +23,7 @@ graph LR
gfx1201, 32 GiB each]
B2[vllm-radiance 0.9.3, TP=2
frank-medium, port 7770]
C[Qwen3.8-27B FP8 + DFlash2 drafter
C[Qwen3.8-27B MXFP4 + DFlash2 drafter
ONE process on both cards, 262144 ctx, up to 8 seqs]
A --> B2 --> C
end
@@ -57,16 +57,15 @@ frank-small, port 7771]
## Active Deployment
- [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook since
2026-09-20: vllm-radiance (stilldeadcode/vllm-radiance:0.9.3) serving
orcarouter/Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, 30.9 GB / 7
shards) as ONE tensor-parallel process (TP=2) behind :7770, with MTP
self-speculation (the model's own single-layer MTP head). The quantization-
aware entrypoint serves the plain FP8 path; the MXFP4 checkpoint
(just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq + the z-lab DFlash2-FP8
drafter) remains on disk as the rollback target. The three-arm battery that
chose TP=2, the auto-sized KV pool, and verified capabilities (tools,
reasoning, vision)
- [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook:
vllm-radiance (stilldeadcode/vllm-radiance:0.9.3) serving
just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4/quark W4A8) with the
z-lab DFlash2-FP8 drafter as ONE tensor-parallel process (TP=2) behind
:7770, now on unquantized bf16 KV (`KV_DTYPE=auto`, cut 2026-09-21 to test
whether the endpoint's historical fp8 KV had been degrading W4A8 quality).
The FP8 checkpoint (orcarouter/Qwen3.8-27B-Uncensored-FP8, the 09-20→09-21
shape) is the on-disk rollback. The three-arm battery that chose TP=2, the
auto-sized KV pool, and verified capabilities (tools, reasoning)
- [[frank-medium-model-swap]] — how to swap the checkpoint behind frank-medium:
the streamlining toolchain (hf-manifest.py, hf-fetch.sh,
swap-frank-medium.sh), the quantization-aware entrypoint contract, the
serverAppConfigs-docs/index.md
+7 -7
View File
@@ -115,13 +115,13 @@ The LLM inference fleet on the two hosts. Full index:
- [[deployment]] — unified deploy and update process for both servers
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
- [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook since
2026-09-20: vllm-radiance as ONE tensor-parallel process (TP=2 over the R4D
one-shot allreduce) serving orcarouter/Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3
block-wise, MTP self-speculation) on Dankserver's 2x Radeon AI PRO R9700
behind the :7770 endpoint — the quantization-aware entrypoint, the
three-arm battery that chose TP=2, verified capabilities, and the record of
the 2026-09-16 two-replica era
- [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook:
vllm-radiance as ONE tensor-parallel process (TP=2 over the R4D one-shot
allreduce) serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4
W4A8, DFlash2 drafter, unquantized bf16 KV since 2026-09-21) on Dankserver's
2x Radeon AI PRO R9700 behind the :7770 endpoint — the quantization-aware
entrypoint, the three-arm battery that chose TP=2, verified capabilities,
and the record of the 2026-09-16 two-replica era
- [[frank-medium-model-swap]] — how to swap the checkpoint behind frank-medium:
the streamlining toolchain (hf-manifest.py, hf-fetch.sh,
swap-frank-medium.sh), the quantization-aware entrypoint contract, the
serverAppConfigs-docs/log.md
+22
View File
@@ -11,6 +11,28 @@ timestamp: 2026-09-14
# Bundle Change Log
## 2026-09-21 — frank-medium cut back to MXFP4 with unquantized bf16 KV
- **The change.** frank-medium (dankserver :7770) served the FP8 checkpoint
(`orcarouter/Qwen3.8-27B-Uncensored-FP8`) with fp8 KV from 2026-09-20 to
09-21; it is now cut back to the MXFP4 checkpoint
(`just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq`, quark/W4A8) with
`--kv-cache-dtype auto` — unquantized bf16 KV instead of the endpoint's
historical fp8. `KV_DTYPE` is now a quadlet tunable (entrypoint default fp8);
`MAXLEN` stays at the native 262,144. Commit a45f396.
- **Why.** The operator's read that fp8 KV was hurting MXFP4. The nuance: fp8
KV sat on BOTH the FP8 and MXFP4 arms (hardcoded on the common exec line in
every commit since the MXFP4 era), so it could not explain the FP8-vs-MXFP4
*throughput* gap — that was weight precision. But MXFP4 is W4A8, and fp8 KV
on top compounds attention-path precision loss; clean KV tests whether that
was degrading *output quality*. No verdict yet — this is the setup.
- **Measured on deploy.** `kv_cache_dtype=auto`, no fp8 banner; pool auto-sized
to **486,464 tokens** (`Maximum concurrency for 262,144 tokens: 1`) — exactly
half MXFP4's 943,581-token fp8 pool (the 2x f16/fp8 ratio), and MXFP4's freed
weights (19.4 vs 30.9 GiB) keep the full 262,144 window fitting. Vision
re-verified on MXFP4 (a solid red 48x48 PNG named "Red"); the quadlet's old
"TEXT ONLY" comment was wrong and is corrected. e2e 32/32.
## 2026-09-21 — Calibre Content Server deployed (dankserver :8084)
- **The change.** The Calibre Content Server (`calibre-server`) is now a