docs(frank-medium): record the MXFP4 + clean-KV cut; correct vision claim

Knowledge-base follow-up to a45f396. The runbook, both index one-liners, and
the AGENTS.md topology row all still declared FP8 active; flipped to
just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq with unquantized bf16 KV (pool
486,464 tokens, full 262,144 window), and recorded the causal nuance: fp8 KV
was on both arms, so the FP8-vs-MXFP4 gap was weight precision, not KV - clean
KV is a quality hypothesis for W4A8, not a throughput one. No verdict yet.

The quadlet's "TEXT ONLY ... 400 here" comment was wrong: vision works on the
MXFP4 checkpoint (Qwen3_5ForConditionalGeneration carries the vision tower
through the Gemma4 merge path, unquantized) - verified by naming a solid red
48x48 PNG "Red". Comment corrected; harness models.yml input stays [text,image].
Docs-only; no re-deploy (the endpoint already serves vision).

AGENTS.md

@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
 
 | Target | Service | Runtime | Model | Port |
 |--------|---------|---------|-------|------|
-| Dankserver | `frank-medium` | vllm-radiance 0.9.3 (rootless Quadlet, image `localhost/radiance-vllm:0.9.3`, gfx1201) | orcarouter Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, 30.9 GB / 7 shards) with the z-lab DFlash2 external drafter (depth 7, live mean accept length 4.24/7 — A/B tested and beats MTP self-speculation on decode: 100.0 vs 85.9 t/s), 2x Radeon AI PRO R9700 in ONE tensor-parallel process (TP=2 over the R4D one-shot allreduce — no RCCL collective; both cards on CPU root ports at Gen5 x8 since the 2026-09-20 ProArt B850-Creator swap), 262144 ctx over up to 8 conversations (auto-sized 384,238-token KV pool = 1.47x the window), fp8 KV, prefix caching, vision | 7770 |
+| Dankserver | `frank-medium` | vllm-radiance 0.9.3 (rootless Quadlet, image `localhost/radiance-vllm:0.9.3`, gfx1201) | just1moremodel Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4/quark W4A8, 19.37 GiB) with the z-lab DFlash2 external drafter (depth 7), 2x Radeon AI PRO R9700 in ONE tensor-parallel process (TP=2 over the R4D one-shot allreduce — no RCCL collective; both cards on CPU root ports at Gen5 x8 since the 2026-09-20 ProArt B850-Creator swap), 262144 ctx over up to 8 conversations, UNQUANTIZED bf16 KV (`KV_DTYPE=auto`, cut 2026-09-21 from the FP8/fp8-KV shape to test whether fp8 KV had been degrading W4A8 output quality; auto-sized 486,464-token pool = 1.86x the window), prefix caching, vision | 7770 |
 | Dankclaw | `frank-large` | unsloth llama.cpp b11030 prebuilt (ROCm gfx1151, root systemd unit) | unsloth Qwen3.8-Flash-Next UD-IQ4_XS on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 200000 ctx single stream (1 slot), MTP depth 2 via the explicit `-md` shared-Q8_0 head, text only | 7770 |
 | Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 |
 

dankserver/frank-medium/quadlet/frank-medium.container

@@ -110,10 +110,12 @@ Environment=MAXLEN=262144
 # excludes the others until it finishes (vLLM preempts rather than refuses).
 # The llama deployment it replaced advertised 327,680 across 2 slots.
 #
-# TEXT ONLY. The MXFP4 checkpoint carries language_model_only=true and the
+# VISION WORKS (verified 2026-09-21: a solid red 48x48 PNG is named "Red").
-# launcher loads no vision projector, so an input_audio/image request is a
+# config.json sets language_model_only=true and there is no separate mmproj,
-# 400 here. Vision traffic routes to frank-small on dankclaw (harness
+# but the Qwen3_5ForConditionalGeneration architecture carries the vision tower
-# modelRoles vision -> frank-small).
+# through the Gemma4 merge path and it is not MXFP4-quantized, so vLLM serves
+# image input on either checkpoint. harness models.yml keeps frank-medium
+# input: [text, image].
 Environment=GPU_UTIL=0.98
 # Speculative decoding: DFlash2 external drafter at depth 7. A/B tested
 # against MTP self-speculation (depth 8) on this FP8 checkpoint: DFlash2

serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md

@@ -1,7 +1,7 @@
 ---
 type: configuration
 title: frank-medium vllm-radiance MXFP4 Runbook
-description: ACTIVE frank-medium runbook since 2026-09-20 — vllm-radiance serving orcarouter/Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, MTP self-speculation) as ONE tensor-parallel process (TP=2) on Dankserver's two AMD Radeon AI PRO R9700 (gfx1201). The quantization-aware entrypoint serves the plain FP8 path; the MXFP4 checkpoint (just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq + the z-lab DFlash2-FP8 drafter) remains on disk as the rollback target. Model-swap process: [[frank-medium-model-swap]]. The sections below describe the 2026-09-16 two-replica era.
+description: ACTIVE frank-medium runbook — vllm-radiance serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4/quark W4A8, DFlash2 drafter, unquantized bf16 KV) as ONE tensor-parallel process (TP=2) on Dankserver's two AMD Radeon AI PRO R9700 (gfx1201). Returned to MXFP4 on 2026-09-21 with KV_DTYPE=auto (clean KV) to test whether the endpoint's historical fp8 KV had been degrading W4A8 attention quality; the FP8 checkpoint (orcarouter/Qwen3.8-27B-Uncensored-FP8, the 2026-09-20 to 09-21 shape) is the on-disk rollback. Model-swap process: [[frank-medium-model-swap]]. The sections below describe the 2026-09-16 two-replica era.
 tags:
   - frank-medium
   - dankserver
@@ -28,15 +28,21 @@ It replaced the mainline-llama.cpp layer-split deployment
 model had earlier run here as two single-GPU replicas, which is the shape most
 of this page describes.
 
-**Checkpoint swapped to FP8 on 2026-09-20.** The endpoint now serves
+**Serving shape since 2026-09-21: MXFP4 with unquantized (bf16) KV.** This
-`orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3 block-wise, 30.9 GB / 7
+endpoint briefly served `orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3,
-shards) with MTP self-speculation — the quantization-aware entrypoint reads
+`--quantization fp8`, DFlash2 drafter) from 2026-09-20 to 2026-09-21; it has
-`quant_method` from `config.json` and serves the plain FP8 path (no MXFP4
+since been cut back to the MXFP4 checkpoint above, now with
-patches, `--quantization fp8`, `ROCM_AITER_UNIFIED_ATTN`). The MXFP4
+`--kv-cache-dtype auto` (unquantized bf16 KV) instead of the endpoint's
-checkpoint above remains on disk as the rollback target. The swap process,
+historical `fp8`. Hypothesis: fp8 KV sat on *both* the FP8 and MXFP4 arms (the
-the toolchain, and the measured trade-off are in [[frank-medium-model-swap]];
+flag was on the common exec path), so it could not explain the FP8-vs-MXFP4
-the "Deployed flags" table below still shows the MXFP4 shape (the rollback
+*throughput* gap — but MXFP4's weights are already W4A8, and fp8 KV on top
-target).
+compounds attention-path precision loss, which shows up as degraded *output
+quality*, not t/s. **Measured on deploy**: `kv_cache_dtype=auto`, no fp8
+banner, pool **486,464 tokens** (`Maximum concurrency for 262,144 tokens per
+request: 1`) — exactly half MXFP4's 943,581-token fp8 pool, still clearing the
+full 262,144 window so `MAXLEN` stays native. No quality verdict yet; this is
+the setup. Rollback: flip `SNAP` + `SPEC` back to the FP8 path with
+`KV_DTYPE=fp8` (that shape auto-sized to ~384k and held the full window).
 
 **Deployed shape since 2026-09-20: one rootless Quadlet, `--tensor-parallel-size 2`.**
 The ProArt B850-Creator WiFi NEO board swap put both R9700s on CPU GPP root
@@ -53,12 +59,12 @@ From `dankserver/frank-medium/quadlet/frank-medium.container`:
 
 | Setting | Value |
 |---|---|
-| Model | `orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3 block-wise 128×128, 30.9 GB / 7 shards) |
+| Model | `just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` (MXFP4 / quark, W4A8, 19.37 GiB single shard) |
 | Speculative | `--speculative-config {"method":"dflash","model":"/models/Qwen3.8-27B-DFlash2-FP8","num_speculative_tokens":7}` (the z-lab DFlash2 external drafter; mean acceptance 4.24/7. A/B tested against MTP self-speculation — DFlash2 wins on decode, see below) |
 | Parallelism | `--tensor-parallel-size 2` (`ROCR_VISIBLE_DEVICES=0,1`) |
 | Context | `--max-model-len 262144` |
 | Concurrency | `--max-num-seqs 8`, chunk 8192 |
-| KV | fp8 (`--kv-cache-dtype fp8`), **auto-sized — no `KV_CACHE_MEMORY_MB`** |
+| KV | auto = **unquantized bf16** (`--kv-cache-dtype auto`), **auto-sized — pool 486,464 tokens**; the endpoint's historical fp8 KV halved precision on top of MXFP4's W4A8 (rollback: `KV_DTYPE=fp8` -> 943,581 tokens) |
 | Prompt cache | `--enable-prefix-caching` |
 | GPU fill | `--gpu-memory-utilization 0.98` |
 | Parsers | `--enable-auto-tool-choice --tool-call-parser qwen3_coder`, `--reasoning-parser qwen3` |
@@ -68,15 +74,16 @@ From `dankserver/frank-medium/quadlet/frank-medium.container`:
 
 **The KV pool is auto-sized, not pinned.** Production ships no `KV_MEM` pin
 (`KV_MEM=0`): the pool comes up at whatever VRAM the weights + drafter leave.
-The FP8 checkpoint is bigger than the MXFP4 one (30.9 vs 19.4 GiB of
+MXFP4's weights are smaller than FP8's (19.4 vs 30.9 GiB), which frees VRAM for
-weights), and the DFlash2 drafter is a separate 2.1 GiB model, so the pool
+the pool — so even with unquantized bf16 KV the full window fits. The 2026-09-21
-is the smallest of the three shapes: the 2026-09-20 DFlash2 start auto-sized
+clean-KV start auto-sized to **486,464 tokens** (`GPU KV cache size: 486,464
-to **384,238 tokens = 1.47× the full window** (`GPU KV cache size: 384,238
+tokens`, `Maximum concurrency for 262,144 tokens per request: 1`) — 1.86x the
-tokens`, `Maximum concurrency for 262,144 tokens per request: 1.47x`). The
+full window, and exactly half MXFP4's fp8-KV pool of 943,581 (the 2x f16-vs-fp8
-MTP shape (no external drafter) auto-sized to 461,540 (1.76×); the MXFP4
+KV ratio). For reference the FP8 checkpoint (now the rollback) auto-sized to
-rollback target to 943,581 (3.59×). After the first healthy start,
+384,238 with fp8 KV (1.47x) and would only reach ~192,000 unquantized — which
-recalibrate with `calibrate-kv.sh` to use the extra VRAM on the 34.2 GiB
+is why FP8 and MXFP4 differ on the window, and why MXFP4 is the shape that
-cards.
+affords clean KV at the native 262,144. After the first healthy start,
+recalibrate with `calibrate-kv.sh`.
 
 ## DFlash2 vs MTP — A/B on the FP8 checkpoint (2026-09-20)
 

serverAppConfigs-docs/LLM/index.md

@@ -2,8 +2,8 @@
 
 Documentation for the LLM inference fleet on the two hosts (2026-09-21):
 **vllm-radiance on Dankserver** serving frank-medium as ONE tensor-parallel
-process across both Radeon AI PRO R9700 cards (orcarouter
+process across both Radeon AI PRO R9700 cards (just1moremodel
-Qwen3.8-27B-Uncensored-FP8 + the z-lab DFlash2 drafter, TP=2 over the
+Qwen3.8-27B-Uncensored-MXFP4 + the z-lab DFlash2 drafter, bf16 KV, TP=2 over the
 R4D one-shot allreduce), and TWO endpoints on Dankclaw — the unsloth
 llama.cpp prebuilt (frank-large, Qwen3.8-Flash-Next UD-IQ4_XS on the gfx1151
 iGPU, port 7770) and mainline llama.cpp (frank-small, HauhauCS Gemma4-12B QAT
@@ -23,7 +23,7 @@ graph LR
 gfx1201, 32 GiB each]
         B2[vllm-radiance 0.9.3, TP=2
 frank-medium, port 7770]
-        C[Qwen3.8-27B FP8 + DFlash2 drafter
+        C[Qwen3.8-27B MXFP4 + DFlash2 drafter
 ONE process on both cards, 262144 ctx, up to 8 seqs]
         A --> B2 --> C
     end
@@ -57,16 +57,15 @@ frank-small, port 7771]
 
 ## Active Deployment
 
-- [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook since
+- [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook:
-  2026-09-20: vllm-radiance (stilldeadcode/vllm-radiance:0.9.3) serving
+  vllm-radiance (stilldeadcode/vllm-radiance:0.9.3) serving
-  orcarouter/Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, 30.9 GB / 7
+  just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4/quark W4A8) with the
-  shards) as ONE tensor-parallel process (TP=2) behind :7770, with MTP
+  z-lab DFlash2-FP8 drafter as ONE tensor-parallel process (TP=2) behind
-  self-speculation (the model's own single-layer MTP head). The quantization-
+  :7770, now on unquantized bf16 KV (`KV_DTYPE=auto`, cut 2026-09-21 to test
-  aware entrypoint serves the plain FP8 path; the MXFP4 checkpoint
+  whether the endpoint's historical fp8 KV had been degrading W4A8 quality).
-  (just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq + the z-lab DFlash2-FP8
+  The FP8 checkpoint (orcarouter/Qwen3.8-27B-Uncensored-FP8, the 09-20→09-21
-  drafter) remains on disk as the rollback target. The three-arm battery that
+  shape) is the on-disk rollback. The three-arm battery that chose TP=2, the
-  chose TP=2, the auto-sized KV pool, and verified capabilities (tools,
+  auto-sized KV pool, and verified capabilities (tools, reasoning)
-  reasoning, vision)
 - [[frank-medium-model-swap]] — how to swap the checkpoint behind frank-medium:
   the streamlining toolchain (hf-manifest.py, hf-fetch.sh,
   swap-frank-medium.sh), the quantization-aware entrypoint contract, the

serverAppConfigs-docs/index.md

@@ -115,13 +115,13 @@ The LLM inference fleet on the two hosts. Full index:
 
 - [[deployment]] — unified deploy and update process for both servers
 - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
-- [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook since
+- [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook:
-  2026-09-20: vllm-radiance as ONE tensor-parallel process (TP=2 over the R4D
+  vllm-radiance as ONE tensor-parallel process (TP=2 over the R4D one-shot
-  one-shot allreduce) serving orcarouter/Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3
+  allreduce) serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4
-  block-wise, MTP self-speculation) on Dankserver's 2x Radeon AI PRO R9700
+  W4A8, DFlash2 drafter, unquantized bf16 KV since 2026-09-21) on Dankserver's
-  behind the :7770 endpoint — the quantization-aware entrypoint, the
+  2x Radeon AI PRO R9700 behind the :7770 endpoint — the quantization-aware
-  three-arm battery that chose TP=2, verified capabilities, and the record of
+  entrypoint, the three-arm battery that chose TP=2, verified capabilities,
-  the 2026-09-16 two-replica era
+  and the record of the 2026-09-16 two-replica era
 - [[frank-medium-model-swap]] — how to swap the checkpoint behind frank-medium:
   the streamlining toolchain (hf-manifest.py, hf-fetch.sh,
   swap-frank-medium.sh), the quantization-aware entrypoint contract, the

serverAppConfigs-docs/log.md

@@ -11,6 +11,28 @@ timestamp: 2026-09-14
 
 # Bundle Change Log
 
+## 2026-09-21 — frank-medium cut back to MXFP4 with unquantized bf16 KV
+
+- **The change.** frank-medium (dankserver :7770) served the FP8 checkpoint
+  (`orcarouter/Qwen3.8-27B-Uncensored-FP8`) with fp8 KV from 2026-09-20 to
+  09-21; it is now cut back to the MXFP4 checkpoint
+  (`just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq`, quark/W4A8) with
+  `--kv-cache-dtype auto` — unquantized bf16 KV instead of the endpoint's
+  historical fp8. `KV_DTYPE` is now a quadlet tunable (entrypoint default fp8);
+  `MAXLEN` stays at the native 262,144. Commit a45f396.
+- **Why.** The operator's read that fp8 KV was hurting MXFP4. The nuance: fp8
+  KV sat on BOTH the FP8 and MXFP4 arms (hardcoded on the common exec line in
+  every commit since the MXFP4 era), so it could not explain the FP8-vs-MXFP4
+  *throughput* gap — that was weight precision. But MXFP4 is W4A8, and fp8 KV
+  on top compounds attention-path precision loss; clean KV tests whether that
+  was degrading *output quality*. No verdict yet — this is the setup.
+- **Measured on deploy.** `kv_cache_dtype=auto`, no fp8 banner; pool auto-sized
+  to **486,464 tokens** (`Maximum concurrency for 262,144 tokens: 1`) — exactly
+  half MXFP4's 943,581-token fp8 pool (the 2x f16/fp8 ratio), and MXFP4's freed
+  weights (19.4 vs 30.9 GiB) keep the full 262,144 window fitting. Vision
+  re-verified on MXFP4 (a solid red 48x48 PNG named "Red"); the quadlet's old
+  "TEXT ONLY" comment was wrong and is corrected. e2e 32/32.
+
 ## 2026-09-21 — Calibre Content Server deployed (dankserver :8084)
 
 - **The change.** The Calibre Content Server (`calibre-server`) is now a