docs(frank-medium): record the MXFP4 + clean-KV cut; correct vision claim
Knowledge-base follow-up to a45f396. The runbook, both index one-liners, and
the AGENTS.md topology row all still declared FP8 active; flipped to
just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq with unquantized bf16 KV (pool
486,464 tokens, full 262,144 window), and recorded the causal nuance: fp8 KV
was on both arms, so the FP8-vs-MXFP4 gap was weight precision, not KV - clean
KV is a quality hypothesis for W4A8, not a throughput one. No verdict yet.
The quadlet's "TEXT ONLY ... 400 here" comment was wrong: vision works on the
MXFP4 checkpoint (Qwen3_5ForConditionalGeneration carries the vision tower
through the Gemma4 merge path, unquantized) - verified by naming a solid red
48x48 PNG "Red". Comment corrected; harness models.yml input stays [text,image].
Docs-only; no re-deploy (the endpoint already serves vision).
This commit is contained in:
1 parent
a45f396c39
commit
d8dbd4f5bc
6 files changed
+76
-46
No files matched your search
@@ -11,6 +11,28 @@ timestamp: 2026-09-14
|
||||
|
||||
# Bundle Change Log
|
||||
|
||||
## 2026-09-21 — frank-medium cut back to MXFP4 with unquantized bf16 KV
|
||||
|
||||
- **The change.** frank-medium (dankserver :7770) served the FP8 checkpoint
|
||||
(`orcarouter/Qwen3.8-27B-Uncensored-FP8`) with fp8 KV from 2026-09-20 to
|
||||
09-21; it is now cut back to the MXFP4 checkpoint
|
||||
(`just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq`, quark/W4A8) with
|
||||
`--kv-cache-dtype auto` — unquantized bf16 KV instead of the endpoint's
|
||||
historical fp8. `KV_DTYPE` is now a quadlet tunable (entrypoint default fp8);
|
||||
`MAXLEN` stays at the native 262,144. Commit a45f396.
|
||||
- **Why.** The operator's read that fp8 KV was hurting MXFP4. The nuance: fp8
|
||||
KV sat on BOTH the FP8 and MXFP4 arms (hardcoded on the common exec line in
|
||||
every commit since the MXFP4 era), so it could not explain the FP8-vs-MXFP4
|
||||
*throughput* gap — that was weight precision. But MXFP4 is W4A8, and fp8 KV
|
||||
on top compounds attention-path precision loss; clean KV tests whether that
|
||||
was degrading *output quality*. No verdict yet — this is the setup.
|
||||
- **Measured on deploy.** `kv_cache_dtype=auto`, no fp8 banner; pool auto-sized
|
||||
to **486,464 tokens** (`Maximum concurrency for 262,144 tokens: 1`) — exactly
|
||||
half MXFP4's 943,581-token fp8 pool (the 2x f16/fp8 ratio), and MXFP4's freed
|
||||
weights (19.4 vs 30.9 GiB) keep the full 262,144 window fitting. Vision
|
||||
re-verified on MXFP4 (a solid red 48x48 PNG named "Red"); the quadlet's old
|
||||
"TEXT ONLY" comment was wrong and is corrected. e2e 32/32.
|
||||
|
||||
## 2026-09-21 — Calibre Content Server deployed (dankserver :8084)
|
||||
|
||||
- **The change.** The Calibre Content Server (`calibre-server`) is now a
|
||||
|
||||
Reference in new issue
Block a user