docs(frank-medium): record the MXFP4 + clean-KV cut; correct vision claim

Knowledge-base follow-up to a45f396. The runbook, both index one-liners, and
the AGENTS.md topology row all still declared FP8 active; flipped to
just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq with unquantized bf16 KV (pool
486,464 tokens, full 262,144 window), and recorded the causal nuance: fp8 KV
was on both arms, so the FP8-vs-MXFP4 gap was weight precision, not KV - clean
KV is a quality hypothesis for W4A8, not a throughput one. No verdict yet.

The quadlet's "TEXT ONLY ... 400 here" comment was wrong: vision works on the
MXFP4 checkpoint (Qwen3_5ForConditionalGeneration carries the vision tower
through the Gemma4 merge path, unquantized) - verified by naming a solid red
48x48 PNG "Red". Comment corrected; harness models.yml input stays [text,image].
Docs-only; no re-deploy (the endpoint already serves vision).
This commit is contained in:
Frank committed 2026-09-21 17:08:33 -07:00
1 parent a45f396c39
commit d8dbd4f5bc
6 files changed
+76 -46

No files matched your search

+12 -13
View File
@@ -2,8 +2,8 @@
Documentation for the LLM inference fleet on the two hosts (2026-09-21):
**vllm-radiance on Dankserver** serving frank-medium as ONE tensor-parallel
process across both Radeon AI PRO R9700 cards (orcarouter
Qwen3.8-27B-Uncensored-FP8 + the z-lab DFlash2 drafter, TP=2 over the
process across both Radeon AI PRO R9700 cards (just1moremodel
Qwen3.8-27B-Uncensored-MXFP4 + the z-lab DFlash2 drafter, bf16 KV, TP=2 over the
R4D one-shot allreduce), and TWO endpoints on Dankclaw — the unsloth
llama.cpp prebuilt (frank-large, Qwen3.8-Flash-Next UD-IQ4_XS on the gfx1151
iGPU, port 7770) and mainline llama.cpp (frank-small, HauhauCS Gemma4-12B QAT
@@ -23,7 +23,7 @@ graph LR
gfx1201, 32 GiB each]
B2[vllm-radiance 0.9.3, TP=2
frank-medium, port 7770]
C[Qwen3.8-27B FP8 + DFlash2 drafter
C[Qwen3.8-27B MXFP4 + DFlash2 drafter
ONE process on both cards, 262144 ctx, up to 8 seqs]
A --> B2 --> C
end
@@ -57,16 +57,15 @@ frank-small, port 7771]
## Active Deployment
- [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook since
2026-09-20: vllm-radiance (stilldeadcode/vllm-radiance:0.9.3) serving
orcarouter/Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, 30.9 GB / 7
shards) as ONE tensor-parallel process (TP=2) behind :7770, with MTP
self-speculation (the model's own single-layer MTP head). The quantization-
aware entrypoint serves the plain FP8 path; the MXFP4 checkpoint
(just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq + the z-lab DFlash2-FP8
drafter) remains on disk as the rollback target. The three-arm battery that
chose TP=2, the auto-sized KV pool, and verified capabilities (tools,
reasoning, vision)
- [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook:
vllm-radiance (stilldeadcode/vllm-radiance:0.9.3) serving
just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4/quark W4A8) with the
z-lab DFlash2-FP8 drafter as ONE tensor-parallel process (TP=2) behind
:7770, now on unquantized bf16 KV (`KV_DTYPE=auto`, cut 2026-09-21 to test
whether the endpoint's historical fp8 KV had been degrading W4A8 quality).
The FP8 checkpoint (orcarouter/Qwen3.8-27B-Uncensored-FP8, the 09-20→09-21
shape) is the on-disk rollback. The three-arm battery that chose TP=2, the
auto-sized KV pool, and verified capabilities (tools, reasoning)
- [[frank-medium-model-swap]] — how to swap the checkpoint behind frank-medium:
the streamlining toolchain (hf-manifest.py, hf-fetch.sh,
swap-frank-medium.sh), the quantization-aware entrypoint contract, the