docs(frank-medium): record the MXFP4 + clean-KV cut; correct vision claim
Knowledge-base follow-up to a45f396. The runbook, both index one-liners, and
the AGENTS.md topology row all still declared FP8 active; flipped to
just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq with unquantized bf16 KV (pool
486,464 tokens, full 262,144 window), and recorded the causal nuance: fp8 KV
was on both arms, so the FP8-vs-MXFP4 gap was weight precision, not KV - clean
KV is a quality hypothesis for W4A8, not a throughput one. No verdict yet.
The quadlet's "TEXT ONLY ... 400 here" comment was wrong: vision works on the
MXFP4 checkpoint (Qwen3_5ForConditionalGeneration carries the vision tower
through the Gemma4 merge path, unquantized) - verified by naming a solid red
48x48 PNG "Red". Comment corrected; harness models.yml input stays [text,image].
Docs-only; no re-deploy (the endpoint already serves vision).
This commit is contained in:
@@ -115,13 +115,13 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
|
||||
- [[deployment]] — unified deploy and update process for both servers
|
||||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||
- [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook since
|
||||
2026-09-20: vllm-radiance as ONE tensor-parallel process (TP=2 over the R4D
|
||||
one-shot allreduce) serving orcarouter/Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3
|
||||
block-wise, MTP self-speculation) on Dankserver's 2x Radeon AI PRO R9700
|
||||
behind the :7770 endpoint — the quantization-aware entrypoint, the
|
||||
three-arm battery that chose TP=2, verified capabilities, and the record of
|
||||
the 2026-09-16 two-replica era
|
||||
- [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook:
|
||||
vllm-radiance as ONE tensor-parallel process (TP=2 over the R4D one-shot
|
||||
allreduce) serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4
|
||||
W4A8, DFlash2 drafter, unquantized bf16 KV since 2026-09-21) on Dankserver's
|
||||
2x Radeon AI PRO R9700 behind the :7770 endpoint — the quantization-aware
|
||||
entrypoint, the three-arm battery that chose TP=2, verified capabilities,
|
||||
and the record of the 2026-09-16 two-replica era
|
||||
- [[frank-medium-model-swap]] — how to swap the checkpoint behind frank-medium:
|
||||
the streamlining toolchain (hf-manifest.py, hf-fetch.sh,
|
||||
swap-frank-medium.sh), the quantization-aware entrypoint contract, the
|
||||
|
||||
Reference in New Issue
Block a user