docs(frank-medium): record the MXFP4 + clean-KV cut; correct vision claim

Knowledge-base follow-up to a45f396. The runbook, both index one-liners, and
the AGENTS.md topology row all still declared FP8 active; flipped to
just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq with unquantized bf16 KV (pool
486,464 tokens, full 262,144 window), and recorded the causal nuance: fp8 KV
was on both arms, so the FP8-vs-MXFP4 gap was weight precision, not KV - clean
KV is a quality hypothesis for W4A8, not a throughput one. No verdict yet.

The quadlet's "TEXT ONLY ... 400 here" comment was wrong: vision works on the
MXFP4 checkpoint (Qwen3_5ForConditionalGeneration carries the vision tower
through the Gemma4 merge path, unquantized) - verified by naming a solid red
48x48 PNG "Red". Comment corrected; harness models.yml input stays [text,image].
Docs-only; no re-deploy (the endpoint already serves vision).
This commit is contained in:
Frank
2026-09-21 17:08:33 -07:00
parent a45f396c39
commit d8dbd4f5bc
6 changed files with 76 additions and 46 deletions
@@ -110,10 +110,12 @@ Environment=MAXLEN=262144
# excludes the others until it finishes (vLLM preempts rather than refuses).
# The llama deployment it replaced advertised 327,680 across 2 slots.
#
# TEXT ONLY. The MXFP4 checkpoint carries language_model_only=true and the
# launcher loads no vision projector, so an input_audio/image request is a
# 400 here. Vision traffic routes to frank-small on dankclaw (harness
# modelRoles vision -> frank-small).
# VISION WORKS (verified 2026-09-21: a solid red 48x48 PNG is named "Red").
# config.json sets language_model_only=true and there is no separate mmproj,
# but the Qwen3_5ForConditionalGeneration architecture carries the vision tower
# through the Gemma4 merge path and it is not MXFP4-quantized, so vLLM serves
# image input on either checkpoint. harness models.yml keeps frank-medium
# input: [text, image].
Environment=GPU_UTIL=0.98
# Speculative decoding: DFlash2 external drafter at depth 7. A/B tested
# against MTP self-speculation (depth 8) on this FP8 checkpoint: DFlash2