dankclaw: revert MTP+ngram layering (slower); ubatch 8192
MTP+ngram benched 12.0-13.6 t/s (vs MTP alone 15.3-19.5) and is parallel-safe, but slower - layering does not pay off. Back to MTP alone. Test ubatch 8192 (== batch) using the VRAM headroom; watch for OOM during load.
This commit is contained in:
1 parent
5dd6949909
commit
0d09756430
1 file changed
+2
-2
@@ -14,7 +14,7 @@ PARALLEL=3
|
||||
# draft-mtp - MTP sidecar draft (a4lg/Qwen3.6-27B-MTP-ONLY-Q8_0.gguf)
|
||||
# ngram-mod - draftless ngram matching (no draft model, no shared ctx)
|
||||
# Benched 2026-08-14: see docs/frank-default-large-benchmark.md for the winner.
|
||||
SPEC_TYPE=draft-mtp,ngram-mod
|
||||
SPEC_TYPE=draft-mtp
|
||||
# Draft model file (only used when SPEC_TYPE contains 'draft-').
|
||||
# DFlash: Qwen3.6-27B-DFlash-Q8_0.gguf ; MTP: Qwen3.6-27B-MTP-ONLY-Q8_0.gguf
|
||||
DRAFT_FILE=Qwen3.6-27B-MTP-ONLY-Q8_0.gguf
|
||||
@@ -39,7 +39,7 @@ CACHE_TYPE_V=q8_0
|
||||
# ubatch is the compute-graph driver; larger ubatch = more VRAM, faster prefill.
|
||||
# Re-verified 2026-08-14 during the spec swap (see benchmark doc).
|
||||
BATCH_SIZE=8192
|
||||
UBATCH_SIZE=4096
|
||||
UBATCH_SIZE=8192
|
||||
# Sampling settings — per HauhauCS Qwen3.6-27B model card.
|
||||
# Thinking-general: temp=1.0, top_p=0.95, top_k=20, min_p=0.
|
||||
# Clients may override per-request; these are the server-side defaults.
|
||||
|
||||
Reference in new issue
Block a user