dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

dankclaw: revert MTP+ngram layering (slower); ubatch 8192

Browse Source 
MTP+ngram benched 12.0-13.6 t/s (vs MTP alone 15.3-19.5) and is
parallel-safe, but slower - layering does not pay off. Back to MTP
alone. Test ubatch 8192 (== batch) using the VRAM headroom; watch for
OOM during load.
This commit is contained in:
Frank committed 2026-08-13 22:46:55 -07:00
1 parent 5dd6949909
commit 0d09756430
1 file changed
+2 -2
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-default-large.env
+2 -2
View File
@@ -14,7 +14,7 @@ PARALLEL=3
# draft-mtp - MTP sidecar draft (a4lg/Qwen3.6-27B-MTP-ONLY-Q8_0.gguf) # draft-mtp - MTP sidecar draft (a4lg/Qwen3.6-27B-MTP-ONLY-Q8_0.gguf)
# ngram-mod - draftless ngram matching (no draft model, no shared ctx) # ngram-mod - draftless ngram matching (no draft model, no shared ctx)
# Benched 2026-08-14: see docs/frank-default-large-benchmark.md for the winner. # Benched 2026-08-14: see docs/frank-default-large-benchmark.md for the winner.
SPEC_TYPE=draft-mtp,ngram-mod SPEC_TYPE=draft-mtp
# Draft model file (only used when SPEC_TYPE contains 'draft-'). # Draft model file (only used when SPEC_TYPE contains 'draft-').
# DFlash: Qwen3.6-27B-DFlash-Q8_0.gguf ; MTP: Qwen3.6-27B-MTP-ONLY-Q8_0.gguf # DFlash: Qwen3.6-27B-DFlash-Q8_0.gguf ; MTP: Qwen3.6-27B-MTP-ONLY-Q8_0.gguf
DRAFT_FILE=Qwen3.6-27B-MTP-ONLY-Q8_0.gguf DRAFT_FILE=Qwen3.6-27B-MTP-ONLY-Q8_0.gguf
@@ -39,7 +39,7 @@ CACHE_TYPE_V=q8_0
# ubatch is the compute-graph driver; larger ubatch = more VRAM, faster prefill. # ubatch is the compute-graph driver; larger ubatch = more VRAM, faster prefill.
# Re-verified 2026-08-14 during the spec swap (see benchmark doc). # Re-verified 2026-08-14 during the spec swap (see benchmark doc).
BATCH_SIZE=8192 BATCH_SIZE=8192
UBATCH_SIZE=4096 UBATCH_SIZE=8192
# Sampling settings — per HauhauCS Qwen3.6-27B model card. # Sampling settings — per HauhauCS Qwen3.6-27B model card.
# Thinking-general: temp=1.0, top_p=0.95, top_k=20, min_p=0. # Thinking-general: temp=1.0, top_p=0.95, top_k=20, min_p=0.
# Clients may override per-request; these are the server-side defaults. # Clients may override per-request; these are the server-side defaults.