dankclaw Qwen3.8: batch 8192/ubatch 256 - fixes MTP collapse + premature-EOS
Found: large ubatch (512) collapses MTP acceptance to ~25% (gen ~9 t/s); ubatch 128 with batch>=2048 triggers premature instant-EOS under 2 concurrent slots. ubatch 256 = ~30 t/s gen, concurrent-safe, ~330 t/s prefill.
This commit is contained in:
@@ -37,10 +37,13 @@ N_GPU_LAYERS=all
|
||||
# mixed types silently disable FA (long-context decode collapses).
|
||||
CACHE_TYPE_K=q8_0
|
||||
CACHE_TYPE_V=q8_0
|
||||
# Batch sizing (start conservative; tune after verifying memory):
|
||||
# ubatch is the compute-graph driver; larger = more VRAM, faster prefill.
|
||||
BATCH_SIZE=2048
|
||||
UBATCH_SIZE=512
|
||||
# Batch sizing (verified 2026-08-14 on Qwen3.8 UD-Q8_K_XL):
|
||||
# ubatch 128 with batch>=2048 -> premature-EOS under concurrency (unsafe).
|
||||
# ubatch 512 -> MTP acceptance collapses (~25%) -> gen ~9 t/s (slow).
|
||||
# ubatch 256 -> ~30 t/s gen, concurrent-safe, ~330 t/s prefill. WINNER.
|
||||
# batch 8192 keeps single-shot prefill flexibility; ubatch drives the graph.
|
||||
BATCH_SIZE=8192
|
||||
UBATCH_SIZE=256
|
||||
# Sampling settings — per Qwen3.8-27B model card thinking mode.
|
||||
# Thinking: temp=1.0, top_p=0.95, top_k=20, min_p=0. Clients may override.
|
||||
# WARNING: do not name this variable TEMP. The ROCm HIP runtime reads the
|
||||
|
||||
Reference in New Issue
Block a user