dankclaw Qwen3.8: batch 8192/ubatch 256 - fixes MTP collapse + premature-EOS
Found: large ubatch (512) collapses MTP acceptance to ~25% (gen ~9 t/s); ubatch 128 with batch>=2048 triggers premature instant-EOS under 2 concurrent slots. ubatch 256 = ~30 t/s gen, concurrent-safe, ~330 t/s prefill.
This commit is contained in:
1 parent
57b0be037c
commit
75bb9bb53e
1 file changed
+7
-4
@@ -37,10 +37,13 @@ N_GPU_LAYERS=all
|
|||||||
# mixed types silently disable FA (long-context decode collapses).
|
# mixed types silently disable FA (long-context decode collapses).
|
||||||
CACHE_TYPE_K=q8_0
|
CACHE_TYPE_K=q8_0
|
||||||
CACHE_TYPE_V=q8_0
|
CACHE_TYPE_V=q8_0
|
||||||
# Batch sizing (start conservative; tune after verifying memory):
|
# Batch sizing (verified 2026-08-14 on Qwen3.8 UD-Q8_K_XL):
|
||||||
# ubatch is the compute-graph driver; larger = more VRAM, faster prefill.
|
# ubatch 128 with batch>=2048 -> premature-EOS under concurrency (unsafe).
|
||||||
BATCH_SIZE=2048
|
# ubatch 512 -> MTP acceptance collapses (~25%) -> gen ~9 t/s (slow).
|
||||||
UBATCH_SIZE=512
|
# ubatch 256 -> ~30 t/s gen, concurrent-safe, ~330 t/s prefill. WINNER.
|
||||||
|
# batch 8192 keeps single-shot prefill flexibility; ubatch drives the graph.
|
||||||
|
BATCH_SIZE=8192
|
||||||
|
UBATCH_SIZE=256
|
||||||
# Sampling settings — per Qwen3.8-27B model card thinking mode.
|
# Sampling settings — per Qwen3.8-27B model card thinking mode.
|
||||||
# Thinking: temp=1.0, top_p=0.95, top_k=20, min_p=0. Clients may override.
|
# Thinking: temp=1.0, top_p=0.95, top_k=20, min_p=0. Clients may override.
|
||||||
# WARNING: do not name this variable TEMP. The ROCm HIP runtime reads the
|
# WARNING: do not name this variable TEMP. The ROCm HIP runtime reads the
|
||||||
|
|||||||
Reference in new issue
Block a user