frank-small: double the compute-buffer pair to 4096/2048
BATCH_SIZE 2048 -> 4096, UBATCH_SIZE 1024 -> 2048. Why the ubatch move is right on THIS card: the prefill curve is card-specific. On the 2x R9700 layer-split sibling it peaks at ubatch 512 and every step up costs GiB for no speed; on this CUDA 3090 it climbs monotonically to 2048 (710 -> 1188 t/s on the 35B MoE sweep, 1944 -> 2423 t/s on the Ornith 8k sweep) with decode flat to under 1 t/s on every rung. The logical batch is free either way: two sweeps agree that at a fixed ubatch, 2048/4096/8192 give identical prefill speed and identical peak VRAM, because the compute buffers are sized by the ubatch alone. The risk is all in the ubatch, and it is documented rather than assumed: the step cost ~1,244 MiB of peak graph VRAM on the 35B MoE, the shape this replaces is 2,815 MiB free, and the Q5_K_M predecessor recorded "ubatch 2048 failed the scratch boot" with MORE free than that. Compute buffers are allocated at graph capture during boot, so a failure is loud and fast; the step-down rungs (4096/1536, then 4096/1024) are named in the env. The seam stops hard-coding a rung: serve_argv now carries BATCH_SIZE and UBATCH_SIZE the way it already carried CTX_SIZE et al., and the pair test asserts the argv follows the shipped env plus the llama.cpp invariant ubatch <= batch. Verified both ways: retuning the env to 8192/4096 keeps the seam green, inverting the pair to 2048/4096 turns it red.
This commit is contained in:
1 parent
b06b47c7b4
commit
e5893dc30d
3 files changed
+58
-16
No files matched your search
@@ -45,10 +45,13 @@ fi
|
||||
: "${CTX_SIZE:=401408}"
|
||||
: "${PARALLEL:=2}"
|
||||
# Compute-buffer geometry. Before 2026-10-03 this endpoint never passed either
|
||||
# flag, so it ran on llama.cpp's own defaults (2048/512); they are tunables now
|
||||
# because the 1024 ubatch is what this shape was booted with.
|
||||
: "${BATCH_SIZE:=2048}"
|
||||
: "${UBATCH_SIZE:=1024}"
|
||||
# flag, so it ran on llama.cpp's own defaults (2048/512). The pair is 4096/2048
|
||||
# since 2026-10-05: on this CUDA card the prefill curve climbs all the way to
|
||||
# ubatch 2048 (unlike the R9700 layer-split sibling, where 512 is the peak),
|
||||
# and the logical batch is VRAM- and throughput-neutral. frank-small.env
|
||||
# carries the sweep tables and the VRAM risk note.
|
||||
: "${BATCH_SIZE:=4096}"
|
||||
: "${UBATCH_SIZE:=2048}"
|
||||
: "${CACHE_TYPE_K:=f16}"
|
||||
: "${CACHE_TYPE_V:=f16}"
|
||||
: "${NGL:=99}"
|
||||
|
||||
Reference in new issue
Block a user