BATCH_SIZE 2048 -> 4096, UBATCH_SIZE 1024 -> 2048.
Why the ubatch move is right on THIS card: the prefill curve is
card-specific. On the 2x R9700 layer-split sibling it peaks at ubatch 512
and every step up costs GiB for no speed; on this CUDA 3090 it climbs
monotonically to 2048 (710 -> 1188 t/s on the 35B MoE sweep, 1944 -> 2423
t/s on the Ornith 8k sweep) with decode flat to under 1 t/s on every rung.
The logical batch is free either way: two sweeps agree that at a fixed
ubatch, 2048/4096/8192 give identical prefill speed and identical peak
VRAM, because the compute buffers are sized by the ubatch alone.
The risk is all in the ubatch, and it is documented rather than assumed:
the step cost ~1,244 MiB of peak graph VRAM on the 35B MoE, the shape this
replaces is 2,815 MiB free, and the Q5_K_M predecessor recorded "ubatch
2048 failed the scratch boot" with MORE free than that. Compute buffers
are allocated at graph capture during boot, so a failure is loud and fast;
the step-down rungs (4096/1536, then 4096/1024) are named in the env.
The seam stops hard-coding a rung: serve_argv now carries BATCH_SIZE and
UBATCH_SIZE the way it already carried CTX_SIZE et al., and the pair test
asserts the argv follows the shipped env plus the llama.cpp invariant
ubatch <= batch. Verified both ways: retuning the env to 8192/4096 keeps
the seam green, inverting the pair to 2048/4096 turns it red.
Frank
e5893dc30d