dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-secondary: ubatch 2048 (the second rung inside the 112k headroom)

Browse Source 
Frank's order the same night. The 1024->2048 rung measured +4.6 %
prefill for 616 MiB on the 3090 (2026-10-05 boot pair); the 112k cut
left 1,751 MiB free, so it rides with ~1.1 GiB predicted margin.
Decode is ubatch-invariant; batch 4096 >= 2048 keeps the seam happy.
This commit is contained in:
Frank
2026-10-06 18:30:35 -07:00
parent 44e7a8dfaf
commit d8da7a9eb7
2 changed files with 9 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-secondary.env
+6 -4
View File
@@ -68,11 +68,13 @@ PARALLEL=2
# Compute-buffer geometry. Ubatch is a prefill knob (decode never moves);
# the 1024 rung was the first A/B back once the per-slot context dropped
# to 112k (2026-10-06 evening) and freed ~1 GiB of KV. The pair is a
# measurement, not a law (loud failure at graph capture if it does not
# fit).
# to 112k (2026-10-06 evening), and 2048 is the second (Frank, same night)
# — the rung measured +4.6 % prefill for 616 MiB on the 3090 (2026-10-05
# boot pair), affordable inside the 1,751 MiB the 112k cut freed. The pair
# is a measurement, not a law (loud failure at graph capture if it does
# not fit).
BATCH_SIZE=4096
UBATCH_SIZE=1024
UBATCH_SIZE=2048
# The KV contract for this swap (Frank, 2026-10-05): q8_0 K + q8_0 V —
# llama.cpp's FP8 class. The flags stay explicit so the choice is visible in