dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-secondary: ubatch 2048 (the second rung inside the 112k headroom)

Browse Source 
Frank's order the same night. The 1024->2048 rung measured +4.6 %
prefill for 616 MiB on the 3090 (2026-10-05 boot pair); the 112k cut
left 1,751 MiB free, so it rides with ~1.1 GiB predicted margin.
Decode is ubatch-invariant; batch 4096 >= 2048 keeps the seam happy.
This commit is contained in:
Frank
2026-10-06 18:30:35 -07:00
parent 44e7a8dfaf
commit d8da7a9eb7
2 changed files with 9 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-secondary.env
+6 -4
View File
@@ -68,11 +68,13 @@ PARALLEL=2
# Compute-buffer geometry. Ubatch is a prefill knob (decode never moves); # Compute-buffer geometry. Ubatch is a prefill knob (decode never moves);
# the 1024 rung was the first A/B back once the per-slot context dropped # the 1024 rung was the first A/B back once the per-slot context dropped
# to 112k (2026-10-06 evening) and freed ~1 GiB of KV. The pair is a # to 112k (2026-10-06 evening), and 2048 is the second (Frank, same night)
# measurement, not a law (loud failure at graph capture if it does not # — the rung measured +4.6 % prefill for 616 MiB on the 3090 (2026-10-05
# fit). # boot pair), affordable inside the 1,751 MiB the 112k cut freed. The pair
# is a measurement, not a law (loud failure at graph capture if it does
# not fit).
BATCH_SIZE=4096 BATCH_SIZE=4096
UBATCH_SIZE=1024 UBATCH_SIZE=2048
# The KV contract for this swap (Frank, 2026-10-05): q8_0 K + q8_0 V — # The KV contract for this swap (Frank, 2026-10-05): q8_0 K + q8_0 V —
# llama.cpp's FP8 class. The flags stay explicit so the choice is visible in # llama.cpp's FP8 class. The flags stay explicit so the choice is visible in
tests/frank_secondary_unit.bats
+3 -3
View File
@@ -146,10 +146,10 @@ argv_value() {
# RCO-IQ3_XXS weights 10,148 MiB (10,442,827,392 B) — 1.56 GiB LIGHTER # RCO-IQ3_XXS weights 10,148 MiB (10,442,827,392 B) — 1.56 GiB LIGHTER
# than the IQ3_S-mtp predecessor — + vision tower ~1.4 GiB (931,145,888 # than the IQ3_S-mtp predecessor — + vision tower ~1.4 GiB (931,145,888
# B BF16 mmproj at the measured ~1.45x cost law) + compute buffers at # B BF16 mmproj at the measured ~1.45x cost law) + compute buffers at
# ubatch 1024 + CUDA graphs + TWO DeltaNet states (~150 MiB each) + the # ubatch 2048 + CUDA graphs + TWO DeltaNet states (~150 MiB each) + the
# MTP draft ctx => 14 GiB budgeted. Two 114,688 slots are the # MTP draft ctx => 14 GiB budgeted. Two 114,688 slots are the
# 2026-10-06 evening shape (the lighter per-slot funds the ubatch # 2026-10-06 late shape (the lighter per-slot funds the ubatch
# increase); a THIRD slot must not fit. Arithmetic, not a boot log: # rungs); a THIRD slot must not fit.
# change CTX_SIZE and this bites before the deploy does; the boot's # change CTX_SIZE and this bites before the deploy does; the boot's
# nvidia-smi free is the measured truth. # nvidia-smi free is the measured truth.
ctx="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")" ctx="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"