perf: raise frank-default-large to batch 8192 / ubatch 2048

Empirically verified on 2026-08-13 (caches dropped): batch 8192 / ubatch
2048 loads and serves at ctx 602100 with 61 GiB VRAM peak. The benchmark
doc's 'ubatch 2048 OOMs' results were confounded by page-cache pressure -
with the model's pages fully cached the KV/graph allocation stalls at
28 GiB VRAM. Drop caches (sync; echo 3 > /proc/sys/vm/drop_caches) before
starting a load to make restarts reliable.
This commit is contained in:
Frank committed 2026-08-13 18:56:12 -07:00
1 parent 447286c2a1
commit 317610a540
2 files changed
+10 -9

No files matched your search

+2 -2
View File
@@ -35,8 +35,8 @@ fi
: "${N_GPU_LAYERS:=all}"
: "${CACHE_TYPE_K:=q8_0}"
: "${CACHE_TYPE_V:=q8_0}"
: "${BATCH_SIZE:=2048}"
: "${UBATCH_SIZE:=512}"
: "${BATCH_SIZE:=8192}"
: "${UBATCH_SIZE:=2048}"
: "${TEMPERATURE:=1.0}"
: "${TOP_K:=20}"
: "${TOP_P:=0.95}"