perf: raise frank-default-large to batch 8192 / ubatch 2048
Empirically verified on 2026-08-13 (caches dropped): batch 8192 / ubatch 2048 loads and serves at ctx 602100 with 61 GiB VRAM peak. The benchmark doc's 'ubatch 2048 OOMs' results were confounded by page-cache pressure - with the model's pages fully cached the KV/graph allocation stalls at 28 GiB VRAM. Drop caches (sync; echo 3 > /proc/sys/vm/drop_caches) before starting a load to make restarts reliable.
This commit is contained in:
1 parent
447286c2a1
commit
317610a540
2 files changed
+10
-9
No files matched your search
@@ -35,8 +35,8 @@ fi
|
||||
: "${N_GPU_LAYERS:=all}"
|
||||
: "${CACHE_TYPE_K:=q8_0}"
|
||||
: "${CACHE_TYPE_V:=q8_0}"
|
||||
: "${BATCH_SIZE:=2048}"
|
||||
: "${UBATCH_SIZE:=512}"
|
||||
: "${BATCH_SIZE:=8192}"
|
||||
: "${UBATCH_SIZE:=2048}"
|
||||
: "${TEMPERATURE:=1.0}"
|
||||
: "${TOP_K:=20}"
|
||||
: "${TOP_P:=0.95}"
|
||||
|
||||
Reference in new issue
Block a user