dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

perf: raise frank-default-large to batch 8192 / ubatch 2048

Browse Source 
Empirically verified on 2026-08-13 (caches dropped): batch 8192 / ubatch
2048 loads and serves at ctx 602100 with 61 GiB VRAM peak. The benchmark
doc's 'ubatch 2048 OOMs' results were confounded by page-cache pressure -
with the model's pages fully cached the KV/graph allocation stalls at
28 GiB VRAM. Drop caches (sync; echo 3 > /proc/sys/vm/drop_caches) before
starting a load to make restarts reliable.
This commit is contained in:
Frank committed 2026-08-13 18:56:12 -07:00
1 parent 447286c2a1
commit 317610a540
2 files changed
+10 -9
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

scripts/frank-default-large.env
+8 -7
View File
@@ -26,13 +26,14 @@ N_GPU_LAYERS=all
# long-context decode to ~1 t/s at 200k ctx. q8_0 V halves V memory too. # long-context decode to ~1 t/s at 200k ctx. q8_0 V halves V memory too.
CACHE_TYPE_K=q8_0 CACHE_TYPE_K=q8_0
CACHE_TYPE_V=q8_0 CACHE_TYPE_V=q8_0
# Batch sizing constrained by ROCm memory at 602100 ctx (3x200700): # Batch sizing: empirically verified 2026-08-13 with caches dropped.
# KV ~50GB + model 29.77GB leave only ~16GB for compute graphs. The compute # batch 8192 / ubatch 2048 loads and serves (61 GiB VRAM peak, ctx 602100).
# graph scales with UBATCH, so ubatch must stay ~512; ubatch 2048 OOMs even # The earlier "ubatch 2048 OOMs" notes in frank-default-large-benchmark.md
# at batch 2048. Proven-fits config: batch 2048 / ubatch 512. # were confounded by page-cache pressure: with the model's pages fully
# User-specified ubatch 4096 / batch 12288 is not feasible here (see bench). # cached, the KV/graph allocation stalls (VRAM stuck at 28 GiB).
BATCH_SIZE=2048 # Drop caches before starting a load: sync && echo 3 > /proc/sys/vm/drop_caches.
UBATCH_SIZE=512 BATCH_SIZE=8192
UBATCH_SIZE=2048
# Sampling settings — per HauhauCS Qwen3.6-27B model card. # Sampling settings — per HauhauCS Qwen3.6-27B model card.
# Thinking-general: temp=1.0, top_p=0.95, top_k=20, min_p=0. # Thinking-general: temp=1.0, top_p=0.95, top_k=20, min_p=0.
# Clients may override per-request; these are the server-side defaults. # Clients may override per-request; these are the server-side defaults.
scripts/serve-llama-frank-large.sh
+2 -2
View File
@@ -35,8 +35,8 @@ fi
: "${N_GPU_LAYERS:=all}" : "${N_GPU_LAYERS:=all}"
: "${CACHE_TYPE_K:=q8_0}" : "${CACHE_TYPE_K:=q8_0}"
: "${CACHE_TYPE_V:=q8_0}" : "${CACHE_TYPE_V:=q8_0}"
: "${BATCH_SIZE:=2048}" : "${BATCH_SIZE:=8192}"
: "${UBATCH_SIZE:=512}" : "${UBATCH_SIZE:=2048}"
: "${TEMPERATURE:=1.0}" : "${TEMPERATURE:=1.0}"
: "${TOP_K:=20}" : "${TOP_K:=20}"
: "${TOP_P:=0.95}" : "${TOP_P:=0.95}"