dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

perf: frank-default-large ubatch 512 (compute graph scales with ubatch; 602100 ctx KV leaves ~16GB, ubatch 2048 OOM)

Browse Source
This commit is contained in:
Frank
2026-08-11 17:19:57 -07:00
parent 62cc2b850a
commit 668083fe76
2 changed files with 6 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-default-large.env
+5 -4
View File
@@ -24,10 +24,11 @@ N_GPU_LAYERS=all
CACHE_TYPE_K=q8_0
CACHE_TYPE_V=f16
# Batch sizing constrained by ROCm memory at 602100 ctx (3x200700):
# KV cache alone ~50GB, model 29.77GB; only ~16GB left for compute graphs.
# Batch 12288 (80.7GB buffer) and 6144 (40.9GB) both OOM; 2048 fits.
# Tried per request (ubatch 4096 / batch 12288) -> OOM. See bench notes.
# KV ~50GB + model 29.77GB leave only ~16GB for compute graphs. The compute
# graph scales with UBATCH, so ubatch must stay ~512; ubatch 2048 OOMs even
# at batch 2048. Proven-fits config: batch 2048 / ubatch 512.
# User-specified ubatch 4096 / batch 12288 is not feasible here (see bench).
BATCH_SIZE=2048
UBATCH_SIZE=2048
UBATCH_SIZE=512
THREADS=8
THREADS_HTTP=8
scripts/serve-llama-frank-large.sh
+1 -1
View File
@@ -36,7 +36,7 @@ fi
: "${CACHE_TYPE_K:=q8_0}"
: "${CACHE_TYPE_V:=f16}"
: "${BATCH_SIZE:=2048}"
: "${UBATCH_SIZE:=2048}"
: "${UBATCH_SIZE:=512}"
: "${THREADS:=8}"
: "${THREADS_HTTP:=8}"