dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

perf: frank-default-large ubatch 512 (compute graph scales with ubatch; 602100 ctx KV leaves ~16GB, ubatch 2048 OOM)

Browse Source
This commit is contained in:
Frank committed 2026-08-11 17:19:57 -07:00
1 parent 62cc2b850a
commit 668083fe76
2 files changed
+6 -5
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

scripts/frank-default-large.env
+5 -4
View File
@@ -24,10 +24,11 @@ N_GPU_LAYERS=all
CACHE_TYPE_K=q8_0 CACHE_TYPE_K=q8_0
CACHE_TYPE_V=f16 CACHE_TYPE_V=f16
# Batch sizing constrained by ROCm memory at 602100 ctx (3x200700): # Batch sizing constrained by ROCm memory at 602100 ctx (3x200700):
# KV cache alone ~50GB, model 29.77GB; only ~16GB left for compute graphs. # KV ~50GB + model 29.77GB leave only ~16GB for compute graphs. The compute
# Batch 12288 (80.7GB buffer) and 6144 (40.9GB) both OOM; 2048 fits. # graph scales with UBATCH, so ubatch must stay ~512; ubatch 2048 OOMs even
# Tried per request (ubatch 4096 / batch 12288) -> OOM. See bench notes. # at batch 2048. Proven-fits config: batch 2048 / ubatch 512.
# User-specified ubatch 4096 / batch 12288 is not feasible here (see bench).
BATCH_SIZE=2048 BATCH_SIZE=2048
UBATCH_SIZE=2048 UBATCH_SIZE=512
THREADS=8 THREADS=8
THREADS_HTTP=8 THREADS_HTTP=8
scripts/serve-llama-frank-large.sh
+1 -1
View File
@@ -36,7 +36,7 @@ fi
: "${CACHE_TYPE_K:=q8_0}" : "${CACHE_TYPE_K:=q8_0}"
: "${CACHE_TYPE_V:=f16}" : "${CACHE_TYPE_V:=f16}"
: "${BATCH_SIZE:=2048}" : "${BATCH_SIZE:=2048}"
: "${UBATCH_SIZE:=2048}" : "${UBATCH_SIZE:=512}"
: "${THREADS:=8}" : "${THREADS:=8}"
: "${THREADS_HTTP:=8}" : "${THREADS_HTTP:=8}"