dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-default-small: ubatch=1024 (tight 24GB VRAM, 786k ctx pool)

Browse Source
This commit is contained in:
Frank committed 2026-08-20 16:45:20 -07:00
1 parent 26b42d2d6e
commit 87dd4224b7
2 files changed
+3 -3
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

scripts/frank-default-small.env
+2 -2
View File
@@ -43,9 +43,9 @@ VERBOSITY=4
# Store SWA/context checkpoints in RAM to hold peak GPU memory down. # Store SWA/context checkpoints in RAM to hold peak GPU memory down.
CTX_CHECKPOINTS=4 CTX_CHECKPOINTS=4
# Big prefill batches (ubatch 2048, batch 4096). # Big prefill batches (ubatch 1024, batch 4096).
BATCH_SIZE=4096 BATCH_SIZE=4096
UBATCH_SIZE=2048 UBATCH_SIZE=1024
# Eager RAM load (MoE experts on CPU; ~31 GB RAM). # Eager RAM load (MoE experts on CPU; ~31 GB RAM).
NO_MMAP=1 NO_MMAP=1
scripts/serve-frank-small.sh
+1 -1
View File
@@ -39,7 +39,7 @@ fi
: "${N_CPU_MOE:=36}" : "${N_CPU_MOE:=36}"
: "${N_GPU_LAYERS:=all}" : "${N_GPU_LAYERS:=all}"
: "${BATCH_SIZE:=4096}" : "${BATCH_SIZE:=4096}"
: "${UBATCH_SIZE:=2048}" : "${UBATCH_SIZE:=1024}"
: "${CTX_CHECKPOINTS:=4}" : "${CTX_CHECKPOINTS:=4}"
: "${NO_MMAP:=1}" : "${NO_MMAP:=1}"
: "${TEMPERATURE:=1.0}" : "${TEMPERATURE:=1.0}"