dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

ling: ubatch 512 -> 3072 (batch 3072) - using heap headroom, watch for OOM

Browse Source
This commit is contained in:
Frank committed 2026-08-14 01:46:36 -07:00
1 parent a5b12e1062
commit 321f946660
1 file changed
+2 -2
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-default-large.ling.env
+2 -2
View File
@@ -32,8 +32,8 @@ DEVICE=Vulkan0
N_GPU_LAYERS=all N_GPU_LAYERS=all
FLASH_ATTN=auto FLASH_ATTN=auto
# Batch kept small-leaning for the tight memory budget; tune with CTX_SIZE. # Batch kept small-leaning for the tight memory budget; tune with CTX_SIZE.
BATCH_SIZE=512 BATCH_SIZE=3072
UBATCH_SIZE=512 UBATCH_SIZE=3072
# Sampling defaults (model card: reasoning-capable; leave temp/top to default) # Sampling defaults (model card: reasoning-capable; leave temp/top to default)
TEMPERATURE=1.0 TEMPERATURE=1.0
TOP_K=20 TOP_K=20