dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-large: EXPERIMENT - HALOGEN_MAX_TOK back to upstream default 32768

Browse Source 
Operator call. Prediction from the measured model: 524288 positions cost
35.0 GiB against a 29.5 budget, so HALOGEN_KV_POOL_FIT clamps the pool to
262144 (27.8). Rollback is a single revert; 16384 was the config that made
the doubling real.
This commit is contained in:
Frank
2026-09-12 20:06:25 -07:00
parent 41ac1f83f6
commit 75ef47c342
1 changed files with 5 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
dankclaw/frank-large/quadlet/frank-large.container
+5 -1
View File
@@ -126,7 +126,11 @@ Environment=HALOGEN_KV_SLOTS=4
# not shrink the pool, it stops the server booting at all — and the unit still
# reads active with the container "Up (starting)", so the health check waits
# out LLM_WAIT rather than failing fast.
Environment=HALOGEN_MAX_TOK=16384
# EXPERIMENT 2026-09-12 (operator call): back to upstream's default 32768.
# Model predicts 524288 needs 35.0 against a 29.5 budget, so HALOGEN_KV_POOL_FIT
# should clamp the pool to 262144 (~27.8) at boot. Watch the ledger.
Environment=HALOGEN_MAX_TOK=32768
# RAM left for the n-gram table the engine reads through the page cache.
# Above the vendor's 20 because a dozen containers share this host; too low
# and long prompts crawl with the disk busy.