dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

Revert "frank-large: EXPERIMENT - raise HALOGEN_MAX_TOK to 65536"

Browse Source 
This reverts commit 6851940b4d.
This commit is contained in:
Frank
2026-09-12 19:34:56 -07:00
parent 6851940b4d
commit 258cccbdcf
1 changed files with 11 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files
dankclaw/frank-large/quadlet/frank-large.container
+11 -14
View File
@@ -104,20 +104,17 @@ Environment=HALOGEN_CTX=262144
# used to contend for one 262k pool. # used to contend for one 262k pool.
Environment=HALOGEN_KV_POOL_POSITIONS=524288 Environment=HALOGEN_KV_POOL_POSITIONS=524288
Environment=HALOGEN_KV_SLOTS=4 Environment=HALOGEN_KV_SLOTS=4
# EXPERIMENT 2026-09-12 (operator call): raised to 65536 to measure what the # LOAD-BEARING. This line is what makes the doubling above real; do not
# arena actually costs, against upstream's warning that it "sizes a ~4 GB # "clean it up". The device budget here is 121 GiB - 67.7 resident weights
# scratch arena, so do not raise it to the native context". # - 24.0 reserved for the n-gram page cache and the OS = 29.5 GiB, and that
# # one budget carries BOTH the KV pool and the prefill arena. At the upstream
# Background on why this line matters at all. The device budget here is # HALOGEN_MAX_TOK=32768 the arena is ~20.6 GiB, so 524288 positions need
# 121 GiB - 67.7 resident weights - 24.0 reserved for the n-gram page cache # 35.0 GiB against 29.5 available and HALOGEN_KV_POOL_FIT silently lowers
# and the OS = 29.5 GiB, and that ONE budget carries both the KV pool and the # the pool back to 262144 — the deploy still reports healthy, so the
# prefill arena. Measured arena-side cost ("everything but the pool"): # regression is invisible unless you read the ledger. Halving the arena to
# max_tok 16384 -> 12.2 GiB (524288 pool fits: 26.6 total) # ~10.5 GiB brings the 524288 need to ~24.9 GiB, which fits with ~3 GiB
# max_tok 32768 -> 20.6 GiB (524288 pool REFUSED at 35.0, fit lowers to 262144) # of margin, at about 8-9% of prefill throughput.
# HALOGEN_KV_POOL_FIT only ever LOWERS, never raises, and never below Environment=HALOGEN_MAX_TOK=16384
# HALOGEN_CTX — and the deploy still reports healthy either way, so the pool
# you asked for is not necessarily the pool you got. Read the `kv pool:` lines.
Environment=HALOGEN_MAX_TOK=65536
# RAM left for the n-gram table the engine reads through the page cache. # RAM left for the n-gram table the engine reads through the page cache.
# Above the vendor's 20 because a dozen containers share this host; too low # Above the vendor's 20 because a dozen containers share this host; too low
# and long prompts crawl with the disk busy. # and long prompts crawl with the disk busy.