frank-large: reserve 24 -> 15, buying the doubled pool AND full prefill speed
Neither lever alone could give both features. At reserve 24 the device budget
is 29.5 GiB and max_tok 32768's 37.0 GiB need always lost the doubling to the
fit-check clamp; at max_tok 16384 the doubling survived but prefill paid 8-9%.
Dropping the reserve to 15 moves the budget to 38.5:
121.2 host - 67.7 resident weights - 15 reserve = 38.5
need = 14.4 pool + 20.6 arena + 1.97 vision = 37.0 -> 1.5 GiB spare
So the payment is page cache for the n-gram lookup table rather than either
feature. 15 GiB, against a table of 47.7 GiB read through the file cache.
Two reasons this is a survivable risk rather than a cliff:
- The engine's own ledger reported only 12-17 GiB left at the previous 24
setting, so the declared reserve was never held literally; 15 may cost
less real cache than it reads as.
- The failure mode is legible: long prompts fault heavily against a busy
disk, and raising the reserve is the documented remedy. Not the silent
clamp, and not the 65536 no-boot mode.
1.5 GiB of headroom is thin and deliberate. Documented in the unit as a
matched pair with the pool: raise the reserve back to 24 and the doubling
quietly goes away again.
This commit is contained in:
@@ -97,44 +97,44 @@ Environment=HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next
|
||||
Environment=HALOGEN_MODEL_ID=frank-large
|
||||
# Native context of the model. Matches what both clients declare.
|
||||
Environment=HALOGEN_CTX=262144
|
||||
# Requested 2026-09-12 as double the 262144 this had run since 09-11, to get
|
||||
# two full-length conversations resident instead of one. NOTE THE ASYMMETRY
|
||||
# BELOW: with HALOGEN_MAX_TOK at 32768 (current) the fit check clamps this to
|
||||
# 262144 at every boot, so the EFFECTIVE pool is 262144 and only one
|
||||
# full-length conversation stays resident. It is left at 524288 deliberately:
|
||||
# it is a "grow if the RAM frees up" declaration, and HALOGEN_KV_POOL_FIT
|
||||
# raises nothing but honours it the moment the arena stops eating the budget.
|
||||
# Set HALOGEN_MAX_TOK=16384 to actually get the doubled pool today.
|
||||
# Doubled 2026-09-12 from the 262144 this had run since 09-11: two full-length
|
||||
# conversations resident at once, or four at 131k, instead of one.
|
||||
Environment=HALOGEN_KV_POOL_POSITIONS=524288
|
||||
Environment=HALOGEN_KV_SLOTS=4
|
||||
# The device budget is ONE budget, covering the KV pool, the prefill arena and
|
||||
# the vision tower:
|
||||
# Making the doubling stick is budget arithmetic, all of it. ONE budget covers
|
||||
# the KV pool, the prefill arena and the vision tower together:
|
||||
#
|
||||
# budget = 121 host - 67.7 resident weights - 24.0 reserve = 29.5 GiB
|
||||
# need = KV(~28 KiB/position) + arena + vision(1.97 measured)
|
||||
# budget = 121.2 host - 67.7 resident weights - reserve
|
||||
# need = KV (~28 KiB/position) + arena + vision (1.97 measured)
|
||||
# arena = 3.80 GiB + 8.40 GiB x (HALOGEN_MAX_TOK / 16384)
|
||||
#
|
||||
# That arena term is linear in max_tok and was checked against five restarts
|
||||
# and against upstream's three published "on the device" figures, all to
|
||||
# 0.1 GiB. Measured on this host:
|
||||
# max_tok 16384 -> arena 12.2, pool 524288 total 28.6 -> FITS, 0.9 spare
|
||||
# max_tok 32768 -> arena 20.6, pool 524288 total 37.0 -> clamps to 262144
|
||||
# max_tok 32768 -> pool floor 262144 total 29.8 -> 0.3 OVER budget, boots
|
||||
# max_tok 65536 -> arena 37.3, even the floor needs 46.5 -> OOM, no boot
|
||||
# So 16384 is the ceiling for a doubled pool, not a conservative setting. The
|
||||
# 0.3 over-budget at the 32768/262144 corner is where this box has been ever
|
||||
# since it was deployed on 09-11; the 24 GiB reserve is generous, so it boots
|
||||
# and serves with 12-14 GiB genuinely left. It is a soft floor, not a wall.
|
||||
# The arena term is linear in max_tok, and it reproduces five live restarts
|
||||
# AND upstream's three published "on the device" figures to within 0.1 GiB —
|
||||
# note those published figures are pool+arena totals, not pool alone, which is
|
||||
# the misreading that first made a doubling look affordable here. Every corner
|
||||
# this box offered at a reserve of 24:
|
||||
#
|
||||
# Do NOT raise max_tok past 16384 while asking for the doubled pool: 65536 did
|
||||
# not shrink the pool, it stopped the server booting — and the unit still reads
|
||||
# active with the container "Up (starting)", so the health check waits out
|
||||
# LLM_WAIT rather than failing fast.
|
||||
# 16384 -> need 28.6 vs 29.5 FITS, 0.9 spare (doubling, 8-9% prefill cost)
|
||||
# 32768 -> need 37.0 vs 29.5 POOL CLAMPED to 262144, deploy still green
|
||||
# 65536 -> need 46.5 vs 29.5 OOM, will not boot; unit still reads ACTIVE
|
||||
#
|
||||
# None of those three gave both features. Dropping the reserve to 15 moves the
|
||||
# budget to 38.5, so 32768's need of 37.0 fits with 1.5 spare: full prefill
|
||||
# AND the doubled pool, paid for out of the n-gram page cache instead of out of
|
||||
# either feature. THE TWO SETTINGS ARE A MATCHED PAIR, NOT independent tunables
|
||||
# — raise the reserve back to 24 and the pool silently clamps to 262144 again.
|
||||
Environment=HALOGEN_MAX_TOK=32768
|
||||
# RAM left for the n-gram table the engine reads through the page cache.
|
||||
# Above the vendor's 20 because a dozen containers share this host; too low
|
||||
# and long prompts crawl with the disk busy.
|
||||
Environment=HALOGEN_HOST_RESERVE_GIB=24
|
||||
# LOWERED 2026-09-12 from 24 (vendor default is 20) to 15 to buy the doubled
|
||||
# pool at full prefill speed:
|
||||
# 121.2 host - 67.7 weights - 15 reserve = 38.5 budget
|
||||
# need = 14.4 pool + 20.6 arena + 1.97 vision = 37.0 -> 1.5 GiB headroom
|
||||
# Thin on purpose; do not go below 15 without re-doing that arithmetic. The
|
||||
# failure mode here is legible rather than fatal — long prompts fault heavily
|
||||
# against a busy disk, and the remedy is to raise this number until the pool
|
||||
# hands the memory back. Note the reserve was never held literally: at the
|
||||
# previous 24 setting the engine's own ledger reported only 12-17 GiB left.
|
||||
Environment=HALOGEN_HOST_RESERVE_GIB=15
|
||||
# Image input (the sidecar ships inside the weights repo).
|
||||
Environment=HALOGEN_VISION_TOWER=1
|
||||
# Verbose engine logging: the vendor's first-line evidence for load,
|
||||
|
||||
Reference in New Issue
Block a user