frank-large: reserve 24 -> 15, buying the doubled pool AND full prefill speed

Neither lever alone could give both features. At reserve 24 the device budget
is 29.5 GiB and max_tok 32768's 37.0 GiB need always lost the doubling to the
fit-check clamp; at max_tok 16384 the doubling survived but prefill paid 8-9%.

Dropping the reserve to 15 moves the budget to 38.5:
  121.2 host - 67.7 resident weights - 15 reserve = 38.5
  need = 14.4 pool + 20.6 arena + 1.97 vision    = 37.0   -> 1.5 GiB spare

So the payment is page cache for the n-gram lookup table rather than either
feature. 15 GiB, against a table of 47.7 GiB read through the file cache.
Two reasons this is a survivable risk rather than a cliff:

  - The engine's own ledger reported only 12-17 GiB left at the previous 24
    setting, so the declared reserve was never held literally; 15 may cost
    less real cache than it reads as.
  - The failure mode is legible: long prompts fault heavily against a busy
    disk, and raising the reserve is the documented remedy. Not the silent
    clamp, and not the 65536 no-boot mode.

1.5 GiB of headroom is thin and deliberate. Documented in the unit as a
matched pair with the pool: raise the reserve back to 24 and the doubling
quietly goes away again.
This commit is contained in:
Frank
2026-09-12 21:17:16 -07:00
parent 65d1be06ac
commit 8a79f9a709
@@ -97,44 +97,44 @@ Environment=HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next
Environment=HALOGEN_MODEL_ID=frank-large
# Native context of the model. Matches what both clients declare.
Environment=HALOGEN_CTX=262144
# Requested 2026-09-12 as double the 262144 this had run since 09-11, to get
# two full-length conversations resident instead of one. NOTE THE ASYMMETRY
# BELOW: with HALOGEN_MAX_TOK at 32768 (current) the fit check clamps this to
# 262144 at every boot, so the EFFECTIVE pool is 262144 and only one
# full-length conversation stays resident. It is left at 524288 deliberately:
# it is a "grow if the RAM frees up" declaration, and HALOGEN_KV_POOL_FIT
# raises nothing but honours it the moment the arena stops eating the budget.
# Set HALOGEN_MAX_TOK=16384 to actually get the doubled pool today.
# Doubled 2026-09-12 from the 262144 this had run since 09-11: two full-length
# conversations resident at once, or four at 131k, instead of one.
Environment=HALOGEN_KV_POOL_POSITIONS=524288
Environment=HALOGEN_KV_SLOTS=4
# The device budget is ONE budget, covering the KV pool, the prefill arena and
# the vision tower:
# Making the doubling stick is budget arithmetic, all of it. ONE budget covers
# the KV pool, the prefill arena and the vision tower together:
#
# budget = 121 host - 67.7 resident weights - 24.0 reserve = 29.5 GiB
# need = KV(~28 KiB/position) + arena + vision(1.97 measured)
# budget = 121.2 host - 67.7 resident weights - reserve
# need = KV (~28 KiB/position) + arena + vision (1.97 measured)
# arena = 3.80 GiB + 8.40 GiB x (HALOGEN_MAX_TOK / 16384)
#
# That arena term is linear in max_tok and was checked against five restarts
# and against upstream's three published "on the device" figures, all to
# 0.1 GiB. Measured on this host:
# max_tok 16384 -> arena 12.2, pool 524288 total 28.6 -> FITS, 0.9 spare
# max_tok 32768 -> arena 20.6, pool 524288 total 37.0 -> clamps to 262144
# max_tok 32768 -> pool floor 262144 total 29.8 -> 0.3 OVER budget, boots
# max_tok 65536 -> arena 37.3, even the floor needs 46.5 -> OOM, no boot
# So 16384 is the ceiling for a doubled pool, not a conservative setting. The
# 0.3 over-budget at the 32768/262144 corner is where this box has been ever
# since it was deployed on 09-11; the 24 GiB reserve is generous, so it boots
# and serves with 12-14 GiB genuinely left. It is a soft floor, not a wall.
# The arena term is linear in max_tok, and it reproduces five live restarts
# AND upstream's three published "on the device" figures to within 0.1 GiB —
# note those published figures are pool+arena totals, not pool alone, which is
# the misreading that first made a doubling look affordable here. Every corner
# this box offered at a reserve of 24:
#
# Do NOT raise max_tok past 16384 while asking for the doubled pool: 65536 did
# not shrink the pool, it stopped the server booting — and the unit still reads
# active with the container "Up (starting)", so the health check waits out
# LLM_WAIT rather than failing fast.
# 16384 -> need 28.6 vs 29.5 FITS, 0.9 spare (doubling, 8-9% prefill cost)
# 32768 -> need 37.0 vs 29.5 POOL CLAMPED to 262144, deploy still green
# 65536 -> need 46.5 vs 29.5 OOM, will not boot; unit still reads ACTIVE
#
# None of those three gave both features. Dropping the reserve to 15 moves the
# budget to 38.5, so 32768's need of 37.0 fits with 1.5 spare: full prefill
# AND the doubled pool, paid for out of the n-gram page cache instead of out of
# either feature. THE TWO SETTINGS ARE A MATCHED PAIR, NOT independent tunables
# — raise the reserve back to 24 and the pool silently clamps to 262144 again.
Environment=HALOGEN_MAX_TOK=32768
# RAM left for the n-gram table the engine reads through the page cache.
# Above the vendor's 20 because a dozen containers share this host; too low
# and long prompts crawl with the disk busy.
Environment=HALOGEN_HOST_RESERVE_GIB=24
# LOWERED 2026-09-12 from 24 (vendor default is 20) to 15 to buy the doubled
# pool at full prefill speed:
# 121.2 host - 67.7 weights - 15 reserve = 38.5 budget
# need = 14.4 pool + 20.6 arena + 1.97 vision = 37.0 -> 1.5 GiB headroom
# Thin on purpose; do not go below 15 without re-doing that arithmetic. The
# failure mode here is legible rather than fatal — long prompts fault heavily
# against a busy disk, and the remedy is to raise this number until the pool
# hands the memory back. Note the reserve was never held literally: at the
# previous 24 setting the engine's own ledger reported only 12-17 GiB left.
Environment=HALOGEN_HOST_RESERVE_GIB=15
# Image input (the sidecar ships inside the weights repo).
Environment=HALOGEN_VISION_TOWER=1
# Verbose engine logging: the vendor's first-line evidence for load,