diff --git a/dankclaw/frank-large/quadlet/frank-large.container b/dankclaw/frank-large/quadlet/frank-large.container index e545b84..a449dbc 100644 --- a/dankclaw/frank-large/quadlet/frank-large.container +++ b/dankclaw/frank-large/quadlet/frank-large.container @@ -97,44 +97,44 @@ Environment=HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next Environment=HALOGEN_MODEL_ID=frank-large # Native context of the model. Matches what both clients declare. Environment=HALOGEN_CTX=262144 -# Requested 2026-09-12 as double the 262144 this had run since 09-11, to get -# two full-length conversations resident instead of one. NOTE THE ASYMMETRY -# BELOW: with HALOGEN_MAX_TOK at 32768 (current) the fit check clamps this to -# 262144 at every boot, so the EFFECTIVE pool is 262144 and only one -# full-length conversation stays resident. It is left at 524288 deliberately: -# it is a "grow if the RAM frees up" declaration, and HALOGEN_KV_POOL_FIT -# raises nothing but honours it the moment the arena stops eating the budget. -# Set HALOGEN_MAX_TOK=16384 to actually get the doubled pool today. +# Doubled 2026-09-12 from the 262144 this had run since 09-11: two full-length +# conversations resident at once, or four at 131k, instead of one. Environment=HALOGEN_KV_POOL_POSITIONS=524288 Environment=HALOGEN_KV_SLOTS=4 -# The device budget is ONE budget, covering the KV pool, the prefill arena and -# the vision tower: +# Making the doubling stick is budget arithmetic, all of it. ONE budget covers +# the KV pool, the prefill arena and the vision tower together: # -# budget = 121 host - 67.7 resident weights - 24.0 reserve = 29.5 GiB -# need = KV(~28 KiB/position) + arena + vision(1.97 measured) +# budget = 121.2 host - 67.7 resident weights - reserve +# need = KV (~28 KiB/position) + arena + vision (1.97 measured) # arena = 3.80 GiB + 8.40 GiB x (HALOGEN_MAX_TOK / 16384) # -# That arena term is linear in max_tok and was checked against five restarts -# and against upstream's three published "on the device" figures, all to -# 0.1 GiB. Measured on this host: -# max_tok 16384 -> arena 12.2, pool 524288 total 28.6 -> FITS, 0.9 spare -# max_tok 32768 -> arena 20.6, pool 524288 total 37.0 -> clamps to 262144 -# max_tok 32768 -> pool floor 262144 total 29.8 -> 0.3 OVER budget, boots -# max_tok 65536 -> arena 37.3, even the floor needs 46.5 -> OOM, no boot -# So 16384 is the ceiling for a doubled pool, not a conservative setting. The -# 0.3 over-budget at the 32768/262144 corner is where this box has been ever -# since it was deployed on 09-11; the 24 GiB reserve is generous, so it boots -# and serves with 12-14 GiB genuinely left. It is a soft floor, not a wall. +# The arena term is linear in max_tok, and it reproduces five live restarts +# AND upstream's three published "on the device" figures to within 0.1 GiB — +# note those published figures are pool+arena totals, not pool alone, which is +# the misreading that first made a doubling look affordable here. Every corner +# this box offered at a reserve of 24: # -# Do NOT raise max_tok past 16384 while asking for the doubled pool: 65536 did -# not shrink the pool, it stopped the server booting — and the unit still reads -# active with the container "Up (starting)", so the health check waits out -# LLM_WAIT rather than failing fast. +# 16384 -> need 28.6 vs 29.5 FITS, 0.9 spare (doubling, 8-9% prefill cost) +# 32768 -> need 37.0 vs 29.5 POOL CLAMPED to 262144, deploy still green +# 65536 -> need 46.5 vs 29.5 OOM, will not boot; unit still reads ACTIVE +# +# None of those three gave both features. Dropping the reserve to 15 moves the +# budget to 38.5, so 32768's need of 37.0 fits with 1.5 spare: full prefill +# AND the doubled pool, paid for out of the n-gram page cache instead of out of +# either feature. THE TWO SETTINGS ARE A MATCHED PAIR, NOT independent tunables +# — raise the reserve back to 24 and the pool silently clamps to 262144 again. Environment=HALOGEN_MAX_TOK=32768 # RAM left for the n-gram table the engine reads through the page cache. -# Above the vendor's 20 because a dozen containers share this host; too low -# and long prompts crawl with the disk busy. -Environment=HALOGEN_HOST_RESERVE_GIB=24 +# LOWERED 2026-09-12 from 24 (vendor default is 20) to 15 to buy the doubled +# pool at full prefill speed: +# 121.2 host - 67.7 weights - 15 reserve = 38.5 budget +# need = 14.4 pool + 20.6 arena + 1.97 vision = 37.0 -> 1.5 GiB headroom +# Thin on purpose; do not go below 15 without re-doing that arithmetic. The +# failure mode here is legible rather than fatal — long prompts fault heavily +# against a busy disk, and the remedy is to raise this number until the pool +# hands the memory back. Note the reserve was never held literally: at the +# previous 24 setting the engine's own ledger reported only 12-17 GiB left. +Environment=HALOGEN_HOST_RESERVE_GIB=15 # Image input (the sidecar ships inside the weights repo). Environment=HALOGEN_VISION_TOWER=1 # Verbose engine logging: the vendor's first-line evidence for load,