frank-large: halve the prefill arena so the doubled pool actually fits

The previous commit's doubling was a silent no-op. HALOGEN_KV_POOL_FIT
measured the device budget at startup and lowered the pool straight back
to 262144, while the deploy still reported healthy in 12s.

The engine's arithmetic: 121 GiB host - 67.7 resident weights - 24.0
reserved for the n-gram page cache and the OS = 29.5 GiB of device
budget, and that single budget carries the KV pool AND the prefill arena.
At HALOGEN_MAX_TOK=32768 the arena is ~20.6 GiB, so 524288 positions
need 35.0 GiB against 29.5 available.

HALOGEN_MAX_TOK=16384 halves the arena to ~10.5 GiB, bringing the
524288 need to ~24.9 GiB, which fits with ~3 GiB of margin. Cost is
about 8-9% of prefill throughput, per upstream. This is the engine's own
prescribed fix, quoted verbatim in its refusal message.

Lowering HALOGEN_CTX was the other option the engine offered; rejected
because both clients declare 262144 as the context window, so it would
break the client contract rather than pay for the pool.
This commit is contained in:
Frank
2026-09-12 17:58:08 -07:00
parent 5ace29e7a0
commit c489b204ea
@@ -39,17 +39,18 @@
# engine lowers it itself and prints what it settled on. Read that ledger on
# every restart — it is the ground truth for sizing, not this comment.
#
# DOUBLED 2026-09-12: pool 524,288 positions (~14.4 GiB at the measured
# 28 KiB/position). Measured startup ledger for the prior 262,144 config was
# 68.0 weights + 7.2 pool + 21.1 working = 96.3 GiB held, 12.1 GiB left on
# the host. The doubled pool puts that remainder at roughly 5 GiB against a
# ~4.3 GiB co-tenant working set, and the extra 7.2 GiB comes out of the
# page cache the 47.7 GiB n-gram lookup table reads through. The table still
# fits the ~69 GiB of cache that remains, so this is a tighter-but-sane
# position rather than an overcommit. If long prompts start crawling with the
# disk busy, the compensating lever is HALOGEN_MAX_TOK=16384: it shrinks the
# prefill arena 16.7 -> 8.4 GiB, which pays for the doubling outright at
# about 8% of prefill throughput.
# DOUBLED 2026-09-12: pool 524,288 positions (~14.4 GiB of KV at the
# measured 28 KiB/position). This did NOT fit on the first attempt. The
# engine's own ledger for the prior 262,144 config was 68.0 weights + 7.2
# pool + 21.1 working = 96.3 GiB held, 12.1 GiB left on the host, and its
# pre-flight arithmetic is: 121 GiB - 67.7 resident weights - 24.0 reserve
# = 29.5 GiB of device budget, which has to cover the pool AND the prefill
# arena together. 524,288 with the 32,768-token arena needs 35.0 GiB, so
# HALOGEN_KV_POOL_FIT lowered the pool straight back to 262,144 and the
# deploy still reported healthy. The doubling only becomes real together
# with HALOGEN_MAX_TOK=16384 further down, which halves the arena and brings
# the need to ~24.9 GiB. Read the `kv pool:` lines after every restart —
# a green health check does NOT mean the pool you asked for is the pool you got.
#
# WEIGHTS. HALOGEN_DOWNLOAD fetches the whole weights repo on FIRST START ONLY
# (the entrypoint checks whether the checkpoint is absent) into the bind mount
@@ -103,6 +104,17 @@ Environment=HALOGEN_CTX=262144
# used to contend for one 262k pool.
Environment=HALOGEN_KV_POOL_POSITIONS=524288
Environment=HALOGEN_KV_SLOTS=4
# LOAD-BEARING. This line is what makes the doubling above real; do not
# "clean it up". The device budget here is 121 GiB - 67.7 resident weights
# - 24.0 reserved for the n-gram page cache and the OS = 29.5 GiB, and that
# one budget carries BOTH the KV pool and the prefill arena. At the upstream
# HALOGEN_MAX_TOK=32768 the arena is ~20.6 GiB, so 524288 positions need
# 35.0 GiB against 29.5 available and HALOGEN_KV_POOL_FIT silently lowers
# the pool back to 262144 — the deploy still reports healthy, so the
# regression is invisible unless you read the ledger. Halving the arena to
# ~10.5 GiB brings the 524288 need to ~24.9 GiB, which fits with ~3 GiB
# of margin, at about 8-9% of prefill throughput.
Environment=HALOGEN_MAX_TOK=16384
# RAM left for the n-gram table the engine reads through the page cache.
# Above the vendor's 20 because a dozen containers share this host; too low
# and long prompts crawl with the disk busy.