frank-large: halve the prefill arena so the doubled pool actually fits
The previous commit's doubling was a silent no-op. HALOGEN_KV_POOL_FIT measured the device budget at startup and lowered the pool straight back to 262144, while the deploy still reported healthy in 12s. The engine's arithmetic: 121 GiB host - 67.7 resident weights - 24.0 reserved for the n-gram page cache and the OS = 29.5 GiB of device budget, and that single budget carries the KV pool AND the prefill arena. At HALOGEN_MAX_TOK=32768 the arena is ~20.6 GiB, so 524288 positions need 35.0 GiB against 29.5 available. HALOGEN_MAX_TOK=16384 halves the arena to ~10.5 GiB, bringing the 524288 need to ~24.9 GiB, which fits with ~3 GiB of margin. Cost is about 8-9% of prefill throughput, per upstream. This is the engine's own prescribed fix, quoted verbatim in its refusal message. Lowering HALOGEN_CTX was the other option the engine offered; rejected because both clients declare 262144 as the context window, so it would break the client contract rather than pay for the pool.
This commit is contained in:
@@ -39,17 +39,18 @@
|
||||
# engine lowers it itself and prints what it settled on. Read that ledger on
|
||||
# every restart — it is the ground truth for sizing, not this comment.
|
||||
#
|
||||
# DOUBLED 2026-09-12: pool 524,288 positions (~14.4 GiB at the measured
|
||||
# 28 KiB/position). Measured startup ledger for the prior 262,144 config was
|
||||
# 68.0 weights + 7.2 pool + 21.1 working = 96.3 GiB held, 12.1 GiB left on
|
||||
# the host. The doubled pool puts that remainder at roughly 5 GiB against a
|
||||
# ~4.3 GiB co-tenant working set, and the extra 7.2 GiB comes out of the
|
||||
# page cache the 47.7 GiB n-gram lookup table reads through. The table still
|
||||
# fits the ~69 GiB of cache that remains, so this is a tighter-but-sane
|
||||
# position rather than an overcommit. If long prompts start crawling with the
|
||||
# disk busy, the compensating lever is HALOGEN_MAX_TOK=16384: it shrinks the
|
||||
# prefill arena 16.7 -> 8.4 GiB, which pays for the doubling outright at
|
||||
# about 8% of prefill throughput.
|
||||
# DOUBLED 2026-09-12: pool 524,288 positions (~14.4 GiB of KV at the
|
||||
# measured 28 KiB/position). This did NOT fit on the first attempt. The
|
||||
# engine's own ledger for the prior 262,144 config was 68.0 weights + 7.2
|
||||
# pool + 21.1 working = 96.3 GiB held, 12.1 GiB left on the host, and its
|
||||
# pre-flight arithmetic is: 121 GiB - 67.7 resident weights - 24.0 reserve
|
||||
# = 29.5 GiB of device budget, which has to cover the pool AND the prefill
|
||||
# arena together. 524,288 with the 32,768-token arena needs 35.0 GiB, so
|
||||
# HALOGEN_KV_POOL_FIT lowered the pool straight back to 262,144 and the
|
||||
# deploy still reported healthy. The doubling only becomes real together
|
||||
# with HALOGEN_MAX_TOK=16384 further down, which halves the arena and brings
|
||||
# the need to ~24.9 GiB. Read the `kv pool:` lines after every restart —
|
||||
# a green health check does NOT mean the pool you asked for is the pool you got.
|
||||
#
|
||||
# WEIGHTS. HALOGEN_DOWNLOAD fetches the whole weights repo on FIRST START ONLY
|
||||
# (the entrypoint checks whether the checkpoint is absent) into the bind mount
|
||||
@@ -103,6 +104,17 @@ Environment=HALOGEN_CTX=262144
|
||||
# used to contend for one 262k pool.
|
||||
Environment=HALOGEN_KV_POOL_POSITIONS=524288
|
||||
Environment=HALOGEN_KV_SLOTS=4
|
||||
# LOAD-BEARING. This line is what makes the doubling above real; do not
|
||||
# "clean it up". The device budget here is 121 GiB - 67.7 resident weights
|
||||
# - 24.0 reserved for the n-gram page cache and the OS = 29.5 GiB, and that
|
||||
# one budget carries BOTH the KV pool and the prefill arena. At the upstream
|
||||
# HALOGEN_MAX_TOK=32768 the arena is ~20.6 GiB, so 524288 positions need
|
||||
# 35.0 GiB against 29.5 available and HALOGEN_KV_POOL_FIT silently lowers
|
||||
# the pool back to 262144 — the deploy still reports healthy, so the
|
||||
# regression is invisible unless you read the ledger. Halving the arena to
|
||||
# ~10.5 GiB brings the 524288 need to ~24.9 GiB, which fits with ~3 GiB
|
||||
# of margin, at about 8-9% of prefill throughput.
|
||||
Environment=HALOGEN_MAX_TOK=16384
|
||||
# RAM left for the n-gram table the engine reads through the page cache.
|
||||
# Above the vendor's 20 because a dozen containers share this host; too low
|
||||
# and long prompts crawl with the disk busy.
|
||||
|
||||
Reference in New Issue
Block a user