frank-large: lower KV slots 4 -> 3
Concurrency cap trimmed to the real fleet size. Each slot is 111.0 MiB of O(1) state, so this hands ~111 MiB back on a host sitting at 7.1 GiB left for everything else. Note the slots already oversubscribed the pool (4 x 262144 > 524288), so admission was pool-bound regardless: at most two full-length conversations, or four at 131k. This does not change how much context fits -- it caps simultaneous requests at three, and a fourth now queues.
This commit is contained in:
@@ -100,7 +100,15 @@ Environment=HALOGEN_CTX=262144
|
||||
# Doubled 2026-09-12 from the 262144 this had run since 09-11: two full-length
|
||||
# conversations resident at once, or four at 131k, instead of one.
|
||||
Environment=HALOGEN_KV_POOL_POSITIONS=524288
|
||||
Environment=HALOGEN_KV_SLOTS=4
|
||||
# Concurrency cap. Each slot is 111.0 MiB of O(1) state; the KV is ONE
|
||||
# 524288-position pool carved into per-request regions, so slots cap how many
|
||||
# requests run at once while the pool binds how many positions exist. These
|
||||
# slots OVERSUBSCRIBE the pool (4 x 262144 > 524288), so admission was always
|
||||
# pool-bound -- at most two full-length conversations, or four at 131k.
|
||||
# LOWERED 4 -> 3 on 2026-09-12: frees ~111 MiB on a host with 7.1 GiB left,
|
||||
# and three users is the real fleet size. A fourth concurrent request now
|
||||
# queues behind the three rather than contending.
|
||||
Environment=HALOGEN_KV_SLOTS=3
|
||||
# Making the doubling stick is budget arithmetic, all of it. ONE budget covers
|
||||
# the KV pool, the prefill arena and the vision tower together:
|
||||
#
|
||||
|
||||
Reference in New Issue
Block a user