frank-large: trim KV pool 524288 -> 401408 for co-tenant headroom

The 09-12 doubled pool fit only at reserve 15, leaving 7.1 GiB / 310
contiguous 2 MiB blocks for the ~28 co-tenant containers and making the
engine print its minutes-long-stall warning on every boot. 401,408
positions (~10.8 GiB KV) hands ~3.4 GiB back while keeping MAX_TOK=32768
and reserve 15: modelled need ~33.4 GiB vs budget 38.5, so the fit check
should not clamp. Capacity becomes one full 262k conversation resident
beside a ~139k one, or three concurrent 131k chats (three slots).

Docs: AGENTS.md fleet table + halogen section, runbook flags table /
co-tenancy tense / admission-arithmetic paragraph updated for the trim.

dankclaw/frank-large/quadlet/frank-large.container

@@ -39,18 +39,17 @@
 # engine lowers it itself and prints what it settled on. Read that ledger on
 # every restart — it is the ground truth for sizing, not this comment.
 #
-# DOUBLED 2026-09-12: pool 524,288 positions (~14.4 GiB of KV at the
-# measured 28 KiB/position). This did NOT fit on the first attempt. The
-# engine's own ledger for the prior 262,144 config was 68.0 weights + 7.2
-# pool + 21.1 working = 96.3 GiB held, 12.1 GiB left on the host, and its
-# pre-flight arithmetic is: 121 GiB - 67.7 resident weights - 24.0 reserve
-# = 29.5 GiB of device budget, which has to cover the pool AND the prefill
-# arena together. 524,288 with the 32,768-token arena needs 35.0 GiB, so
-# HALOGEN_KV_POOL_FIT lowered the pool straight back to 262,144 and the
-# deploy still reported healthy. The doubling only becomes real together
-# with HALOGEN_MAX_TOK=16384 further down, which halves the arena and brings
-# the need to ~24.9 GiB. Read the `kv pool:` lines after every restart —
-# a green health check does NOT mean the pool you asked for is the pool you got.
+# POOL TRIMMED 2026-09-13: 401,408 positions (~10.8 GiB of KV at the measured
+# 28 KiB/position), down from the 524,288 the 2026-09-12 doubling deployed.
+# The doubling fit only at reserve 15, which left the host 7.1 GiB / 310
+# contiguous 2 MiB blocks and made the engine print its co-tenant stall
+# warning on every boot. 401,408 hands ~3.4 GiB back (expect ~10.5 GiB left)
+# while keeping HALOGEN_MAX_TOK=32768 and the reserve at 15. Capacity: one
+# full 262k conversation resident beside a ~139k one, or three concurrent
+# 131k chats -- the fleet's real shape at three slots. Fit check: modelled
+# need ~33.4 GiB (KV 10.8 + arena 20.6 + vision 2.0) vs budget 38.5. The
+# 09-12 doubling story (silent clamp at reserve 24, the arena term, the
+# reserve ceiling) is preserved in the budget block below and the runbook.
 #
 # WEIGHTS. HALOGEN_DOWNLOAD fetches the whole weights repo on FIRST START ONLY
 # (the entrypoint checks whether the checkpoint is absent) into the bind mount
@@ -97,19 +96,21 @@ Environment=HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next
 Environment=HALOGEN_MODEL_ID=frank-large
 # Native context of the model. Matches what both clients declare.
 Environment=HALOGEN_CTX=262144
-# Doubled 2026-09-12 from the 262144 this had run since 09-11: two full-length
-# conversations resident at once, or four at 131k, instead of one.
-Environment=HALOGEN_KV_POOL_POSITIONS=524288
+# Trimmed 2026-09-13 from the 524,288 the 09-12 doubling deployed: buys
+# ~3.4 GiB of co-tenant headroom, keeps one full 262k conversation resident
+# beside a ~139k one, or three concurrent 131k chats.
+Environment=HALOGEN_KV_POOL_POSITIONS=401408
 # Concurrency cap. Each slot is 111.0 MiB of O(1) state; the KV is ONE
-# 524288-position pool carved into per-request regions, so slots cap how many
-# requests run at once while the pool binds how many positions exist. These
-# slots OVERSUBSCRIBE the pool (4 x 262144 > 524288), so admission was always
-# pool-bound -- at most two full-length conversations, or four at 131k.
+# 401408-position pool carved into per-request regions, so slots cap how many
+# requests run at once while the pool binds how many positions exist. The
+# slots still OVERSUBSCRIBE the pool (3 x 262144 > 401408), so admission is
+# pool-bound -- one full-length conversation plus a ~139k second, or three
+# at 131k.
 # LOWERED 4 -> 3 on 2026-09-12: frees ~111 MiB on a host with 7.1 GiB left,
 # and three users is the real fleet size. A fourth concurrent request now
 # queues behind the three rather than contending.
 Environment=HALOGEN_KV_SLOTS=3
-# Making the doubling stick is budget arithmetic, all of it. ONE budget covers
+# Sizing is budget arithmetic, all of it. ONE budget covers
 # the KV pool, the prefill arena and the vision tower together:
 #
 #   budget = 121.2 host - 67.7 resident weights - reserve
@@ -126,11 +127,12 @@ Environment=HALOGEN_KV_SLOTS=3
 #   32768 -> need 37.0 vs 29.5  POOL CLAMPED to 262144, deploy still green
 #   65536 -> need 46.5 vs 29.5  OOM, will not boot; unit still reads ACTIVE
 #
-# None of those three gave both features. Dropping the reserve to 15 moves the
-# budget to 38.5, so 32768's need of 37.0 fits with 1.5 spare: full prefill
-# AND the doubled pool, paid for out of the n-gram page cache instead of out of
-# either feature. THE TWO SETTINGS ARE A MATCHED PAIR, NOT independent tunables
-# — raise the reserve back to 24 and the pool silently clamps to 262144 again.
+# None of those three gave both features; reserve 15 made the doubled pool
+# fit (38.5 vs 37.0) but at 7.1 GiB of host headroom. On 2026-09-13 the pool
+# came down to 401,408 to restore co-tenant headroom: need ~33.4 vs 38.5.
+# THE POOL, ARENA AND RESERVE ARE A MATCHED SET, NOT independent tunables --
+# the fit check's printed ledger on each boot is the arbiter, not these
+# comments.
 Environment=HALOGEN_MAX_TOK=32768
 # RAM left for the n-gram table the engine reads through the page cache.
 # LOWERED 2026-09-12 from 24 (vendor default is 20) to buy the doubled pool at
@@ -138,10 +140,12 @@ Environment=HALOGEN_MAX_TOK=32768
 # read from a clamped boot, not modelled:
 #   "524288 positions need ~35.0 GiB (plus 1.5 of margin)"  ->  budget >= 36.5
 #   budget = 121.2 host - 67.7 weights - reserve
-# reserve 16 -> 37.5 (1.0 spare); 15 -> 38.5 (2.0 spare, deployed); 18 -> 35.5
-# CLAMPS the pool to 262144. Do not raise past 16.
+# reserve 16 -> 37.5 (1.0 spare); 15 -> 38.5 (2.0 spare, deployed). At the
+# old 524,288 pool, 18 -> 35.5 CLAMPED it to 262144, so the ceiling there was
+# 16; at 401,408 (need ~33.4) it loosens, but a higher reserve still buys
+# nothing.
 #
-# CEILING IS 16, AND 16 BUYS NOTHING. Tested live: 15 -> 16 changed the fit
+# AT THE DOUBLED POOL THE CEILING WAS 16, AND 16 BOUGHT NOTHING. Tested live at 524,288: 15 -> 16 changed the fit
 # margin only. The server held 103.5 GiB and reported "7.1 GiB left for
 # everything else" at BOTH settings, because this reserve feeds the fit-check
 # budget and is not an allocation — actual footprint is weights + KV + arena +
@@ -154,14 +158,15 @@ Environment=HALOGEN_MAX_TOK=32768
 # (7.1 GiB total)" — against 2081 blocks / 12.1 GiB at the 24-reserve
 # baseline. All 28 co-tenant containers stayed up (~4.3 GiB working set).
 #
-# The price is that the engine volunteers this note on every boot now:
+# The price at 524,288 was that the engine volunteered this note on every boot:
 #   when it runs out, BOTH they and this server can stall for minutes at a
 #   time at full CPU with no disk activity and no output, which is not a
 #   crash and needs no restart.
 # That is the failure mode to watch, and it is NOT the legible one predicted
 # above: a minutes-long stall that looks like a hang is harder to diagnose
-# than a slow prompt. If it happens, the reserve has no headroom left to give
-# — the remedy is HALOGEN_KV_POOL_POSITIONS=262144 (hands back 7.2 GiB), or
+# than a slow prompt. The 09-13 trim to 401,408 was the first application of
+# the remedy. If it happens again, cut further:
+# HALOGEN_KV_POOL_POSITIONS=262144 (hands back another 3.6 GiB), or
 # HALOGEN_MAX_TOK=16384 (hands back 8.4) with reserve then free to rise to 24.
 Environment=HALOGEN_HOST_RESERVE_GIB=15
 # Image input (the sidecar ships inside the weights repo).