frank-large: double the KV pool to 524288 positions

HALOGEN_KV_POOL_POSITIONS 262144 -> 524288, which is upstream's own
default (2 x HALOGEN_CTX). The old pool was the binding constraint on
concurrency, not context: a request reserves prompt + max_tokens against
the pool, so with pool == CTX four concurrent 65k chats were contending
for one full-length conversation's worth of positions. Two full-length
conversations, or four at 131k, now stay resident.

Cost is ~7.2 GiB more device RAM (measured 28 KiB/position), taken from
the page cache the 47.7 GiB n-gram lookup table reads through. Live
startup ledger for the prior config: 68.0 weights + 7.2 pool + 21.1
working = 96.3 GiB held, 12.1 GiB left on the host. The doubled pool
puts the remainder near 5 GiB against a ~4.3 GiB co-tenant working set,
and cache at ~69 GiB -- still above the lookup table, so the table can
stay hot. Compensating lever if long prompts start crawling:
HALOGEN_MAX_TOK=16384, which frees 8.3 GiB of prefill arena for ~8%
of prefill throughput.

Docs updated to match, including the admission-trap paragraph, which was
only true while pool == CTX.

AGENTS.md

@@ -32,7 +32,7 @@ host.
 | Target | Runtime | Service | Model | Port |
 |--------|---------|---------|-------|------|
 | Dankserver | FreeToken v0.1.2 | frank-small | batsclamp Huihui-Qwen3.6-35B-A3B-abliterated-FP8 (block-fp8 e4m3 128x128, ~35 GB safetensors), experts host-RAM LRU, 262144 ctx / 4 concurrent, reasoning on | 7770 |
-| Dankclaw | halogen-flash-server 0.5.8 (rootless Quadlet container) | frank-large | peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b .hgn 115.55 GiB + quality overlay, ~68 GiB pinned, 262144 ctx / 4 slots, pool 262144, vision tower on, greedy for silent clients) | 7770 |
+| Dankclaw | halogen-flash-server 0.5.8 (rootless Quadlet container) | frank-large | peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b .hgn 115.55 GiB + quality overlay, ~68 GiB pinned, 262144 ctx / 4 slots, pool 524288, vision tower on, greedy for silent clients) | 7770 |
 | Dankclaw | *(retired 2026-09-11)* FastFlowLM v1.0.1 on the XDNA2 NPU | frank-npu | Gemma4-E4B-IT-NPU2 pack (vision + audio). Retired with the `amd_iommu=off` decision: the driver's SVA/PASID bind needs an enabled IOMMU | 7771 |
 
 History: the 2026-08-17 redo retired the pipeline-parallel split and the
@@ -395,8 +395,9 @@ both halves — the engine on container loopback :8730 (unauthenticated, never
 published) and the OpenAI front-end on :8731 published as host **7770** —
 with `HALOGEN_MODEL_ID=frank-large`, so no client wiring changed.
 
-Deployed flags: `HALOGEN_CTX=262144`, `HALOGEN_KV_POOL_POSITIONS=262144`
-(conservative for a shared host), `HALOGEN_KV_SLOTS=4`,
+Deployed flags: `HALOGEN_CTX=262144`, `HALOGEN_KV_POOL_POSITIONS=524288`
+(doubled 2026-09-12 to upstream's default — two full-length conversations
+resident instead of one, ~14.4 GiB), `HALOGEN_KV_SLOTS=4`,
 `HALOGEN_HOST_RESERVE_GIB=24`, `HALOGEN_VISION_TOWER=1`, and
 `HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next` for the first-start
 fetch; everything else is upstream's default. The weights stay pinned

dankclaw/frank-large/quadlet/frank-large.container

@@ -34,12 +34,22 @@
 # that then takes the remainder can stall both for minutes at full CPU with no
 # disk activity and no output. That is documented behaviour, not a crash, and
 # the escape hatch is HALOGEN_FLASH_PIN_TRUNK=0 (at several times the decode
-# cost). Sizing chosen 2026-09-11 for a SHARED 121 GiB host: KV pool 262,144
-# positions (~7.2 GiB) + the baked 32,768-token prefill arena (16.7 GiB) sits
-# well inside the device budget once the ~68 GiB of weights and the ~5 GiB of
-# co-tenant containers are accounted for. HALOGEN_KV_POOL_FIT stays on as the
-# safety net: if the machine refuses, the engine lowers the pool itself and
-# prints what it chose.
+# cost). Sizing is chosen for a SHARED 121 GiB host, and HALOGEN_KV_POOL_FIT
+# stays on as the safety net: if the machine refuses the configured pool the
+# engine lowers it itself and prints what it settled on. Read that ledger on
+# every restart — it is the ground truth for sizing, not this comment.
+#
+# DOUBLED 2026-09-12: pool 524,288 positions (~14.4 GiB at the measured
+# 28 KiB/position). Measured startup ledger for the prior 262,144 config was
+# 68.0 weights + 7.2 pool + 21.1 working = 96.3 GiB held, 12.1 GiB left on
+# the host. The doubled pool puts that remainder at roughly 5 GiB against a
+# ~4.3 GiB co-tenant working set, and the extra 7.2 GiB comes out of the
+# page cache the 47.7 GiB n-gram lookup table reads through. The table still
+# fits the ~69 GiB of cache that remains, so this is a tighter-but-sane
+# position rather than an overcommit. If long prompts start crawling with the
+# disk busy, the compensating lever is HALOGEN_MAX_TOK=16384: it shrinks the
+# prefill arena 16.7 -> 8.4 GiB, which pays for the doubling outright at
+# about 8% of prefill throughput.
 #
 # WEIGHTS. HALOGEN_DOWNLOAD fetches the whole weights repo on FIRST START ONLY
 # (the entrypoint checks whether the checkpoint is absent) into the bind mount
@@ -86,10 +96,12 @@ Environment=HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next
 Environment=HALOGEN_MODEL_ID=frank-large
 # Native context of the model. Matches what both clients declare.
 Environment=HALOGEN_CTX=262144
-# Conservative pool for a shared host: one full-length conversation, or four
-# at 65k. Raising it to 524288 doubles the resident conversations for ~7 GiB
-# more device memory — do that only after reading the startup ledger.
-Environment=HALOGEN_KV_POOL_POSITIONS=262144
+# Doubled 2026-09-12 from 262144 to upstream's own default (2 x HALOGEN_CTX):
+# two full-length conversations resident at once, or four at 131k, instead of
+# one. The old pool was the binding constraint on concurrency, not CTX — a
+# request reserves prompt + max_tokens against the pool, so four 65k chats
+# used to contend for one 262k pool.
+Environment=HALOGEN_KV_POOL_POSITIONS=524288
 Environment=HALOGEN_KV_SLOTS=4
 # RAM left for the n-gram table the engine reads through the page cache.
 # Above the vendor's 20 because a dozen containers share this host; too low

serverAppConfigs-docs/LLM/halogen-flash-server.md

@@ -11,7 +11,7 @@ tags:
   - gfx1151
   - qwen4exp
   - rocm
-timestamp: 2026-09-11
+timestamp: 2026-09-12
 ---
 
 # halogen-flash-server on Dankclaw (frank-large)
@@ -78,7 +78,7 @@ upstream's defaults:
 |---|---|---|
 | `HALOGEN_MODEL_ID` | `frank-large` | fleet identity; no client rename |
 | `HALOGEN_CTX` | `262144` | the model's native context, and what both clients declare |
-| `HALOGEN_KV_POOL_POSITIONS` | `262144` | conservative for a **shared** 121 GiB host: one full-length conversation, or four at 65k, at ~7.2 GiB |
+| `HALOGEN_KV_POOL_POSITIONS` | `524288` | **doubled 2026-09-12** to upstream's own default (2 × `HALOGEN_CTX`): two full-length conversations resident, or four at 131k, at ~14.4 GiB |
 | `HALOGEN_KV_SLOTS` | `4` | four concurrent generations; slots share the pool and cost ~115 MiB each |
 | `HALOGEN_HOST_RESERVE_GIB` | `24` | RAM kept for the n-gram table the engine reads through the page cache — above upstream's 20 because a dozen containers share this host |
 | `HALOGEN_VISION_TOWER` | `1` | image input; the fleet's only image path since [[frank-npu]] was retired |
@@ -103,7 +103,8 @@ Three numbers explain the whole box:
   file cache, so `free`, `MemAvailable` and every dashboard built on them
   **overstate free memory by the size of the model**. The server prints the
   corrected figure at startup; believe that line, not Grafana.
-- **~7.2 GiB per 262,144 KV positions**, plus the prefill arena (16.7 GiB at
+- **~28 KiB per KV position** (measured: 262,144 positions = 6,656 MiB of KV,
+  ~7.2 GiB with block scratch), plus the prefill arena (16.7 GiB at
   `HALOGEN_MAX_TOK=32768`, 8.4 GiB at 16384). The engine's device-side budget
   stops at roughly 47 GiB on a 128 GB machine *with the weights pinned* — and
   lower on a machine carrying more besides this server, which Dankclaw is.
@@ -116,6 +117,23 @@ Three numbers explain the whole box:
 the pool itself if the configured one will not fit, printing what it chose.
 Read those lines on every restart; they are the ground truth for sizing.
 
+**Measured ledger (prior 262,144 config, read live 2026-09-12).** The
+engine's own startup account: 68.0 GiB of weights locked + 7.2 GiB KV pool +
+21.1 GiB working memory = **96.3 GiB held**, with **12.1 GiB left on the
+host**, only ~7.3 GiB of that in contiguous 2 MiB pieces. The 47.7 GiB n-gram
+lookup table is never held in RAM — it is read through the file cache — and
+the 2.0 GiB firmware carves for the iGPU appears in no `/proc/meminfo`
+figure.
+
+Doubling to 524,288 puts the held total near **103.5 GiB** and the host
+remainder at roughly **5 GiB**, against a measured ~4.3 GiB working set for
+the 25 co-tenant containers on this box. The extra 7.2 GiB comes out of the
+page cache, which falls from ~76 GiB to ~69 GiB — still above the 47.7 GiB
+lookup table, so the table can stay hot. Tight, not an overcommit. If long
+prompts start crawling with the disk busy, `HALOGEN_MAX_TOK=16384` shrinks
+the prefill arena 16.7 → 8.4 GiB and pays for the doubling outright at about
+8 % of prefill throughput.
+
 **Co-tenancy.** `HALOGEN_FLASH_PIN_TRUNK=1` (default) is what the published
 decode numbers were measured with, and it means this container holds most of
 the host. If a co-tenant then takes the remainder, both it and the engine can
@@ -156,9 +174,12 @@ deployment:
 **plus** `max_tokens` positions when it is admitted; if the pool cannot hold it
 the request waits in arrival order, and after the queue timeout (3600 s) it
 gets a `503 engine_busy`. A prompt large enough that prompt + budget exceeds
-the pool can therefore never be admitted at all. The 262,144-position pool is
-the binding constraint on how large a conversation can get, not `HALOGEN_CTX`
-and not the client's declared `context_length`.
+the pool can therefore never be admitted at all. With the pool at 524,288 the
+per-conversation ceiling is `HALOGEN_CTX` (262,144) — the pool binds **how
+many** conversations stay resident, not how long any one of them can be. Under
+the previous 262,144 pool the two numbers were identical, which is why four
+concurrent 65k chats were contending for one conversation's worth of
+positions.
 
 ## Vision