dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-large: trim KV pool 524288 -> 401408 for co-tenant headroom

Browse Source 
The 09-12 doubled pool fit only at reserve 15, leaving 7.1 GiB / 310
contiguous 2 MiB blocks for the ~28 co-tenant containers and making the
engine print its minutes-long-stall warning on every boot. 401,408
positions (~10.8 GiB KV) hands ~3.4 GiB back while keeping MAX_TOK=32768
and reserve 15: modelled need ~33.4 GiB vs budget 38.5, so the fit check
should not clamp. Capacity becomes one full 262k conversation resident
beside a ~139k one, or three concurrent 131k chats (three slots).

Docs: AGENTS.md fleet table + halogen section, runbook flags table /
co-tenancy tense / admission-arithmetic paragraph updated for the trim.
This commit is contained in:
Frank
2026-09-13 22:57:25 -07:00
parent 025216fc84
commit 91406af2a1
3 changed files with 76 additions and 63 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/halogen-flash-server.md
+21 -17
View File
@@ -11,7 +11,7 @@ tags:
- gfx1151
- qwen4exp
- rocm
timestamp: 2026-09-12
timestamp: 2026-09-13
---
# halogen-flash-server on Dankclaw (frank-large)
@@ -78,10 +78,10 @@ upstream's defaults:
|---|---|---|
| `HALOGEN_MODEL_ID` | `frank-large` | fleet identity; no client rename |
| `HALOGEN_CTX` | `262144` | the model's native context, and what both clients declare |
| `HALOGEN_KV_POOL_POSITIONS` | `524288` | **in force** — two full-length conversations resident, or four at 131k, 13,312 MiB of KV. Was silently clamped to 262,144 until the reserve below dropped; `HALOGEN_KV_POOL_FIT` still guards the boot, so read the ledger rather than trusting this column |
| `HALOGEN_KV_SLOTS` | `3` | Concurrency cap — **lowered from 4 on 2026-09-12** to the real fleet size. Each slot is 111.0 MiB of O(1) state, so this freed ~111 MiB (held 103.5 → 103.4 GiB). Slots share the ONE pool and **oversubscribe** it (4 × 262144 > 524288), so admission was always pool-bound: at most two full-length conversations, or four at 131k. This trims simultaneous requests, not how much context fits |
| `HALOGEN_MAX_TOK` | `32768` | upstream's default, kept. The arena is 20.6 GiB here against 12.2 at 16384; it only fits alongside the doubled pool because the reserve dropped to 15. At 65536 the arena alone exceeds the budget and the server **will not boot** |
| `HALOGEN_HOST_RESERVE_GIB` | `15` | **lowered from 24 (vendor default 20) on 2026-09-12** to buy both features. The only term you own in the device budget (24 → 29.5 GiB, 15 → 38.5 GiB), and **the ceiling for the doubled pool at `max_tok=32768` is 16** — the engine needs budget ≥ 36.5, and 18 gives 35.5, which silently re-clamps the pool. Matched pair with the pool. It is **not** an allocation: raising it cannot return memory to the host, only take the pool away |
| `HALOGEN_KV_POOL_POSITIONS` | `401408` | **trimmed 2026-09-13** from the 524,288 the doubling deployed: one full 262k conversation resident beside a ~139k one, or three at 131k, ~10.8 GiB of KV. The trim hands ~3.4 GiB back to the ~28 co-tenants — the doubled pool left only ~7.1 GiB on the host and the engine warned of co-tenant stalls on every boot. `HALOGEN_KV_POOL_FIT` still guards the boot, so read the ledger rather than trusting this column |
| `HALOGEN_KV_SLOTS` | `3` | Concurrency cap — **lowered from 4 on 2026-09-12** to the real fleet size. Each slot is 111.0 MiB of O(1) state, so this freed ~111 MiB (held 103.5 → 103.4 GiB). Slots share the ONE pool and **oversubscribe** it (3 × 262144 > 401408), so admission is pool-bound: one full-length conversation plus a ~139k second, or three at 131k. This trims simultaneous requests, not how much context fits |
| `HALOGEN_MAX_TOK` | `32768` | upstream's default, kept. The arena is 20.6 GiB here against 12.2 at 16384; it fit alongside the doubled pool only because the reserve dropped to 15, and it fits the trimmed 401,408 pool with room to spare. At 65536 the arena alone exceeds the budget and the server **will not boot** |
| `HALOGEN_HOST_RESERVE_GIB` | `15` | **lowered from 24 (vendor default 20) on 2026-09-12** to buy the doubled pool. The only term you own in the device budget (24 → 29.5 GiB, 15 → 38.5 GiB). Against the doubled pool the ceiling was 16 — the engine needed budget ≥ 36.5, and 18 gave 35.5, silently re-clamping; at 401,408 the need is ~33.4 so the ceiling loosens, but a bigger reserve still buys nothing. Matched pair with the pool. It is **not** an allocation: raising it cannot return memory to the host, only take the pool away |
| `HALOGEN_VISION_TOWER` | `1` | image input; the fleet's only image path since [[frank-npu]] was retired. Charges 1.97 GiB against the budget — the term missing from my first estimate |
| `HALOGEN_DOWNLOAD` | `peonist-ai/halogen-qwen3.8-flash-next` | first-start fetch |
| `HALOGEN_VERBOSE` | `1` | verbose engine logging — the fullest startup account, on permanently so an incident report needs no restart-into-debug |
@@ -174,8 +174,8 @@ fits unmodified.
### What the reserve actually costs
The device budget is not the whole story. Freeing the doubled pool by cutting
the reserve moves the machine's memory position considerably:
The device budget is not the whole story. Memory position by configuration
(the 09-13 trim added its row after the deploy was measured — see below):
| config | held | host left | contiguous 2 MiB blocks |
|---|---|---|---|
@@ -184,8 +184,8 @@ the reserve moves the machine's memory position considerably:
| **524288 / 32768, reserve 15** | **103.5** | **7.1 GiB** | **310** |
| 524288 / 32768, reserve **16** | **103.5** | **7.1 GiB** | 293 |
At the deployed setting the server holds 103.5 of 121 GiB and the engine now
volunteers a note it did not print before:
At the doubled setting (2026-09-12) the server held 103.5 of 121 GiB and the
engine volunteered a note it had not printed before:
```
halogen: NOTE this server now holds most of the machine, and 310 contiguous
@@ -228,8 +228,10 @@ thing a bigger reserve can do is take the doubling away.
**To actually free host memory, change the pool or the arena**, and the reserve
then becomes free to rise: `HALOGEN_KV_POOL_POSITIONS=262144` hands back
7.2 GiB, `HALOGEN_MAX_TOK=16384` hands back 8.4. Both are one-line edits that
the fit check will confirm on the next boot.
7.2 GiB from the doubled pool, `HALOGEN_MAX_TOK=16384` hands back 8.4. Both
are one-line edits that the fit check will confirm on the next boot. The
2026-09-13 trim to 401,408 was this remedy applied at half strength: ~3.4 GiB
back with `HALOGEN_MAX_TOK` untouched.
### The `WARNING` line only appears when the fit check had to move
@@ -320,12 +322,14 @@ deployment:
**plus** `max_tokens` positions when it is admitted; if the pool cannot hold it
the request waits in arrival order, and after the queue timeout (3600 s) it
gets a `503 engine_busy`. A prompt large enough that prompt + budget exceeds
the pool can therefore never be admitted at all. With the pool at 524,288 the
per-conversation ceiling is `HALOGEN_CTX` (262,144) — the pool binds **how
many** conversations stay resident, not how long any one of them can be. Under
the previous 262,144 pool the two numbers were identical, which is why four
concurrent 65k chats were contending for one conversation's worth of
positions.
the pool can therefore never be admitted at all. With the pool at 401,408
(since 2026-09-13) the per-conversation ceiling is still `HALOGEN_CTX`
(262,144) — even a full-context prompt plus the 65,536 cap is 327,680 ≤
401,408. The pool binds **how many** conversations stay resident: at 524,288
two full-length ones fit; at 401,408 it is one full-length beside a ~139k
second, or three at 131k. Under the pre-09-12 262,144 pool the two numbers
were identical, which is why four concurrent 65k chats were contending for
one conversation's worth of positions.
## Vision