diff --git a/dankclaw/frank-large/quadlet/frank-large.container b/dankclaw/frank-large/quadlet/frank-large.container index 7aed5bb..16942eb 100644 --- a/dankclaw/frank-large/quadlet/frank-large.container +++ b/dankclaw/frank-large/quadlet/frank-large.container @@ -107,13 +107,25 @@ Environment=HALOGEN_KV_SLOTS=4 # LOAD-BEARING. This line is what makes the doubling above real; do not # "clean it up". The device budget here is 121 GiB - 67.7 resident weights # - 24.0 reserved for the n-gram page cache and the OS = 29.5 GiB, and that -# one budget carries BOTH the KV pool and the prefill arena. At the upstream -# HALOGEN_MAX_TOK=32768 the arena is ~20.6 GiB, so 524288 positions need -# 35.0 GiB against 29.5 available and HALOGEN_KV_POOL_FIT silently lowers -# the pool back to 262144 — the deploy still reports healthy, so the -# regression is invisible unless you read the ledger. Halving the arena to -# ~10.5 GiB brings the 524288 need to ~24.9 GiB, which fits with ~3 GiB -# of margin, at about 8-9% of prefill throughput. +# one budget carries BOTH the KV pool and the prefill arena. +# +# Measured across three restarts on 2026-09-12. The arena (the engine's +# "everything but the pool" figure) is linear in this flag: 3.80 GiB of base +# plus 8.40 GiB per 16384 tokens. +# 16384 -> arena 12.2, 524288 needs 26.6 -> FITS +# 32768 -> arena 20.6, 524288 needs 35.0 -> fit lowers the pool to 262144 +# 65536 -> arena 37.3, even 262144 needs 44.5 -> OOM, dmalloc FAILED at 39.97 +# So 16384 is not conservative, it is the ceiling: 29.5 budget - 1.5 margin - +# 14.4 pool leaves 13.6 for the arena, i.e. max_tok <= 19,115, and 16384 is +# the largest power of two under that. The reverse checks too — put max_tok +# back to 32768 and only 7.4 GiB is left for the pool, about 269k positions, +# which is exactly the 262144 config this replaced. Those two corners are the +# only points of this box that have ever run here. +# +# Do NOT raise this flag past 16384 while the pool stays doubled: 65536 does +# not shrink the pool, it stops the server booting at all — and the unit still +# reads active with the container "Up (starting)", so the health check waits +# out LLM_WAIT rather than failing fast. Environment=HALOGEN_MAX_TOK=16384 # RAM left for the n-gram table the engine reads through the page cache. # Above the vendor's 20 because a dozen containers share this host; too low diff --git a/serverAppConfigs-docs/LLM/halogen-flash-server.md b/serverAppConfigs-docs/LLM/halogen-flash-server.md index bd0243e..ee070c3 100644 --- a/serverAppConfigs-docs/LLM/halogen-flash-server.md +++ b/serverAppConfigs-docs/LLM/halogen-flash-server.md @@ -80,7 +80,7 @@ upstream's defaults: | `HALOGEN_CTX` | `262144` | the model's native context, and what both clients declare | | `HALOGEN_KV_POOL_POSITIONS` | `524288` | **doubled 2026-09-12** to upstream's own default (2 × `HALOGEN_CTX`): two full-length conversations resident, or four at 131k — 13,312 MiB of KV | | `HALOGEN_KV_SLOTS` | `4` | four concurrent generations; slots share the pool and cost ~115 MiB each | -| `HALOGEN_MAX_TOK` | `16384` | **load-bearing for the doubling.** Halves the prefill arena so the pool fits the 29.5 GiB device budget, at ~8–9 % of prefill throughput. Leave it at upstream's 32768 and `HALOGEN_KV_POOL_FIT` silently reverts the pool to 262144 while the deploy still reports healthy | +| `HALOGEN_MAX_TOK` | `16384` | **load-bearing for the doubling, and already at the ceiling.** Halves the prefill arena so the pool fits the 29.5 GiB device budget, at ~8–9 % of prefill throughput. At 32768 `HALOGEN_KV_POOL_FIT` silently reverts the pool to 262144; at 65536 the arena alone exceeds the budget and **the server will not boot**. See the sizing rule below | | `HALOGEN_VISION_TOWER` | `1` | image input; the fleet's only image path since [[frank-npu]] was retired | | `HALOGEN_DOWNLOAD` | `peonist-ai/halogen-qwen3.8-flash-next` | first-start fetch | | `HALOGEN_VERBOSE` | `1` | verbose engine logging — the fullest startup account, on permanently so an incident report needs no restart-into-debug | @@ -116,33 +116,74 @@ Three numbers explain the whole box: the pool itself if the configured one will not fit, printing what it chose. Read those lines on every restart; they are the ground truth for sizing. -**Measured ledger, both configs, read live 2026-09-12.** The engine's own +**Measured ledger, four restarts, read live 2026-09-12.** The engine's own startup account: -| config | weights | KV pool | working mem | held | host left | -|---|---|---|---|---|---| -| pool 262144 / max_tok 32768 (prior) | 68.0 | 7.2 | 21.1 | **96.3** | 12.1 | -| pool 524288 / max_tok 16384 (now) | 68.0 | 14.4 | 12.3 | **94.6** | 13.8 | +| pool | max_tok | KV pool | arena | held | host left | verdict | +|---|---|---|---|---|---|---| +| 262144 | 32768 | 7.2 | 20.6 | 96.3 | 12.1 | the prior config | +| 524288 | 16384 | 14.4 | 12.2 | **94.6** | 13.8 | **deployed** | +| 524288 | 32768 | 14.4 | 20.6 | — | — | pool silently lowered to 262144 | +| 524288 | 65536 | — | 37.3 | — | — | **would not boot — OOM** | -The doubled pool is **1.7 GiB cheaper in total** than the config it replaced. -Halving the prefill arena gives back more than the extra pool takes: the -arena is ~20.6 GiB regardless of context, while the KV costs only ~28 KiB a -position. +### The sizing rule -The first attempt — 524288 with the arena left at 32768 — did not fit, and -the way it failed is the trap on this host: +One budget covers the pool **and** the arena together: + +``` +device budget = host RAM - resident weights - HALOGEN_HOST_RESERVE_GIB + = 121 - 67.7 - 24 = 29.5 GiB +``` + +The arena is **linear in `HALOGEN_MAX_TOK`**: measured at 3.80 GiB of base plus +8.40 GiB per 16,384 tokens, which predicts all three measured arenas to within +0.1 GiB. The KV costs ~28 KiB a position, so 524,288 of them cost 14.4 GiB. +Solving for the arena with the doubled pool in place: + +``` +29.5 budget - 1.5 margin - 14.4 pool = 13.6 GiB left for the arena + -> HALOGEN_MAX_TOK <= 19,115 +``` + +16384 is the largest power of two under that, so the deployed setting is not +conservative — **it is the ceiling**. The corner check confirms it: put +`max_tok` back to 32768 and only 7.4 GiB remains for the pool, about 269k +positions, which is precisely the 262,144 config that ran here before. Those +two corners are the only points of this box the host has ever served. + +Note the consequence for the doubling: it is **1.7 GiB cheaper overall** than +the config it replaced, because halving the arena gives back more than the +extra pool takes. + +### Two failure modes, and only one is dangerous + +Raising `max_tok` past the ceiling does not shrink the pool — the arena alone +outgrows the budget, and the server cannot allocate at all: + +``` +kv pool: 524288 positions need ~51.7 GiB ... LOWERING THE POOL TO 262144 (~44.5 GiB). +kv pool: WARNING even 262144 positions (the context, the smallest pool that can + serve one request) needs ~44.5 GiB and host RAM cannot spare it. +dmalloc: FAILED requesting 0.250 GiB after 39.970 GiB in 566 allocations (out of memory) +``` + +This is the quiet one: the unit stays `active`, the container reports +`Up (starting)`, and `deploy.sh` sits in its health wait for the whole +`LLM_WAIT` rather than failing fast. It confirms upstream's warning that a +large `HALOGEN_MAX_TOK` means "the server will not start", not "it will be +slower". + +Under-running the ceiling is far more benign and far easier to miss. At 524288 +with the arena back at 32768, `HALOGEN_KV_POOL_FIT` lowered the pool and +`deploy.sh` still reported healthy in 12 seconds: ``` -kv pool: host RAM 121 GiB, less 67.7 resident weights and 24.0 reserved - for the n-gram page cache and the OS = 29.5 GiB for the device kv pool: 524288 positions need ~35.0 GiB (plus 1.5 of margin) and host RAM cannot spare it. LOWERING THE POOL TO 262144 (~27.8 GiB). ``` -One budget covers the pool **and** the arena together. `HALOGEN_KV_POOL_FIT` -lowered the pool, and `deploy.sh` still reported healthy in 12 seconds. A -green health check does not mean the pool you asked for is the pool you got — -always read the `kv pool:` lines after a restart. +A green health check does not mean the pool you asked for is the pool you got — +read the `kv pool:` lines after every restart. The 47.7 GiB n-gram lookup table is never held in RAM; it is read through the file cache. The 2.0 GiB the firmware carves for the iGPU appears in no