From 41ac1f83f6ea8bba13cbf2a2ba1c828d7352acc2 Mon Sep 17 00:00:00 2001 From: Frank Date: Sat, 12 Sep 2026 19:38:13 -0700 Subject: [PATCH] docs(halogen): record the measured arena model; the 65536 experiment was worth doing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Raising HALOGEN_MAX_TOK to 65536 did not shrink the pool, it OOMed the boot: the arena alone (37.3 GiB) exceeds the 29.5 GiB device budget, so even the minimum pool faulted and dmalloc failed at 39.97 GiB. Rolled back to 16384, which restored the 524288 pool and a clean completion. The three measured arenas are exactly linear in the flag — 3.80 GiB base plus 8.40 GiB per 16384 tokens, predicting all three to 0.1 GiB. That gives a real sizing rule instead of guesswork, and it says the deployed 16384 is the CEILING, not a conservative choice: 29.5 - 1.5 margin - 14.4 pool leaves 13.6 for the arena, i.e. max_tok <= 19,115. The corner check falls out of the same model: set max_tok back to 32768 and only 7.4 GiB remains for the pool, ~269k positions, which is precisely the 262144 config this host ran before. The two corners of the feasible box are the only two points it has ever served. No functional change — the quadlet edit is comments only; the running service is already at 16384 from the rollback deploy (258cccb). --- .../frank-large/quadlet/frank-large.container | 26 +++++-- .../LLM/halogen-flash-server.md | 77 ++++++++++++++----- 2 files changed, 78 insertions(+), 25 deletions(-) diff --git a/dankclaw/frank-large/quadlet/frank-large.container b/dankclaw/frank-large/quadlet/frank-large.container index 7aed5bb..16942eb 100644 --- a/dankclaw/frank-large/quadlet/frank-large.container +++ b/dankclaw/frank-large/quadlet/frank-large.container @@ -107,13 +107,25 @@ Environment=HALOGEN_KV_SLOTS=4 # LOAD-BEARING. This line is what makes the doubling above real; do not # "clean it up". The device budget here is 121 GiB - 67.7 resident weights # - 24.0 reserved for the n-gram page cache and the OS = 29.5 GiB, and that -# one budget carries BOTH the KV pool and the prefill arena. At the upstream -# HALOGEN_MAX_TOK=32768 the arena is ~20.6 GiB, so 524288 positions need -# 35.0 GiB against 29.5 available and HALOGEN_KV_POOL_FIT silently lowers -# the pool back to 262144 — the deploy still reports healthy, so the -# regression is invisible unless you read the ledger. Halving the arena to -# ~10.5 GiB brings the 524288 need to ~24.9 GiB, which fits with ~3 GiB -# of margin, at about 8-9% of prefill throughput. +# one budget carries BOTH the KV pool and the prefill arena. +# +# Measured across three restarts on 2026-09-12. The arena (the engine's +# "everything but the pool" figure) is linear in this flag: 3.80 GiB of base +# plus 8.40 GiB per 16384 tokens. +# 16384 -> arena 12.2, 524288 needs 26.6 -> FITS +# 32768 -> arena 20.6, 524288 needs 35.0 -> fit lowers the pool to 262144 +# 65536 -> arena 37.3, even 262144 needs 44.5 -> OOM, dmalloc FAILED at 39.97 +# So 16384 is not conservative, it is the ceiling: 29.5 budget - 1.5 margin - +# 14.4 pool leaves 13.6 for the arena, i.e. max_tok <= 19,115, and 16384 is +# the largest power of two under that. The reverse checks too — put max_tok +# back to 32768 and only 7.4 GiB is left for the pool, about 269k positions, +# which is exactly the 262144 config this replaced. Those two corners are the +# only points of this box that have ever run here. +# +# Do NOT raise this flag past 16384 while the pool stays doubled: 65536 does +# not shrink the pool, it stops the server booting at all — and the unit still +# reads active with the container "Up (starting)", so the health check waits +# out LLM_WAIT rather than failing fast. Environment=HALOGEN_MAX_TOK=16384 # RAM left for the n-gram table the engine reads through the page cache. # Above the vendor's 20 because a dozen containers share this host; too low diff --git a/serverAppConfigs-docs/LLM/halogen-flash-server.md b/serverAppConfigs-docs/LLM/halogen-flash-server.md index bd0243e..ee070c3 100644 --- a/serverAppConfigs-docs/LLM/halogen-flash-server.md +++ b/serverAppConfigs-docs/LLM/halogen-flash-server.md @@ -80,7 +80,7 @@ upstream's defaults: | `HALOGEN_CTX` | `262144` | the model's native context, and what both clients declare | | `HALOGEN_KV_POOL_POSITIONS` | `524288` | **doubled 2026-09-12** to upstream's own default (2 × `HALOGEN_CTX`): two full-length conversations resident, or four at 131k — 13,312 MiB of KV | | `HALOGEN_KV_SLOTS` | `4` | four concurrent generations; slots share the pool and cost ~115 MiB each | -| `HALOGEN_MAX_TOK` | `16384` | **load-bearing for the doubling.** Halves the prefill arena so the pool fits the 29.5 GiB device budget, at ~8–9 % of prefill throughput. Leave it at upstream's 32768 and `HALOGEN_KV_POOL_FIT` silently reverts the pool to 262144 while the deploy still reports healthy | +| `HALOGEN_MAX_TOK` | `16384` | **load-bearing for the doubling, and already at the ceiling.** Halves the prefill arena so the pool fits the 29.5 GiB device budget, at ~8–9 % of prefill throughput. At 32768 `HALOGEN_KV_POOL_FIT` silently reverts the pool to 262144; at 65536 the arena alone exceeds the budget and **the server will not boot**. See the sizing rule below | | `HALOGEN_VISION_TOWER` | `1` | image input; the fleet's only image path since [[frank-npu]] was retired | | `HALOGEN_DOWNLOAD` | `peonist-ai/halogen-qwen3.8-flash-next` | first-start fetch | | `HALOGEN_VERBOSE` | `1` | verbose engine logging — the fullest startup account, on permanently so an incident report needs no restart-into-debug | @@ -116,33 +116,74 @@ Three numbers explain the whole box: the pool itself if the configured one will not fit, printing what it chose. Read those lines on every restart; they are the ground truth for sizing. -**Measured ledger, both configs, read live 2026-09-12.** The engine's own +**Measured ledger, four restarts, read live 2026-09-12.** The engine's own startup account: -| config | weights | KV pool | working mem | held | host left | -|---|---|---|---|---|---| -| pool 262144 / max_tok 32768 (prior) | 68.0 | 7.2 | 21.1 | **96.3** | 12.1 | -| pool 524288 / max_tok 16384 (now) | 68.0 | 14.4 | 12.3 | **94.6** | 13.8 | +| pool | max_tok | KV pool | arena | held | host left | verdict | +|---|---|---|---|---|---|---| +| 262144 | 32768 | 7.2 | 20.6 | 96.3 | 12.1 | the prior config | +| 524288 | 16384 | 14.4 | 12.2 | **94.6** | 13.8 | **deployed** | +| 524288 | 32768 | 14.4 | 20.6 | — | — | pool silently lowered to 262144 | +| 524288 | 65536 | — | 37.3 | — | — | **would not boot — OOM** | -The doubled pool is **1.7 GiB cheaper in total** than the config it replaced. -Halving the prefill arena gives back more than the extra pool takes: the -arena is ~20.6 GiB regardless of context, while the KV costs only ~28 KiB a -position. +### The sizing rule -The first attempt — 524288 with the arena left at 32768 — did not fit, and -the way it failed is the trap on this host: +One budget covers the pool **and** the arena together: + +``` +device budget = host RAM - resident weights - HALOGEN_HOST_RESERVE_GIB + = 121 - 67.7 - 24 = 29.5 GiB +``` + +The arena is **linear in `HALOGEN_MAX_TOK`**: measured at 3.80 GiB of base plus +8.40 GiB per 16,384 tokens, which predicts all three measured arenas to within +0.1 GiB. The KV costs ~28 KiB a position, so 524,288 of them cost 14.4 GiB. +Solving for the arena with the doubled pool in place: + +``` +29.5 budget - 1.5 margin - 14.4 pool = 13.6 GiB left for the arena + -> HALOGEN_MAX_TOK <= 19,115 +``` + +16384 is the largest power of two under that, so the deployed setting is not +conservative — **it is the ceiling**. The corner check confirms it: put +`max_tok` back to 32768 and only 7.4 GiB remains for the pool, about 269k +positions, which is precisely the 262,144 config that ran here before. Those +two corners are the only points of this box the host has ever served. + +Note the consequence for the doubling: it is **1.7 GiB cheaper overall** than +the config it replaced, because halving the arena gives back more than the +extra pool takes. + +### Two failure modes, and only one is dangerous + +Raising `max_tok` past the ceiling does not shrink the pool — the arena alone +outgrows the budget, and the server cannot allocate at all: + +``` +kv pool: 524288 positions need ~51.7 GiB ... LOWERING THE POOL TO 262144 (~44.5 GiB). +kv pool: WARNING even 262144 positions (the context, the smallest pool that can + serve one request) needs ~44.5 GiB and host RAM cannot spare it. +dmalloc: FAILED requesting 0.250 GiB after 39.970 GiB in 566 allocations (out of memory) +``` + +This is the quiet one: the unit stays `active`, the container reports +`Up (starting)`, and `deploy.sh` sits in its health wait for the whole +`LLM_WAIT` rather than failing fast. It confirms upstream's warning that a +large `HALOGEN_MAX_TOK` means "the server will not start", not "it will be +slower". + +Under-running the ceiling is far more benign and far easier to miss. At 524288 +with the arena back at 32768, `HALOGEN_KV_POOL_FIT` lowered the pool and +`deploy.sh` still reported healthy in 12 seconds: ``` -kv pool: host RAM 121 GiB, less 67.7 resident weights and 24.0 reserved - for the n-gram page cache and the OS = 29.5 GiB for the device kv pool: 524288 positions need ~35.0 GiB (plus 1.5 of margin) and host RAM cannot spare it. LOWERING THE POOL TO 262144 (~27.8 GiB). ``` -One budget covers the pool **and** the arena together. `HALOGEN_KV_POOL_FIT` -lowered the pool, and `deploy.sh` still reported healthy in 12 seconds. A -green health check does not mean the pool you asked for is the pool you got — -always read the `kv pool:` lines after a restart. +A green health check does not mean the pool you asked for is the pool you got — +read the `kv pool:` lines after every restart. The 47.7 GiB n-gram lookup table is never held in RAM; it is read through the file cache. The 2.0 GiB the firmware carves for the iGPU appears in no