From a45edbcc6d14d4a535047803927ae11dc893c92d Mon Sep 17 00:00:00 2001 From: Frank Date: Sat, 12 Sep 2026 21:32:08 -0700 Subject: [PATCH] docs(halogen): record the reserve ceiling and that the reserve is not an allocation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Seven restarts measured. New section "The reserve cannot buy air": reserve 16 was deployed and held exactly what 15 holds — 103.5 GiB, 7.1 GiB left to the host. The flag is a term in the fit-check inequality, not a reservation, so it has only two regimes: below the pool's requirement it changes nothing observable, above it the pool silently disappears. Sizing rule now defers to the engine's printed requirement (budget >= 36.5 for 524288 at max_tok 32768, from "need ~35.0 GiB plus 1.5 of margin") over my modelled 37.0, and the ledger table carries the reserve-16 row. --- AGENTS.md | 9 +++- .../LLM/halogen-flash-server.md | 47 ++++++++++++++++--- 2 files changed, 47 insertions(+), 9 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 77cc286..45ab55a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -403,8 +403,13 @@ four at 131k), `HALOGEN_KV_SLOTS=4`, `HALOGEN_MAX_TOK=32768`, fetch; everything else is upstream's default. **Pool and reserve are a matched pair:** the budget is `121.2 host − 67.7 weights − reserve`, and that one budget carries the KV pool, the prefill arena and the 1.97 GiB vision tower -together. Raise the reserve back to 24 and `HALOGEN_KV_POOL_FIT` silently -clamps the pool to 262144 while the deploy still reports healthy. At 15 the +together. Raise the reserve past **16** and `HALOGEN_KV_POOL_FIT` silently +clamps the pool to 262144 while the deploy still reports healthy (the engine +needs budget ≥ 36.5; reserve 18 yields 35.5). The reserve is **not an +allocation** — 15 and 16 were both measured holding 103.5 GiB with 7.1 GiB left +on the host, so a bigger reserve cannot free memory for co-tenants, it can only +remove the pool. To free host RAM, cut `HALOGEN_KV_POOL_POSITIONS` (262144 +hands back 7.2 GiB) or `HALOGEN_MAX_TOK` (16384 hands back 8.4). At 15 the server holds 103.5 of 121 GiB, leaving ~7.1 GiB on the host, and the engine warns that when it runs out frank-large and its neighbours can stall for minutes at full CPU with no disk activity — not a crash, and invisible to diff --git a/serverAppConfigs-docs/LLM/halogen-flash-server.md b/serverAppConfigs-docs/LLM/halogen-flash-server.md index 675ed0e..4808c7c 100644 --- a/serverAppConfigs-docs/LLM/halogen-flash-server.md +++ b/serverAppConfigs-docs/LLM/halogen-flash-server.md @@ -81,7 +81,7 @@ upstream's defaults: | `HALOGEN_KV_POOL_POSITIONS` | `524288` | **in force** — two full-length conversations resident, or four at 131k, 13,312 MiB of KV. Was silently clamped to 262,144 until the reserve below dropped; `HALOGEN_KV_POOL_FIT` still guards the boot, so read the ledger rather than trusting this column | | `HALOGEN_KV_SLOTS` | `4` | four concurrent generations; slots share the pool and cost ~115 MiB each | | `HALOGEN_MAX_TOK` | `32768` | upstream's default, kept. The arena is 20.6 GiB here against 12.2 at 16384; it only fits alongside the doubled pool because the reserve dropped to 15. At 65536 the arena alone exceeds the budget and the server **will not boot** | -| `HALOGEN_HOST_RESERVE_GIB` | `15` | **lowered from 24 (vendor default 20) on 2026-09-12** to buy both features. The one term you own: it sets the device budget (24 → 29.5 GiB, 15 → 38.5 GiB). Matched pair with the pool — raise it back to 24 and the pool silently clamps to 262,144 again. Cost is page cache for the n-gram table; see "What the reserve actually costs" | +| `HALOGEN_HOST_RESERVE_GIB` | `15` | **lowered from 24 (vendor default 20) on 2026-09-12** to buy both features. The only term you own in the device budget (24 → 29.5 GiB, 15 → 38.5 GiB), and **the ceiling for the doubled pool at `max_tok=32768` is 16** — the engine needs budget ≥ 36.5, and 18 gives 35.5, which silently re-clamps the pool. Matched pair with the pool. It is **not** an allocation: raising it cannot return memory to the host, only take the pool away | | `HALOGEN_VISION_TOWER` | `1` | image input; the fleet's only image path since [[frank-npu]] was retired. Charges 1.97 GiB against the budget — the term missing from my first estimate | | `HALOGEN_DOWNLOAD` | `peonist-ai/halogen-qwen3.8-flash-next` | first-start fetch | | `HALOGEN_VERBOSE` | `1` | verbose engine logging — the fullest startup account, on permanently so an incident report needs no restart-into-debug | @@ -117,8 +117,8 @@ Three numbers explain the whole box: the pool itself if the configured one will not fit, printing what it chose. Read those lines on every restart; they are the ground truth for sizing. -**Measured ledger, six restarts, read live 2026-09-12.** The engine's own -startup account, including two deliberate experiments: +**Measured ledger, seven restarts, read live 2026-09-12.** The engine's own +startup account, including three deliberate experiments: | pool | max_tok | reserve | KV | arena | need | budget | verdict | |---|---|---|---|---|---|---|---| @@ -127,6 +127,7 @@ startup account, including two deliberate experiments: | 524288 | 32768 | 24 | 14.4 | 20.6 | 37.0 | 29.5 | clamped to 262144, deploy still green | | 524288 | 65536 | 24 | — | 37.3 | 46.5 | 29.5 | **OOM — will not boot** | | 524288 | 32768 | **15** | 14.4 | 20.6 | 37.0 | **38.5** | **deployed — both features** | +| 524288 | 32768 | 16 | 14.4 | 20.6 | 37.0 | 37.5 | fits, footprint unchanged — see the ceiling below | ### The sizing rule @@ -145,8 +146,15 @@ arena = 3.80 GiB + 8.40 GiB x (HALOGEN_MAX_TOK / 16384) The `121` the engine prints is rounded **down** from `MemTotal 127104060 kB = 121.2159 GiB`, so subtracting the displayed figures gives 29.3 rather than 29.5. Use the unrounded value. Two of the three terms are fixed hardware and -checkpoint facts; `HALOGEN_HOST_RESERVE_GIB` is the only one you own, which -makes it the lever that buys both features — see the cost below. +checkpoint facts; `HALOGEN_HOST_RESERVE_GIB` is the only one you own — which +buys the *fit margin*, not the host's memory. It is a term in this inequality, +not an allocation, and the section at the end of this page shows the live test +that proves it. + +Trust the engine's printed requirement over this model wherever they disagree: +a boot clamped at reserve 24 stated `524288 positions need ~35.0 GiB (plus 1.5 +of margin)`, i.e. **budget >= 36.5**, against a modelled 37.0 here. Close +enough to plan with, and the authoritative number to cut the other way. The arena term is **linear in `HALOGEN_MAX_TOK`** and predicts every measured point to 0.1 GiB — including upstream's three published "on the device" figures @@ -174,6 +182,7 @@ the reserve moves the machine's memory position considerably: | 262144 / 32768, reserve 24 | 96.3 | 12.1 GiB | 2081 | | 524288 / 16384, reserve 24 | 94.6 | 13.8 GiB | 3742 | | **524288 / 32768, reserve 15** | **103.5** | **7.1 GiB** | **310** | +| 524288 / 32768, reserve **16** | **103.5** | **7.1 GiB** | 293 | At the deployed setting the server holds 103.5 of 121 GiB and the engine now volunteers a note it did not print before: @@ -195,8 +204,32 @@ page cache counts as reclaimable. Watch the `host memory left for everything else` line and the contiguous-block count at boot instead. 28 co-tenant containers (~4.3 GiB working set) stayed up through the change, -so it is survivable. The mitigation if anything goes quiet with the CPU -pinned: raise the reserve, or drop `HALOGEN_KV_POOL_POSITIONS` back to 262144. +so it is survivable. + +### The reserve cannot buy air — ceiling 16 + +The last row above is the experiment that proves it. Raising the reserve from +15 to 16 left `held` and `host left` **byte-for-byte identical** (103.5 / 7.1 +GiB). `HALOGEN_HOST_RESERVE_GIB` feeds the fit-check budget and is *not* an +allocation; the actual footprint is `weights + KV + arena + vision`, none of +which the reserve touches. So the knob has exactly two regimes: + +| reserve | budget | vs required 36.5 | effect | +|---|---|---|---| +| 15 | 38.5 | +2.0 | deployed | +| 16 | 37.5 | +1.0 | no change to footprint, thinner margin — pointless | +| 17 | 36.5 | +0.02 | knife edge, inside the noise of the 67.7 term | +| **18** | **35.5** | **−1.0** | **clamps the pool to 262144** | +| 24 | 29.5 | −7.0 | clamps the pool to 262144 | + +The 36.5 requirement is the engine's own, read from a clamped boot: +`524288 positions need ~35.0 GiB (plus 1.5 of margin)`. Above 16 the only +thing a bigger reserve can do is take the doubling away. + +**To actually free host memory, change the pool or the arena**, and the reserve +then becomes free to rise: `HALOGEN_KV_POOL_POSITIONS=262144` hands back +7.2 GiB, `HALOGEN_MAX_TOK=16384` hands back 8.4. Both are one-line edits that +the fit check will confirm on the next boot. ### The `WARNING` line only appears when the fit check had to move