dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

docs(halogen): record the reserve ceiling and that the reserve is not an allocation

Browse Source 
Seven restarts measured. New section "The reserve cannot buy air": reserve 16
was deployed and held exactly what 15 holds — 103.5 GiB, 7.1 GiB left to the
host. The flag is a term in the fit-check inequality, not a reservation, so it
has only two regimes: below the pool's requirement it changes nothing
observable, above it the pool silently disappears.

Sizing rule now defers to the engine's printed requirement (budget >= 36.5 for
524288 at max_tok 32768, from "need ~35.0 GiB plus 1.5 of margin") over my
modelled 37.0, and the ledger table carries the reserve-16 row.
This commit is contained in:
Frank
2026-09-12 21:32:08 -07:00
parent 5c4e8a9600
commit a45edbcc6d
2 changed files with 47 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+7 -2
View File
@@ -403,8 +403,13 @@ four at 131k), `HALOGEN_KV_SLOTS=4`, `HALOGEN_MAX_TOK=32768`,
fetch; everything else is upstream's default. **Pool and reserve are a matched
pair:** the budget is `121.2 host − 67.7 weights − reserve`, and that one
budget carries the KV pool, the prefill arena and the 1.97 GiB vision tower
together. Raise the reserve back to 24 and `HALOGEN_KV_POOL_FIT` silently
clamps the pool to 262144 while the deploy still reports healthy. At 15 the
together. Raise the reserve past **16** and `HALOGEN_KV_POOL_FIT` silently
clamps the pool to 262144 while the deploy still reports healthy (the engine
needs budget ≥ 36.5; reserve 18 yields 35.5). The reserve is **not an
allocation** — 15 and 16 were both measured holding 103.5 GiB with 7.1 GiB left
on the host, so a bigger reserve cannot free memory for co-tenants, it can only
remove the pool. To free host RAM, cut `HALOGEN_KV_POOL_POSITIONS` (262144
hands back 7.2 GiB) or `HALOGEN_MAX_TOK` (16384 hands back 8.4). At 15 the
server holds 103.5 of 121 GiB, leaving ~7.1 GiB on the host, and the engine
warns that when it runs out frank-large and its neighbours can stall for
minutes at full CPU with no disk activity — not a crash, and invisible to
serverAppConfigs-docs/LLM/halogen-flash-server.md
+40 -7
View File
@@ -81,7 +81,7 @@ upstream's defaults:
| `HALOGEN_KV_POOL_POSITIONS` | `524288` | **in force** — two full-length conversations resident, or four at 131k, 13,312 MiB of KV. Was silently clamped to 262,144 until the reserve below dropped; `HALOGEN_KV_POOL_FIT` still guards the boot, so read the ledger rather than trusting this column |
| `HALOGEN_KV_SLOTS` | `4` | four concurrent generations; slots share the pool and cost ~115 MiB each |
| `HALOGEN_MAX_TOK` | `32768` | upstream's default, kept. The arena is 20.6 GiB here against 12.2 at 16384; it only fits alongside the doubled pool because the reserve dropped to 15. At 65536 the arena alone exceeds the budget and the server **will not boot** |
| `HALOGEN_HOST_RESERVE_GIB` | `15` | **lowered from 24 (vendor default 20) on 2026-09-12** to buy both features. The one term you own: it sets the device budget (24 → 29.5 GiB, 15 → 38.5 GiB). Matched pair with the pool — raise it back to 24 and the pool silently clamps to 262,144 again. Cost is page cache for the n-gram table; see "What the reserve actually costs" |
| `HALOGEN_HOST_RESERVE_GIB` | `15` | **lowered from 24 (vendor default 20) on 2026-09-12** to buy both features. The only term you own in the device budget (24 → 29.5 GiB, 15 → 38.5 GiB), and **the ceiling for the doubled pool at `max_tok=32768` is 16** — the engine needs budget ≥ 36.5, and 18 gives 35.5, which silently re-clamps the pool. Matched pair with the pool. It is **not** an allocation: raising it cannot return memory to the host, only take the pool away |
| `HALOGEN_VISION_TOWER` | `1` | image input; the fleet's only image path since [[frank-npu]] was retired. Charges 1.97 GiB against the budget — the term missing from my first estimate |
| `HALOGEN_DOWNLOAD` | `peonist-ai/halogen-qwen3.8-flash-next` | first-start fetch |
| `HALOGEN_VERBOSE` | `1` | verbose engine logging — the fullest startup account, on permanently so an incident report needs no restart-into-debug |
@@ -117,8 +117,8 @@ Three numbers explain the whole box:
the pool itself if the configured one will not fit, printing what it chose.
Read those lines on every restart; they are the ground truth for sizing.
**Measured ledger, six restarts, read live 2026-09-12.** The engine's own
startup account, including two deliberate experiments:
**Measured ledger, seven restarts, read live 2026-09-12.** The engine's own
startup account, including three deliberate experiments:
| pool | max_tok | reserve | KV | arena | need | budget | verdict |
|---|---|---|---|---|---|---|---|
@@ -127,6 +127,7 @@ startup account, including two deliberate experiments:
| 524288 | 32768 | 24 | 14.4 | 20.6 | 37.0 | 29.5 | clamped to 262144, deploy still green |
| 524288 | 65536 | 24 | — | 37.3 | 46.5 | 29.5 | **OOM — will not boot** |
| 524288 | 32768 | **15** | 14.4 | 20.6 | 37.0 | **38.5** | **deployed — both features** |
| 524288 | 32768 | 16 | 14.4 | 20.6 | 37.0 | 37.5 | fits, footprint unchanged — see the ceiling below |
### The sizing rule
@@ -145,8 +146,15 @@ arena = 3.80 GiB + 8.40 GiB x (HALOGEN_MAX_TOK / 16384)
The `121` the engine prints is rounded **down** from `MemTotal 127104060 kB =
121.2159 GiB`, so subtracting the displayed figures gives 29.3 rather than 29.5.
Use the unrounded value. Two of the three terms are fixed hardware and
checkpoint facts; `HALOGEN_HOST_RESERVE_GIB` is the only one you own, which
makes it the lever that buys both features — see the cost below.
checkpoint facts; `HALOGEN_HOST_RESERVE_GIB` is the only one you own — which
buys the *fit margin*, not the host's memory. It is a term in this inequality,
not an allocation, and the section at the end of this page shows the live test
that proves it.
Trust the engine's printed requirement over this model wherever they disagree:
a boot clamped at reserve 24 stated `524288 positions need ~35.0 GiB (plus 1.5
of margin)`, i.e. **budget >= 36.5**, against a modelled 37.0 here. Close
enough to plan with, and the authoritative number to cut the other way.
The arena term is **linear in `HALOGEN_MAX_TOK`** and predicts every measured
point to 0.1 GiB — including upstream's three published "on the device" figures
@@ -174,6 +182,7 @@ the reserve moves the machine's memory position considerably:
| 262144 / 32768, reserve 24 | 96.3 | 12.1 GiB | 2081 |
| 524288 / 16384, reserve 24 | 94.6 | 13.8 GiB | 3742 |
| **524288 / 32768, reserve 15** | **103.5** | **7.1 GiB** | **310** |
| 524288 / 32768, reserve **16** | **103.5** | **7.1 GiB** | 293 |
At the deployed setting the server holds 103.5 of 121 GiB and the engine now
volunteers a note it did not print before:
@@ -195,8 +204,32 @@ page cache counts as reclaimable. Watch the `host memory left for everything
else` line and the contiguous-block count at boot instead.
28 co-tenant containers (~4.3 GiB working set) stayed up through the change,
so it is survivable. The mitigation if anything goes quiet with the CPU
pinned: raise the reserve, or drop `HALOGEN_KV_POOL_POSITIONS` back to 262144.
so it is survivable.
### The reserve cannot buy air — ceiling 16
The last row above is the experiment that proves it. Raising the reserve from
15 to 16 left `held` and `host left` **byte-for-byte identical** (103.5 / 7.1
GiB). `HALOGEN_HOST_RESERVE_GIB` feeds the fit-check budget and is *not* an
allocation; the actual footprint is `weights + KV + arena + vision`, none of
which the reserve touches. So the knob has exactly two regimes:
| reserve | budget | vs required 36.5 | effect |
|---|---|---|---|
| 15 | 38.5 | +2.0 | deployed |
| 16 | 37.5 | +1.0 | no change to footprint, thinner margin — pointless |
| 17 | 36.5 | +0.02 | knife edge, inside the noise of the 67.7 term |
| **18** | **35.5** | **−1.0** | **clamps the pool to 262144** |
| 24 | 29.5 | −7.0 | clamps the pool to 262144 |
The 36.5 requirement is the engine's own, read from a clamped boot:
`524288 positions need ~35.0 GiB (plus 1.5 of margin)`. Above 16 the only
thing a bigger reserve can do is take the doubling away.
**To actually free host memory, change the pool or the arena**, and the reserve
then becomes free to rise: `HALOGEN_KV_POOL_POSITIONS=262144` hands back
7.2 GiB, `HALOGEN_MAX_TOK=16384` hands back 8.4. Both are one-line edits that
the fit check will confirm on the next boot.
### The `WARNING` line only appears when the fit check had to move