dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-large: document reserve 15 result — both features in force, stall risk logged

Browse Source 
Ledger confirms the doubled pool is actually reserved (524288, 13312.0 MiB)
at max_tok 32768 with no clamp and no WARNING: 38.5 budget vs 37.0 needed.

Recorded the cost honestly. The server now holds 103.5 of 121.2 GiB and the
engine reports only 310 contiguous 2 MiB blocks / 7.1 GiB left for everything
else (24-reserve baseline: 2081 blocks / 12.1 GiB). All 28 co-tenants stayed
up, but the engine volunteers a new note: when memory runs out, frank-large
AND its neighbours "can stall for minutes at a time at full CPU with no disk
activity and no output, which is not a crash and needs no restart". That is
the least diagnosable of the three failure modes seen today, and `free` hides
it — MemAvailable still reads ~74 GiB because the pinned weights count as
reclaimable page cache.

Sizing rule now expresses the budget as a function of reserve, shows both
endpoints (24 -> 29.5, 15 -> 38.5), and the max_tok ceiling is labelled as
reserve-conditional (<= 18,130 at 24; <= 35,700 at 15).
This commit is contained in:
Frank
2026-09-12 21:22:36 -07:00
parent 8a79f9a709
commit 2969e08616
3 changed files with 93 additions and 31 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/halogen-flash-server.md
+62 -15
View File
@@ -78,10 +78,11 @@ upstream's defaults:
|---|---|---|
| `HALOGEN_MODEL_ID` | `frank-large` | fleet identity; no client rename |
| `HALOGEN_CTX` | `262144` | the model's native context, and what both clients declare |
| `HALOGEN_KV_POOL_POSITIONS` | `524288` | requested; **the engine actually serves 262,144 right now.** 524,288 is kept as a "grow if the RAM frees up" declaration and `HALOGEN_KV_POOL_FIT` clamps it to the floor whenever the arena leaves no room. Read the ledger, not this column |
| `HALOGEN_KV_POOL_POSITIONS` | `524288` | **in force** — two full-length conversations resident, or four at 131k, 13,312 MiB of KV. Was silently clamped to 262,144 until the reserve below dropped; `HALOGEN_KV_POOL_FIT` still guards the boot, so read the ledger rather than trusting this column |
| `HALOGEN_KV_SLOTS` | `4` | four concurrent generations; slots share the pool and cost ~115 MiB each |
| `HALOGEN_MAX_TOK` | `32768` | upstream's default, restored 2026-09-12 by operator call in exchange for the doubling. The arena is 20.6 GiB here against 12.2 at 16384, and that is what pushes a 524,288 pool to 37.0 against a 29.5 budget and forces the clamp. **Set `16384` to make the doubled pool real**, at ~8–9 % of prefill throughput. At 65536 the arena alone exceeds the budget and the server **will not boot** |
| `HALOGEN_VISION_TOWER` | `1` | image input; the fleet's only image path since [[frank-npu]] was retired |
| `HALOGEN_MAX_TOK` | `32768` | upstream's default, kept. The arena is 20.6 GiB here against 12.2 at 16384; it only fits alongside the doubled pool because the reserve dropped to 15. At 65536 the arena alone exceeds the budget and the server **will not boot** |
| `HALOGEN_HOST_RESERVE_GIB` | `15` | **lowered from 24 (vendor default 20) on 2026-09-12** to buy both features. The one term you own: it sets the device budget (24 → 29.5 GiB, 15 → 38.5 GiB). Matched pair with the pool — raise it back to 24 and the pool silently clamps to 262,144 again. Cost is page cache for the n-gram table; see "What the reserve actually costs" |
| `HALOGEN_VISION_TOWER` | `1` | image input; the fleet's only image path since [[frank-npu]] was retired. Charges 1.97 GiB against the budget — the term missing from my first estimate |
| `HALOGEN_DOWNLOAD` | `peonist-ai/halogen-qwen3.8-flash-next` | first-start fetch |
| `HALOGEN_VERBOSE` | `1` | verbose engine logging — the fullest startup account, on permanently so an incident report needs no restart-into-debug |
@@ -116,15 +117,16 @@ Three numbers explain the whole box:
the pool itself if the configured one will not fit, printing what it chose.
Read those lines on every restart; they are the ground truth for sizing.
**Measured ledger, five restarts, read live 2026-09-12.** The engine's own
**Measured ledger, six restarts, read live 2026-09-12.** The engine's own
startup account, including two deliberate experiments:
| configured pool | max_tok | KV | arena | + vision | verdict |
|---|---|---|---|---|---|
| 262144 (09-11 baseline) | 32768 | 7.2 | 20.6 | **29.8** | boots — prints **no** sizing lines |
| 524288 | 16384 | 14.4 | 12.2 | **28.6** | **fits clean**, ~0.9 headroom |
| 524288 | 32768 | 14.4 | 20.6 | 37.0 | clamped to 262144, deploy still green |
| 524288 | 65536 | — | 37.3 | 46.5 | **OOM — will not boot** |
| pool | max_tok | reserve | KV | arena | need | budget | verdict |
|---|---|---|---|---|---|---|---|
| 262144 (baseline) | 32768 | 24 | 7.2 | 20.6 | 29.8 | 29.5 | boots 0.3 over — prints **no** sizing lines |
| 524288 | 16384 | 24 | 14.4 | 12.2 | 28.6 | 29.5 | fits, 0.9 spare |
| 524288 | 32768 | 24 | 14.4 | 20.6 | 37.0 | 29.5 | clamped to 262144, deploy still green |
| 524288 | 65536 | 24 | — | 37.3 | 46.5 | 29.5 | **OOM — will not boot** |
| 524288 | 32768 | **15** | 14.4 | 20.6 | 37.0 | **38.5** | **deployed — both features** |
### The sizing rule
@@ -132,12 +134,20 @@ One budget covers the pool, the arena **and** the vision tower together:
```
device budget = host RAM - resident weights - HALOGEN_HOST_RESERVE_GIB
= 121 - 67.7 - 24 = 29.5 GiB
= 121.2 - 67.7 - reserve
reserve 24 -> 29.5 GiB
reserve 15 -> 38.5 GiB
need = KV (28 KiB/position) + arena + vision (1.97 measured)
arena = 3.80 GiB + 8.40 GiB x (HALOGEN_MAX_TOK / 16384)
```
The `121` the engine prints is rounded **down** from `MemTotal 127104060 kB =
121.2159 GiB`, so subtracting the displayed figures gives 29.3 rather than 29.5.
Use the unrounded value. Two of the three terms are fixed hardware and
checkpoint facts; `HALOGEN_HOST_RESERVE_GIB` is the only one you own, which
makes it the lever that buys both features — see the cost below.
The arena term is **linear in `HALOGEN_MAX_TOK`** and predicts every measured
point to 0.1 GiB — including upstream's three published "on the device" figures
(524,288 → ~35, 786,432 → ~42, 1,048,576 at `max_tok=16384` → ~41), which are
@@ -146,10 +156,47 @@ their largest pool "needs `HALOGEN_MAX_TOK=16384`": at 32768 a 1M pool would be
49.4. Reading those figures as pool-only is the trap that makes a doubling look
affordable when it is not.
For the doubled pool: `29.5 - 1.97 vision - 14.4 KV = 13.1` for the arena, so
`max_tok <= 18,130` — **16384 is the ceiling, not a conservative setting.** The
corner check agrees: at 32768 only 7.0 GiB is left for the KV, about 254k
positions, so the pool drops back toward 262144.
At the baseline reserve of 24 the doubled pool leaves `29.5 - 1.97 - 14.4 =
13.1` for the arena, so `max_tok <= 18,130` and **16384 is that reserve's
ceiling, not a conservative setting.** The corner check agrees: at 32768 only
7.0 GiB is left for the KV, ~254k positions, and the pool drops back toward
262144. Cut the reserve to 15 and the same arithmetic gives `38.5 - 1.97 - 14.4
= 22.1` for the arena — `max_tok <= 35,700`, which is exactly why 32768 now
fits unmodified.
### What the reserve actually costs
The device budget is not the whole story. Freeing the doubled pool by cutting
the reserve moves the machine's memory position considerably:
| config | held | host left | contiguous 2 MiB blocks |
|---|---|---|---|
| 262144 / 32768, reserve 24 | 96.3 | 12.1 GiB | 2081 |
| 524288 / 16384, reserve 24 | 94.6 | 13.8 GiB | 3742 |
| **524288 / 32768, reserve 15** | **103.5** | **7.1 GiB** | **310** |
At the deployed setting the server holds 103.5 of 121 GiB and the engine now
volunteers a note it did not print before:
```
halogen: NOTE this server now holds most of the machine, and 310 contiguous
2 MiB blocks remain. It runs best on a host of its own. If other large
processes share this machine they will compete for what is left, and when it
runs out BOTH they and this server can stall for minutes at a time at full
CPU with no disk activity and no output, which is not a crash and needs no
restart.
```
That is the failure mode to watch, and it is the least diagnosable of the
three: not a silent clamp, not a boot refusal, but a minutes-long freeze that
looks like a hang, for the neighbours as much as for frank-large. `free` is
useless for spotting it — `MemAvailable` still reads ~74 GiB because the
page cache counts as reclaimable. Watch the `host memory left for everything
else` line and the contiguous-block count at boot instead.
28 co-tenant containers (~4.3 GiB working set) stayed up through the change,
so it is survivable. The mitigation if anything goes quiet with the CPU
pinned: raise the reserve, or drop `HALOGEN_KV_POOL_POSITIONS` back to 262144.
### The `WARNING` line only appears when the fit check had to move