dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-large: record the 32k outcome — the doubling is clamped away, docs resynced

Browse Source 
Deployed HALOGEN_MAX_TOK=32768 by operator call (upstream default, full prefill
speed) and confirmed the prediction from the ledger:

  kv pool: 524288 positions need ~35.0 GiB ... LOWERING THE POOL TO 262144 (~27.8 GiB)
  memory: 68.0 weights + 7.2 KV pool + 21.1 working = 96.3 GiB in all

so the effective pool is 262144 again and the doubling this task set out to do
is not in force. Healthy, completion verified. Left the pool requesting 524288
deliberately: with KV_POOL_FIT on that is a grow-if-RAM-frees-up declaration,
not a mistake.

Two corrections to what I wrote earlier today:

1. The engine's sizing block, and its "WARNING even 262144 ... host RAM cannot
   spare it" line, print ONLY when the fit check had to clamp something. The
   09-11 baseline boot prints no kv pool: sizing lines at all. So the absence
   of a warning in old logs was never evidence of headroom.

2. The budget needs a vision term. Actual need is pool + arena + 1.97 GiB of
   vision tower, not pool + a 1.5 GiB nominal margin. That makes the ceiling
   for a doubled pool max_tok <= 18,130 rather than 19,115, and it means the
   32768/262144 corner this box has run in since 09-11 is ~0.3 GiB OVER its own
   stated budget -- and has been the whole time. It boots and serves with
   12-14 GiB genuinely free because the 24 GiB reserve is generous. A soft
   floor, not a wall.

The arena model also reproduces upstream's three published "on the device"
figures exactly (35 / 42 / 41 GiB), confirming they are pool+arena totals at
the default max_tok and not pool-only -- which is the misreading that made the
doubling look affordable in my first estimate.
This commit is contained in:
Frank
2026-09-12 20:12:03 -07:00
parent 75ef47c342
commit 65d1be06ac
3 changed files with 72 additions and 59 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/halogen-flash-server.md
+38 -27
View File
@@ -78,9 +78,9 @@ upstream's defaults:
|---|---|---|
| `HALOGEN_MODEL_ID` | `frank-large` | fleet identity; no client rename |
| `HALOGEN_CTX` | `262144` | the model's native context, and what both clients declare |
| `HALOGEN_KV_POOL_POSITIONS` | `524288` | **doubled 2026-09-12** to upstream's own default (2 × `HALOGEN_CTX`): two full-length conversations resident, or four at 131k — 13,312 MiB of KV |
| `HALOGEN_KV_POOL_POSITIONS` | `524288` | requested; **the engine actually serves 262,144 right now.** 524,288 is kept as a "grow if the RAM frees up" declaration and `HALOGEN_KV_POOL_FIT` clamps it to the floor whenever the arena leaves no room. Read the ledger, not this column |
| `HALOGEN_KV_SLOTS` | `4` | four concurrent generations; slots share the pool and cost ~115 MiB each |
| `HALOGEN_MAX_TOK` | `16384` | **load-bearing for the doubling, and already at the ceiling.** Halves the prefill arena so the pool fits the 29.5 GiB device budget, at ~8–9 % of prefill throughput. At 32768 `HALOGEN_KV_POOL_FIT` silently reverts the pool to 262144; at 65536 the arena alone exceeds the budget and **the server will not boot**. See the sizing rule below |
| `HALOGEN_MAX_TOK` | `32768` | upstream's default, restored 2026-09-12 by operator call in exchange for the doubling. The arena is 20.6 GiB here against 12.2 at 16384, and that is what pushes a 524,288 pool to 37.0 against a 29.5 budget and forces the clamp. **Set `16384` to make the doubled pool real**, at ~8–9 % of prefill throughput. At 65536 the arena alone exceeds the budget and the server **will not boot** |
| `HALOGEN_VISION_TOWER` | `1` | image input; the fleet's only image path since [[frank-npu]] was retired |
| `HALOGEN_DOWNLOAD` | `peonist-ai/halogen-qwen3.8-flash-next` | first-start fetch |
| `HALOGEN_VERBOSE` | `1` | verbose engine logging — the fullest startup account, on permanently so an incident report needs no restart-into-debug |
@@ -116,44 +116,55 @@ Three numbers explain the whole box:
the pool itself if the configured one will not fit, printing what it chose.
Read those lines on every restart; they are the ground truth for sizing.
**Measured ledger, four restarts, read live 2026-09-12.** The engine's own
startup account:
**Measured ledger, five restarts, read live 2026-09-12.** The engine's own
startup account, including two deliberate experiments:
| pool | max_tok | KV pool | arena | held | host left | verdict |
|---|---|---|---|---|---|---|
| 262144 | 32768 | 7.2 | 20.6 | 96.3 | 12.1 | the prior config |
| 524288 | 16384 | 14.4 | 12.2 | **94.6** | 13.8 | **deployed** |
| 524288 | 32768 | 14.4 | 20.6 | — | — | pool silently lowered to 262144 |
| 524288 | 65536 | — | 37.3 | — | — | **would not boot — OOM** |
| configured pool | max_tok | KV | arena | + vision | verdict |
|---|---|---|---|---|---|
| 262144 (09-11 baseline) | 32768 | 7.2 | 20.6 | **29.8** | boots — prints **no** sizing lines |
| 524288 | 16384 | 14.4 | 12.2 | **28.6** | **fits clean**, ~0.9 headroom |
| 524288 | 32768 | 14.4 | 20.6 | 37.0 | clamped to 262144, deploy still green |
| 524288 | 65536 | — | 37.3 | 46.5 | **OOM — will not boot** |
### The sizing rule
One budget covers the pool **and** the arena together:
One budget covers the pool, the arena **and** the vision tower together:
```
device budget = host RAM - resident weights - HALOGEN_HOST_RESERVE_GIB
= 121 - 67.7 - 24 = 29.5 GiB
need = KV (28 KiB/position) + arena + vision (1.97 measured)
arena = 3.80 GiB + 8.40 GiB x (HALOGEN_MAX_TOK / 16384)
```
The arena is **linear in `HALOGEN_MAX_TOK`**: measured at 3.80 GiB of base plus
8.40 GiB per 16,384 tokens, which predicts all three measured arenas to within
0.1 GiB. The KV costs ~28 KiB a position, so 524,288 of them cost 14.4 GiB.
Solving for the arena with the doubled pool in place:
The arena term is **linear in `HALOGEN_MAX_TOK`** and predicts every measured
point to 0.1 GiB — including upstream's three published "on the device" figures
(524,288 → ~35, 786,432 → ~42, 1,048,576 at `max_tok=16384` → ~41), which are
**pool + arena totals at the default `max_tok`**, not pool alone. That is why
their largest pool "needs `HALOGEN_MAX_TOK=16384`": at 32768 a 1M pool would be
49.4. Reading those figures as pool-only is the trap that makes a doubling look
affordable when it is not.
```
29.5 budget - 1.5 margin - 14.4 pool = 13.6 GiB left for the arena
-> HALOGEN_MAX_TOK <= 19,115
```
For the doubled pool: `29.5 - 1.97 vision - 14.4 KV = 13.1` for the arena, so
`max_tok <= 18,130` — **16384 is the ceiling, not a conservative setting.** The
corner check agrees: at 32768 only 7.0 GiB is left for the KV, about 254k
positions, so the pool drops back toward 262144.
16384 is the largest power of two under that, so the deployed setting is not
conservative — **it is the ceiling**. The corner check confirms it: put
`max_tok` back to 32768 and only 7.4 GiB remains for the pool, about 269k
positions, which is precisely the 262,144 config that ran here before. Those
two corners are the only points of this box the host has ever served.
### The `WARNING` line only appears when the fit check had to move
Note the consequence for the doubling: it is **1.7 GiB cheaper overall** than
the config it replaced, because halving the arena gives back more than the
extra pool takes.
This one cost me a wrong conclusion, so it is worth stating plainly. When the
configured pool already fits, the engine reserves it **and prints nothing** —
the 09-11 baseline boot goes straight from the pre-flight line to
`checkpoint: mapped` with no `kv pool:` sizing block at all. The sizing block and
its `WARNING even 262144 positions ... host RAM cannot spare it` appear **only
after a clamp**.
So the absence of a warning in old logs is not evidence of headroom. Both
`262144/32768` rows above need 29.8 against a 29.5 budget — 0.27 over — and the
baseline has lived in that state since it was deployed without incident,
because the 24 GiB reserve is itself generous (the host reports 12–14 GiB free
at that point). The engine's budget is a conservative floor, not a hard wall.
### Two failure modes, and only one is dangerous