dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(halogen): correct the memory model to the measured ledger

Browse Source 
The first draft of these notes estimated the doubled pool at 103.5 GiB
held / ~5 GiB host remainder. Both numbers were wrong — they costed the
KV alone and ignored that the arena shrinks when MAX_TOK drops. Measured:

  pool 262144 / max_tok 32768 -> 96.3 GiB held, 12.1 GiB host left
  pool 524288 / max_tok 16384 -> 94.6 GiB held, 13.8 GiB host left

The doubled pool is 1.7 GiB cheaper overall. Also records the failed
first attempt verbatim, since the silent-revert failure mode is the thing
a future operator most needs to recognise, and adds the HALOGEN_MAX_TOK
row to the deployed-flags table (the "baked 32768" line it overrode was
stale).
This commit is contained in:
Frank
2026-09-12 18:04:02 -07:00
parent c489b204ea
commit d1544f3191
1 changed files with 33 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/halogen-flash-server.md
+33 -19
View File
@@ -78,9 +78,9 @@ upstream's defaults:
|---|---|---|
| `HALOGEN_MODEL_ID` | `frank-large` | fleet identity; no client rename |
| `HALOGEN_CTX` | `262144` | the model's native context, and what both clients declare |
| `HALOGEN_KV_POOL_POSITIONS` | `524288` | **doubled 2026-09-12** to upstream's own default (2 × `HALOGEN_CTX`): two full-length conversations resident, or four at 131k, at ~14.4 GiB |
| `HALOGEN_KV_POOL_POSITIONS` | `524288` | **doubled 2026-09-12** to upstream's own default (2 × `HALOGEN_CTX`): two full-length conversations resident, or four at 131k — 13,312 MiB of KV |
| `HALOGEN_KV_SLOTS` | `4` | four concurrent generations; slots share the pool and cost ~115 MiB each |
| `HALOGEN_HOST_RESERVE_GIB` | `24` | RAM kept for the n-gram table the engine reads through the page cache — above upstream's 20 because a dozen containers share this host |
| `HALOGEN_MAX_TOK` | `16384` | **load-bearing for the doubling.** Halves the prefill arena so the pool fits the 29.5 GiB device budget, at ~8–9 % of prefill throughput. Leave it at upstream's 32768 and `HALOGEN_KV_POOL_FIT` silently reverts the pool to 262144 while the deploy still reports healthy |
| `HALOGEN_VISION_TOWER` | `1` | image input; the fleet's only image path since [[frank-npu]] was retired |
| `HALOGEN_DOWNLOAD` | `peonist-ai/halogen-qwen3.8-flash-next` | first-start fetch |
| `HALOGEN_VERBOSE` | `1` | verbose engine logging — the fullest startup account, on permanently so an incident report needs no restart-into-debug |
@@ -91,8 +91,7 @@ already covers the shared-prefix case), `HALOGEN_CACHE_INPLACE=1` (the snapshot
stays ~115 MiB at any depth), the speculative drafter `1` (byte-identical at
temperature 0, and it stands down whenever a second conversation is
generating), the quality sidecar, `HALOGEN_ADMIT_CHUNK=0` (a non-zero value
gives up the byte-identity property), the baked `HALOGEN_MAX_TOK=32768`
(16.7 GiB prefill arena), and the default `HALOGEN_MAX_TOKENS_CAP=65536`.
gives up the byte-identity property), and the default `HALOGEN_MAX_TOKENS_CAP=65536`.
## Memory model
@@ -117,22 +116,37 @@ Three numbers explain the whole box:
the pool itself if the configured one will not fit, printing what it chose.
Read those lines on every restart; they are the ground truth for sizing.
**Measured ledger (prior 262,144 config, read live 2026-09-12).** The
engine's own startup account: 68.0 GiB of weights locked + 7.2 GiB KV pool +
21.1 GiB working memory = **96.3 GiB held**, with **12.1 GiB left on the
host**, only ~7.3 GiB of that in contiguous 2 MiB pieces. The 47.7 GiB n-gram
lookup table is never held in RAM — it is read through the file cache — and
the 2.0 GiB firmware carves for the iGPU appears in no `/proc/meminfo`
figure.
**Measured ledger, both configs, read live 2026-09-12.** The engine's own
startup account:
Doubling to 524,288 puts the held total near **103.5 GiB** and the host
remainder at roughly **5 GiB**, against a measured ~4.3 GiB working set for
the 25 co-tenant containers on this box. The extra 7.2 GiB comes out of the
page cache, which falls from ~76 GiB to ~69 GiB — still above the 47.7 GiB
lookup table, so the table can stay hot. Tight, not an overcommit. If long
prompts start crawling with the disk busy, `HALOGEN_MAX_TOK=16384` shrinks
the prefill arena 16.7 → 8.4 GiB and pays for the doubling outright at about
8 % of prefill throughput.
| config | weights | KV pool | working mem | held | host left |
|---|---|---|---|---|---|
| pool 262144 / max_tok 32768 (prior) | 68.0 | 7.2 | 21.1 | **96.3** | 12.1 |
| pool 524288 / max_tok 16384 (now) | 68.0 | 14.4 | 12.3 | **94.6** | 13.8 |
The doubled pool is **1.7 GiB cheaper in total** than the config it replaced.
Halving the prefill arena gives back more than the extra pool takes: the
arena is ~20.6 GiB regardless of context, while the KV costs only ~28 KiB a
position.
The first attempt — 524288 with the arena left at 32768 — did not fit, and
the way it failed is the trap on this host:
```
kv pool: host RAM 121 GiB, less 67.7 resident weights and 24.0 reserved
for the n-gram page cache and the OS = 29.5 GiB for the device
kv pool: 524288 positions need ~35.0 GiB (plus 1.5 of margin) and host
RAM cannot spare it. LOWERING THE POOL TO 262144 (~27.8 GiB).
```
One budget covers the pool **and** the arena together. `HALOGEN_KV_POOL_FIT`
lowered the pool, and `deploy.sh` still reported healthy in 12 seconds. A
green health check does not mean the pool you asked for is the pool you got —
always read the `kv pool:` lines after a restart.
The 47.7 GiB n-gram lookup table is never held in RAM; it is read through
the file cache. The 2.0 GiB the firmware carves for the iGPU appears in no
`/proc/meminfo` figure at all.
**Co-tenancy.** `HALOGEN_FLASH_PIN_TRUNK=1` (default) is what the published
decode numbers were measured with, and it means this container holds most of