diff --git a/serverAppConfigs-docs/LLM/halogen-flash-server.md b/serverAppConfigs-docs/LLM/halogen-flash-server.md index 8c554d2..bd0243e 100644 --- a/serverAppConfigs-docs/LLM/halogen-flash-server.md +++ b/serverAppConfigs-docs/LLM/halogen-flash-server.md @@ -78,9 +78,9 @@ upstream's defaults: |---|---|---| | `HALOGEN_MODEL_ID` | `frank-large` | fleet identity; no client rename | | `HALOGEN_CTX` | `262144` | the model's native context, and what both clients declare | -| `HALOGEN_KV_POOL_POSITIONS` | `524288` | **doubled 2026-09-12** to upstream's own default (2 × `HALOGEN_CTX`): two full-length conversations resident, or four at 131k, at ~14.4 GiB | +| `HALOGEN_KV_POOL_POSITIONS` | `524288` | **doubled 2026-09-12** to upstream's own default (2 × `HALOGEN_CTX`): two full-length conversations resident, or four at 131k — 13,312 MiB of KV | | `HALOGEN_KV_SLOTS` | `4` | four concurrent generations; slots share the pool and cost ~115 MiB each | -| `HALOGEN_HOST_RESERVE_GIB` | `24` | RAM kept for the n-gram table the engine reads through the page cache — above upstream's 20 because a dozen containers share this host | +| `HALOGEN_MAX_TOK` | `16384` | **load-bearing for the doubling.** Halves the prefill arena so the pool fits the 29.5 GiB device budget, at ~8–9 % of prefill throughput. Leave it at upstream's 32768 and `HALOGEN_KV_POOL_FIT` silently reverts the pool to 262144 while the deploy still reports healthy | | `HALOGEN_VISION_TOWER` | `1` | image input; the fleet's only image path since [[frank-npu]] was retired | | `HALOGEN_DOWNLOAD` | `peonist-ai/halogen-qwen3.8-flash-next` | first-start fetch | | `HALOGEN_VERBOSE` | `1` | verbose engine logging — the fullest startup account, on permanently so an incident report needs no restart-into-debug | @@ -91,8 +91,7 @@ already covers the shared-prefix case), `HALOGEN_CACHE_INPLACE=1` (the snapshot stays ~115 MiB at any depth), the speculative drafter `1` (byte-identical at temperature 0, and it stands down whenever a second conversation is generating), the quality sidecar, `HALOGEN_ADMIT_CHUNK=0` (a non-zero value -gives up the byte-identity property), the baked `HALOGEN_MAX_TOK=32768` -(16.7 GiB prefill arena), and the default `HALOGEN_MAX_TOKENS_CAP=65536`. +gives up the byte-identity property), and the default `HALOGEN_MAX_TOKENS_CAP=65536`. ## Memory model @@ -117,22 +116,37 @@ Three numbers explain the whole box: the pool itself if the configured one will not fit, printing what it chose. Read those lines on every restart; they are the ground truth for sizing. -**Measured ledger (prior 262,144 config, read live 2026-09-12).** The -engine's own startup account: 68.0 GiB of weights locked + 7.2 GiB KV pool + -21.1 GiB working memory = **96.3 GiB held**, with **12.1 GiB left on the -host**, only ~7.3 GiB of that in contiguous 2 MiB pieces. The 47.7 GiB n-gram -lookup table is never held in RAM — it is read through the file cache — and -the 2.0 GiB firmware carves for the iGPU appears in no `/proc/meminfo` -figure. +**Measured ledger, both configs, read live 2026-09-12.** The engine's own +startup account: -Doubling to 524,288 puts the held total near **103.5 GiB** and the host -remainder at roughly **5 GiB**, against a measured ~4.3 GiB working set for -the 25 co-tenant containers on this box. The extra 7.2 GiB comes out of the -page cache, which falls from ~76 GiB to ~69 GiB — still above the 47.7 GiB -lookup table, so the table can stay hot. Tight, not an overcommit. If long -prompts start crawling with the disk busy, `HALOGEN_MAX_TOK=16384` shrinks -the prefill arena 16.7 → 8.4 GiB and pays for the doubling outright at about -8 % of prefill throughput. +| config | weights | KV pool | working mem | held | host left | +|---|---|---|---|---|---| +| pool 262144 / max_tok 32768 (prior) | 68.0 | 7.2 | 21.1 | **96.3** | 12.1 | +| pool 524288 / max_tok 16384 (now) | 68.0 | 14.4 | 12.3 | **94.6** | 13.8 | + +The doubled pool is **1.7 GiB cheaper in total** than the config it replaced. +Halving the prefill arena gives back more than the extra pool takes: the +arena is ~20.6 GiB regardless of context, while the KV costs only ~28 KiB a +position. + +The first attempt — 524288 with the arena left at 32768 — did not fit, and +the way it failed is the trap on this host: + +``` +kv pool: host RAM 121 GiB, less 67.7 resident weights and 24.0 reserved + for the n-gram page cache and the OS = 29.5 GiB for the device +kv pool: 524288 positions need ~35.0 GiB (plus 1.5 of margin) and host + RAM cannot spare it. LOWERING THE POOL TO 262144 (~27.8 GiB). +``` + +One budget covers the pool **and** the arena together. `HALOGEN_KV_POOL_FIT` +lowered the pool, and `deploy.sh` still reported healthy in 12 seconds. A +green health check does not mean the pool you asked for is the pool you got — +always read the `kv pool:` lines after a restart. + +The 47.7 GiB n-gram lookup table is never held in RAM; it is read through +the file cache. The 2.0 GiB the firmware carves for the iGPU appears in no +`/proc/meminfo` figure at all. **Co-tenancy.** `HALOGEN_FLASH_PIN_TRUNK=1` (default) is what the published decode numbers were measured with, and it means this container holds most of