diff --git a/AGENTS.md b/AGENTS.md index b776f09..18dfbe9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -32,7 +32,7 @@ host. | Target | Runtime | Service | Model | Port | |--------|---------|---------|-------|------| | Dankserver | FreeToken v0.1.2 | frank-small | batsclamp Huihui-Qwen3.6-35B-A3B-abliterated-FP8 (block-fp8 e4m3 128x128, ~35 GB safetensors), experts host-RAM LRU, 262144 ctx / 4 concurrent, reasoning on | 7770 | -| Dankclaw | halogen-flash-server 0.5.8 (rootless Quadlet container) | frank-large | peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b .hgn 115.55 GiB + quality overlay, ~68 GiB pinned, 262144 ctx / 4 slots, pool 524288, vision tower on, greedy for silent clients) | 7770 | +| Dankclaw|halogen-flash-server 0.5.8 (rootless Quadlet container)|frank-large|peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b .hgn 115.55 GiB + quality overlay, ~68 GiB pinned, 262144 ctx / 4 slots, pool 524288 requested / 262144 effective, vision tower on, greedy for silent clients)|7770| | Dankclaw | *(retired 2026-09-11)* FastFlowLM v1.0.1 on the XDNA2 NPU | frank-npu | Gemma4-E4B-IT-NPU2 pack (vision + audio). Retired with the `amd_iommu=off` decision: the driver's SVA/PASID bind needs an enabled IOMMU | 7771 | History: the 2026-08-17 redo retired the pipeline-parallel split and the @@ -396,8 +396,10 @@ published) and the OpenAI front-end on :8731 published as host **7770** — with `HALOGEN_MODEL_ID=frank-large`, so no client wiring changed. Deployed flags: `HALOGEN_CTX=262144`, `HALOGEN_KV_POOL_POSITIONS=524288` -(doubled 2026-09-12 to upstream's default — two full-length conversations -resident instead of one, ~14.4 GiB), `HALOGEN_KV_SLOTS=4`, +(requested 2026-09-12, but **the engine serves 262144** — with +`HALOGEN_MAX_TOK=32768` the prefill arena eats the device budget and +`HALOGEN_KV_POOL_FIT` clamps the pool to the floor at every boot; set +`HALOGEN_MAX_TOK=16384` to make the doubled pool real), `HALOGEN_KV_SLOTS=4`, `HALOGEN_HOST_RESERVE_GIB=24`, `HALOGEN_VISION_TOWER=1`, and `HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next` for the first-start fetch; everything else is upstream's default. The weights stay pinned diff --git a/dankclaw/frank-large/quadlet/frank-large.container b/dankclaw/frank-large/quadlet/frank-large.container index 3a4f3f7..e545b84 100644 --- a/dankclaw/frank-large/quadlet/frank-large.container +++ b/dankclaw/frank-large/quadlet/frank-large.container @@ -97,39 +97,39 @@ Environment=HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next Environment=HALOGEN_MODEL_ID=frank-large # Native context of the model. Matches what both clients declare. Environment=HALOGEN_CTX=262144 -# Doubled 2026-09-12 from 262144 to upstream's own default (2 x HALOGEN_CTX): -# two full-length conversations resident at once, or four at 131k, instead of -# one. The old pool was the binding constraint on concurrency, not CTX — a -# request reserves prompt + max_tokens against the pool, so four 65k chats -# used to contend for one 262k pool. +# Requested 2026-09-12 as double the 262144 this had run since 09-11, to get +# two full-length conversations resident instead of one. NOTE THE ASYMMETRY +# BELOW: with HALOGEN_MAX_TOK at 32768 (current) the fit check clamps this to +# 262144 at every boot, so the EFFECTIVE pool is 262144 and only one +# full-length conversation stays resident. It is left at 524288 deliberately: +# it is a "grow if the RAM frees up" declaration, and HALOGEN_KV_POOL_FIT +# raises nothing but honours it the moment the arena stops eating the budget. +# Set HALOGEN_MAX_TOK=16384 to actually get the doubled pool today. Environment=HALOGEN_KV_POOL_POSITIONS=524288 Environment=HALOGEN_KV_SLOTS=4 -# LOAD-BEARING. This line is what makes the doubling above real; do not -# "clean it up". The device budget here is 121 GiB - 67.7 resident weights -# - 24.0 reserved for the n-gram page cache and the OS = 29.5 GiB, and that -# one budget carries BOTH the KV pool and the prefill arena. +# The device budget is ONE budget, covering the KV pool, the prefill arena and +# the vision tower: # -# Measured across three restarts on 2026-09-12. The arena (the engine's -# "everything but the pool" figure) is linear in this flag: 3.80 GiB of base -# plus 8.40 GiB per 16384 tokens. -# 16384 -> arena 12.2, 524288 needs 26.6 -> FITS -# 32768 -> arena 20.6, 524288 needs 35.0 -> fit lowers the pool to 262144 -# 65536 -> arena 37.3, even 262144 needs 44.5 -> OOM, dmalloc FAILED at 39.97 -# So 16384 is not conservative, it is the ceiling: 29.5 budget - 1.5 margin - -# 14.4 pool leaves 13.6 for the arena, i.e. max_tok <= 19,115, and 16384 is -# the largest power of two under that. The reverse checks too — put max_tok -# back to 32768 and only 7.4 GiB is left for the pool, about 269k positions, -# which is exactly the 262144 config this replaced. Those two corners are the -# only points of this box that have ever run here. +# budget = 121 host - 67.7 resident weights - 24.0 reserve = 29.5 GiB +# need = KV(~28 KiB/position) + arena + vision(1.97 measured) +# arena = 3.80 GiB + 8.40 GiB x (HALOGEN_MAX_TOK / 16384) # -# Do NOT raise this flag past 16384 while the pool stays doubled: 65536 does -# not shrink the pool, it stops the server booting at all — and the unit still -# reads active with the container "Up (starting)", so the health check waits -# out LLM_WAIT rather than failing fast. - -# EXPERIMENT 2026-09-12 (operator call): back to upstream's default 32768. -# Model predicts 524288 needs 35.0 against a 29.5 budget, so HALOGEN_KV_POOL_FIT -# should clamp the pool to 262144 (~27.8) at boot. Watch the ledger. +# That arena term is linear in max_tok and was checked against five restarts +# and against upstream's three published "on the device" figures, all to +# 0.1 GiB. Measured on this host: +# max_tok 16384 -> arena 12.2, pool 524288 total 28.6 -> FITS, 0.9 spare +# max_tok 32768 -> arena 20.6, pool 524288 total 37.0 -> clamps to 262144 +# max_tok 32768 -> pool floor 262144 total 29.8 -> 0.3 OVER budget, boots +# max_tok 65536 -> arena 37.3, even the floor needs 46.5 -> OOM, no boot +# So 16384 is the ceiling for a doubled pool, not a conservative setting. The +# 0.3 over-budget at the 32768/262144 corner is where this box has been ever +# since it was deployed on 09-11; the 24 GiB reserve is generous, so it boots +# and serves with 12-14 GiB genuinely left. It is a soft floor, not a wall. +# +# Do NOT raise max_tok past 16384 while asking for the doubled pool: 65536 did +# not shrink the pool, it stopped the server booting — and the unit still reads +# active with the container "Up (starting)", so the health check waits out +# LLM_WAIT rather than failing fast. Environment=HALOGEN_MAX_TOK=32768 # RAM left for the n-gram table the engine reads through the page cache. # Above the vendor's 20 because a dozen containers share this host; too low diff --git a/serverAppConfigs-docs/LLM/halogen-flash-server.md b/serverAppConfigs-docs/LLM/halogen-flash-server.md index ee070c3..f8ec23c 100644 --- a/serverAppConfigs-docs/LLM/halogen-flash-server.md +++ b/serverAppConfigs-docs/LLM/halogen-flash-server.md @@ -78,9 +78,9 @@ upstream's defaults: |---|---|---| | `HALOGEN_MODEL_ID` | `frank-large` | fleet identity; no client rename | | `HALOGEN_CTX` | `262144` | the model's native context, and what both clients declare | -| `HALOGEN_KV_POOL_POSITIONS` | `524288` | **doubled 2026-09-12** to upstream's own default (2 × `HALOGEN_CTX`): two full-length conversations resident, or four at 131k — 13,312 MiB of KV | +| `HALOGEN_KV_POOL_POSITIONS` | `524288` | requested; **the engine actually serves 262,144 right now.** 524,288 is kept as a "grow if the RAM frees up" declaration and `HALOGEN_KV_POOL_FIT` clamps it to the floor whenever the arena leaves no room. Read the ledger, not this column | | `HALOGEN_KV_SLOTS` | `4` | four concurrent generations; slots share the pool and cost ~115 MiB each | -| `HALOGEN_MAX_TOK` | `16384` | **load-bearing for the doubling, and already at the ceiling.** Halves the prefill arena so the pool fits the 29.5 GiB device budget, at ~8–9 % of prefill throughput. At 32768 `HALOGEN_KV_POOL_FIT` silently reverts the pool to 262144; at 65536 the arena alone exceeds the budget and **the server will not boot**. See the sizing rule below | +| `HALOGEN_MAX_TOK` | `32768` | upstream's default, restored 2026-09-12 by operator call in exchange for the doubling. The arena is 20.6 GiB here against 12.2 at 16384, and that is what pushes a 524,288 pool to 37.0 against a 29.5 budget and forces the clamp. **Set `16384` to make the doubled pool real**, at ~8–9 % of prefill throughput. At 65536 the arena alone exceeds the budget and the server **will not boot** | | `HALOGEN_VISION_TOWER` | `1` | image input; the fleet's only image path since [[frank-npu]] was retired | | `HALOGEN_DOWNLOAD` | `peonist-ai/halogen-qwen3.8-flash-next` | first-start fetch | | `HALOGEN_VERBOSE` | `1` | verbose engine logging — the fullest startup account, on permanently so an incident report needs no restart-into-debug | @@ -116,44 +116,55 @@ Three numbers explain the whole box: the pool itself if the configured one will not fit, printing what it chose. Read those lines on every restart; they are the ground truth for sizing. -**Measured ledger, four restarts, read live 2026-09-12.** The engine's own -startup account: +**Measured ledger, five restarts, read live 2026-09-12.** The engine's own +startup account, including two deliberate experiments: -| pool | max_tok | KV pool | arena | held | host left | verdict | -|---|---|---|---|---|---|---| -| 262144 | 32768 | 7.2 | 20.6 | 96.3 | 12.1 | the prior config | -| 524288 | 16384 | 14.4 | 12.2 | **94.6** | 13.8 | **deployed** | -| 524288 | 32768 | 14.4 | 20.6 | — | — | pool silently lowered to 262144 | -| 524288 | 65536 | — | 37.3 | — | — | **would not boot — OOM** | +| configured pool | max_tok | KV | arena | + vision | verdict | +|---|---|---|---|---|---| +| 262144 (09-11 baseline) | 32768 | 7.2 | 20.6 | **29.8** | boots — prints **no** sizing lines | +| 524288 | 16384 | 14.4 | 12.2 | **28.6** | **fits clean**, ~0.9 headroom | +| 524288 | 32768 | 14.4 | 20.6 | 37.0 | clamped to 262144, deploy still green | +| 524288 | 65536 | — | 37.3 | 46.5 | **OOM — will not boot** | ### The sizing rule -One budget covers the pool **and** the arena together: +One budget covers the pool, the arena **and** the vision tower together: ``` device budget = host RAM - resident weights - HALOGEN_HOST_RESERVE_GIB = 121 - 67.7 - 24 = 29.5 GiB + +need = KV (28 KiB/position) + arena + vision (1.97 measured) +arena = 3.80 GiB + 8.40 GiB x (HALOGEN_MAX_TOK / 16384) ``` -The arena is **linear in `HALOGEN_MAX_TOK`**: measured at 3.80 GiB of base plus -8.40 GiB per 16,384 tokens, which predicts all three measured arenas to within -0.1 GiB. The KV costs ~28 KiB a position, so 524,288 of them cost 14.4 GiB. -Solving for the arena with the doubled pool in place: +The arena term is **linear in `HALOGEN_MAX_TOK`** and predicts every measured +point to 0.1 GiB — including upstream's three published "on the device" figures +(524,288 → ~35, 786,432 → ~42, 1,048,576 at `max_tok=16384` → ~41), which are +**pool + arena totals at the default `max_tok`**, not pool alone. That is why +their largest pool "needs `HALOGEN_MAX_TOK=16384`": at 32768 a 1M pool would be +49.4. Reading those figures as pool-only is the trap that makes a doubling look +affordable when it is not. -``` -29.5 budget - 1.5 margin - 14.4 pool = 13.6 GiB left for the arena - -> HALOGEN_MAX_TOK <= 19,115 -``` +For the doubled pool: `29.5 - 1.97 vision - 14.4 KV = 13.1` for the arena, so +`max_tok <= 18,130` — **16384 is the ceiling, not a conservative setting.** The +corner check agrees: at 32768 only 7.0 GiB is left for the KV, about 254k +positions, so the pool drops back toward 262144. -16384 is the largest power of two under that, so the deployed setting is not -conservative — **it is the ceiling**. The corner check confirms it: put -`max_tok` back to 32768 and only 7.4 GiB remains for the pool, about 269k -positions, which is precisely the 262,144 config that ran here before. Those -two corners are the only points of this box the host has ever served. +### The `WARNING` line only appears when the fit check had to move -Note the consequence for the doubling: it is **1.7 GiB cheaper overall** than -the config it replaced, because halving the arena gives back more than the -extra pool takes. +This one cost me a wrong conclusion, so it is worth stating plainly. When the +configured pool already fits, the engine reserves it **and prints nothing** — +the 09-11 baseline boot goes straight from the pre-flight line to +`checkpoint: mapped` with no `kv pool:` sizing block at all. The sizing block and +its `WARNING even 262144 positions ... host RAM cannot spare it` appear **only +after a clamp**. + +So the absence of a warning in old logs is not evidence of headroom. Both +`262144/32768` rows above need 29.8 against a 29.5 budget — 0.27 over — and the +baseline has lived in that state since it was deployed without incident, +because the 24 GiB reserve is itself generous (the host reports 12–14 GiB free +at that point). The engine's budget is a conservative floor, not a hard wall. ### Two failure modes, and only one is dangerous