dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-large: record the 32k outcome — the doubling is clamped away, docs resynced

Browse Source 
Deployed HALOGEN_MAX_TOK=32768 by operator call (upstream default, full prefill
speed) and confirmed the prediction from the ledger:

  kv pool: 524288 positions need ~35.0 GiB ... LOWERING THE POOL TO 262144 (~27.8 GiB)
  memory: 68.0 weights + 7.2 KV pool + 21.1 working = 96.3 GiB in all

so the effective pool is 262144 again and the doubling this task set out to do
is not in force. Healthy, completion verified. Left the pool requesting 524288
deliberately: with KV_POOL_FIT on that is a grow-if-RAM-frees-up declaration,
not a mistake.

Two corrections to what I wrote earlier today:

1. The engine's sizing block, and its "WARNING even 262144 ... host RAM cannot
   spare it" line, print ONLY when the fit check had to clamp something. The
   09-11 baseline boot prints no kv pool: sizing lines at all. So the absence
   of a warning in old logs was never evidence of headroom.

2. The budget needs a vision term. Actual need is pool + arena + 1.97 GiB of
   vision tower, not pool + a 1.5 GiB nominal margin. That makes the ceiling
   for a doubled pool max_tok <= 18,130 rather than 19,115, and it means the
   32768/262144 corner this box has run in since 09-11 is ~0.3 GiB OVER its own
   stated budget -- and has been the whole time. It boots and serves with
   12-14 GiB genuinely free because the 24 GiB reserve is generous. A soft
   floor, not a wall.

The arena model also reproduces upstream's three published "on the device"
figures exactly (35 / 42 / 41 GiB), confirming they are pool+arena totals at
the default max_tok and not pool-only -- which is the misreading that made the
doubling look affordable in my first estimate.
This commit is contained in:
Frank
2026-09-12 20:12:03 -07:00
parent 75ef47c342
commit 65d1be06ac
3 changed files with 72 additions and 59 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+5 -3
View File
@@ -32,7 +32,7 @@ host.
| Target | Runtime | Service | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | FreeToken v0.1.2 | frank-small | batsclamp Huihui-Qwen3.6-35B-A3B-abliterated-FP8 (block-fp8 e4m3 128x128, ~35 GB safetensors), experts host-RAM LRU, 262144 ctx / 4 concurrent, reasoning on | 7770 |
| Dankclaw | halogen-flash-server 0.5.8 (rootless Quadlet container) | frank-large | peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b .hgn 115.55 GiB + quality overlay, ~68 GiB pinned, 262144 ctx / 4 slots, pool 524288, vision tower on, greedy for silent clients) | 7770 |
| Dankclaw|halogen-flash-server 0.5.8 (rootless Quadlet container)|frank-large|peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b .hgn 115.55 GiB + quality overlay, ~68 GiB pinned, 262144 ctx / 4 slots, pool 524288 requested / 262144 effective, vision tower on, greedy for silent clients)|7770|
| Dankclaw | *(retired 2026-09-11)* FastFlowLM v1.0.1 on the XDNA2 NPU | frank-npu | Gemma4-E4B-IT-NPU2 pack (vision + audio). Retired with the `amd_iommu=off` decision: the driver's SVA/PASID bind needs an enabled IOMMU | 7771 |
History: the 2026-08-17 redo retired the pipeline-parallel split and the
@@ -396,8 +396,10 @@ published) and the OpenAI front-end on :8731 published as host **7770** —
with `HALOGEN_MODEL_ID=frank-large`, so no client wiring changed.
Deployed flags: `HALOGEN_CTX=262144`, `HALOGEN_KV_POOL_POSITIONS=524288`
(doubled 2026-09-12 to upstream's default — two full-length conversations
resident instead of one, ~14.4 GiB), `HALOGEN_KV_SLOTS=4`,
(requested 2026-09-12, but **the engine serves 262144** — with
`HALOGEN_MAX_TOK=32768` the prefill arena eats the device budget and
`HALOGEN_KV_POOL_FIT` clamps the pool to the floor at every boot; set
`HALOGEN_MAX_TOK=16384` to make the doubled pool real), `HALOGEN_KV_SLOTS=4`,
`HALOGEN_HOST_RESERVE_GIB=24`, `HALOGEN_VISION_TOWER=1`, and
`HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next` for the first-start
fetch; everything else is upstream's default. The weights stay pinned
dankclaw/frank-large/quadlet/frank-large.container
+29 -29
View File
@@ -97,39 +97,39 @@ Environment=HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next
Environment=HALOGEN_MODEL_ID=frank-large
# Native context of the model. Matches what both clients declare.
Environment=HALOGEN_CTX=262144
# Doubled 2026-09-12 from 262144 to upstream's own default (2 x HALOGEN_CTX):
# two full-length conversations resident at once, or four at 131k, instead of
# one. The old pool was the binding constraint on concurrency, not CTX — a
# request reserves prompt + max_tokens against the pool, so four 65k chats
# used to contend for one 262k pool.
# Requested 2026-09-12 as double the 262144 this had run since 09-11, to get
# two full-length conversations resident instead of one. NOTE THE ASYMMETRY
# BELOW: with HALOGEN_MAX_TOK at 32768 (current) the fit check clamps this to
# 262144 at every boot, so the EFFECTIVE pool is 262144 and only one
# full-length conversation stays resident. It is left at 524288 deliberately:
# it is a "grow if the RAM frees up" declaration, and HALOGEN_KV_POOL_FIT
# raises nothing but honours it the moment the arena stops eating the budget.
# Set HALOGEN_MAX_TOK=16384 to actually get the doubled pool today.
Environment=HALOGEN_KV_POOL_POSITIONS=524288
Environment=HALOGEN_KV_SLOTS=4
# LOAD-BEARING. This line is what makes the doubling above real; do not
# "clean it up". The device budget here is 121 GiB - 67.7 resident weights
# - 24.0 reserved for the n-gram page cache and the OS = 29.5 GiB, and that
# one budget carries BOTH the KV pool and the prefill arena.
# The device budget is ONE budget, covering the KV pool, the prefill arena and
# the vision tower:
#
# Measured across three restarts on 2026-09-12. The arena (the engine's
# "everything but the pool" figure) is linear in this flag: 3.80 GiB of base
# plus 8.40 GiB per 16384 tokens.
# 16384 -> arena 12.2, 524288 needs 26.6 -> FITS
# 32768 -> arena 20.6, 524288 needs 35.0 -> fit lowers the pool to 262144
# 65536 -> arena 37.3, even 262144 needs 44.5 -> OOM, dmalloc FAILED at 39.97
# So 16384 is not conservative, it is the ceiling: 29.5 budget - 1.5 margin -
# 14.4 pool leaves 13.6 for the arena, i.e. max_tok <= 19,115, and 16384 is
# the largest power of two under that. The reverse checks too — put max_tok
# back to 32768 and only 7.4 GiB is left for the pool, about 269k positions,
# which is exactly the 262144 config this replaced. Those two corners are the
# only points of this box that have ever run here.
# budget = 121 host - 67.7 resident weights - 24.0 reserve = 29.5 GiB
# need = KV(~28 KiB/position) + arena + vision(1.97 measured)
# arena = 3.80 GiB + 8.40 GiB x (HALOGEN_MAX_TOK / 16384)
#
# Do NOT raise this flag past 16384 while the pool stays doubled: 65536 does
# not shrink the pool, it stops the server booting at all — and the unit still
# reads active with the container "Up (starting)", so the health check waits
# out LLM_WAIT rather than failing fast.
# EXPERIMENT 2026-09-12 (operator call): back to upstream's default 32768.
# Model predicts 524288 needs 35.0 against a 29.5 budget, so HALOGEN_KV_POOL_FIT
# should clamp the pool to 262144 (~27.8) at boot. Watch the ledger.
# That arena term is linear in max_tok and was checked against five restarts
# and against upstream's three published "on the device" figures, all to
# 0.1 GiB. Measured on this host:
# max_tok 16384 -> arena 12.2, pool 524288 total 28.6 -> FITS, 0.9 spare
# max_tok 32768 -> arena 20.6, pool 524288 total 37.0 -> clamps to 262144
# max_tok 32768 -> pool floor 262144 total 29.8 -> 0.3 OVER budget, boots
# max_tok 65536 -> arena 37.3, even the floor needs 46.5 -> OOM, no boot
# So 16384 is the ceiling for a doubled pool, not a conservative setting. The
# 0.3 over-budget at the 32768/262144 corner is where this box has been ever
# since it was deployed on 09-11; the 24 GiB reserve is generous, so it boots
# and serves with 12-14 GiB genuinely left. It is a soft floor, not a wall.
#
# Do NOT raise max_tok past 16384 while asking for the doubled pool: 65536 did
# not shrink the pool, it stopped the server booting — and the unit still reads
# active with the container "Up (starting)", so the health check waits out
# LLM_WAIT rather than failing fast.
Environment=HALOGEN_MAX_TOK=32768
# RAM left for the n-gram table the engine reads through the page cache.
# Above the vendor's 20 because a dozen containers share this host; too low
serverAppConfigs-docs/LLM/halogen-flash-server.md
+38 -27
View File
@@ -78,9 +78,9 @@ upstream's defaults:
|---|---|---|
| `HALOGEN_MODEL_ID` | `frank-large` | fleet identity; no client rename |
| `HALOGEN_CTX` | `262144` | the model's native context, and what both clients declare |
| `HALOGEN_KV_POOL_POSITIONS` | `524288` | **doubled 2026-09-12** to upstream's own default (2 × `HALOGEN_CTX`): two full-length conversations resident, or four at 131k — 13,312 MiB of KV |
| `HALOGEN_KV_POOL_POSITIONS` | `524288` | requested; **the engine actually serves 262,144 right now.** 524,288 is kept as a "grow if the RAM frees up" declaration and `HALOGEN_KV_POOL_FIT` clamps it to the floor whenever the arena leaves no room. Read the ledger, not this column |
| `HALOGEN_KV_SLOTS` | `4` | four concurrent generations; slots share the pool and cost ~115 MiB each |
| `HALOGEN_MAX_TOK` | `16384` | **load-bearing for the doubling, and already at the ceiling.** Halves the prefill arena so the pool fits the 29.5 GiB device budget, at ~8–9 % of prefill throughput. At 32768 `HALOGEN_KV_POOL_FIT` silently reverts the pool to 262144; at 65536 the arena alone exceeds the budget and **the server will not boot**. See the sizing rule below |
| `HALOGEN_MAX_TOK` | `32768` | upstream's default, restored 2026-09-12 by operator call in exchange for the doubling. The arena is 20.6 GiB here against 12.2 at 16384, and that is what pushes a 524,288 pool to 37.0 against a 29.5 budget and forces the clamp. **Set `16384` to make the doubled pool real**, at ~8–9 % of prefill throughput. At 65536 the arena alone exceeds the budget and the server **will not boot** |
| `HALOGEN_VISION_TOWER` | `1` | image input; the fleet's only image path since [[frank-npu]] was retired |
| `HALOGEN_DOWNLOAD` | `peonist-ai/halogen-qwen3.8-flash-next` | first-start fetch |
| `HALOGEN_VERBOSE` | `1` | verbose engine logging — the fullest startup account, on permanently so an incident report needs no restart-into-debug |
@@ -116,44 +116,55 @@ Three numbers explain the whole box:
the pool itself if the configured one will not fit, printing what it chose.
Read those lines on every restart; they are the ground truth for sizing.
**Measured ledger, four restarts, read live 2026-09-12.** The engine's own
startup account:
**Measured ledger, five restarts, read live 2026-09-12.** The engine's own
startup account, including two deliberate experiments:
| pool | max_tok | KV pool | arena | held | host left | verdict |
|---|---|---|---|---|---|---|
| 262144 | 32768 | 7.2 | 20.6 | 96.3 | 12.1 | the prior config |
| 524288 | 16384 | 14.4 | 12.2 | **94.6** | 13.8 | **deployed** |
| 524288 | 32768 | 14.4 | 20.6 | — | — | pool silently lowered to 262144 |
| 524288 | 65536 | — | 37.3 | — | — | **would not boot — OOM** |
| configured pool | max_tok | KV | arena | + vision | verdict |
|---|---|---|---|---|---|
| 262144 (09-11 baseline) | 32768 | 7.2 | 20.6 | **29.8** | boots — prints **no** sizing lines |
| 524288 | 16384 | 14.4 | 12.2 | **28.6** | **fits clean**, ~0.9 headroom |
| 524288 | 32768 | 14.4 | 20.6 | 37.0 | clamped to 262144, deploy still green |
| 524288 | 65536 | — | 37.3 | 46.5 | **OOM — will not boot** |
### The sizing rule
One budget covers the pool **and** the arena together:
One budget covers the pool, the arena **and** the vision tower together:
```
device budget = host RAM - resident weights - HALOGEN_HOST_RESERVE_GIB
= 121 - 67.7 - 24 = 29.5 GiB
need = KV (28 KiB/position) + arena + vision (1.97 measured)
arena = 3.80 GiB + 8.40 GiB x (HALOGEN_MAX_TOK / 16384)
```
The arena is **linear in `HALOGEN_MAX_TOK`**: measured at 3.80 GiB of base plus
8.40 GiB per 16,384 tokens, which predicts all three measured arenas to within
0.1 GiB. The KV costs ~28 KiB a position, so 524,288 of them cost 14.4 GiB.
Solving for the arena with the doubled pool in place:
The arena term is **linear in `HALOGEN_MAX_TOK`** and predicts every measured
point to 0.1 GiB — including upstream's three published "on the device" figures
(524,288 → ~35, 786,432 → ~42, 1,048,576 at `max_tok=16384` → ~41), which are
**pool + arena totals at the default `max_tok`**, not pool alone. That is why
their largest pool "needs `HALOGEN_MAX_TOK=16384`": at 32768 a 1M pool would be
49.4. Reading those figures as pool-only is the trap that makes a doubling look
affordable when it is not.
```
29.5 budget - 1.5 margin - 14.4 pool = 13.6 GiB left for the arena
-> HALOGEN_MAX_TOK <= 19,115
```
For the doubled pool: `29.5 - 1.97 vision - 14.4 KV = 13.1` for the arena, so
`max_tok <= 18,130` — **16384 is the ceiling, not a conservative setting.** The
corner check agrees: at 32768 only 7.0 GiB is left for the KV, about 254k
positions, so the pool drops back toward 262144.
16384 is the largest power of two under that, so the deployed setting is not
conservative — **it is the ceiling**. The corner check confirms it: put
`max_tok` back to 32768 and only 7.4 GiB remains for the pool, about 269k
positions, which is precisely the 262,144 config that ran here before. Those
two corners are the only points of this box the host has ever served.
### The `WARNING` line only appears when the fit check had to move
Note the consequence for the doubling: it is **1.7 GiB cheaper overall** than
the config it replaced, because halving the arena gives back more than the
extra pool takes.
This one cost me a wrong conclusion, so it is worth stating plainly. When the
configured pool already fits, the engine reserves it **and prints nothing** —
the 09-11 baseline boot goes straight from the pre-flight line to
`checkpoint: mapped` with no `kv pool:` sizing block at all. The sizing block and
its `WARNING even 262144 positions ... host RAM cannot spare it` appear **only
after a clamp**.
So the absence of a warning in old logs is not evidence of headroom. Both
`262144/32768` rows above need 29.8 against a 29.5 budget — 0.27 over — and the
baseline has lived in that state since it was deployed without incident,
because the 24 GiB reserve is itself generous (the host reports 12–14 GiB free
at that point). The engine's budget is a conservative floor, not a hard wall.
### Two failure modes, and only one is dangerous