dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-large: document reserve 15 result — both features in force, stall risk logged

Browse Source 
Ledger confirms the doubled pool is actually reserved (524288, 13312.0 MiB)
at max_tok 32768 with no clamp and no WARNING: 38.5 budget vs 37.0 needed.

Recorded the cost honestly. The server now holds 103.5 of 121.2 GiB and the
engine reports only 310 contiguous 2 MiB blocks / 7.1 GiB left for everything
else (24-reserve baseline: 2081 blocks / 12.1 GiB). All 28 co-tenants stayed
up, but the engine volunteers a new note: when memory runs out, frank-large
AND its neighbours "can stall for minutes at a time at full CPU with no disk
activity and no output, which is not a crash and needs no restart". That is
the least diagnosable of the three failure modes seen today, and `free` hides
it — MemAvailable still reads ~74 GiB because the pinned weights count as
reclaimable page cache.

Sizing rule now expresses the budget as a function of reserve, shows both
endpoints (24 -> 29.5, 15 -> 38.5), and the max_tok ceiling is labelled as
reserve-conditional (<= 18,130 at 24; <= 35,700 at 15).
This commit is contained in:
Frank
2026-09-12 21:22:36 -07:00
parent 8a79f9a709
commit 2969e08616
3 changed files with 93 additions and 31 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+17 -11
View File
@@ -32,7 +32,7 @@ host.
| Target | Runtime | Service | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | FreeToken v0.1.2 | frank-small | batsclamp Huihui-Qwen3.6-35B-A3B-abliterated-FP8 (block-fp8 e4m3 128x128, ~35 GB safetensors), experts host-RAM LRU, 262144 ctx / 4 concurrent, reasoning on | 7770 |
| Dankclaw|halogen-flash-server 0.5.8 (rootless Quadlet container)|frank-large|peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b .hgn 115.55 GiB + quality overlay, ~68 GiB pinned, 262144 ctx / 4 slots, pool 524288 requested / 262144 effective, vision tower on, greedy for silent clients)|7770|
|Dankclaw|halogen-flash-server 0.5.8 (rootless Quadlet container)|frank-large|peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b .hgn 115.55 GiB + quality overlay, ~68 GiB pinned, 262144 ctx / 4 slots, pool 524288 / max_tok 32768 / reserve 15, vision tower on, greedy for silent clients)|7770|
| Dankclaw | *(retired 2026-09-11)* FastFlowLM v1.0.1 on the XDNA2 NPU | frank-npu | Gemma4-E4B-IT-NPU2 pack (vision + audio). Retired with the `amd_iommu=off` decision: the driver's SVA/PASID bind needs an enabled IOMMU | 7771 |
History: the 2026-08-17 redo retired the pipeline-parallel split and the
@@ -396,17 +396,23 @@ published) and the OpenAI front-end on :8731 published as host **7770** —
with `HALOGEN_MODEL_ID=frank-large`, so no client wiring changed.
Deployed flags: `HALOGEN_CTX=262144`, `HALOGEN_KV_POOL_POSITIONS=524288`
(requested 2026-09-12, but **the engine serves 262144** — with
`HALOGEN_MAX_TOK=32768` the prefill arena eats the device budget and
`HALOGEN_KV_POOL_FIT` clamps the pool to the floor at every boot; set
`HALOGEN_MAX_TOK=16384` to make the doubled pool real), `HALOGEN_KV_SLOTS=4`,
`HALOGEN_HOST_RESERVE_GIB=24`, `HALOGEN_VISION_TOWER=1`, and
(**in force** since 2026-09-12 — two full-length conversations resident, or
four at 131k), `HALOGEN_KV_SLOTS=4`, `HALOGEN_MAX_TOK=32768`,
`HALOGEN_HOST_RESERVE_GIB=15`, `HALOGEN_VISION_TOWER=1`, and
`HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next` for the first-start
fetch; everything else is upstream's default. The weights stay pinned
(~68 GiB) and the kernel counts those locked pages as reclaimable page cache,
so `free`, `MemAvailable` and anything built on them (Grafana) overstate free
memory by the size of the model — believe the server's own startup ledger,
not the dashboards.
fetch; everything else is upstream's default. **Pool and reserve are a matched
pair:** the budget is `121.2 host − 67.7 weights − reserve`, and that one
budget carries the KV pool, the prefill arena and the 1.97 GiB vision tower
together. Raise the reserve back to 24 and `HALOGEN_KV_POOL_FIT` silently
clamps the pool to 262144 while the deploy still reports healthy. At 15 the
server holds 103.5 of 121 GiB, leaving ~7.1 GiB on the host, and the engine
warns that when it runs out frank-large and its neighbours can stall for
minutes at full CPU with no disk activity — not a crash, and invisible to
`free`, which still reads ~74 GiB available. The weights stay pinned (~68 GiB)
and the kernel counts those locked pages as reclaimable page cache, so `free`,
`MemAvailable` and anything built on them (Grafana) overstate free memory by
the size of the model — believe the server's own startup ledger, not the
dashboards.
Client-visible contract: **no server-side sampler defaults** (a request that
sends no `temperature` decodes greedy); the per-request budget defaults to
dankclaw/frank-large/quadlet/frank-large.container
+14 -5
View File
@@ -129,11 +129,20 @@ Environment=HALOGEN_MAX_TOK=32768
# pool at full prefill speed:
# 121.2 host - 67.7 weights - 15 reserve = 38.5 budget
# need = 14.4 pool + 20.6 arena + 1.97 vision = 37.0 -> 1.5 GiB headroom
# Thin on purpose; do not go below 15 without re-doing that arithmetic. The
# failure mode here is legible rather than fatal — long prompts fault heavily
# against a busy disk, and the remedy is to raise this number until the pool
# hands the memory back. Note the reserve was never held literally: at the
# previous 24 setting the engine's own ledger reported only 12-17 GiB left.
# VERIFIED at boot: budget 38.5, need 37.0, pool reserved at 524288 with no
# clamp and no WARNING. But the server now holds 103.5 GiB and the ledger
# reads "host memory left for everything else: 310 contiguous 2 MiB blocks
# (7.1 GiB total)" — against 2081 blocks / 12.1 GiB at the 24-reserve
# baseline. All 28 co-tenant containers stayed up (~4.3 GiB working set).
#
# The price is that the engine volunteers this note on every boot now:
# when it runs out, BOTH they and this server can stall for minutes at a
# time at full CPU with no disk activity and no output, which is not a
# crash and needs no restart.
# That is the failure mode to watch, and it is NOT the legible one predicted
# above: a minutes-long stall that looks like a hang is harder to diagnose
# than a slow prompt. If frank-large or a neighbour ever goes quiet with the
# CPU pinned, raise this reserve or drop the pool back to 262144.
Environment=HALOGEN_HOST_RESERVE_GIB=15
# Image input (the sidecar ships inside the weights repo).
Environment=HALOGEN_VISION_TOWER=1
serverAppConfigs-docs/LLM/halogen-flash-server.md
+62 -15
View File
@@ -78,10 +78,11 @@ upstream's defaults:
|---|---|---|
| `HALOGEN_MODEL_ID` | `frank-large` | fleet identity; no client rename |
| `HALOGEN_CTX` | `262144` | the model's native context, and what both clients declare |
| `HALOGEN_KV_POOL_POSITIONS` | `524288` | requested; **the engine actually serves 262,144 right now.** 524,288 is kept as a "grow if the RAM frees up" declaration and `HALOGEN_KV_POOL_FIT` clamps it to the floor whenever the arena leaves no room. Read the ledger, not this column |
| `HALOGEN_KV_POOL_POSITIONS` | `524288` | **in force** — two full-length conversations resident, or four at 131k, 13,312 MiB of KV. Was silently clamped to 262,144 until the reserve below dropped; `HALOGEN_KV_POOL_FIT` still guards the boot, so read the ledger rather than trusting this column |
| `HALOGEN_KV_SLOTS` | `4` | four concurrent generations; slots share the pool and cost ~115 MiB each |
| `HALOGEN_MAX_TOK` | `32768` | upstream's default, restored 2026-09-12 by operator call in exchange for the doubling. The arena is 20.6 GiB here against 12.2 at 16384, and that is what pushes a 524,288 pool to 37.0 against a 29.5 budget and forces the clamp. **Set `16384` to make the doubled pool real**, at ~8–9 % of prefill throughput. At 65536 the arena alone exceeds the budget and the server **will not boot** |
| `HALOGEN_VISION_TOWER` | `1` | image input; the fleet's only image path since [[frank-npu]] was retired |
| `HALOGEN_MAX_TOK` | `32768` | upstream's default, kept. The arena is 20.6 GiB here against 12.2 at 16384; it only fits alongside the doubled pool because the reserve dropped to 15. At 65536 the arena alone exceeds the budget and the server **will not boot** |
| `HALOGEN_HOST_RESERVE_GIB` | `15` | **lowered from 24 (vendor default 20) on 2026-09-12** to buy both features. The one term you own: it sets the device budget (24 → 29.5 GiB, 15 → 38.5 GiB). Matched pair with the pool — raise it back to 24 and the pool silently clamps to 262,144 again. Cost is page cache for the n-gram table; see "What the reserve actually costs" |
| `HALOGEN_VISION_TOWER` | `1` | image input; the fleet's only image path since [[frank-npu]] was retired. Charges 1.97 GiB against the budget — the term missing from my first estimate |
| `HALOGEN_DOWNLOAD` | `peonist-ai/halogen-qwen3.8-flash-next` | first-start fetch |
| `HALOGEN_VERBOSE` | `1` | verbose engine logging — the fullest startup account, on permanently so an incident report needs no restart-into-debug |
@@ -116,15 +117,16 @@ Three numbers explain the whole box:
the pool itself if the configured one will not fit, printing what it chose.
Read those lines on every restart; they are the ground truth for sizing.
**Measured ledger, five restarts, read live 2026-09-12.** The engine's own
**Measured ledger, six restarts, read live 2026-09-12.** The engine's own
startup account, including two deliberate experiments:
| configured pool | max_tok | KV | arena | + vision | verdict |
|---|---|---|---|---|---|
| 262144 (09-11 baseline) | 32768 | 7.2 | 20.6 | **29.8** | boots — prints **no** sizing lines |
| 524288 | 16384 | 14.4 | 12.2 | **28.6** | **fits clean**, ~0.9 headroom |
| 524288 | 32768 | 14.4 | 20.6 | 37.0 | clamped to 262144, deploy still green |
| 524288 | 65536 | — | 37.3 | 46.5 | **OOM — will not boot** |
| pool | max_tok | reserve | KV | arena | need | budget | verdict |
|---|---|---|---|---|---|---|---|
| 262144 (baseline) | 32768 | 24 | 7.2 | 20.6 | 29.8 | 29.5 | boots 0.3 over — prints **no** sizing lines |
| 524288 | 16384 | 24 | 14.4 | 12.2 | 28.6 | 29.5 | fits, 0.9 spare |
| 524288 | 32768 | 24 | 14.4 | 20.6 | 37.0 | 29.5 | clamped to 262144, deploy still green |
| 524288 | 65536 | 24 | — | 37.3 | 46.5 | 29.5 | **OOM — will not boot** |
| 524288 | 32768 | **15** | 14.4 | 20.6 | 37.0 | **38.5** | **deployed — both features** |
### The sizing rule
@@ -132,12 +134,20 @@ One budget covers the pool, the arena **and** the vision tower together:
```
device budget = host RAM - resident weights - HALOGEN_HOST_RESERVE_GIB
= 121 - 67.7 - 24 = 29.5 GiB
= 121.2 - 67.7 - reserve
reserve 24 -> 29.5 GiB
reserve 15 -> 38.5 GiB
need = KV (28 KiB/position) + arena + vision (1.97 measured)
arena = 3.80 GiB + 8.40 GiB x (HALOGEN_MAX_TOK / 16384)
```
The `121` the engine prints is rounded **down** from `MemTotal 127104060 kB =
121.2159 GiB`, so subtracting the displayed figures gives 29.3 rather than 29.5.
Use the unrounded value. Two of the three terms are fixed hardware and
checkpoint facts; `HALOGEN_HOST_RESERVE_GIB` is the only one you own, which
makes it the lever that buys both features — see the cost below.
The arena term is **linear in `HALOGEN_MAX_TOK`** and predicts every measured
point to 0.1 GiB — including upstream's three published "on the device" figures
(524,288 → ~35, 786,432 → ~42, 1,048,576 at `max_tok=16384` → ~41), which are
@@ -146,10 +156,47 @@ their largest pool "needs `HALOGEN_MAX_TOK=16384`": at 32768 a 1M pool would be
49.4. Reading those figures as pool-only is the trap that makes a doubling look
affordable when it is not.
For the doubled pool: `29.5 - 1.97 vision - 14.4 KV = 13.1` for the arena, so
`max_tok <= 18,130` — **16384 is the ceiling, not a conservative setting.** The
corner check agrees: at 32768 only 7.0 GiB is left for the KV, about 254k
positions, so the pool drops back toward 262144.
At the baseline reserve of 24 the doubled pool leaves `29.5 - 1.97 - 14.4 =
13.1` for the arena, so `max_tok <= 18,130` and **16384 is that reserve's
ceiling, not a conservative setting.** The corner check agrees: at 32768 only
7.0 GiB is left for the KV, ~254k positions, and the pool drops back toward
262144. Cut the reserve to 15 and the same arithmetic gives `38.5 - 1.97 - 14.4
= 22.1` for the arena — `max_tok <= 35,700`, which is exactly why 32768 now
fits unmodified.
### What the reserve actually costs
The device budget is not the whole story. Freeing the doubled pool by cutting
the reserve moves the machine's memory position considerably:
| config | held | host left | contiguous 2 MiB blocks |
|---|---|---|---|
| 262144 / 32768, reserve 24 | 96.3 | 12.1 GiB | 2081 |
| 524288 / 16384, reserve 24 | 94.6 | 13.8 GiB | 3742 |
| **524288 / 32768, reserve 15** | **103.5** | **7.1 GiB** | **310** |
At the deployed setting the server holds 103.5 of 121 GiB and the engine now
volunteers a note it did not print before:
```
halogen: NOTE this server now holds most of the machine, and 310 contiguous
2 MiB blocks remain. It runs best on a host of its own. If other large
processes share this machine they will compete for what is left, and when it
runs out BOTH they and this server can stall for minutes at a time at full
CPU with no disk activity and no output, which is not a crash and needs no
restart.
```
That is the failure mode to watch, and it is the least diagnosable of the
three: not a silent clamp, not a boot refusal, but a minutes-long freeze that
looks like a hang, for the neighbours as much as for frank-large. `free` is
useless for spotting it — `MemAvailable` still reads ~74 GiB because the
page cache counts as reclaimable. Watch the `host memory left for everything
else` line and the contiguous-block count at boot instead.
28 co-tenant containers (~4.3 GiB working set) stayed up through the change,
so it is survivable. The mitigation if anything goes quiet with the CPU
pinned: raise the reserve, or drop `HALOGEN_KV_POOL_POSITIONS` back to 262144.
### The `WARNING` line only appears when the fit check had to move