From 2969e08616ee153d04d40e63531dc6205446756e Mon Sep 17 00:00:00 2001 From: Frank Date: Sat, 12 Sep 2026 21:22:36 -0700 Subject: [PATCH] =?UTF-8?q?frank-large:=20document=20reserve=2015=20result?= =?UTF-8?q?=20=E2=80=94=20both=20features=20in=20force,=20stall=20risk=20l?= =?UTF-8?q?ogged?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ledger confirms the doubled pool is actually reserved (524288, 13312.0 MiB) at max_tok 32768 with no clamp and no WARNING: 38.5 budget vs 37.0 needed. Recorded the cost honestly. The server now holds 103.5 of 121.2 GiB and the engine reports only 310 contiguous 2 MiB blocks / 7.1 GiB left for everything else (24-reserve baseline: 2081 blocks / 12.1 GiB). All 28 co-tenants stayed up, but the engine volunteers a new note: when memory runs out, frank-large AND its neighbours "can stall for minutes at a time at full CPU with no disk activity and no output, which is not a crash and needs no restart". That is the least diagnosable of the three failure modes seen today, and `free` hides it — MemAvailable still reads ~74 GiB because the pinned weights count as reclaimable page cache. Sizing rule now expresses the budget as a function of reserve, shows both endpoints (24 -> 29.5, 15 -> 38.5), and the max_tok ceiling is labelled as reserve-conditional (<= 18,130 at 24; <= 35,700 at 15). --- AGENTS.md | 28 ++++--- .../frank-large/quadlet/frank-large.container | 19 +++-- .../LLM/halogen-flash-server.md | 77 +++++++++++++++---- 3 files changed, 93 insertions(+), 31 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 18dfbe9..77cc286 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -32,7 +32,7 @@ host. | Target | Runtime | Service | Model | Port | |--------|---------|---------|-------|------| | Dankserver | FreeToken v0.1.2 | frank-small | batsclamp Huihui-Qwen3.6-35B-A3B-abliterated-FP8 (block-fp8 e4m3 128x128, ~35 GB safetensors), experts host-RAM LRU, 262144 ctx / 4 concurrent, reasoning on | 7770 | -| Dankclaw|halogen-flash-server 0.5.8 (rootless Quadlet container)|frank-large|peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b .hgn 115.55 GiB + quality overlay, ~68 GiB pinned, 262144 ctx / 4 slots, pool 524288 requested / 262144 effective, vision tower on, greedy for silent clients)|7770| +|Dankclaw|halogen-flash-server 0.5.8 (rootless Quadlet container)|frank-large|peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b .hgn 115.55 GiB + quality overlay, ~68 GiB pinned, 262144 ctx / 4 slots, pool 524288 / max_tok 32768 / reserve 15, vision tower on, greedy for silent clients)|7770| | Dankclaw | *(retired 2026-09-11)* FastFlowLM v1.0.1 on the XDNA2 NPU | frank-npu | Gemma4-E4B-IT-NPU2 pack (vision + audio). Retired with the `amd_iommu=off` decision: the driver's SVA/PASID bind needs an enabled IOMMU | 7771 | History: the 2026-08-17 redo retired the pipeline-parallel split and the @@ -396,17 +396,23 @@ published) and the OpenAI front-end on :8731 published as host **7770** — with `HALOGEN_MODEL_ID=frank-large`, so no client wiring changed. Deployed flags: `HALOGEN_CTX=262144`, `HALOGEN_KV_POOL_POSITIONS=524288` -(requested 2026-09-12, but **the engine serves 262144** — with -`HALOGEN_MAX_TOK=32768` the prefill arena eats the device budget and -`HALOGEN_KV_POOL_FIT` clamps the pool to the floor at every boot; set -`HALOGEN_MAX_TOK=16384` to make the doubled pool real), `HALOGEN_KV_SLOTS=4`, -`HALOGEN_HOST_RESERVE_GIB=24`, `HALOGEN_VISION_TOWER=1`, and +(**in force** since 2026-09-12 — two full-length conversations resident, or +four at 131k), `HALOGEN_KV_SLOTS=4`, `HALOGEN_MAX_TOK=32768`, +`HALOGEN_HOST_RESERVE_GIB=15`, `HALOGEN_VISION_TOWER=1`, and `HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next` for the first-start -fetch; everything else is upstream's default. The weights stay pinned -(~68 GiB) and the kernel counts those locked pages as reclaimable page cache, -so `free`, `MemAvailable` and anything built on them (Grafana) overstate free -memory by the size of the model — believe the server's own startup ledger, -not the dashboards. +fetch; everything else is upstream's default. **Pool and reserve are a matched +pair:** the budget is `121.2 host − 67.7 weights − reserve`, and that one +budget carries the KV pool, the prefill arena and the 1.97 GiB vision tower +together. Raise the reserve back to 24 and `HALOGEN_KV_POOL_FIT` silently +clamps the pool to 262144 while the deploy still reports healthy. At 15 the +server holds 103.5 of 121 GiB, leaving ~7.1 GiB on the host, and the engine +warns that when it runs out frank-large and its neighbours can stall for +minutes at full CPU with no disk activity — not a crash, and invisible to +`free`, which still reads ~74 GiB available. The weights stay pinned (~68 GiB) +and the kernel counts those locked pages as reclaimable page cache, so `free`, +`MemAvailable` and anything built on them (Grafana) overstate free memory by +the size of the model — believe the server's own startup ledger, not the +dashboards. Client-visible contract: **no server-side sampler defaults** (a request that sends no `temperature` decodes greedy); the per-request budget defaults to diff --git a/dankclaw/frank-large/quadlet/frank-large.container b/dankclaw/frank-large/quadlet/frank-large.container index a449dbc..6e90554 100644 --- a/dankclaw/frank-large/quadlet/frank-large.container +++ b/dankclaw/frank-large/quadlet/frank-large.container @@ -129,11 +129,20 @@ Environment=HALOGEN_MAX_TOK=32768 # pool at full prefill speed: # 121.2 host - 67.7 weights - 15 reserve = 38.5 budget # need = 14.4 pool + 20.6 arena + 1.97 vision = 37.0 -> 1.5 GiB headroom -# Thin on purpose; do not go below 15 without re-doing that arithmetic. The -# failure mode here is legible rather than fatal — long prompts fault heavily -# against a busy disk, and the remedy is to raise this number until the pool -# hands the memory back. Note the reserve was never held literally: at the -# previous 24 setting the engine's own ledger reported only 12-17 GiB left. +# VERIFIED at boot: budget 38.5, need 37.0, pool reserved at 524288 with no +# clamp and no WARNING. But the server now holds 103.5 GiB and the ledger +# reads "host memory left for everything else: 310 contiguous 2 MiB blocks +# (7.1 GiB total)" — against 2081 blocks / 12.1 GiB at the 24-reserve +# baseline. All 28 co-tenant containers stayed up (~4.3 GiB working set). +# +# The price is that the engine volunteers this note on every boot now: +# when it runs out, BOTH they and this server can stall for minutes at a +# time at full CPU with no disk activity and no output, which is not a +# crash and needs no restart. +# That is the failure mode to watch, and it is NOT the legible one predicted +# above: a minutes-long stall that looks like a hang is harder to diagnose +# than a slow prompt. If frank-large or a neighbour ever goes quiet with the +# CPU pinned, raise this reserve or drop the pool back to 262144. Environment=HALOGEN_HOST_RESERVE_GIB=15 # Image input (the sidecar ships inside the weights repo). Environment=HALOGEN_VISION_TOWER=1 diff --git a/serverAppConfigs-docs/LLM/halogen-flash-server.md b/serverAppConfigs-docs/LLM/halogen-flash-server.md index f8ec23c..675ed0e 100644 --- a/serverAppConfigs-docs/LLM/halogen-flash-server.md +++ b/serverAppConfigs-docs/LLM/halogen-flash-server.md @@ -78,10 +78,11 @@ upstream's defaults: |---|---|---| | `HALOGEN_MODEL_ID` | `frank-large` | fleet identity; no client rename | | `HALOGEN_CTX` | `262144` | the model's native context, and what both clients declare | -| `HALOGEN_KV_POOL_POSITIONS` | `524288` | requested; **the engine actually serves 262,144 right now.** 524,288 is kept as a "grow if the RAM frees up" declaration and `HALOGEN_KV_POOL_FIT` clamps it to the floor whenever the arena leaves no room. Read the ledger, not this column | +| `HALOGEN_KV_POOL_POSITIONS` | `524288` | **in force** — two full-length conversations resident, or four at 131k, 13,312 MiB of KV. Was silently clamped to 262,144 until the reserve below dropped; `HALOGEN_KV_POOL_FIT` still guards the boot, so read the ledger rather than trusting this column | | `HALOGEN_KV_SLOTS` | `4` | four concurrent generations; slots share the pool and cost ~115 MiB each | -| `HALOGEN_MAX_TOK` | `32768` | upstream's default, restored 2026-09-12 by operator call in exchange for the doubling. The arena is 20.6 GiB here against 12.2 at 16384, and that is what pushes a 524,288 pool to 37.0 against a 29.5 budget and forces the clamp. **Set `16384` to make the doubled pool real**, at ~8–9 % of prefill throughput. At 65536 the arena alone exceeds the budget and the server **will not boot** | -| `HALOGEN_VISION_TOWER` | `1` | image input; the fleet's only image path since [[frank-npu]] was retired | +| `HALOGEN_MAX_TOK` | `32768` | upstream's default, kept. The arena is 20.6 GiB here against 12.2 at 16384; it only fits alongside the doubled pool because the reserve dropped to 15. At 65536 the arena alone exceeds the budget and the server **will not boot** | +| `HALOGEN_HOST_RESERVE_GIB` | `15` | **lowered from 24 (vendor default 20) on 2026-09-12** to buy both features. The one term you own: it sets the device budget (24 → 29.5 GiB, 15 → 38.5 GiB). Matched pair with the pool — raise it back to 24 and the pool silently clamps to 262,144 again. Cost is page cache for the n-gram table; see "What the reserve actually costs" | +| `HALOGEN_VISION_TOWER` | `1` | image input; the fleet's only image path since [[frank-npu]] was retired. Charges 1.97 GiB against the budget — the term missing from my first estimate | | `HALOGEN_DOWNLOAD` | `peonist-ai/halogen-qwen3.8-flash-next` | first-start fetch | | `HALOGEN_VERBOSE` | `1` | verbose engine logging — the fullest startup account, on permanently so an incident report needs no restart-into-debug | @@ -116,15 +117,16 @@ Three numbers explain the whole box: the pool itself if the configured one will not fit, printing what it chose. Read those lines on every restart; they are the ground truth for sizing. -**Measured ledger, five restarts, read live 2026-09-12.** The engine's own +**Measured ledger, six restarts, read live 2026-09-12.** The engine's own startup account, including two deliberate experiments: -| configured pool | max_tok | KV | arena | + vision | verdict | -|---|---|---|---|---|---| -| 262144 (09-11 baseline) | 32768 | 7.2 | 20.6 | **29.8** | boots — prints **no** sizing lines | -| 524288 | 16384 | 14.4 | 12.2 | **28.6** | **fits clean**, ~0.9 headroom | -| 524288 | 32768 | 14.4 | 20.6 | 37.0 | clamped to 262144, deploy still green | -| 524288 | 65536 | — | 37.3 | 46.5 | **OOM — will not boot** | +| pool | max_tok | reserve | KV | arena | need | budget | verdict | +|---|---|---|---|---|---|---|---| +| 262144 (baseline) | 32768 | 24 | 7.2 | 20.6 | 29.8 | 29.5 | boots 0.3 over — prints **no** sizing lines | +| 524288 | 16384 | 24 | 14.4 | 12.2 | 28.6 | 29.5 | fits, 0.9 spare | +| 524288 | 32768 | 24 | 14.4 | 20.6 | 37.0 | 29.5 | clamped to 262144, deploy still green | +| 524288 | 65536 | 24 | — | 37.3 | 46.5 | 29.5 | **OOM — will not boot** | +| 524288 | 32768 | **15** | 14.4 | 20.6 | 37.0 | **38.5** | **deployed — both features** | ### The sizing rule @@ -132,12 +134,20 @@ One budget covers the pool, the arena **and** the vision tower together: ``` device budget = host RAM - resident weights - HALOGEN_HOST_RESERVE_GIB - = 121 - 67.7 - 24 = 29.5 GiB + = 121.2 - 67.7 - reserve + reserve 24 -> 29.5 GiB + reserve 15 -> 38.5 GiB need = KV (28 KiB/position) + arena + vision (1.97 measured) arena = 3.80 GiB + 8.40 GiB x (HALOGEN_MAX_TOK / 16384) ``` +The `121` the engine prints is rounded **down** from `MemTotal 127104060 kB = +121.2159 GiB`, so subtracting the displayed figures gives 29.3 rather than 29.5. +Use the unrounded value. Two of the three terms are fixed hardware and +checkpoint facts; `HALOGEN_HOST_RESERVE_GIB` is the only one you own, which +makes it the lever that buys both features — see the cost below. + The arena term is **linear in `HALOGEN_MAX_TOK`** and predicts every measured point to 0.1 GiB — including upstream's three published "on the device" figures (524,288 → ~35, 786,432 → ~42, 1,048,576 at `max_tok=16384` → ~41), which are @@ -146,10 +156,47 @@ their largest pool "needs `HALOGEN_MAX_TOK=16384`": at 32768 a 1M pool would be 49.4. Reading those figures as pool-only is the trap that makes a doubling look affordable when it is not. -For the doubled pool: `29.5 - 1.97 vision - 14.4 KV = 13.1` for the arena, so -`max_tok <= 18,130` — **16384 is the ceiling, not a conservative setting.** The -corner check agrees: at 32768 only 7.0 GiB is left for the KV, about 254k -positions, so the pool drops back toward 262144. +At the baseline reserve of 24 the doubled pool leaves `29.5 - 1.97 - 14.4 = +13.1` for the arena, so `max_tok <= 18,130` and **16384 is that reserve's +ceiling, not a conservative setting.** The corner check agrees: at 32768 only +7.0 GiB is left for the KV, ~254k positions, and the pool drops back toward +262144. Cut the reserve to 15 and the same arithmetic gives `38.5 - 1.97 - 14.4 += 22.1` for the arena — `max_tok <= 35,700`, which is exactly why 32768 now +fits unmodified. + +### What the reserve actually costs + +The device budget is not the whole story. Freeing the doubled pool by cutting +the reserve moves the machine's memory position considerably: + +| config | held | host left | contiguous 2 MiB blocks | +|---|---|---|---| +| 262144 / 32768, reserve 24 | 96.3 | 12.1 GiB | 2081 | +| 524288 / 16384, reserve 24 | 94.6 | 13.8 GiB | 3742 | +| **524288 / 32768, reserve 15** | **103.5** | **7.1 GiB** | **310** | + +At the deployed setting the server holds 103.5 of 121 GiB and the engine now +volunteers a note it did not print before: + +``` +halogen: NOTE this server now holds most of the machine, and 310 contiguous +2 MiB blocks remain. It runs best on a host of its own. If other large +processes share this machine they will compete for what is left, and when it +runs out BOTH they and this server can stall for minutes at a time at full +CPU with no disk activity and no output, which is not a crash and needs no +restart. +``` + +That is the failure mode to watch, and it is the least diagnosable of the +three: not a silent clamp, not a boot refusal, but a minutes-long freeze that +looks like a hang, for the neighbours as much as for frank-large. `free` is +useless for spotting it — `MemAvailable` still reads ~74 GiB because the +page cache counts as reclaimable. Watch the `host memory left for everything +else` line and the contiguous-block count at boot instead. + +28 co-tenant containers (~4.3 GiB working set) stayed up through the change, +so it is survivable. The mitigation if anything goes quiet with the CPU +pinned: raise the reserve, or drop `HALOGEN_KV_POOL_POSITIONS` back to 262144. ### The `WARNING` line only appears when the fit check had to move