diff --git a/AGENTS.md b/AGENTS.md index 63dfd5c..5fbbeb9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -32,7 +32,7 @@ host. | Target | Runtime | Service | Model | Port | |--------|---------|---------|-------|------| | Dankserver | FreeToken v0.1.2 | frank-small | batsclamp Huihui-Qwen3.6-35B-A3B-abliterated-FP8 (block-fp8 e4m3 128x128, ~35 GB safetensors), experts host-RAM LRU, 262144 ctx / 4 concurrent, reasoning on | 7770 | -|Dankclaw|halogen-flash-server 0.5.8 (rootless Quadlet container)|frank-large|peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b .hgn 115.55 GiB + quality overlay, ~68 GiB pinned, 262144 ctx / 3 slots, pool 524288 / max_tok 32768 / reserve 15, vision tower on, greedy for silent clients)|7770| +|Dankclaw|halogen-flash-server 0.5.8 (rootless Quadlet container)|frank-large|peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b .hgn 115.55 GiB + quality overlay, ~68 GiB pinned, 262144 ctx / 3 slots, pool 401408 / max_tok 32768 / reserve 15, vision tower on, greedy for silent clients)|7770| | Dankclaw | *(retired 2026-09-11)* FastFlowLM v1.0.1 on the XDNA2 NPU | frank-npu | Gemma4-E4B-IT-NPU2 pack (vision + audio). Retired with the `amd_iommu=off` decision: the driver's SVA/PASID bind needs an enabled IOMMU | 7771 | History: the 2026-08-17 redo retired the pipeline-parallel split and the @@ -395,24 +395,28 @@ both halves — the engine on container loopback :8730 (unauthenticated, never published) and the OpenAI front-end on :8731 published as host **7770** — with `HALOGEN_MODEL_ID=frank-large`, so no client wiring changed. -Deployed flags: `HALOGEN_CTX=262144`, `HALOGEN_KV_POOL_POSITIONS=524288` -(**in force** since 2026-09-12 — two full-length conversations resident, or -four at 131k), `HALOGEN_KV_SLOTS=3` (lowered from 4 on 2026-09-12 to the real -fleet size; 111 MiB of O(1) state each), `HALOGEN_MAX_TOK=32768`, -`HALOGEN_HOST_RESERVE_GIB=15`, `HALOGEN_VISION_TOWER=1`, and +Deployed flags: `HALOGEN_CTX=262144`, `HALOGEN_KV_POOL_POSITIONS=401408` +(**trimmed 2026-09-13** from the 524,288 the 09-12 doubling deployed — one +full 262k conversation resident beside a ~139k one, or three at 131k; hands +~3.4 GiB back for the ~28 co-tenants after the doubled pool left only +~7.1 GiB and the engine warned of co-tenant stalls), `HALOGEN_KV_SLOTS=3` +(lowered from 4 on 2026-09-12 to the real fleet size; 111 MiB of O(1) state +each), `HALOGEN_MAX_TOK=32768`, `HALOGEN_HOST_RESERVE_GIB=15`, +`HALOGEN_VISION_TOWER=1`, and `HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next` for the first-start fetch; everything else is upstream's default. **Pool and reserve are a matched pair:** the budget is `121.2 host − 67.7 weights − reserve`, and that one budget carries the KV pool, the prefill arena and the 1.97 GiB vision tower -together. Raise the reserve past **16** and `HALOGEN_KV_POOL_FIT` silently -clamps the pool to 262144 while the deploy still reports healthy (the engine -needs budget ≥ 36.5; reserve 18 yields 35.5). The reserve is **not an -allocation** — 15 and 16 were both measured holding 103.5 GiB with 7.1 GiB left -on the host, so a bigger reserve cannot free memory for co-tenants, it can only -remove the pool. To free host RAM, cut `HALOGEN_KV_POOL_POSITIONS` (262144 -hands back 7.2 GiB) or `HALOGEN_MAX_TOK` (16384 hands back 8.4). At 15 the -server holds 103.5 of 121 GiB, leaving ~7.1 GiB on the host, and the engine -warns that when it runs out frank-large and its neighbours can stall for +together. The reserve is **not an allocation** — 15 and 16 were both measured +holding 103.5 GiB with 7.1 GiB left on the host, so a bigger reserve cannot +free memory for co-tenants, it can only remove the pool. Against the old +524,288 pool the reserve ceiling was 16 (engine needs budget ≥ 36.5; reserve +18 yields 35.5 and silently clamps); at 401,408 the need is ~33.4 so the +ceiling loosens, but a bigger reserve still buys nothing. To free host RAM, +cut `HALOGEN_KV_POOL_POSITIONS` (262144 hands back another 3.6 GiB) or +`HALOGEN_MAX_TOK` (16384 hands back 8.4). At the trimmed pool expect roughly +10.5 GiB left on the host — confirm on the boot ledger, not here. If a +co-tenant still runs it dry, frank-large and its neighbours stall for minutes at full CPU with no disk activity — not a crash, and invisible to `free`, which still reads ~74 GiB available. The weights stay pinned (~68 GiB) and the kernel counts those locked pages as reclaimable page cache, so `free`, diff --git a/dankclaw/frank-large/quadlet/frank-large.container b/dankclaw/frank-large/quadlet/frank-large.container index 199091e..f1c6c16 100644 --- a/dankclaw/frank-large/quadlet/frank-large.container +++ b/dankclaw/frank-large/quadlet/frank-large.container @@ -39,18 +39,17 @@ # engine lowers it itself and prints what it settled on. Read that ledger on # every restart — it is the ground truth for sizing, not this comment. # -# DOUBLED 2026-09-12: pool 524,288 positions (~14.4 GiB of KV at the -# measured 28 KiB/position). This did NOT fit on the first attempt. The -# engine's own ledger for the prior 262,144 config was 68.0 weights + 7.2 -# pool + 21.1 working = 96.3 GiB held, 12.1 GiB left on the host, and its -# pre-flight arithmetic is: 121 GiB - 67.7 resident weights - 24.0 reserve -# = 29.5 GiB of device budget, which has to cover the pool AND the prefill -# arena together. 524,288 with the 32,768-token arena needs 35.0 GiB, so -# HALOGEN_KV_POOL_FIT lowered the pool straight back to 262,144 and the -# deploy still reported healthy. The doubling only becomes real together -# with HALOGEN_MAX_TOK=16384 further down, which halves the arena and brings -# the need to ~24.9 GiB. Read the `kv pool:` lines after every restart — -# a green health check does NOT mean the pool you asked for is the pool you got. +# POOL TRIMMED 2026-09-13: 401,408 positions (~10.8 GiB of KV at the measured +# 28 KiB/position), down from the 524,288 the 2026-09-12 doubling deployed. +# The doubling fit only at reserve 15, which left the host 7.1 GiB / 310 +# contiguous 2 MiB blocks and made the engine print its co-tenant stall +# warning on every boot. 401,408 hands ~3.4 GiB back (expect ~10.5 GiB left) +# while keeping HALOGEN_MAX_TOK=32768 and the reserve at 15. Capacity: one +# full 262k conversation resident beside a ~139k one, or three concurrent +# 131k chats -- the fleet's real shape at three slots. Fit check: modelled +# need ~33.4 GiB (KV 10.8 + arena 20.6 + vision 2.0) vs budget 38.5. The +# 09-12 doubling story (silent clamp at reserve 24, the arena term, the +# reserve ceiling) is preserved in the budget block below and the runbook. # # WEIGHTS. HALOGEN_DOWNLOAD fetches the whole weights repo on FIRST START ONLY # (the entrypoint checks whether the checkpoint is absent) into the bind mount @@ -97,19 +96,21 @@ Environment=HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next Environment=HALOGEN_MODEL_ID=frank-large # Native context of the model. Matches what both clients declare. Environment=HALOGEN_CTX=262144 -# Doubled 2026-09-12 from the 262144 this had run since 09-11: two full-length -# conversations resident at once, or four at 131k, instead of one. -Environment=HALOGEN_KV_POOL_POSITIONS=524288 +# Trimmed 2026-09-13 from the 524,288 the 09-12 doubling deployed: buys +# ~3.4 GiB of co-tenant headroom, keeps one full 262k conversation resident +# beside a ~139k one, or three concurrent 131k chats. +Environment=HALOGEN_KV_POOL_POSITIONS=401408 # Concurrency cap. Each slot is 111.0 MiB of O(1) state; the KV is ONE -# 524288-position pool carved into per-request regions, so slots cap how many -# requests run at once while the pool binds how many positions exist. These -# slots OVERSUBSCRIBE the pool (4 x 262144 > 524288), so admission was always -# pool-bound -- at most two full-length conversations, or four at 131k. +# 401408-position pool carved into per-request regions, so slots cap how many +# requests run at once while the pool binds how many positions exist. The +# slots still OVERSUBSCRIBE the pool (3 x 262144 > 401408), so admission is +# pool-bound -- one full-length conversation plus a ~139k second, or three +# at 131k. # LOWERED 4 -> 3 on 2026-09-12: frees ~111 MiB on a host with 7.1 GiB left, # and three users is the real fleet size. A fourth concurrent request now # queues behind the three rather than contending. Environment=HALOGEN_KV_SLOTS=3 -# Making the doubling stick is budget arithmetic, all of it. ONE budget covers +# Sizing is budget arithmetic, all of it. ONE budget covers # the KV pool, the prefill arena and the vision tower together: # # budget = 121.2 host - 67.7 resident weights - reserve @@ -126,11 +127,12 @@ Environment=HALOGEN_KV_SLOTS=3 # 32768 -> need 37.0 vs 29.5 POOL CLAMPED to 262144, deploy still green # 65536 -> need 46.5 vs 29.5 OOM, will not boot; unit still reads ACTIVE # -# None of those three gave both features. Dropping the reserve to 15 moves the -# budget to 38.5, so 32768's need of 37.0 fits with 1.5 spare: full prefill -# AND the doubled pool, paid for out of the n-gram page cache instead of out of -# either feature. THE TWO SETTINGS ARE A MATCHED PAIR, NOT independent tunables -# — raise the reserve back to 24 and the pool silently clamps to 262144 again. +# None of those three gave both features; reserve 15 made the doubled pool +# fit (38.5 vs 37.0) but at 7.1 GiB of host headroom. On 2026-09-13 the pool +# came down to 401,408 to restore co-tenant headroom: need ~33.4 vs 38.5. +# THE POOL, ARENA AND RESERVE ARE A MATCHED SET, NOT independent tunables -- +# the fit check's printed ledger on each boot is the arbiter, not these +# comments. Environment=HALOGEN_MAX_TOK=32768 # RAM left for the n-gram table the engine reads through the page cache. # LOWERED 2026-09-12 from 24 (vendor default is 20) to buy the doubled pool at @@ -138,10 +140,12 @@ Environment=HALOGEN_MAX_TOK=32768 # read from a clamped boot, not modelled: # "524288 positions need ~35.0 GiB (plus 1.5 of margin)" -> budget >= 36.5 # budget = 121.2 host - 67.7 weights - reserve -# reserve 16 -> 37.5 (1.0 spare); 15 -> 38.5 (2.0 spare, deployed); 18 -> 35.5 -# CLAMPS the pool to 262144. Do not raise past 16. +# reserve 16 -> 37.5 (1.0 spare); 15 -> 38.5 (2.0 spare, deployed). At the +# old 524,288 pool, 18 -> 35.5 CLAMPED it to 262144, so the ceiling there was +# 16; at 401,408 (need ~33.4) it loosens, but a higher reserve still buys +# nothing. # -# CEILING IS 16, AND 16 BUYS NOTHING. Tested live: 15 -> 16 changed the fit +# AT THE DOUBLED POOL THE CEILING WAS 16, AND 16 BOUGHT NOTHING. Tested live at 524,288: 15 -> 16 changed the fit # margin only. The server held 103.5 GiB and reported "7.1 GiB left for # everything else" at BOTH settings, because this reserve feeds the fit-check # budget and is not an allocation — actual footprint is weights + KV + arena + @@ -154,14 +158,15 @@ Environment=HALOGEN_MAX_TOK=32768 # (7.1 GiB total)" — against 2081 blocks / 12.1 GiB at the 24-reserve # baseline. All 28 co-tenant containers stayed up (~4.3 GiB working set). # -# The price is that the engine volunteers this note on every boot now: +# The price at 524,288 was that the engine volunteered this note on every boot: # when it runs out, BOTH they and this server can stall for minutes at a # time at full CPU with no disk activity and no output, which is not a # crash and needs no restart. # That is the failure mode to watch, and it is NOT the legible one predicted # above: a minutes-long stall that looks like a hang is harder to diagnose -# than a slow prompt. If it happens, the reserve has no headroom left to give -# — the remedy is HALOGEN_KV_POOL_POSITIONS=262144 (hands back 7.2 GiB), or +# than a slow prompt. The 09-13 trim to 401,408 was the first application of +# the remedy. If it happens again, cut further: +# HALOGEN_KV_POOL_POSITIONS=262144 (hands back another 3.6 GiB), or # HALOGEN_MAX_TOK=16384 (hands back 8.4) with reserve then free to rise to 24. Environment=HALOGEN_HOST_RESERVE_GIB=15 # Image input (the sidecar ships inside the weights repo). diff --git a/serverAppConfigs-docs/LLM/halogen-flash-server.md b/serverAppConfigs-docs/LLM/halogen-flash-server.md index 4c28c71..ccf06c3 100644 --- a/serverAppConfigs-docs/LLM/halogen-flash-server.md +++ b/serverAppConfigs-docs/LLM/halogen-flash-server.md @@ -11,7 +11,7 @@ tags: - gfx1151 - qwen4exp - rocm -timestamp: 2026-09-12 +timestamp: 2026-09-13 --- # halogen-flash-server on Dankclaw (frank-large) @@ -78,10 +78,10 @@ upstream's defaults: |---|---|---| | `HALOGEN_MODEL_ID` | `frank-large` | fleet identity; no client rename | | `HALOGEN_CTX` | `262144` | the model's native context, and what both clients declare | -| `HALOGEN_KV_POOL_POSITIONS` | `524288` | **in force** — two full-length conversations resident, or four at 131k, 13,312 MiB of KV. Was silently clamped to 262,144 until the reserve below dropped; `HALOGEN_KV_POOL_FIT` still guards the boot, so read the ledger rather than trusting this column | -| `HALOGEN_KV_SLOTS` | `3` | Concurrency cap — **lowered from 4 on 2026-09-12** to the real fleet size. Each slot is 111.0 MiB of O(1) state, so this freed ~111 MiB (held 103.5 → 103.4 GiB). Slots share the ONE pool and **oversubscribe** it (4 × 262144 > 524288), so admission was always pool-bound: at most two full-length conversations, or four at 131k. This trims simultaneous requests, not how much context fits | -| `HALOGEN_MAX_TOK` | `32768` | upstream's default, kept. The arena is 20.6 GiB here against 12.2 at 16384; it only fits alongside the doubled pool because the reserve dropped to 15. At 65536 the arena alone exceeds the budget and the server **will not boot** | -| `HALOGEN_HOST_RESERVE_GIB` | `15` | **lowered from 24 (vendor default 20) on 2026-09-12** to buy both features. The only term you own in the device budget (24 → 29.5 GiB, 15 → 38.5 GiB), and **the ceiling for the doubled pool at `max_tok=32768` is 16** — the engine needs budget ≥ 36.5, and 18 gives 35.5, which silently re-clamps the pool. Matched pair with the pool. It is **not** an allocation: raising it cannot return memory to the host, only take the pool away | +| `HALOGEN_KV_POOL_POSITIONS` | `401408` | **trimmed 2026-09-13** from the 524,288 the doubling deployed: one full 262k conversation resident beside a ~139k one, or three at 131k, ~10.8 GiB of KV. The trim hands ~3.4 GiB back to the ~28 co-tenants — the doubled pool left only ~7.1 GiB on the host and the engine warned of co-tenant stalls on every boot. `HALOGEN_KV_POOL_FIT` still guards the boot, so read the ledger rather than trusting this column | +| `HALOGEN_KV_SLOTS` | `3` | Concurrency cap — **lowered from 4 on 2026-09-12** to the real fleet size. Each slot is 111.0 MiB of O(1) state, so this freed ~111 MiB (held 103.5 → 103.4 GiB). Slots share the ONE pool and **oversubscribe** it (3 × 262144 > 401408), so admission is pool-bound: one full-length conversation plus a ~139k second, or three at 131k. This trims simultaneous requests, not how much context fits | +| `HALOGEN_MAX_TOK` | `32768` | upstream's default, kept. The arena is 20.6 GiB here against 12.2 at 16384; it fit alongside the doubled pool only because the reserve dropped to 15, and it fits the trimmed 401,408 pool with room to spare. At 65536 the arena alone exceeds the budget and the server **will not boot** | +| `HALOGEN_HOST_RESERVE_GIB` | `15` | **lowered from 24 (vendor default 20) on 2026-09-12** to buy the doubled pool. The only term you own in the device budget (24 → 29.5 GiB, 15 → 38.5 GiB). Against the doubled pool the ceiling was 16 — the engine needed budget ≥ 36.5, and 18 gave 35.5, silently re-clamping; at 401,408 the need is ~33.4 so the ceiling loosens, but a bigger reserve still buys nothing. Matched pair with the pool. It is **not** an allocation: raising it cannot return memory to the host, only take the pool away | | `HALOGEN_VISION_TOWER` | `1` | image input; the fleet's only image path since [[frank-npu]] was retired. Charges 1.97 GiB against the budget — the term missing from my first estimate | | `HALOGEN_DOWNLOAD` | `peonist-ai/halogen-qwen3.8-flash-next` | first-start fetch | | `HALOGEN_VERBOSE` | `1` | verbose engine logging — the fullest startup account, on permanently so an incident report needs no restart-into-debug | @@ -174,8 +174,8 @@ fits unmodified. ### What the reserve actually costs -The device budget is not the whole story. Freeing the doubled pool by cutting -the reserve moves the machine's memory position considerably: +The device budget is not the whole story. Memory position by configuration +(the 09-13 trim added its row after the deploy was measured — see below): | config | held | host left | contiguous 2 MiB blocks | |---|---|---|---| @@ -184,8 +184,8 @@ the reserve moves the machine's memory position considerably: | **524288 / 32768, reserve 15** | **103.5** | **7.1 GiB** | **310** | | 524288 / 32768, reserve **16** | **103.5** | **7.1 GiB** | 293 | -At the deployed setting the server holds 103.5 of 121 GiB and the engine now -volunteers a note it did not print before: +At the doubled setting (2026-09-12) the server held 103.5 of 121 GiB and the +engine volunteered a note it had not printed before: ``` halogen: NOTE this server now holds most of the machine, and 310 contiguous @@ -228,8 +228,10 @@ thing a bigger reserve can do is take the doubling away. **To actually free host memory, change the pool or the arena**, and the reserve then becomes free to rise: `HALOGEN_KV_POOL_POSITIONS=262144` hands back -7.2 GiB, `HALOGEN_MAX_TOK=16384` hands back 8.4. Both are one-line edits that -the fit check will confirm on the next boot. +7.2 GiB from the doubled pool, `HALOGEN_MAX_TOK=16384` hands back 8.4. Both +are one-line edits that the fit check will confirm on the next boot. The +2026-09-13 trim to 401,408 was this remedy applied at half strength: ~3.4 GiB +back with `HALOGEN_MAX_TOK` untouched. ### The `WARNING` line only appears when the fit check had to move @@ -320,12 +322,14 @@ deployment: **plus** `max_tokens` positions when it is admitted; if the pool cannot hold it the request waits in arrival order, and after the queue timeout (3600 s) it gets a `503 engine_busy`. A prompt large enough that prompt + budget exceeds -the pool can therefore never be admitted at all. With the pool at 524,288 the -per-conversation ceiling is `HALOGEN_CTX` (262,144) — the pool binds **how -many** conversations stay resident, not how long any one of them can be. Under -the previous 262,144 pool the two numbers were identical, which is why four -concurrent 65k chats were contending for one conversation's worth of -positions. +the pool can therefore never be admitted at all. With the pool at 401,408 +(since 2026-09-13) the per-conversation ceiling is still `HALOGEN_CTX` +(262,144) — even a full-context prompt plus the 65,536 cap is 327,680 ≤ +401,408. The pool binds **how many** conversations stay resident: at 524,288 +two full-length ones fit; at 401,408 it is one full-length beside a ~139k +second, or three at 131k. Under the pre-09-12 262,144 pool the two numbers +were identical, which is why four concurrent 65k chats were contending for +one conversation's worth of positions. ## Vision