From 27e63359d68a233639d9597600c2e610799b6702 Mon Sep 17 00:00:00 2001 From: Frank Date: Sun, 13 Sep 2026 23:02:16 -0700 Subject: [PATCH] frank-large: correct docs to measured 401408 ledger figures Predicted ~10.8 GiB KV / ~33.4 need / ~10.5 host-left; the boot ledger printed ~11.0 / ~31.5 / 100.0 GiB held with 9.1 GiB + 1396 contiguous 2 MiB blocks left. Stall NOTE no longer prints at this footprint. Also documents the ~45-60 s first-request PING silence after a restart (the healthcheck's HealthStartPeriod covers it; do not restart the unit). --- AGENTS.md | 9 ++++--- .../frank-large/quadlet/frank-large.container | 27 ++++++++++++------- .../LLM/halogen-flash-server.md | 16 ++++++----- 3 files changed, 33 insertions(+), 19 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 5fbbeb9..b359f45 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -413,9 +413,12 @@ free memory for co-tenants, it can only remove the pool. Against the old 524,288 pool the reserve ceiling was 16 (engine needs budget ≥ 36.5; reserve 18 yields 35.5 and silently clamps); at 401,408 the need is ~33.4 so the ceiling loosens, but a bigger reserve still buys nothing. To free host RAM, -cut `HALOGEN_KV_POOL_POSITIONS` (262144 hands back another 3.6 GiB) or -`HALOGEN_MAX_TOK` (16384 hands back 8.4). At the trimmed pool expect roughly -10.5 GiB left on the host — confirm on the boot ledger, not here. If a +cut `HALOGEN_KV_POOL_POSITIONS` (262144 hands back another ~3.6 GiB) or +`HALOGEN_MAX_TOK` (16384 hands back 8.4). Measured at 401,408 (2026-09-13 +deploy ledger): 100.0 GiB held (68.0 weights + 11.0 KV + 21.0 working), ~9.1 +GiB / 1396 contiguous 2 MiB blocks left on the host, engine need printed +~31.5 GiB against the 38.5 budget, and the co-tenant stall NOTE no longer +prints — confirm on the boot ledger, not here. If a co-tenant still runs it dry, frank-large and its neighbours stall for minutes at full CPU with no disk activity — not a crash, and invisible to `free`, which still reads ~74 GiB available. The weights stay pinned (~68 GiB) diff --git a/dankclaw/frank-large/quadlet/frank-large.container b/dankclaw/frank-large/quadlet/frank-large.container index f1c6c16..698fb8f 100644 --- a/dankclaw/frank-large/quadlet/frank-large.container +++ b/dankclaw/frank-large/quadlet/frank-large.container @@ -39,17 +39,24 @@ # engine lowers it itself and prints what it settled on. Read that ledger on # every restart — it is the ground truth for sizing, not this comment. # -# POOL TRIMMED 2026-09-13: 401,408 positions (~10.8 GiB of KV at the measured -# 28 KiB/position), down from the 524,288 the 2026-09-12 doubling deployed. -# The doubling fit only at reserve 15, which left the host 7.1 GiB / 310 +# POOL TRIMMED 2026-09-13: 401,408 positions (~11.0 GiB of KV as reserved by +# the engine), down from the 524,288 the 2026-09-12 doubling deployed. The +# doubling fit only at reserve 15, which left the host 7.1 GiB / 310 # contiguous 2 MiB blocks and made the engine print its co-tenant stall -# warning on every boot. 401,408 hands ~3.4 GiB back (expect ~10.5 GiB left) -# while keeping HALOGEN_MAX_TOK=32768 and the reserve at 15. Capacity: one -# full 262k conversation resident beside a ~139k one, or three concurrent -# 131k chats -- the fleet's real shape at three slots. Fit check: modelled -# need ~33.4 GiB (KV 10.8 + arena 20.6 + vision 2.0) vs budget 38.5. The -# 09-12 doubling story (silent clamp at reserve 24, the arena term, the -# reserve ceiling) is preserved in the budget block below and the runbook. +# warning on every boot. MEASURED AT 401,408 (deploy ledger): 68.0 weights + +# 11.0 KV + 21.0 working = 100.0 GiB held, 9.1 GiB / 1396 contiguous 2 MiB +# blocks left, engine need printed ~31.5 vs the modelled ~33.4, and NO +# stall-warning NOTE. Capacity: one full 262k conversation resident beside a +# ~139k one, or three concurrent 131k chats -- the fleet's real shape at +# three slots. Fit check budget: 121.2 - 67.7 - 15 = 38.5. The 09-12 doubling +# story (silent clamp at reserve 24, the arena term, the reserve ceiling) is +# preserved in the budget block below and the runbook. +# +# BOOT NOTE: on the first request after a (re)start the engine can go silent +# for ~45-60 s ("has not answered PING") while it first-touches the freshly +# reserved pool; /health reads engine_unresponsive during it and recovers by +# itself. The quadlet health check's HealthStartPeriod=300s covers it; do not +# restart the unit over it. # # WEIGHTS. HALOGEN_DOWNLOAD fetches the whole weights repo on FIRST START ONLY # (the entrypoint checks whether the checkpoint is absent) into the bind mount diff --git a/serverAppConfigs-docs/LLM/halogen-flash-server.md b/serverAppConfigs-docs/LLM/halogen-flash-server.md index ccf06c3..2b5af55 100644 --- a/serverAppConfigs-docs/LLM/halogen-flash-server.md +++ b/serverAppConfigs-docs/LLM/halogen-flash-server.md @@ -78,7 +78,7 @@ upstream's defaults: |---|---|---| | `HALOGEN_MODEL_ID` | `frank-large` | fleet identity; no client rename | | `HALOGEN_CTX` | `262144` | the model's native context, and what both clients declare | -| `HALOGEN_KV_POOL_POSITIONS` | `401408` | **trimmed 2026-09-13** from the 524,288 the doubling deployed: one full 262k conversation resident beside a ~139k one, or three at 131k, ~10.8 GiB of KV. The trim hands ~3.4 GiB back to the ~28 co-tenants — the doubled pool left only ~7.1 GiB on the host and the engine warned of co-tenant stalls on every boot. `HALOGEN_KV_POOL_FIT` still guards the boot, so read the ledger rather than trusting this column | +| `HALOGEN_KV_POOL_POSITIONS` | `401408` | **trimmed 2026-09-13** from the 524,288 the doubling deployed: one full 262k conversation resident beside a ~139k one, or three at 131k, ~11.0 GiB of KV as reserved. Measured against the doubled pool the host went from 7.1 GiB / 310 contiguous 2 MiB blocks to 9.1 / 1396, and the co-tenant stall NOTE no longer prints. `HALOGEN_KV_POOL_FIT` still guards the boot, so read the ledger rather than trusting this column | | `HALOGEN_KV_SLOTS` | `3` | Concurrency cap — **lowered from 4 on 2026-09-12** to the real fleet size. Each slot is 111.0 MiB of O(1) state, so this freed ~111 MiB (held 103.5 → 103.4 GiB). Slots share the ONE pool and **oversubscribe** it (3 × 262144 > 401408), so admission is pool-bound: one full-length conversation plus a ~139k second, or three at 131k. This trims simultaneous requests, not how much context fits | | `HALOGEN_MAX_TOK` | `32768` | upstream's default, kept. The arena is 20.6 GiB here against 12.2 at 16384; it fit alongside the doubled pool only because the reserve dropped to 15, and it fits the trimmed 401,408 pool with room to spare. At 65536 the arena alone exceeds the budget and the server **will not boot** | | `HALOGEN_HOST_RESERVE_GIB` | `15` | **lowered from 24 (vendor default 20) on 2026-09-12** to buy the doubled pool. The only term you own in the device budget (24 → 29.5 GiB, 15 → 38.5 GiB). Against the doubled pool the ceiling was 16 — the engine needed budget ≥ 36.5, and 18 gave 35.5, silently re-clamping; at 401,408 the need is ~33.4 so the ceiling loosens, but a bigger reserve still buys nothing. Matched pair with the pool. It is **not** an allocation: raising it cannot return memory to the host, only take the pool away | @@ -175,17 +175,19 @@ fits unmodified. ### What the reserve actually costs The device budget is not the whole story. Memory position by configuration -(the 09-13 trim added its row after the deploy was measured — see below): +(401,408 is the 2026-09-13 deployed row, read from its boot ledger): | config | held | host left | contiguous 2 MiB blocks | |---|---|---|---| | 262144 / 32768, reserve 24 | 96.3 | 12.1 GiB | 2081 | | 524288 / 16384, reserve 24 | 94.6 | 13.8 GiB | 3742 | -| **524288 / 32768, reserve 15** | **103.5** | **7.1 GiB** | **310** | | 524288 / 32768, reserve **16** | **103.5** | **7.1 GiB** | 293 | +| **401408 / 32768, reserve 15** | **100.0** | **9.1 GiB** | **1396** | At the doubled setting (2026-09-12) the server held 103.5 of 121 GiB and the -engine volunteered a note it had not printed before: +engine volunteered a note it had not printed before (it no longer prints at +the 401,408 trim — 9.1 GiB / 1396 blocks sits below whatever threshold +triggers it): ``` halogen: NOTE this server now holds most of the machine, and 310 contiguous @@ -230,8 +232,10 @@ thing a bigger reserve can do is take the doubling away. then becomes free to rise: `HALOGEN_KV_POOL_POSITIONS=262144` hands back 7.2 GiB from the doubled pool, `HALOGEN_MAX_TOK=16384` hands back 8.4. Both are one-line edits that the fit check will confirm on the next boot. The -2026-09-13 trim to 401,408 was this remedy applied at half strength: ~3.4 GiB -back with `HALOGEN_MAX_TOK` untouched. +2026-09-13 trim to 401,408 was this remedy applied at half strength — +`HALOGEN_MAX_TOK` untouched, and on the host-left metric it bought +2.0 GiB +with a 4.5x improvement in contiguous blocks (310 → 1396), less than the +modelled 3.4 because the engine holds the freed KV as working slack. ### The `WARNING` line only appears when the fit check had to move