frank-large: trim KV pool 524288 -> 401408 for co-tenant headroom
The 09-12 doubled pool fit only at reserve 15, leaving 7.1 GiB / 310 contiguous 2 MiB blocks for the ~28 co-tenant containers and making the engine print its minutes-long-stall warning on every boot. 401,408 positions (~10.8 GiB KV) hands ~3.4 GiB back while keeping MAX_TOK=32768 and reserve 15: modelled need ~33.4 GiB vs budget 38.5, so the fit check should not clamp. Capacity becomes one full 262k conversation resident beside a ~139k one, or three concurrent 131k chats (three slots). Docs: AGENTS.md fleet table + halogen section, runbook flags table / co-tenancy tense / admission-arithmetic paragraph updated for the trim.
This commit is contained in:
1 parent
025216fc84
commit
91406af2a1
3 files changed
+76
-63
No files matched your search
@@ -32,7 +32,7 @@ host.
|
||||
| Target | Runtime | Service | Model | Port |
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | FreeToken v0.1.2 | frank-small | batsclamp Huihui-Qwen3.6-35B-A3B-abliterated-FP8 (block-fp8 e4m3 128x128, ~35 GB safetensors), experts host-RAM LRU, 262144 ctx / 4 concurrent, reasoning on | 7770 |
|
||||
|Dankclaw|halogen-flash-server 0.5.8 (rootless Quadlet container)|frank-large|peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b .hgn 115.55 GiB + quality overlay, ~68 GiB pinned, 262144 ctx / 3 slots, pool 524288 / max_tok 32768 / reserve 15, vision tower on, greedy for silent clients)|7770|
|
||||
|Dankclaw|halogen-flash-server 0.5.8 (rootless Quadlet container)|frank-large|peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b .hgn 115.55 GiB + quality overlay, ~68 GiB pinned, 262144 ctx / 3 slots, pool 401408 / max_tok 32768 / reserve 15, vision tower on, greedy for silent clients)|7770|
|
||||
| Dankclaw | *(retired 2026-09-11)* FastFlowLM v1.0.1 on the XDNA2 NPU | frank-npu | Gemma4-E4B-IT-NPU2 pack (vision + audio). Retired with the `amd_iommu=off` decision: the driver's SVA/PASID bind needs an enabled IOMMU | 7771 |
|
||||
|
||||
History: the 2026-08-17 redo retired the pipeline-parallel split and the
|
||||
@@ -395,24 +395,28 @@ both halves — the engine on container loopback :8730 (unauthenticated, never
|
||||
published) and the OpenAI front-end on :8731 published as host **7770** —
|
||||
with `HALOGEN_MODEL_ID=frank-large`, so no client wiring changed.
|
||||
|
||||
Deployed flags: `HALOGEN_CTX=262144`, `HALOGEN_KV_POOL_POSITIONS=524288`
|
||||
(**in force** since 2026-09-12 — two full-length conversations resident, or
|
||||
four at 131k), `HALOGEN_KV_SLOTS=3` (lowered from 4 on 2026-09-12 to the real
|
||||
fleet size; 111 MiB of O(1) state each), `HALOGEN_MAX_TOK=32768`,
|
||||
`HALOGEN_HOST_RESERVE_GIB=15`, `HALOGEN_VISION_TOWER=1`, and
|
||||
Deployed flags: `HALOGEN_CTX=262144`, `HALOGEN_KV_POOL_POSITIONS=401408`
|
||||
(**trimmed 2026-09-13** from the 524,288 the 09-12 doubling deployed — one
|
||||
full 262k conversation resident beside a ~139k one, or three at 131k; hands
|
||||
~3.4 GiB back for the ~28 co-tenants after the doubled pool left only
|
||||
~7.1 GiB and the engine warned of co-tenant stalls), `HALOGEN_KV_SLOTS=3`
|
||||
(lowered from 4 on 2026-09-12 to the real fleet size; 111 MiB of O(1) state
|
||||
each), `HALOGEN_MAX_TOK=32768`, `HALOGEN_HOST_RESERVE_GIB=15`,
|
||||
`HALOGEN_VISION_TOWER=1`, and
|
||||
`HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next` for the first-start
|
||||
fetch; everything else is upstream's default. **Pool and reserve are a matched
|
||||
pair:** the budget is `121.2 host − 67.7 weights − reserve`, and that one
|
||||
budget carries the KV pool, the prefill arena and the 1.97 GiB vision tower
|
||||
together. Raise the reserve past **16** and `HALOGEN_KV_POOL_FIT` silently
|
||||
clamps the pool to 262144 while the deploy still reports healthy (the engine
|
||||
needs budget ≥ 36.5; reserve 18 yields 35.5). The reserve is **not an
|
||||
allocation** — 15 and 16 were both measured holding 103.5 GiB with 7.1 GiB left
|
||||
on the host, so a bigger reserve cannot free memory for co-tenants, it can only
|
||||
remove the pool. To free host RAM, cut `HALOGEN_KV_POOL_POSITIONS` (262144
|
||||
hands back 7.2 GiB) or `HALOGEN_MAX_TOK` (16384 hands back 8.4). At 15 the
|
||||
server holds 103.5 of 121 GiB, leaving ~7.1 GiB on the host, and the engine
|
||||
warns that when it runs out frank-large and its neighbours can stall for
|
||||
together. The reserve is **not an allocation** — 15 and 16 were both measured
|
||||
holding 103.5 GiB with 7.1 GiB left on the host, so a bigger reserve cannot
|
||||
free memory for co-tenants, it can only remove the pool. Against the old
|
||||
524,288 pool the reserve ceiling was 16 (engine needs budget ≥ 36.5; reserve
|
||||
18 yields 35.5 and silently clamps); at 401,408 the need is ~33.4 so the
|
||||
ceiling loosens, but a bigger reserve still buys nothing. To free host RAM,
|
||||
cut `HALOGEN_KV_POOL_POSITIONS` (262144 hands back another 3.6 GiB) or
|
||||
`HALOGEN_MAX_TOK` (16384 hands back 8.4). At the trimmed pool expect roughly
|
||||
10.5 GiB left on the host — confirm on the boot ledger, not here. If a
|
||||
co-tenant still runs it dry, frank-large and its neighbours stall for
|
||||
minutes at full CPU with no disk activity — not a crash, and invisible to
|
||||
`free`, which still reads ~74 GiB available. The weights stay pinned (~68 GiB)
|
||||
and the kernel counts those locked pages as reclaimable page cache, so `free`,
|
||||
|
||||
Reference in new issue
Block a user