frank-large: double the KV pool to 524288 positions
HALOGEN_KV_POOL_POSITIONS 262144 -> 524288, which is upstream's own default (2 x HALOGEN_CTX). The old pool was the binding constraint on concurrency, not context: a request reserves prompt + max_tokens against the pool, so with pool == CTX four concurrent 65k chats were contending for one full-length conversation's worth of positions. Two full-length conversations, or four at 131k, now stay resident. Cost is ~7.2 GiB more device RAM (measured 28 KiB/position), taken from the page cache the 47.7 GiB n-gram lookup table reads through. Live startup ledger for the prior config: 68.0 weights + 7.2 pool + 21.1 working = 96.3 GiB held, 12.1 GiB left on the host. The doubled pool puts the remainder near 5 GiB against a ~4.3 GiB co-tenant working set, and cache at ~69 GiB -- still above the lookup table, so the table can stay hot. Compensating lever if long prompts start crawling: HALOGEN_MAX_TOK=16384, which frees 8.3 GiB of prefill arena for ~8% of prefill throughput. Docs updated to match, including the admission-trap paragraph, which was only true while pool == CTX.
This commit is contained in:
1 parent
dd44223f3f
commit
5ace29e7a0
3 files changed
+53
-19
No files matched your search
@@ -32,7 +32,7 @@ host.
|
||||
| Target | Runtime | Service | Model | Port |
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | FreeToken v0.1.2 | frank-small | batsclamp Huihui-Qwen3.6-35B-A3B-abliterated-FP8 (block-fp8 e4m3 128x128, ~35 GB safetensors), experts host-RAM LRU, 262144 ctx / 4 concurrent, reasoning on | 7770 |
|
||||
| Dankclaw | halogen-flash-server 0.5.8 (rootless Quadlet container) | frank-large | peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b .hgn 115.55 GiB + quality overlay, ~68 GiB pinned, 262144 ctx / 4 slots, pool 262144, vision tower on, greedy for silent clients) | 7770 |
|
||||
| Dankclaw | halogen-flash-server 0.5.8 (rootless Quadlet container) | frank-large | peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b .hgn 115.55 GiB + quality overlay, ~68 GiB pinned, 262144 ctx / 4 slots, pool 524288, vision tower on, greedy for silent clients) | 7770 |
|
||||
| Dankclaw | *(retired 2026-09-11)* FastFlowLM v1.0.1 on the XDNA2 NPU | frank-npu | Gemma4-E4B-IT-NPU2 pack (vision + audio). Retired with the `amd_iommu=off` decision: the driver's SVA/PASID bind needs an enabled IOMMU | 7771 |
|
||||
|
||||
History: the 2026-08-17 redo retired the pipeline-parallel split and the
|
||||
@@ -395,8 +395,9 @@ both halves — the engine on container loopback :8730 (unauthenticated, never
|
||||
published) and the OpenAI front-end on :8731 published as host **7770** —
|
||||
with `HALOGEN_MODEL_ID=frank-large`, so no client wiring changed.
|
||||
|
||||
Deployed flags: `HALOGEN_CTX=262144`, `HALOGEN_KV_POOL_POSITIONS=262144`
|
||||
(conservative for a shared host), `HALOGEN_KV_SLOTS=4`,
|
||||
Deployed flags: `HALOGEN_CTX=262144`, `HALOGEN_KV_POOL_POSITIONS=524288`
|
||||
(doubled 2026-09-12 to upstream's default — two full-length conversations
|
||||
resident instead of one, ~14.4 GiB), `HALOGEN_KV_SLOTS=4`,
|
||||
`HALOGEN_HOST_RESERVE_GIB=24`, `HALOGEN_VISION_TOWER=1`, and
|
||||
`HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next` for the first-start
|
||||
fetch; everything else is upstream's default. The weights stay pinned
|
||||
|
||||
Reference in new issue
Block a user