frank-large: record the 32k outcome — the doubling is clamped away, docs resynced
Deployed HALOGEN_MAX_TOK=32768 by operator call (upstream default, full prefill speed) and confirmed the prediction from the ledger: kv pool: 524288 positions need ~35.0 GiB ... LOWERING THE POOL TO 262144 (~27.8 GiB) memory: 68.0 weights + 7.2 KV pool + 21.1 working = 96.3 GiB in all so the effective pool is 262144 again and the doubling this task set out to do is not in force. Healthy, completion verified. Left the pool requesting 524288 deliberately: with KV_POOL_FIT on that is a grow-if-RAM-frees-up declaration, not a mistake. Two corrections to what I wrote earlier today: 1. The engine's sizing block, and its "WARNING even 262144 ... host RAM cannot spare it" line, print ONLY when the fit check had to clamp something. The 09-11 baseline boot prints no kv pool: sizing lines at all. So the absence of a warning in old logs was never evidence of headroom. 2. The budget needs a vision term. Actual need is pool + arena + 1.97 GiB of vision tower, not pool + a 1.5 GiB nominal margin. That makes the ceiling for a doubled pool max_tok <= 18,130 rather than 19,115, and it means the 32768/262144 corner this box has run in since 09-11 is ~0.3 GiB OVER its own stated budget -- and has been the whole time. It boots and serves with 12-14 GiB genuinely free because the 24 GiB reserve is generous. A soft floor, not a wall. The arena model also reproduces upstream's three published "on the device" figures exactly (35 / 42 / 41 GiB), confirming they are pool+arena totals at the default max_tok and not pool-only -- which is the misreading that made the doubling look affordable in my first estimate.
This commit is contained in:
1 parent
75ef47c342
commit
65d1be06ac
3 files changed
+72
-59
No files matched your search
@@ -32,7 +32,7 @@ host.
|
||||
| Target | Runtime | Service | Model | Port |
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | FreeToken v0.1.2 | frank-small | batsclamp Huihui-Qwen3.6-35B-A3B-abliterated-FP8 (block-fp8 e4m3 128x128, ~35 GB safetensors), experts host-RAM LRU, 262144 ctx / 4 concurrent, reasoning on | 7770 |
|
||||
| Dankclaw | halogen-flash-server 0.5.8 (rootless Quadlet container) | frank-large | peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b .hgn 115.55 GiB + quality overlay, ~68 GiB pinned, 262144 ctx / 4 slots, pool 524288, vision tower on, greedy for silent clients) | 7770 |
|
||||
| Dankclaw|halogen-flash-server 0.5.8 (rootless Quadlet container)|frank-large|peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b .hgn 115.55 GiB + quality overlay, ~68 GiB pinned, 262144 ctx / 4 slots, pool 524288 requested / 262144 effective, vision tower on, greedy for silent clients)|7770|
|
||||
| Dankclaw | *(retired 2026-09-11)* FastFlowLM v1.0.1 on the XDNA2 NPU | frank-npu | Gemma4-E4B-IT-NPU2 pack (vision + audio). Retired with the `amd_iommu=off` decision: the driver's SVA/PASID bind needs an enabled IOMMU | 7771 |
|
||||
|
||||
History: the 2026-08-17 redo retired the pipeline-parallel split and the
|
||||
@@ -396,8 +396,10 @@ published) and the OpenAI front-end on :8731 published as host **7770** —
|
||||
with `HALOGEN_MODEL_ID=frank-large`, so no client wiring changed.
|
||||
|
||||
Deployed flags: `HALOGEN_CTX=262144`, `HALOGEN_KV_POOL_POSITIONS=524288`
|
||||
(doubled 2026-09-12 to upstream's default — two full-length conversations
|
||||
resident instead of one, ~14.4 GiB), `HALOGEN_KV_SLOTS=4`,
|
||||
(requested 2026-09-12, but **the engine serves 262144** — with
|
||||
`HALOGEN_MAX_TOK=32768` the prefill arena eats the device budget and
|
||||
`HALOGEN_KV_POOL_FIT` clamps the pool to the floor at every boot; set
|
||||
`HALOGEN_MAX_TOK=16384` to make the doubled pool real), `HALOGEN_KV_SLOTS=4`,
|
||||
`HALOGEN_HOST_RESERVE_GIB=24`, `HALOGEN_VISION_TOWER=1`, and
|
||||
`HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next` for the first-start
|
||||
fetch; everything else is upstream's default. The weights stay pinned
|
||||
|
||||
Reference in new issue
Block a user