Ledger confirms the doubled pool is actually reserved (524288, 13312.0 MiB)
at max_tok 32768 with no clamp and no WARNING: 38.5 budget vs 37.0 needed.
Recorded the cost honestly. The server now holds 103.5 of 121.2 GiB and the
engine reports only 310 contiguous 2 MiB blocks / 7.1 GiB left for everything
else (24-reserve baseline: 2081 blocks / 12.1 GiB). All 28 co-tenants stayed
up, but the engine volunteers a new note: when memory runs out, frank-large
AND its neighbours "can stall for minutes at a time at full CPU with no disk
activity and no output, which is not a crash and needs no restart". That is
the least diagnosable of the three failure modes seen today, and `free` hides
it — MemAvailable still reads ~74 GiB because the pinned weights count as
reclaimable page cache.
Sizing rule now expresses the budget as a function of reserve, shows both
endpoints (24 -> 29.5, 15 -> 38.5), and the max_tok ceiling is labelled as
reserve-conditional (<= 18,130 at 24; <= 35,700 at 15).
Deployed HALOGEN_MAX_TOK=32768 by operator call (upstream default, full prefill
speed) and confirmed the prediction from the ledger:
kv pool: 524288 positions need ~35.0 GiB ... LOWERING THE POOL TO 262144 (~27.8 GiB)
memory: 68.0 weights + 7.2 KV pool + 21.1 working = 96.3 GiB in all
so the effective pool is 262144 again and the doubling this task set out to do
is not in force. Healthy, completion verified. Left the pool requesting 524288
deliberately: with KV_POOL_FIT on that is a grow-if-RAM-frees-up declaration,
not a mistake.
Two corrections to what I wrote earlier today:
1. The engine's sizing block, and its "WARNING even 262144 ... host RAM cannot
spare it" line, print ONLY when the fit check had to clamp something. The
09-11 baseline boot prints no kv pool: sizing lines at all. So the absence
of a warning in old logs was never evidence of headroom.
2. The budget needs a vision term. Actual need is pool + arena + 1.97 GiB of
vision tower, not pool + a 1.5 GiB nominal margin. That makes the ceiling
for a doubled pool max_tok <= 18,130 rather than 19,115, and it means the
32768/262144 corner this box has run in since 09-11 is ~0.3 GiB OVER its own
stated budget -- and has been the whole time. It boots and serves with
12-14 GiB genuinely free because the 24 GiB reserve is generous. A soft
floor, not a wall.
The arena model also reproduces upstream's three published "on the device"
figures exactly (35 / 42 / 41 GiB), confirming they are pool+arena totals at
the default max_tok and not pool-only -- which is the misreading that made the
doubling look affordable in my first estimate.
Raising HALOGEN_MAX_TOK to 65536 did not shrink the pool, it OOMed the boot:
the arena alone (37.3 GiB) exceeds the 29.5 GiB device budget, so even the
minimum pool faulted and dmalloc failed at 39.97 GiB. Rolled back to 16384,
which restored the 524288 pool and a clean completion.
The three measured arenas are exactly linear in the flag — 3.80 GiB base plus
8.40 GiB per 16384 tokens, predicting all three to 0.1 GiB. That gives a real
sizing rule instead of guesswork, and it says the deployed 16384 is the
CEILING, not a conservative choice: 29.5 - 1.5 margin - 14.4 pool leaves 13.6
for the arena, i.e. max_tok <= 19,115.
The corner check falls out of the same model: set max_tok back to 32768 and
only 7.4 GiB remains for the pool, ~269k positions, which is precisely the
262144 config this host ran before. The two corners of the feasible box are
the only two points it has ever served.
No functional change — the quadlet edit is comments only; the running service
is already at 16384 from the rollback deploy (258cccb).
The first draft of these notes estimated the doubled pool at 103.5 GiB
held / ~5 GiB host remainder. Both numbers were wrong — they costed the
KV alone and ignored that the arena shrinks when MAX_TOK drops. Measured:
pool 262144 / max_tok 32768 -> 96.3 GiB held, 12.1 GiB host left
pool 524288 / max_tok 16384 -> 94.6 GiB held, 13.8 GiB host left
The doubled pool is 1.7 GiB cheaper overall. Also records the failed
first attempt verbatim, since the silent-revert failure mode is the thing
a future operator most needs to recognise, and adds the HALOGEN_MAX_TOK
row to the deployed-flags table (the "baked 32768" line it overrode was
stale).
HALOGEN_KV_POOL_POSITIONS 262144 -> 524288, which is upstream's own
default (2 x HALOGEN_CTX). The old pool was the binding constraint on
concurrency, not context: a request reserves prompt + max_tokens against
the pool, so with pool == CTX four concurrent 65k chats were contending
for one full-length conversation's worth of positions. Two full-length
conversations, or four at 131k, now stay resident.
Cost is ~7.2 GiB more device RAM (measured 28 KiB/position), taken from
the page cache the 47.7 GiB n-gram lookup table reads through. Live
startup ledger for the prior config: 68.0 weights + 7.2 pool + 21.1
working = 96.3 GiB held, 12.1 GiB left on the host. The doubled pool
puts the remainder near 5 GiB against a ~4.3 GiB co-tenant working set,
and cache at ~69 GiB -- still above the lookup table, so the table can
stay hot. Compensating lever if long prompts start crawling:
HALOGEN_MAX_TOK=16384, which frees 8.3 GiB of prefill arena for ~8%
of prefill throughput.
Docs updated to match, including the admission-trap paragraph, which was
only true while pool == CTX.
Verbose engine logging is the vendor's first-line evidence for load,
allocation and admission questions; shipping it on means an incident
report already carries the full startup account instead of needing a
restart-into-debug. HALOGEN_DMALLOC_LOG stays off (every alloc >64MB is
steady-state noise on a shared host). Doc updated with the rootless
podman-logs-vs-journalctl retrieval note.
frank-large moves from the kingjones30/ROCmFPX llama.cpp build to
peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b
.hgn 115.55 GiB + quality overlay). Upstream measures this engine at
roughly 4x the prefill throughput of the llama.cpp family on the same
gfx1151 silicon, and prefill dominates any prompt with real context.
Shape: a rootless Quadlet container (dankclaw/frank-large/quadlet/),
engine on container loopback :8730 (never published), OpenAI front-end
:8731 published as host 7770, HALOGEN_MODEL_ID=frank-large so no client
renamed anything. Context 262144, KV pool 262144 positions (~7.2 GiB,
conservative for a shared 121 GiB host), 4 slots, host reserve 24,
vision tower on, weights pinned, all other flags at vendor defaults.
Weights are fetched by HALOGEN_DOWNLOAD on first start into a writable
/home/dank/models/halogen.
The llama.cpp era (unit, env, serve script) is staged in .AGENTTRASH/
rather than deleted; the STRIX payload and the ROCmFPX build stay on
disk as the rollback target. deploy_llm_large becomes a user-quadlet
handler: it retires the old ROOT unit (moving its checkout symlink into
the home trash, never rm) and gates the first start on the download
rather than reporting a failure for a fetch that is working exactly as
designed.
frank-npu is retired in the same change. The kernel cmdline moves from
iommu=pt to amd_iommu=off for the 13-16% of prefill upstream measured,
and that takes the XDNA2 NPU with it: without PASID/SVA the driver
cannot open /dev/accel/accel0 (err=-19). Its five dependent routes are
repointed (Hermes vision -> frank-large, Hermes title generation ->
frank-small, harness vision -> frank-large, tiny -> frank-small, task ->
frank-large:medium) and its provider blocks are removed from both the
harness config and Hermes's live config. Audio input leaves the fleet
with no replacement; image input survives on halogen's vision tower.
Docs: new halogen-flash-server runbook, new IOMMU decision record, and
frank-npu rewritten as a retirement record. AGENTS.md topology, runtime
sections, benchmarking policy (frank-large is a user unit now) and the
harness model-config section are corrected.
Frank