Dankclaw's endpoints are re-split. frank-large is the halogen-flash-server
quadlet again -- the 2026-09-13 unit file was never touched by the two-day
ik_llama.cpp interlude, so the runtime was still in the repo and its image is
still in local storage -- retuned to the shape asked for: HALOGEN_CTX 200192
with a 400,384-position pool and two slots, exactly two 200k conversations.
frank-small is new: mainline llama.cpp on the RTX 3090 serving
ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF at IQ3_S, using the -mtp build (it carries
blk.64.nextn.*, which is what lets --spec-type draft-mtp run on the main
model's own weights instead of a second 12 GiB copy) plus the BF16 vision
projector, as one 131,072-token slot with f16 KV on port 7771.
The host is shared now, and that is what the halogen retune pays for:
HALOGEN_MAX_TOK 24576 (arena 20.6 -> 16.4 GiB, ~4-5% prefill) and the vision
tower dropped (1.97 GiB) hand ~6 GiB back to the host, and the reserve rises to
20 to keep the fit check honest. Vision moved wholesale to frank-small.
Retired to scripts/.AGENTTRASH/: the ik trio (env, service, serve script), the
Qwen-Sharp template only that runtime read, and four benchmark harnesses bound
to retired runtimes. Because retiring a file makes any test that greps it pass
by absence, the seams assert the archive exists AND the live path does not.
Predicted ~10.8 GiB KV / ~33.4 need / ~10.5 host-left; the boot ledger
printed ~11.0 / ~31.5 / 100.0 GiB held with 9.1 GiB + 1396 contiguous
2 MiB blocks left. Stall NOTE no longer prints at this footprint. Also
documents the ~45-60 s first-request PING silence after a restart (the
healthcheck's HealthStartPeriod covers it; do not restart the unit).
The 09-12 doubled pool fit only at reserve 15, leaving 7.1 GiB / 310
contiguous 2 MiB blocks for the ~28 co-tenant containers and making the
engine print its minutes-long-stall warning on every boot. 401,408
positions (~10.8 GiB KV) hands ~3.4 GiB back while keeping MAX_TOK=32768
and reserve 15: modelled need ~33.4 GiB vs budget 38.5, so the fit check
should not clamp. Capacity becomes one full 262k conversation resident
beside a ~139k one, or three concurrent 131k chats (three slots).
Docs: AGENTS.md fleet table + halogen section, runbook flags table /
co-tenancy tense / admission-arithmetic paragraph updated for the trim.
Concurrency cap trimmed to the real fleet size. Each slot is 111.0 MiB of
O(1) state, so this hands ~111 MiB back on a host sitting at 7.1 GiB left
for everything else.
Note the slots already oversubscribed the pool (4 x 262144 > 524288), so
admission was pool-bound regardless: at most two full-length conversations,
or four at 131k. This does not change how much context fits -- it caps
simultaneous requests at three, and a fourth now queues.
Tested 16 live. Pool stayed 524288 (no clamp), server held 103.5 GiB and
reported "7.1 GiB left for everything else" at BOTH 15 and 16 — identical
footprint. HALOGEN_HOST_RESERVE_GIB feeds the fit-check budget only; it is not
an allocation, so raising it cannot give RAM back to the co-tenants. Its sole
observable effect above 16 is to take the KV pool away.
16 therefore cost half the fit margin (1.0 vs 2.0 against the engine's 36.5 GiB
requirement) and bought nothing. 15 is strictly better at the same cost.
Recorded the ceiling (16) and the real air levers in the unit comment: pool
262144 frees 7.2 GiB, max_tok 16384 frees 8.4.
Operator asked for 18. 18 is not available: the engine's own fit-check line,
read from a clamped boot rather than modelled, requires budget >= 36.5 GiB to
keep 524288 at max_tok 32768 ("524288 positions need ~35.0 GiB (plus 1.5 of
margin)"). budget(18) = 35.5, so 18 silently un-doubles the pool we just paid
the reserve for. 17 fits by 0.02, inside the noise of the 67.7 resident-weights
term. 16 is the largest reserve that survives the check with real margin:
37.5 budget vs 36.5 required.
Recorded the ceiling and both remedies in the unit comment. The reserve is now
spent, so a future stall is answered by the pool or the arena, not here.
Ledger confirms the doubled pool is actually reserved (524288, 13312.0 MiB)
at max_tok 32768 with no clamp and no WARNING: 38.5 budget vs 37.0 needed.
Recorded the cost honestly. The server now holds 103.5 of 121.2 GiB and the
engine reports only 310 contiguous 2 MiB blocks / 7.1 GiB left for everything
else (24-reserve baseline: 2081 blocks / 12.1 GiB). All 28 co-tenants stayed
up, but the engine volunteers a new note: when memory runs out, frank-large
AND its neighbours "can stall for minutes at a time at full CPU with no disk
activity and no output, which is not a crash and needs no restart". That is
the least diagnosable of the three failure modes seen today, and `free` hides
it — MemAvailable still reads ~74 GiB because the pinned weights count as
reclaimable page cache.
Sizing rule now expresses the budget as a function of reserve, shows both
endpoints (24 -> 29.5, 15 -> 38.5), and the max_tok ceiling is labelled as
reserve-conditional (<= 18,130 at 24; <= 35,700 at 15).
Neither lever alone could give both features. At reserve 24 the device budget
is 29.5 GiB and max_tok 32768's 37.0 GiB need always lost the doubling to the
fit-check clamp; at max_tok 16384 the doubling survived but prefill paid 8-9%.
Dropping the reserve to 15 moves the budget to 38.5:
121.2 host - 67.7 resident weights - 15 reserve = 38.5
need = 14.4 pool + 20.6 arena + 1.97 vision = 37.0 -> 1.5 GiB spare
So the payment is page cache for the n-gram lookup table rather than either
feature. 15 GiB, against a table of 47.7 GiB read through the file cache.
Two reasons this is a survivable risk rather than a cliff:
- The engine's own ledger reported only 12-17 GiB left at the previous 24
setting, so the declared reserve was never held literally; 15 may cost
less real cache than it reads as.
- The failure mode is legible: long prompts fault heavily against a busy
disk, and raising the reserve is the documented remedy. Not the silent
clamp, and not the 65536 no-boot mode.
1.5 GiB of headroom is thin and deliberate. Documented in the unit as a
matched pair with the pool: raise the reserve back to 24 and the doubling
quietly goes away again.
Deployed HALOGEN_MAX_TOK=32768 by operator call (upstream default, full prefill
speed) and confirmed the prediction from the ledger:
kv pool: 524288 positions need ~35.0 GiB ... LOWERING THE POOL TO 262144 (~27.8 GiB)
memory: 68.0 weights + 7.2 KV pool + 21.1 working = 96.3 GiB in all
so the effective pool is 262144 again and the doubling this task set out to do
is not in force. Healthy, completion verified. Left the pool requesting 524288
deliberately: with KV_POOL_FIT on that is a grow-if-RAM-frees-up declaration,
not a mistake.
Two corrections to what I wrote earlier today:
1. The engine's sizing block, and its "WARNING even 262144 ... host RAM cannot
spare it" line, print ONLY when the fit check had to clamp something. The
09-11 baseline boot prints no kv pool: sizing lines at all. So the absence
of a warning in old logs was never evidence of headroom.
2. The budget needs a vision term. Actual need is pool + arena + 1.97 GiB of
vision tower, not pool + a 1.5 GiB nominal margin. That makes the ceiling
for a doubled pool max_tok <= 18,130 rather than 19,115, and it means the
32768/262144 corner this box has run in since 09-11 is ~0.3 GiB OVER its own
stated budget -- and has been the whole time. It boots and serves with
12-14 GiB genuinely free because the 24 GiB reserve is generous. A soft
floor, not a wall.
The arena model also reproduces upstream's three published "on the device"
figures exactly (35 / 42 / 41 GiB), confirming they are pool+arena totals at
the default max_tok and not pool-only -- which is the misreading that made the
doubling look affordable in my first estimate.
Operator call. Prediction from the measured model: 524288 positions cost
35.0 GiB against a 29.5 budget, so HALOGEN_KV_POOL_FIT clamps the pool to
262144 (27.8). Rollback is a single revert; 16384 was the config that made
the doubling real.
Raising HALOGEN_MAX_TOK to 65536 did not shrink the pool, it OOMed the boot:
the arena alone (37.3 GiB) exceeds the 29.5 GiB device budget, so even the
minimum pool faulted and dmalloc failed at 39.97 GiB. Rolled back to 16384,
which restored the 524288 pool and a clean completion.
The three measured arenas are exactly linear in the flag — 3.80 GiB base plus
8.40 GiB per 16384 tokens, predicting all three to 0.1 GiB. That gives a real
sizing rule instead of guesswork, and it says the deployed 16384 is the
CEILING, not a conservative choice: 29.5 - 1.5 margin - 14.4 pool leaves 13.6
for the arena, i.e. max_tok <= 19,115.
The corner check falls out of the same model: set max_tok back to 32768 and
only 7.4 GiB remains for the pool, ~269k positions, which is precisely the
262144 config this host ran before. The two corners of the feasible box are
the only two points it has ever served.
No functional change — the quadlet edit is comments only; the running service
is already at 16384 from the rollback deploy (258cccb).
Operator call, measuring what the prefill arena actually costs against
upstream's do-not-raise warning. Rollback is one revert: the settled
config was 16384, which made the 524288 pool fit at 26.6 GiB.
The previous commit's doubling was a silent no-op. HALOGEN_KV_POOL_FIT
measured the device budget at startup and lowered the pool straight back
to 262144, while the deploy still reported healthy in 12s.
The engine's arithmetic: 121 GiB host - 67.7 resident weights - 24.0
reserved for the n-gram page cache and the OS = 29.5 GiB of device
budget, and that single budget carries the KV pool AND the prefill arena.
At HALOGEN_MAX_TOK=32768 the arena is ~20.6 GiB, so 524288 positions
need 35.0 GiB against 29.5 available.
HALOGEN_MAX_TOK=16384 halves the arena to ~10.5 GiB, bringing the
524288 need to ~24.9 GiB, which fits with ~3 GiB of margin. Cost is
about 8-9% of prefill throughput, per upstream. This is the engine's own
prescribed fix, quoted verbatim in its refusal message.
Lowering HALOGEN_CTX was the other option the engine offered; rejected
because both clients declare 262144 as the context window, so it would
break the client contract rather than pay for the pool.
HALOGEN_KV_POOL_POSITIONS 262144 -> 524288, which is upstream's own
default (2 x HALOGEN_CTX). The old pool was the binding constraint on
concurrency, not context: a request reserves prompt + max_tokens against
the pool, so with pool == CTX four concurrent 65k chats were contending
for one full-length conversation's worth of positions. Two full-length
conversations, or four at 131k, now stay resident.
Cost is ~7.2 GiB more device RAM (measured 28 KiB/position), taken from
the page cache the 47.7 GiB n-gram lookup table reads through. Live
startup ledger for the prior config: 68.0 weights + 7.2 pool + 21.1
working = 96.3 GiB held, 12.1 GiB left on the host. The doubled pool
puts the remainder near 5 GiB against a ~4.3 GiB co-tenant working set,
and cache at ~69 GiB -- still above the lookup table, so the table can
stay hot. Compensating lever if long prompts start crawling:
HALOGEN_MAX_TOK=16384, which frees 8.3 GiB of prefill arena for ~8%
of prefill throughput.
Docs updated to match, including the admission-trap paragraph, which was
only true while pool == CTX.
Verbose engine logging is the vendor's first-line evidence for load,
allocation and admission questions; shipping it on means an incident
report already carries the full startup account instead of needing a
restart-into-debug. HALOGEN_DMALLOC_LOG stays off (every alloc >64MB is
steady-state noise on a shared host). Doc updated with the rootless
podman-logs-vs-journalctl retrieval note.
frank-large moves from the kingjones30/ROCmFPX llama.cpp build to
peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b
.hgn 115.55 GiB + quality overlay). Upstream measures this engine at
roughly 4x the prefill throughput of the llama.cpp family on the same
gfx1151 silicon, and prefill dominates any prompt with real context.
Shape: a rootless Quadlet container (dankclaw/frank-large/quadlet/),
engine on container loopback :8730 (never published), OpenAI front-end
:8731 published as host 7770, HALOGEN_MODEL_ID=frank-large so no client
renamed anything. Context 262144, KV pool 262144 positions (~7.2 GiB,
conservative for a shared 121 GiB host), 4 slots, host reserve 24,
vision tower on, weights pinned, all other flags at vendor defaults.
Weights are fetched by HALOGEN_DOWNLOAD on first start into a writable
/home/dank/models/halogen.
The llama.cpp era (unit, env, serve script) is staged in .AGENTTRASH/
rather than deleted; the STRIX payload and the ROCmFPX build stay on
disk as the rollback target. deploy_llm_large becomes a user-quadlet
handler: it retires the old ROOT unit (moving its checkout symlink into
the home trash, never rm) and gates the first start on the download
rather than reporting a failure for a fetch that is working exactly as
designed.
frank-npu is retired in the same change. The kernel cmdline moves from
iommu=pt to amd_iommu=off for the 13-16% of prefill upstream measured,
and that takes the XDNA2 NPU with it: without PASID/SVA the driver
cannot open /dev/accel/accel0 (err=-19). Its five dependent routes are
repointed (Hermes vision -> frank-large, Hermes title generation ->
frank-small, harness vision -> frank-large, tiny -> frank-small, task ->
frank-large:medium) and its provider blocks are removed from both the
harness config and Hermes's live config. Audio input leaves the fleet
with no replacement; image input survives on halogen's vision tower.
Docs: new halogen-flash-server runbook, new IOMMU decision record, and
frank-npu rewritten as a retirement record. AGENTS.md topology, runtime
sections, benchmarking policy (frank-large is a user unit now) and the
harness model-config section are corrected.
Frank