command-center/agent/managed-skills/halogen-kv-pool-fit/SKILL.md

name, description

name	description
halogen-kv-pool-fit	Resize halogen-flash-server's KV pool or prefill arena without getting silently clamped or OOMing the boot: pool, arena and vision share one device budget, HALOGEN_HOST_RESERVE_GIB only scales the fit-check ceiling and is NOT an allocation, HALOGEN_KV_POOL_FIT only ever lowers, and the sizing warning prints only when a clamp happened.

Sizing a halogen-flash-server KV pool

HALOGEN_KV_POOL_FIT=1 (default) computes a device budget at startup and only ever lowers the configured HALOGEN_KV_POOL_POSITIONS. It never raises it, and never below the context.

device budget = host RAM - resident weights - HALOGEN_HOST_RESERVE_GIB
fit check     = needs budget >= (KV + arena + vision) + 1.5 GiB margin
KV            = 28 KiB/position
arena         = 3.80 GiB + 8.40 GiB x (HALOGEN_MAX_TOK / 16384)   # measured, linear
vision        = 1.97 GiB when HALOGEN_VISION_TOWER=1

Prefer the engine's printed requirement over the model: a clamped boot states it directly, e.g. 524288 positions need ~35.0 GiB (plus 1.5 of margin).

The reserve is NOT an allocation

HALOGEN_HOST_RESERVE_GIB is a term in the fit-check inequality, nothing more. Proved by A/B: reserve 15 and reserve 16 both held 103.5 GiB with 7.1 GiB left for everything else — identical footprint. Actual footprint is weights + KV + arena + vision, none of which the reserve touches.

So the knob has only two regimes: below the pool's requirement it changes nothing observable; above it, it silently deletes the pool. Raising the reserve can never give memory back to co-tenants. To free host RAM, cut the pool or the arena:

change	RAM handed back
HALOGEN_KV_POOL_POSITIONS 524288 → 262144	7.2 GiB
HALOGEN_MAX_TOK 32768 → 16384	8.4 GiB

Lowering the reserve DOES work — it buys fit margin for a bigger pool or arena. On a 121.2 GiB host with 67.7 GiB weights, for 524288 + max_tok 32768 (needs 36.5):

reserve	budget	vs 36.5	outcome
24	29.5	−7.0	clamps
18	35.5	−1.0	clamps — looks like a healthy deploy
17	36.5	+0.02	knife edge, inside the noise of the weights term
16	37.5	+1.0	fits, zero benefit over 15
15	38.5	+2.0	deployed

Ceiling here is 16; pick the lowest reserve that fits, not the highest.

Gotchas

• The sizing block and its WARNING print ONLY after a clamp. A fitting boot prints no kv pool: lines at all, so absence of a warning in old logs is NOT evidence of headroom.
• deploy.sh health-passes a clamped pool. Container active, /health 200, in ~12 s. Only the ledger proves the pool size.
• Beware the false positive when grepping for a clamp: the NOTE line contains the sentence "Lowering HALOGEN_KV_POOL_POSITIONS gives some back", which matches LOWERING case-insensitively. Grep LOWERING THE POOL / cannot spare it.
• free/MemAvailable cannot see this server. Weights are pinned and the kernel counts locked pages as reclaimable page cache — MemAvailable read ~74 GiB while the engine held 103.5 of 121.2 GiB. Read the engine's lines instead.
• Host RAM is rounded DOWN when displayed. 121 is MemTotal 127104060 kB = 121.2159; subtracting displayed figures gives 29.3 instead of the true 29.5.
• Vendor "on the device" figures are pool + arena totals, not pool. Reading them as pool-only makes a doubling look affordable.
• A 2.0 GiB firmware iGPU carve-out is reported by the ledger and is not free memory.
• HALOGEN_MAX_TOK is the arena / per-request generation budget (thinking included) — do not conflate with HALOGEN_CTX or the pool. HALOGEN_KV_SLOTS costs only ~111 MiB/slot, irrelevant to sizing.

Three failure modes, worst last

1. Silent clamp — need exceeds budget; pool drops to the floor, service healthy. Caught only in the ledger.
2. Boot OOM — enough HALOGEN_MAX_TOK and the arena alone beats the budget: dmalloc: FAILED requesting 0.250 GiB after 39.970 GiB in 566 allocations (out of memory). Container stays active / Up (starting) forever; health never passes.
3. Minutes-long stall — reserve cut so far that the engine holds most of a shared host. It prints its own note: neighbours and this server "can stall for minutes at a time at full CPU with no disk activity and no output, which is not a crash and needs no restart." Looks like a hang, invisible to free, hits co-tenants too.

Verify

podman logs frank-large 2>&1 | grep -aiE 'for the device|reserving the KV|ONE pool of|memory: [0-9.]+ GiB of weights|left for everything|LOWERING THE POOL|cannot spare it|listening on 127'

Expect the budget line, reserving the KV pool, <N> positions, ONE pool of <N> positions, zero clamp matches, a host memory left for everything else figure, then listening on 127.0.0.1:8730. (A failing 7770 publish on restart is a stale-netns race — self-heals; podman network reload --all forces it.)

Then a real completion — /health 200 alone proves nothing:

curl -s http://127.0.0.1:7770/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"frank-large","messages":[{"role":"user","content":"Reply with exactly: OK"}],"max_tokens":48,"reasoning_effort":"low"}'

Expect finish_reason: stop. A silent client decodes greedy — this server ships no sampler defaults.

Reference points (121.2 GiB host, 67.7 GiB weights, vision on)

pool	max_tok	reserve	need	budget	outcome
262144	32768	24	29.8	29.5	boots (0.3 over), no sizing lines
524288	16384	24	28.6	29.5	fits clean
524288	32768	24	37.0	29.5	clamped, deploy green
524288	65536	24	46.5	29.5	OOM, will not boot
524288	32768	15	37.0	38.5	both features, deployed
524288	32768	16	37.0	37.5	fits, footprint identical

Procedure

Compute need vs budget with unrounded RAM, then confirm against the engine's own printed requirement. After each deploy read the ledger for actual pool, clamp matches, and the host-left line — and A/B it: if two settings report the same held, the knob you turned doesn't do what its name says. Keep reserve and pool documented as a matched pair. Roll back with git revert (never --no-verify).