Frank
4b82f35354
Tracked the harness artifacts accumulated on blankpad (blank@), matching the existing convention for the --home-dank-* memory stores and the 36 already-tracked managed-skills. managed-skills (9): authentik-proxy-provider-app-link, dankclaw-guard-chain-layer-diagnosis, e2e-arr-rawip-reachability-triage, halogen-kv-pool-fit, lan-client-split-dns-fix, noctalia-reference, rootless-quadlet-log-retrieval, ssh-key-bootstrap-no-sshpass, swapfile-resize-safe-ordering. memories (2 stores): --home-blank-.omp-- (dotfiles_git, handy_lifecycle, input_permissions) and --home-blank-Projects-serverAppConfig-- (frank_large_quadlet). Secret sweep before staging: no PEM blocks, no credential literals. Content is fleet topology/IPs only, same class as the tracked dank stores.
91 lines
6.3 KiB
Markdown
91 lines
6.3 KiB
Markdown
---
|
||
name: halogen-kv-pool-fit
|
||
description: "Resize halogen-flash-server's KV pool or prefill arena without getting silently clamped or OOMing the boot: pool, arena and vision share one device budget, HALOGEN_HOST_RESERVE_GIB only scales the fit-check ceiling and is NOT an allocation, HALOGEN_KV_POOL_FIT only ever lowers, and the sizing warning prints only when a clamp happened."
|
||
---
|
||
|
||
# Sizing a halogen-flash-server KV pool
|
||
|
||
`HALOGEN_KV_POOL_FIT=1` (default) computes a device budget at startup and **only ever lowers** the configured `HALOGEN_KV_POOL_POSITIONS`. It never raises it, and never below the context.
|
||
|
||
```
|
||
device budget = host RAM - resident weights - HALOGEN_HOST_RESERVE_GIB
|
||
fit check = needs budget >= (KV + arena + vision) + 1.5 GiB margin
|
||
KV = 28 KiB/position
|
||
arena = 3.80 GiB + 8.40 GiB x (HALOGEN_MAX_TOK / 16384) # measured, linear
|
||
vision = 1.97 GiB when HALOGEN_VISION_TOWER=1
|
||
```
|
||
|
||
Prefer the engine's printed requirement over the model: a clamped boot states
|
||
it directly, e.g. `524288 positions need ~35.0 GiB (plus 1.5 of margin)`.
|
||
|
||
## The reserve is NOT an allocation
|
||
|
||
`HALOGEN_HOST_RESERVE_GIB` is a term in the fit-check inequality, nothing more. Proved by A/B: reserve 15 and reserve 16 both held **103.5 GiB** with **7.1 GiB left for everything else** — identical footprint. Actual footprint is `weights + KV + arena + vision`, none of which the reserve touches.
|
||
|
||
So the knob has only two regimes: below the pool's requirement it changes nothing observable; above it, it silently deletes the pool. **Raising the reserve can never give memory back to co-tenants.** To free host RAM, cut the pool or the arena:
|
||
|
||
| change | RAM handed back |
|
||
|---|---|
|
||
| `HALOGEN_KV_POOL_POSITIONS` 524288 → 262144 | 7.2 GiB |
|
||
| `HALOGEN_MAX_TOK` 32768 → 16384 | 8.4 GiB |
|
||
|
||
Lowering the reserve DOES work — it buys fit margin for a bigger pool or arena. On a 121.2 GiB host with 67.7 GiB weights, for 524288 + max_tok 32768 (needs 36.5):
|
||
|
||
| reserve | budget | vs 36.5 | outcome |
|
||
|---|---|---|---|
|
||
| 24 | 29.5 | −7.0 | clamps |
|
||
| **18** | **35.5** | **−1.0** | **clamps — looks like a healthy deploy** |
|
||
| 17 | 36.5 | +0.02 | knife edge, inside the noise of the weights term |
|
||
| 16 | 37.5 | +1.0 | fits, zero benefit over 15 |
|
||
| **15** | **38.5** | +2.0 | **deployed** |
|
||
|
||
Ceiling here is 16; pick the lowest reserve that fits, not the highest.
|
||
|
||
## Gotchas
|
||
|
||
- **The sizing block and its `WARNING` print ONLY after a clamp.** A fitting boot prints no `kv pool:` lines at all, so absence of a warning in old logs is NOT evidence of headroom.
|
||
- **`deploy.sh` health-passes a clamped pool.** Container `active`, `/health` 200, in ~12 s. Only the ledger proves the pool size.
|
||
- **Beware the false positive when grepping for a clamp**: the NOTE line contains the sentence "Lowering HALOGEN_KV_POOL_POSITIONS gives some back", which matches `LOWERING` case-insensitively. Grep `LOWERING THE POOL` / `cannot spare it`.
|
||
- **`free`/`MemAvailable` cannot see this server.** Weights are pinned and the kernel counts locked pages as reclaimable page cache — `MemAvailable` read ~74 GiB while the engine held 103.5 of 121.2 GiB. Read the engine's lines instead.
|
||
- **Host RAM is rounded DOWN when displayed.** `121` is `MemTotal 127104060 kB` = `121.2159`; subtracting displayed figures gives 29.3 instead of the true 29.5.
|
||
- **Vendor "on the device" figures are pool + arena totals, not pool.** Reading them as pool-only makes a doubling look affordable.
|
||
- **A 2.0 GiB firmware iGPU carve-out** is reported by the ledger and is not free memory.
|
||
- `HALOGEN_MAX_TOK` is the arena / per-request generation budget (thinking included) — do not conflate with `HALOGEN_CTX` or the pool. `HALOGEN_KV_SLOTS` costs only ~111 MiB/slot, irrelevant to sizing.
|
||
|
||
## Three failure modes, worst last
|
||
|
||
1. **Silent clamp** — need exceeds budget; pool drops to the floor, service healthy. Caught only in the ledger.
|
||
2. **Boot OOM** — enough `HALOGEN_MAX_TOK` and the arena alone beats the budget: `dmalloc: FAILED requesting 0.250 GiB after 39.970 GiB in 566 allocations (out of memory)`. Container stays `active / Up (starting)` forever; health never passes.
|
||
3. **Minutes-long stall** — reserve cut so far that the engine holds most of a shared host. It prints its own note: neighbours and this server "can stall for minutes at a time at full CPU with no disk activity and no output, which is not a crash and needs no restart." Looks like a hang, invisible to `free`, hits co-tenants too.
|
||
|
||
## Verify
|
||
|
||
```
|
||
podman logs frank-large 2>&1 | grep -aiE 'for the device|reserving the KV|ONE pool of|memory: [0-9.]+ GiB of weights|left for everything|LOWERING THE POOL|cannot spare it|listening on 127'
|
||
```
|
||
|
||
Expect the budget line, `reserving the KV pool, <N> positions`, `ONE pool of <N> positions`, zero clamp matches, a `host memory left for everything else` figure, then `listening on 127.0.0.1:8730`. (A failing 7770 publish on restart is a stale-netns race — self-heals; `podman network reload --all` forces it.)
|
||
|
||
Then a real completion — `/health` 200 alone proves nothing:
|
||
|
||
```
|
||
curl -s http://127.0.0.1:7770/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"frank-large","messages":[{"role":"user","content":"Reply with exactly: OK"}],"max_tokens":48,"reasoning_effort":"low"}'
|
||
```
|
||
|
||
Expect `finish_reason: stop`. A silent client decodes **greedy** — this server ships no sampler defaults.
|
||
|
||
## Reference points (121.2 GiB host, 67.7 GiB weights, vision on)
|
||
|
||
| pool | max_tok | reserve | need | budget | outcome |
|
||
|---|---|---|---|---|---|
|
||
| 262144 | 32768 | 24 | 29.8 | 29.5 | boots (0.3 over), no sizing lines |
|
||
| 524288 | 16384 | 24 | 28.6 | 29.5 | fits clean |
|
||
| 524288 | 32768 | 24 | 37.0 | 29.5 | clamped, deploy green |
|
||
| 524288 | 65536 | 24 | 46.5 | 29.5 | OOM, will not boot |
|
||
| 524288 | 32768 | 15 | 37.0 | 38.5 | **both features, deployed** |
|
||
| 524288 | 32768 | 16 | 37.0 | 37.5 | fits, footprint identical |
|
||
|
||
## Procedure
|
||
|
||
Compute `need` vs `budget` with unrounded RAM, then confirm against the engine's own printed requirement. After each deploy read the ledger for actual pool, clamp matches, **and the host-left line** — and A/B it: if two settings report the same `held`, the knob you turned doesn't do what its name says. Keep reserve and pool documented as a matched pair. Roll back with `git revert` (never `--no-verify`).
|