--- name: radiance-kv-cache-resize description: "Resize a radiance endpoint's KV cache/VRAM: the resolver's funding formula, which knob actually moves the carve, the pinned-RAM ledger to recompute, and the boot-ledger lines that referee the result." --- ## Method - The endpoint is sized by ONE env file (`scripts/.env`) plus the serve script; the boot ledger prints the split before allocating anything: `VRAM budget, per card` → `claimable` → `elastic` → `experts / kv cache`, then `kv: the cache can grow to X a card -- N tokens of context, M sessions at W`. - **`--vram-weights-mib` / `--vram-kv-mib` no longer exist (1.8.1).** The engine REFUSES to start on them (`core/config.cpp`), so this skill has no direct KV knob; the budget resolver derives everything from the cache bound (`--max-num-seqs` x `--max-model-len`), `--gpu-headroom-mib` and `--host-pool-mib`. ## The funding formula (radiance 1.8.1, source-cited — re-read the pinned tree before trusting it against another build) ``` capacity = card_total - already_held - drift - headroom - arena - step_staging - hw_queues - prefetch elastic = capacity - static_weights expert_floor = min(plane+spare, max(plane + 2*unit_max - host_room, 0)) # planner.cpp expert_vram_floor host_room = host_pool_mib * MiB (less the file tier's staging when disk tiering is on) kv_cap = max_num_seqs * max_model_len # DEMAND, in tokens kv = min(kv_cap, elastic - expert_floor) # what it actually gets tokens = min over paged groups of n_blocks*block_size # the printed number ``` Read consequence: **`--max-num-seqs` raises demand, `--host-pool-mib` raises supply, and both are the same lever** — every MiB added to the host pool lowers the expert floor one-for-one and the cache takes it (up to `kv_cap`); the leftover returns to the experts. `--max-num-seqs` alone does nothing to capacity: it raises demand while the floor is unchanged, so the printed tokens stay put. The page's "funds ~91 % of demand" table rows are that cap, not a policy percentage. - Derive the plane from a live ledger instead of guessing: `plane + 2*unit ≈ printed_experts + host_pool_bytes`. - `--gpu-headroom-mib` lowers go to the cache but the engine's settle floor is built from it — 48 MiB measured decode-neutral, 24 untested. 48 is the first rung. ## Procedure 1. Read `scripts/.env` for the current pool: `GPU_HEADROOM_MIB` (96), `HOST_POOL_MIB` (44,928 per rank), `MAX_SEQS` x `MAX_MODEL_LEN` (demand), `HOST_POOL_MIB`, `PLACEMENT`. 2. Compute the target as tokens: tokens x bytes/token-per-card, where bytes/token = (cache_given_bytes / tokens) from the latest ledger. If the target exceeds `elastic - expert_floor`, ALSO lower `expert_floor` via `--host-pool-mib` (up = more room = fewer resident experts) — a bigger host pool buys cache only while experts remain on the card; once the host absorbs the whole plane, more host RAM buys nothing (measured: +640 MiB moved `vram_kv` byte-identical). 3. Edit ONE knob (`GPU_HEADROOM_MIB`), deploy service-scoped: `./scripts/deploy.sh --service `. The engine reloads everything (~2-4 min); the deployer's `LLM_WAIT` covers it. 4. Re-read the ledger: `vram_kv` must be the new number, byte-exact or the knob lost to another claim; `mover: N slots can be lent` tells what the expert slab re-borrowed from the enlarged pool — that steal is the decode cost of the resize. 5. Re-run the probe (`scripts/probe-frank-archon-kv.py`): depth 1.75x tokens, then the 2x refusal. Update `.md` Laws with the measured pair (tokens, given, backing %). 6. Under-funded? Raise `HOST_POOL_MIB` in 1,024-MiB steps and redeploy; never touch demand to fake the number. ## Pinned-RAM ledger to recompute BEFORE booting (this is where a resize kills the host) `total_pin = tp * HOST_POOL_MIB + PLE/n-gram table + PREFIX_CACHE_MIB + (statics + arena ~2,048) + checkpoint_store`, with `checkpoint_store = slots * 111.98 MiB` and `--checkpoint-slots auto` = `max_num_seqs * (1 + what the retention policy keeps)` — **it grows with the slot count**, so a slot bump raises the pin ledger twice (state + store). The repo's bats guard recomputes this and asserts `TTM_MIN_MIB >= ledger`; when you change pool or slots, update `TTM_MIN_MIB` (and that guard) in the same commit, or the seam fails. The serve script's own gate refuses to start below `TTM_MIN_MIB`; `ttm-preflight.sh` raises the knob to `TTM_TARGET_MIB`, so the target must clear the new floor with slack. ## Traps - The cache is ELASTIC: the ledger's `kv cache` line is the ceiling, and `mover: ... slots can be lent out of the activation arena` (987 MiB) means the slab borrows arena first, then cache blocks. A bigger ceiling only helps if the heat engine stops re-borrowing it — check `flex_bytes`/`flex_capacity` movement after load, not just the budget line. - A resize that shrinks the pool mid-session preempts live sequences (`vllm:num_preemptions_total` jumps); a resize that grows it cannot retroactively un-preempt. Deploy at low traffic. - Do not resize the KV pool AND the expert placement in one deploy: the `experts` and `kv cache` lines move together and the attribution dies. - The `claimable` arithmetic (`capacity`) is computed ONCE before allocation; `card total` differs per card (26.0 vs 26.68 GiB on dankserver), so the SLOWER card's number is the budget — read the `[0]` and `[1]` `pools:` lines separately, not just the shared budget table.