command-center/agent/managed-skills/radiance-kv-cache-resize/SKILL.md

name, description

name	description
radiance-kv-cache-resize	Resize the KV cache / VRAM pair on a radiance (StillDeadcode) endpoint: engine-budget-table ceiling, pool contract arithmetic, slab-slope law, ledger-first verification, free token-count probes

Radiance KV / VRAM pair resize

Use when changing --vram-kv-mib, --vram-weights-mib, --max-num-seqs, --max-model-len or --kv-cache-dtype on a radiance endpoint (frank-archon pattern), or when asked "what can we do with the free VRAM".

1. The ceiling is the engine's, never a hand-picked reserve

rocm-smi free space is NOT spare capacity. The boot prints a VRAM budget, per card table; read it:

card total          31.86 GiB
vram_weights        25.00 GiB   stated by --vram-weights-mib
vram_kv              5.00 GiB   stated by --vram-kv-mib
unclaimed            477.30 MiB  the headroom, the unusable above, rounding
W VRAM budgets total 30.00 GiB of the card's 31.86 GiB, so 1.86 GiB is neither
  budgeted nor accounted. The driver's context, the activation arena and
  --gpu-headroom-mib come out of it; anything past those is throughput on the floor
W card 0 settled at 2.01 GiB free against 209.19 MiB (the 96.00 MiB
  --gpu-headroom-mib and 113.19 MiB kept for the static region)

ceiling = (settled_free_MiB - engine_floor_MiB) + current_pair_sum. Worked (dankserver, 2026-10-13): 29,120 + (2,058 − 209) = 30,969 MiB/card. A pair at the ceiling settles ~330 MiB free at worst case. If a bound cannot be traced to a ledger line, it is a personal reserve — label it and put it in the bats gate as such.

2. Pool contract (the engine enforces it at REQUEST time, not boot)

pool_mib >= seqs*max_model_len*rate + 110*seqs + 448

• rate/token/rank: 7,488 B fp8, 14,144 B bf16 (1.89×, not 2× — the QSA block key is 3.25 KiB at BOTH dtypes; it is an index, not attention state).
• +110 MiB a sequence of recurrent state (kv_gdn_state 54 MiB × 2 copies + conv/tail/ple) and 448 MiB of prefix checkpoints, dtype-independent.

A pool that fails this boots, serves short chats, then 400s every long request as does not fit the 200000 token context, which leaves nothing to generate into — radiance refuses, never truncates. Encode it as a bats gate.

--kv-cache-dtype takes ONLY bf16|fp8, one dtype for the whole cache (no llama.cpp-style exact-K/quantized-V pair).

3. Slab and KV are one budget; the slab is a SLOPE

• Idle KV VRAM is the expert cache's spillover: ledger mover: N slots can be lent out of the KV cache (X GiB). A bigger pool steals residency twice over.
• Slab MiB ↔ resident experts is linear (~1,300 slots per 1,792 MiB measured); the pinned host pool hands back the same slot count as the slab grows (16,235→17,548 slab with 8,893→7,580 pool). The pinned RAM path is the slow path, so slab MiB = throughput.
• Measured law (fp8, 43.5k single-stream, Qwen3.8-Flash-Next w4): 22,208 → 3,397, 24,000 → 3,290, 25,600 → 3,584, 26,071 → 4,379 t/s. Two points inside the ±3 % noise band are NOT a plateau — never infer a threshold from two arms.
• Decode is a separate axis. It follows KV dtype/bandwidth and draft acceptance, not the slab: fp8/slab 26,071 decoded 124.1 t/s cold where bf16/slab 22,208 decoded 94.1 (512 tokens, 51.5k). Slot count moves the batch-shared per-stream floor (3 concurrent ≈ 6 t/s; solo ≈ 90–120).

4. Verify: ledger first, then live probes (no scratch server)

1. Deploy the service; read podman logs <svc>: backs N sequences all at M tokens at once is the capacity referee; check slab slots, unclaimed, sampler: ... R rows (= seqs × (1+spec), must stay inside the 25-row wht6 exactness bound), settled at … free against … floor.
2. Boundary probe: N concurrent prompts sized to max_model_len - gen, each with a unique marker at the head; expect N/N marker retrieval.
3. Prefill/decode probe: repo scripts/probe-frank-archon-kv.py (reads the shape from the deployed env, so it self-resizes).

Free token-count calibration (no prefill burned)

/v1/completions refuses when prompt tokens alone exceed the window — prompt: 202,777 prompt tokens does not fit the 200000 token context — and the refusal names the count BEFORE any prefill. It does NOT fire on prompt + max_tokens. Corpus ratio measured: 2.7617 chars/token (2.5714 for the fox filler), so a 200,000-token window ≈ 514,286–552,286 chars. Calibrate with a probe ABOVE the window; below it the request is accepted and pays real prefill (~22 s).

Probe traps that produced false results

• Generate ≥128 tokens (512 preferred) for any decode number; at 8–9 tokens inter_token is first-token noise — rates like 92.7 vs 85.8 are noise.
• Radiance timings can carry null rates — (t.get(k) or 0), and j.get("timings") or {}.
• The disk prefix cache (64 GiB) survives restarts; put a random nonce at the HEAD of a "cold" prompt. A warm 197k prefill returns in ~10 s and looks like a 60,000 t/s miracle.
• With thinking enabled, a small max_tokens yields empty content + finish_reason: stop on a healthy server; set chat_template_kwargs.enable_thinking=false in probes.
• echo === in an ssh'd zsh dies on =-expansion; it silently truncates the command chain.

5. Ship

scripts/<svc>.env pair + header, serve-script header, bats gates (pair ≤ ceiling; pool ≥ contract), docs (runbook, AGENTS.md fleet row, LLM index, log.md), ./tests/run.sh, deploy.sh <host> --service <svc>, then the probes. Record every prediction as falsifiable in the env comment and write the verdict there when the probe lands — including when it kills the prediction.