live: heartbeat sync 2026-10-18T16:47:59-07:00 (Refs #11)

agent/managed-skills/radiance-kv-cache-resize/SKILL.md

@@ -0,0 +1,111 @@
+---
+name: radiance-kv-cache-resize
+description: "Resize the KV cache / VRAM pair on a radiance (StillDeadcode) endpoint: engine-budget-table ceiling, pool contract arithmetic, slab-slope law, ledger-first verification, free token-count probes"
+---
+
+# Radiance KV / VRAM pair resize
+
+Use when changing `--vram-kv-mib`, `--vram-weights-mib`, `--max-num-seqs`,
+`--max-model-len` or `--kv-cache-dtype` on a radiance endpoint
+(frank-archon pattern), or when asked "what can we do with the free VRAM".
+
+## 1. The ceiling is the engine's, never a hand-picked reserve
+
+`rocm-smi` free space is NOT spare capacity. The boot prints a `VRAM budget,
+per card` table; read it:
+
+```
+card total          31.86 GiB
+vram_weights        25.00 GiB   stated by --vram-weights-mib
+vram_kv              5.00 GiB   stated by --vram-kv-mib
+unclaimed            477.30 MiB  the headroom, the unusable above, rounding
+W VRAM budgets total 30.00 GiB of the card's 31.86 GiB, so 1.86 GiB is neither
+  budgeted nor accounted. The driver's context, the activation arena and
+  --gpu-headroom-mib come out of it; anything past those is throughput on the floor
+W card 0 settled at 2.01 GiB free against 209.19 MiB (the 96.00 MiB
+  --gpu-headroom-mib and 113.19 MiB kept for the static region)
+```
+
+`ceiling = (settled_free_MiB - engine_floor_MiB) + current_pair_sum`.
+Worked (dankserver, 2026-10-13): 29,120 + (2,058 − 209) = **30,969 MiB/card**.
+A pair at the ceiling settles ~330 MiB free at worst case. If a bound cannot be
+traced to a ledger line, it is a personal reserve — label it and put it in the
+bats gate as such.
+
+## 2. Pool contract (the engine enforces it at REQUEST time, not boot)
+
+`pool_mib >= seqs*max_model_len*rate + 110*seqs + 448`
+
+- rate/token/rank: **7,488 B fp8**, **14,144 B bf16** (1.89×, not 2× — the QSA
+  block key is 3.25 KiB at BOTH dtypes; it is an index, not attention state).
+- +110 MiB a sequence of recurrent state (`kv_gdn_state` 54 MiB × 2 copies +
+  conv/tail/ple) and 448 MiB of prefix checkpoints, dtype-independent.
+
+A pool that fails this **boots, serves short chats, then 400s every long
+request** as `does not fit the 200000 token context, which leaves nothing to
+generate into` — radiance refuses, never truncates. Encode it as a bats gate.
+
+`--kv-cache-dtype` takes ONLY `bf16|fp8`, one dtype for the whole cache (no
+llama.cpp-style exact-K/quantized-V pair).
+
+## 3. Slab and KV are one budget; the slab is a SLOPE
+
+- Idle KV VRAM is the expert cache's spillover: ledger
+  `mover: N slots can be lent out of the KV cache (X GiB)`. A bigger pool steals
+  residency twice over.
+- Slab MiB ↔ resident experts is linear (~1,300 slots per 1,792 MiB measured);
+  the pinned host pool hands back the same slot count as the slab grows
+  (16,235→17,548 slab with 8,893→7,580 pool). The pinned RAM path is the slow
+  path, so slab MiB = throughput.
+- Measured law (fp8, 43.5k single-stream, Qwen3.8-Flash-Next w4):
+  22,208 → 3,397, 24,000 → 3,290, 25,600 → **3,584**, 26,071 → **4,379 t/s**.
+  Two points inside the ±3 % noise band are NOT a plateau — never infer a
+  threshold from two arms.
+- **Decode is a separate axis.** It follows KV dtype/bandwidth and draft
+  acceptance, not the slab: fp8/slab 26,071 decoded 124.1 t/s cold where
+  bf16/slab 22,208 decoded 94.1 (512 tokens, 51.5k). Slot count moves the
+  batch-shared per-stream floor (3 concurrent ≈ 6 t/s; solo ≈ 90–120).
+
+## 4. Verify: ledger first, then live probes (no scratch server)
+
+1. Deploy the service; read `podman logs <svc>`: `backs N sequences all at
+   M tokens at once` is the capacity referee; check slab slots, `unclaimed`,
+   `sampler: ... R rows` (= seqs × (1+spec), must stay inside the 25-row wht6
+   exactness bound), `settled at … free against … floor`.
+2. Boundary probe: N concurrent prompts sized to `max_model_len - gen`, each
+   with a unique marker at the head; expect N/N marker retrieval.
+3. Prefill/decode probe: repo `scripts/probe-frank-archon-kv.py` (reads the
+   shape from the deployed env, so it self-resizes).
+
+### Free token-count calibration (no prefill burned)
+
+`/v1/completions` refuses when **prompt tokens alone** exceed the window —
+`prompt: 202,777 prompt tokens does not fit the 200000 token context` — and the
+refusal names the count BEFORE any prefill. It does NOT fire on
+prompt + max_tokens. Corpus ratio measured: **2.7617 chars/token** (2.5714 for
+the fox filler), so a 200,000-token window ≈ 514,286–552,286 chars. Calibrate
+with a probe ABOVE the window; below it the request is accepted and pays real
+prefill (~22 s).
+
+### Probe traps that produced false results
+
+- Generate **≥128 tokens** (512 preferred) for any decode number; at 8–9 tokens
+  `inter_token` is first-token noise — rates like 92.7 vs 85.8 are noise.
+- Radiance `timings` can carry `null` rates — `(t.get(k) or 0)`, and
+  `j.get("timings") or {}`.
+- The disk prefix cache (64 GiB) survives restarts; put a random nonce at the
+  HEAD of a "cold" prompt. A warm 197k prefill returns in ~10 s and looks like
+  a 60,000 t/s miracle.
+- With thinking enabled, a small `max_tokens` yields empty `content` +
+  `finish_reason: stop` on a healthy server; set
+  `chat_template_kwargs.enable_thinking=false` in probes.
+- `echo ===` in an ssh'd zsh dies on `=`-expansion; it silently truncates the
+  command chain.
+
+## 5. Ship
+
+`scripts/<svc>.env` pair + header, serve-script header, bats gates (pair ≤
+ceiling; pool ≥ contract), docs (runbook, `AGENTS.md` fleet row, LLM index,
+log.md), `./tests/run.sh`, `deploy.sh <host> --service <svc>`, then the probes.
+Record every prediction as falsifiable in the env comment and write the verdict
+there when the probe lands — including when it kills the prediction.