--- name: radiance-shape-resize-verification description: "Verify a radiance (frank-archon) KV/slab shape change end-to-end: TTM preflight, boot ledger, boundary probe, decode ladder — with the laws that govern each axis." --- Use when changing `FRANK_ARCHON_KV_DTYPE`, `FRANK_ARCHON_MAX_SEQS`, `FRANK_ARCHON_VRAM_WEIGHTS_MIB`, or `FRANK_ARCHON_VRAM_KV_MIB` in `scripts/frank-archon.env`, and when asked why a shape change did or did not move throughput. ## Sequence 1. Pool sizing first: `pool_mib >= seqs*maxlen*rate + 110*seqs + 448`, rate = 7.3125 KiB/token fp8, 13.8125 KiB/token bf16 (measured page sizes 26.00 vs 52.00 KiB + 3.25 KiB dtype-independent QSA key). 2. Pair ceiling: `weights + kv <= 30,969` MiB/card (card 32,624 − 3,425 engine-side fixed: 2,048 activation arena, 209.19 floor, 960 driver/kernel, 224 NCCL/IPC, 96 headroom, 600 unaccounted). Settle floor is 209.19 MiB, NOT the hand-picked 1 GiB reserve. 3. Update bats seams: pair ceiling (`FRANK_ARCHON_VRAM_CEILING_MIB`), slab floor, KV-pool contract; keep the ceiling seam owning only pair arithmetic. 4. `./tests/run.sh` (needs bats on PATH: `/home/blank/go/bin`), commit, merge `origin/wt/` ff-only into main at **/home/blank/Projects/serverAppConfig**, push main. 5. Deploy: `ssh dankserver 'cd /home/dank/serverAppConfigs && git pull --ff-only && ./scripts/deploy.sh dankserver --service frank-archon'` — boot 3–4 min; `--dry-run` first to confirm the plan selects exactly one service. 6. Read the boot ledger: `journalctl -u frank-archon | grep -E "kv cache|budgeted|expert slab|resident experts|unclaimed|settled at|sampler"` — engine prints backed tokens, slab slots, host-pool slots, settle, rows. 7. Verify contract: `ssh dankserver 'python3 /home/dank/serverAppConfigs/scripts/probe-frank-archon-kv.py'` — legs A (decode at depth), B (N×window capacity), C (prefix restore), D (N-stream decode). ## Laws (measured; trust over recalculation) - Slab is a THRESHOLD: 22,208 → 3,397, 24,000 → 3,290, 25,600 → 3,290 t/s at 43.5k; cliff at ~26 GiB (26,071 → 4,379). - Decode is nearly window-free: 99.0 t/s at 43.5k, 93.3 at 197k. - Slots are paid in decode CONCURRENCY: 3 streams = 18.4 t/s aggregate, per-stream a fused-step lottery (113.9/43.3/10.9). - Idle KV VRAM lends to the expert slab (fp8 pool 6,912 lent 6.08 GiB vs 2.69 at 3,543). - Bigger pool can LOWER prefill: more resident experts = less pinned host-pool traffic. - Oversized prompt is FREE and refused by name — counter `frank_archon_prompt_too_large_total` counts; refusal fires on PROMPT tokens alone ("leaves nothing to generate into"); radiance NEVER truncates. - TTM: serve script raises `ttm.pages_limit` to `FRANK_ARCHON_TTM_TARGET_MIB` itself (reboot resets to half RAM); unit ExecStartPre hook. - Refusal signature: `activating (auto-restart)`, exit 1 in ~50 ms, no container, no engine log. ## Probe law - 512 generated tokens minimum (8–24-token probes report decode noise as rate); greedy; cold prompts (unique nonce) to defeat the disk prefix cache; acceptance ~0.66–0.74 on repo corpus, repetitive filler 0.93–0.97 (never compare across corpora). - Radiance routes thinking to `reasoning_content`: set `chat_template_kwargs.enable_thinking=false`, read content + reasoning_content. - `finish_reason: "stop"` with `completion_tokens: 1` on a capacity request = accepted, not failed. ## Traps - Python heredoc over ssh: `str.maketrans` with quoted numeric literals collapses (`'0123456789'` → `'06789'`) — build alphabets with `chr(48+i)`/`chr(97+i)`. - `FRANK_SUDO_PW` in `/home/dank/.profile` has a CRLF tail; strip `\r` for systemd credentials (`tr -d '\r' | sudo systemd-credential ...`). - Deployer reports DEPLOY INCOMPLETE on a stray port-holder (`ss -ltnp | grep :7770`); never kill a root process over ssh without sudo. - The main checkout on the dev workstation is `/home/blank/Projects/serverAppConfig`, NOT `/home/dank/serverAppConfigs` (that path is on the servers). - `systemctl mask` fails on the relinked unit; `deploy.sh ` with no `--service` is a fleet-wide drain.