frank-primary: shape E — --expert-precise-memory (arena 6,496->7,522 slots), GPU_MAX_HW_QUEUES=2, and a mounted /root/.cache

Measured by boot probe on the shape-D base (production stopped, candidate
booted AS the service), plan numbers read off the engine's own log:

  runtime overhead   7.60 -> 5.25 GiB (precise gfx1201 TP2 registry)
  expert cache      14.87 -> 17.22 GiB
  slots              6,496 -> 7,522 (+1,026, +16%)
  swap region                13.7 GiB / 5,986 slots
  boot                       healthy in 420 s, no OOM

'--expert-precise-memory': True also appears in non-default args, so a silent
argparse drop is visible. The pool is NOT cut (option A) because the pool is
not an arena lever: 3.15e9/rank keeps shape D's measured 1.99x concurrency,
and a 2.2e9 arm probed in the same window (277,686 tokens, 1.38x) is recorded
as the conservative fallback.

Boot-cost bug found while diffing Frank's 3xR9700 recipe: the image ships no
/root/.cache at all, our boot writes 1.1 GB of vLLM AOT/torch_compile
artifacts (4,665 files) plus 229 MB of comgr JIT kernels into the ephemeral
layer, and podman rm -f discards them — so every deploy/restart/reboot re-pays
'torch.compile took 181.20 s in total' (173.19 s of it one graph). Mounting
FRANK_PRIMARY_CACHE_DIR at /root/.cache persists both and hides nothing.

GPU_MAX_HW_QUEUES=2 added at Frank's direction (real HSA knob); not A/B
measured. Retracted my own recipe-diff claim that cudagraph_capture_sizes
needed pinning: capture is 3 PIECEWISE + 2 FULL graphs, 3 s, 0.31 GiB.

Tests: 367/367.

AGENTS.md

@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
 
 | Target | Service | Runtime | Model | Port |
 |--------|---------|---------|-------|------|
-| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 1024, MoE gather 32×64, **3 slots × 200704 ctx** at KV_BYTES 3,150,000,000/rank — the **pool, not the slot count, caps resident conversations at ~2 full-length** (measured live: `GPU KV cache size: 398,658 tokens, Maximum concurrency for 200,704 tokens per request: 1.99×`), and shorter requests use the spare blocks. **`--max-num-seqs` must stay 3: SEQS=2 cannot boot this model** — torch.fx `ConstraintViolationError` on `query_start_loc`, measured at BOTH MNBT 1024 and 2048. The pool cut buys runtime **headroom, NOT arena**: the arena is sized by `max_model_len` via the registry KV reserve (262144 → 6,248 slots; 200704 → 6,496) — util 0.98, and the client contract wired: `--reasoning-parser …
+| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 1024, MoE gather 32×64, **`--expert-precise-memory`** (fitted 5.25 GiB overhead, not the 7.60 GiB default → expert arena **17.22 GiB / 7,522 slots**, +16 %), **`GPU_MAX_HW_QUEUES=2`**, **mounted cache dir** at `/root/.cache` (the image ships no `/root/.cache` at all; unmounted, every boot re-pays `torch.compile took 181.20 s`), **3 slots × 200704 ctx** at KV_BYTES 3,150,000,000/rank — the **pool, not the slot count, caps resident conversations at ~2 full-length** (measured live: `GPU KV cache size: 398,658 tokens, Maximum concurrency for 200,704 tokens per request: 1.99×`), and shorter requests use the spare blocks. **`--max-num-seqs` must stay 3: SEQS=2 cannot boot this model** — torch.fx `ConstraintViolationError` on `query_start_loc`, measured at BOTH MNBT 1024 and 2048. The pool cut buys runtime **headroom, NOT arena**: the arena is sized by `max_model_len` via the registry KV reserve (262144 → 6,248 slots; 200704 → 6,496) plus the overhead term (`--expert-precise-memory` → 7,522) — util 0.98, and the client contract wired: `--reasoning-parser …
 | Dankclaw | `frank-large` | **ROCmFPX llama.cpp fork** (`kingjones30/ROCmFPX` @ `36e9acd`, gfx1151, built by `scripts/build-rocmfpx.sh`; bare `llama-server`, root systemd unit) | `kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF` **`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf`** (26.28 GiB, ftype 115 ≈ 8.39 bpw: 353 F32 + 187 Q8_0 + 311 `GGML_TYPE_Q8_0_ROCMFPX` tensors — only this fork can read them), **MTP spec decode from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` (n-max 4, pinned to ROCm0)**, **1 session × 262,144 (native window)** with **f16 KV**, the bundled BF16 mmproj for vision, thinking on with the card's recipe (temp 1.0 / top_p 0.95 / top_k 20). Measured 2026-10-02: decode **17.0–17.1 t/s** with MTP vs **7.6 without (+123 %)**, draft acceptance 0.462, prefill 325–351 t/s, prompt-cache replay 0.5 s, boot→healthy 16 s, GTT 64.5 GiB. Cut over 2026-10-02 from gufo; rollback tag `frank-large-gufo-q4kxl-20261001` | 7770 |
 | Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** (9B dense qwen3_5 hybrid, 6.4 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **3 × 180,224** (pool 540,672 — NOT the native window; three contexts cost 81,920 tokens per conversation) with **asymmetric KV f16 K + q8_0 V** under flash attention, 21,684 used / **2,441 free** of 24,576 MiB, 106.8 tok/s decode, 3,814 t/s prefill. Resized twice 2026-10-02 (Q6_K → Q5_K_M → 3 slots + vision → this pool); rollback `models/MiMo-Q6K/` or `PARALLEL=2 CTX_SIZE=524288` | 7771 |
 
@@ -453,16 +453,39 @@ sizes that input buffer `max_num_reqs + 1` and `_dummy_run` sets
 ((1,1024) vs (1,2048)). `--enforce-eager` bypasses the path but drops CUDA
 graphs. The same failure class killed the 2026-10-01 bf16 2×262144 shape.
 
-**The expert arena's lever is `max_model_len`, not `--kv-cache-memory-bytes`.**
+**The expert arena has TWO levers: `max_model_len` and the overhead term.**
 The fork's planner sizes expert cache as `budget − non-expert − KV reserve −
 graphs − overhead`, and the KV reserve is registry-priced at 9190 B/token ×
 `max_model_len`: 262144 → 2.48 GiB reserve → arena 14.30 GiB / **6,248 slots**;
 200704 → 1.91 GiB → 14.87 GiB / **6,496 slots**. Δreserve 0.57 GiB = Δarena
 0.57 GiB. The 6.1e9 → 3.15e9 pool cut bought **zero** arena; it buys runtime
 headroom (the shape it replaced died with `0 bytes free` on a real 995-token
-request). Grow the arena by lowering `max_model_len`. The arena (2.344 MiB per
-slot) is the decode-critical region — cutting it for KV headroom collapses the
-offload hit rate.
+request). The second lever is `--expert-precise-memory`, **ON since
+2026-10-02**: it swaps the registry's default overhead (7.60 GiB = fitted +
+compile/graph-capture headroom) for the fitted figure, and the 2.35 GiB
+difference lands in the arena — measured plan lines at 200704 × 3: overhead
+5.25 GiB, expert cache **17.22 GiB / 7,522 slots** (+1,026 / +16 %), swap region
+13.7 GiB / 5,986 slots, boot healthy in 420 s. The pool stayed at 3.15e9
+deliberately: because the pool is not an arena lever, cutting it would cost
+concurrency (a 2.2e9 probe reports 277,686 tokens / **1.38×**) and buy no slots.
+The arena (2.344 MiB per slot) is the decode-critical region — cutting it for KV
+headroom collapses the offload hit rate.
+
+**A container's compile cache does NOT survive `podman rm -f`, and on this
+endpoint that is 181 s per boot.** Measured 2026-10-02: the image ships **no
+`/root/.cache` at all** (fresh container: `No such file or directory`); after a
+boot it holds `/root/.cache/vllm` at **1,154,714,333 B in 4,665 files** plus
+`/root/.cache/comgr` at 229 MB, all in the ephemeral layer, and the engine
+prints `torch.compile took 181.20 s in total` (173.19 s of it one graph,
+`Compiling a graph for compile range (1, 1024)`). Every deploy, restart and
+reboot re-pays it. The serve script now mounts
+`FRANK_PRIMARY_CACHE_DIR` at `/root/.cache` — the parent, so the vLLM AOT /
+torch_compile tree (`VLLM_CACHE_ROOT`, whose default is
+`/root/.cache/vllm`) and the comgr HIP-kernel JIT cache both persist, and
+nothing in the image is hidden because the image has no such directory. The
+compile RANGE is part of the cache key, so an MNBT change costs one
+recompilation, not a permanent tax. Same class of fix as the recipe's
+`/cache/{triton,vllm,inductor}` volumes.
 
 Measured 2026-09-29 tuning campaign (fp8 + MTP4 era, since retired): decode
 **37.4-40.6 tok/s, acceptance ~54 %**; the 256k ctx arm measured 34.1-36.5;