frank-primary: shape E — --expert-precise-memory (arena 6,496->7,522 slots), GPU_MAX_HW_QUEUES=2, and a mounted /root/.cache

Measured by boot probe on the shape-D base (production stopped, candidate
booted AS the service), plan numbers read off the engine's own log:

  runtime overhead   7.60 -> 5.25 GiB (precise gfx1201 TP2 registry)
  expert cache      14.87 -> 17.22 GiB
  slots              6,496 -> 7,522 (+1,026, +16%)
  swap region                13.7 GiB / 5,986 slots
  boot                       healthy in 420 s, no OOM

'--expert-precise-memory': True also appears in non-default args, so a silent
argparse drop is visible. The pool is NOT cut (option A) because the pool is
not an arena lever: 3.15e9/rank keeps shape D's measured 1.99x concurrency,
and a 2.2e9 arm probed in the same window (277,686 tokens, 1.38x) is recorded
as the conservative fallback.

Boot-cost bug found while diffing Frank's 3xR9700 recipe: the image ships no
/root/.cache at all, our boot writes 1.1 GB of vLLM AOT/torch_compile
artifacts (4,665 files) plus 229 MB of comgr JIT kernels into the ephemeral
layer, and podman rm -f discards them — so every deploy/restart/reboot re-pays
'torch.compile took 181.20 s in total' (173.19 s of it one graph). Mounting
FRANK_PRIMARY_CACHE_DIR at /root/.cache persists both and hides nothing.

GPU_MAX_HW_QUEUES=2 added at Frank's direction (real HSA knob); not A/B
measured. Retracted my own recipe-diff claim that cudagraph_capture_sizes
needed pinning: capture is 3 PIECEWISE + 2 FULL graphs, 3 s, 0.31 GiB.

Tests: 367/367.

serverAppConfigs-docs/log.md

@@ -1106,3 +1106,85 @@ and then answers with nothing, and curl reports "Empty reply from server" (52).
 - **Operating lesson: `deploy.sh <target>` with no `--service` is a fleet-wide
   drain.** It restarts every service whose recorded revision is behind HEAD and
   whose paths changed since. To touch one service, name it.
+
+## 2026-10-02 — frank-primary shape E: `--expert-precise-memory` (+1,026 arena slots), a mounted compile cache, and the P2P answer
+
+Shape D left one question open — the plan's overhead term was reserving 2.35 GiB
+of compile headroom the boot never used — and the answer to Frank's free-VRAM
+question arrived together with a boot-cost bug that had been costing 181 s on
+every restart since the cutover.
+
+- **`--expert-precise-memory` is real and it fits (option A).** Boot-probed on
+  the shape-D base with production stopped, the candidate booted AS the service
+  (`FRANK_PRIMARY_ENV_FILE=/tmp/probe-b.env bash scripts/serve-frank-primary.sh`,
+  trap-guaranteed restore), the numbers read off the engine's own plan lines:
+
+  | plan line | shape D | shape E |
+  |---|---|---|
+  | `runtime overhead (non-torch + buffers + activation)` | 7.60 GiB | **5.25 GiB** (`precise (gfx1201 TP2 nbt4096/seqs16/len262144)`) |
+  | `expert cache space` | 14.87 GiB | **17.22 GiB** |
+  | slots | 6,496 | **7,522** (+1,026, +16 %) |
+  | `swap region` | — | 13.7 GiB / 5,986 slots, 133-134 per layer |
+  | boot | — | **healthy in 420 s**, no OOM |
+
+  The flag also prints `'expert_precise_memory': True` in `non-default args`, so
+  a silent argparse drop cannot hide. It was refused on 2026-10-01 only because
+  the 2 x 262144 bf16 shape was already 0.13 GiB under the ceiling — there the
+  flag was the last stack over the line, not the cause.
+- **The pool was NOT cut (option A) because the pool is not an arena lever.**
+  A 2.2e9 arm was probed in the same window and reports `277,686 tokens,
+  1.38x` — measured, rejected as the shipped value, kept as the conservative
+  fallback. With 3.15e9/rank the endpoint keeps shape D's 1.99x concurrency and
+  gains the arena, which is the shape D follow-up had actually been looking for.
+- **THE COMPILE CACHE WAS BEING THROWN AWAY ON EVERY RESTART — 181 s each.**
+  Measured: a fresh container has **no `/root/.cache` at all** (`No such file or
+  directory`); after our boot it holds `/root/.cache/vllm` at
+  **1,154,714,333 B in 4,665 files** plus `/root/.cache/comgr` at 229 MB, inside
+  the ephemeral layer, and `podman rm -f` (every deploy, restart, reboot)
+  discards them. The engine names the loss itself: `Compiling a graph for compile
+  range (1, 1024) takes 173.19 s` ... `torch.compile took 181.20 s in total` ...
+  `init engine (profile, create kv cache, warmup model) took 217.26 s
+  (compilation: 181.20 s)`. The serve script now mounts
+  `FRANK_PRIMARY_CACHE_DIR` at `/root/.cache` — the **parent**, so the vLLM
+  AOT/torch_compile tree (`VLLM_CACHE_ROOT`, default `/root/.cache/vllm`) and
+  the comgr JIT cache both persist, and nothing is hidden because the image has
+  no such directory. Same class of fix as the recipe's
+  `/cache/{triton,vllm,inductor}` volumes; the compile RANGE is part of the
+  cache key, so an MNBT change costs one recompilation, not a permanent tax.
+- **`GPU_MAX_HW_QUEUES=2`** (HSA hardware queues per GPU) added at Frank's
+  direction. Verified a real knob rather than a no-op — the literal is read from
+  `/opt/rocm/core-10.0/lib/libhsa-runtime64.so` — and shipped deliberately
+  alongside the 32x64 gather geometry. **Not A/B measured.**
+- **P2P: the hardware supports it, and the engine is already using it.** The
+  live log shows `r4d all-reduce INSTALLED (rank=0 max=49152KB finegrained=T…)`
+  on BOTH ranks and `Using ['R4D_AR', 'PYNCCL'] all-reduce backends` — R4D's
+  P2P path, with RCCL as fallback, measured at **45.274 GB/s bidirectional**.
+  Root ports `00:01.1`/`00:01.3` both report ACS `ReqRedir+`, so P2P hairpins at
+  the root complex; there is no XGMI. The old FAIL predates the 2026-09-20 board
+  swap. Kernel support confirmed: `CONFIG_HSA_AMD_P2P=y`, `CONFIG_PCI_P2PDMA=y`,
+  `CONFIG_HSA_AMD=y`, `CONFIG_ZONE_DEVICE=y`, `iommu=pt` on the cmdline.
+- **Recipe diff (Frank's 3 x R9700 compose vs this podman run), ranked:**
+  - *Acted on:* persistent compile caches (above — the one real operational
+    gap), and `GPU_MAX_HW_QUEUES`.
+  - *Retracted:* pinned `--compilation-config cudagraph_capture_sizes` was on my
+    list; measured, it is **not** a gap here — capture is 3 PIECEWISE + 2 FULL
+    graphs, `Graph capturing finished in 3 secs, took 0.31 GiB` (0.11 GiB on the
+    lighter rank). No 15-52-size ladder to cut.
+  - *Deliberate divergence, keep ours:* no MTP (the recipe has it; measured
+    fastest here), `qwen3_xml` vs `qwen3_coder` (ours verified live), no
+    `--override-generation-config` (server-side sampling defaults), published
+    port + Caddy rather than `network_mode: host`.
+  - *Non-gaps:* `shm_size 32gb` (ours is `--ipc=host`), `memlock -1` (same),
+    `SYS_PTRACE` (same), `/dev/kfd` + `/dev/dri` (same), `--enable-chunked-prefill`
+    (V1 default), `plecache` volume (the flag is not even passed), and
+    `ROCR_VISIBLE_DEVICES=1,2,3` vs our `HIP_VISIBLE_DEVICES=0,1` — verified
+    equivalent here: `rocm-smi --showid` puts the two R9700s at GPU[0]/GPU[1] and
+    the 7700X iGPU at GPU[2], so our indices are the right two cards.
+- **Harness note, because it cost two probe attempts:** a probe that reported an
+  instantaneous candidate death was a logging artifact of the inline SSH script,
+  not a crash — the container and its serve script must be polled by recorded id
+  and their logs dumped on the exit path, or a failure looks like a bad flag.
+  The flag was fine both times.
+- Live: `health=200`, `NRestarts=0`, arena 17.22 GiB / 7,522 slots, KV 398,658
+  tokens / 1.99x. Docs: [[dankserver-qwen38-fp8-expert-offload-tuning]].
+