frank-primary: shape E — --expert-precise-memory (arena 6,496->7,522 slots), GPU_MAX_HW_QUEUES=2, and a mounted /root/.cache

Measured by boot probe on the shape-D base (production stopped, candidate
booted AS the service), plan numbers read off the engine's own log:

  runtime overhead   7.60 -> 5.25 GiB (precise gfx1201 TP2 registry)
  expert cache      14.87 -> 17.22 GiB
  slots              6,496 -> 7,522 (+1,026, +16%)
  swap region                13.7 GiB / 5,986 slots
  boot                       healthy in 420 s, no OOM

'--expert-precise-memory': True also appears in non-default args, so a silent
argparse drop is visible. The pool is NOT cut (option A) because the pool is
not an arena lever: 3.15e9/rank keeps shape D's measured 1.99x concurrency,
and a 2.2e9 arm probed in the same window (277,686 tokens, 1.38x) is recorded
as the conservative fallback.

Boot-cost bug found while diffing Frank's 3xR9700 recipe: the image ships no
/root/.cache at all, our boot writes 1.1 GB of vLLM AOT/torch_compile
artifacts (4,665 files) plus 229 MB of comgr JIT kernels into the ephemeral
layer, and podman rm -f discards them — so every deploy/restart/reboot re-pays
'torch.compile took 181.20 s in total' (173.19 s of it one graph). Mounting
FRANK_PRIMARY_CACHE_DIR at /root/.cache persists both and hides nothing.

GPU_MAX_HW_QUEUES=2 added at Frank's direction (real HSA knob); not A/B
measured. Retracted my own recipe-diff claim that cudagraph_capture_sizes
needed pinning: capture is 3 PIECEWISE + 2 FULL graphs, 3 s, 0.31 GiB.

Tests: 367/367.

AGENTS.md

@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
 
 | Target | Service | Runtime | Model | Port |
 |--------|---------|---------|-------|------|
-| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 1024, MoE gather 32×64, **3 slots × 200704 ctx** at KV_BYTES 3,150,000,000/rank — the **pool, not the slot count, caps resident conversations at ~2 full-length** (measured live: `GPU KV cache size: 398,658 tokens, Maximum concurrency for 200,704 tokens per request: 1.99×`), and shorter requests use the spare blocks. **`--max-num-seqs` must stay 3: SEQS=2 cannot boot this model** — torch.fx `ConstraintViolationError` on `query_start_loc`, measured at BOTH MNBT 1024 and 2048. The pool cut buys runtime **headroom, NOT arena**: the arena is sized by `max_model_len` via the registry KV reserve (262144 → 6,248 slots; 200704 → 6,496) — util 0.98, and the client contract wired: `--reasoning-parser …
+| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 1024, MoE gather 32×64, **`--expert-precise-memory`** (fitted 5.25 GiB overhead, not the 7.60 GiB default → expert arena **17.22 GiB / 7,522 slots**, +16 %), **`GPU_MAX_HW_QUEUES=2`**, **mounted cache dir** at `/root/.cache` (the image ships no `/root/.cache` at all; unmounted, every boot re-pays `torch.compile took 181.20 s`), **3 slots × 200704 ctx** at KV_BYTES 3,150,000,000/rank — the **pool, not the slot count, caps resident conversations at ~2 full-length** (measured live: `GPU KV cache size: 398,658 tokens, Maximum concurrency for 200,704 tokens per request: 1.99×`), and shorter requests use the spare blocks. **`--max-num-seqs` must stay 3: SEQS=2 cannot boot this model** — torch.fx `ConstraintViolationError` on `query_start_loc`, measured at BOTH MNBT 1024 and 2048. The pool cut buys runtime **headroom, NOT arena**: the arena is sized by `max_model_len` via the registry KV reserve (262144 → 6,248 slots; 200704 → 6,496) plus the overhead term (`--expert-precise-memory` → 7,522) — util 0.98, and the client contract wired: `--reasoning-parser …
 | Dankclaw | `frank-large` | **ROCmFPX llama.cpp fork** (`kingjones30/ROCmFPX` @ `36e9acd`, gfx1151, built by `scripts/build-rocmfpx.sh`; bare `llama-server`, root systemd unit) | `kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF` **`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf`** (26.28 GiB, ftype 115 ≈ 8.39 bpw: 353 F32 + 187 Q8_0 + 311 `GGML_TYPE_Q8_0_ROCMFPX` tensors — only this fork can read them), **MTP spec decode from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` (n-max 4, pinned to ROCm0)**, **1 session × 262,144 (native window)** with **f16 KV**, the bundled BF16 mmproj for vision, thinking on with the card's recipe (temp 1.0 / top_p 0.95 / top_k 20). Measured 2026-10-02: decode **17.0–17.1 t/s** with MTP vs **7.6 without (+123 %)**, draft acceptance 0.462, prefill 325–351 t/s, prompt-cache replay 0.5 s, boot→healthy 16 s, GTT 64.5 GiB. Cut over 2026-10-02 from gufo; rollback tag `frank-large-gufo-q4kxl-20261001` | 7770 |
 | Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** (9B dense qwen3_5 hybrid, 6.4 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **3 × 180,224** (pool 540,672 — NOT the native window; three contexts cost 81,920 tokens per conversation) with **asymmetric KV f16 K + q8_0 V** under flash attention, 21,684 used / **2,441 free** of 24,576 MiB, 106.8 tok/s decode, 3,814 t/s prefill. Resized twice 2026-10-02 (Q6_K → Q5_K_M → 3 slots + vision → this pool); rollback `models/MiMo-Q6K/` or `PARALLEL=2 CTX_SIZE=524288` | 7771 |
 
@@ -453,16 +453,39 @@ sizes that input buffer `max_num_reqs + 1` and `_dummy_run` sets
 ((1,1024) vs (1,2048)). `--enforce-eager` bypasses the path but drops CUDA
 graphs. The same failure class killed the 2026-10-01 bf16 2×262144 shape.
 
-**The expert arena's lever is `max_model_len`, not `--kv-cache-memory-bytes`.**
+**The expert arena has TWO levers: `max_model_len` and the overhead term.**
 The fork's planner sizes expert cache as `budget − non-expert − KV reserve −
 graphs − overhead`, and the KV reserve is registry-priced at 9190 B/token ×
 `max_model_len`: 262144 → 2.48 GiB reserve → arena 14.30 GiB / **6,248 slots**;
 200704 → 1.91 GiB → 14.87 GiB / **6,496 slots**. Δreserve 0.57 GiB = Δarena
 0.57 GiB. The 6.1e9 → 3.15e9 pool cut bought **zero** arena; it buys runtime
 headroom (the shape it replaced died with `0 bytes free` on a real 995-token
-request). Grow the arena by lowering `max_model_len`. The arena (2.344 MiB per
+request). The second lever is `--expert-precise-memory`, **ON since
-slot) is the decode-critical region — cutting it for KV headroom collapses the
+2026-10-02**: it swaps the registry's default overhead (7.60 GiB = fitted +
-offload hit rate.
+compile/graph-capture headroom) for the fitted figure, and the 2.35 GiB
+difference lands in the arena — measured plan lines at 200704 × 3: overhead
+5.25 GiB, expert cache **17.22 GiB / 7,522 slots** (+1,026 / +16 %), swap region
+13.7 GiB / 5,986 slots, boot healthy in 420 s. The pool stayed at 3.15e9
+deliberately: because the pool is not an arena lever, cutting it would cost
+concurrency (a 2.2e9 probe reports 277,686 tokens / **1.38×**) and buy no slots.
+The arena (2.344 MiB per slot) is the decode-critical region — cutting it for KV
+headroom collapses the offload hit rate.
+
+**A container's compile cache does NOT survive `podman rm -f`, and on this
+endpoint that is 181 s per boot.** Measured 2026-10-02: the image ships **no
+`/root/.cache` at all** (fresh container: `No such file or directory`); after a
+boot it holds `/root/.cache/vllm` at **1,154,714,333 B in 4,665 files** plus
+`/root/.cache/comgr` at 229 MB, all in the ephemeral layer, and the engine
+prints `torch.compile took 181.20 s in total` (173.19 s of it one graph,
+`Compiling a graph for compile range (1, 1024)`). Every deploy, restart and
+reboot re-pays it. The serve script now mounts
+`FRANK_PRIMARY_CACHE_DIR` at `/root/.cache` — the parent, so the vLLM AOT /
+torch_compile tree (`VLLM_CACHE_ROOT`, whose default is
+`/root/.cache/vllm`) and the comgr HIP-kernel JIT cache both persist, and
+nothing in the image is hidden because the image has no such directory. The
+compile RANGE is part of the cache key, so an MNBT change costs one
+recompilation, not a permanent tax. Same class of fix as the recipe's
+`/cache/{triton,vllm,inductor}` volumes.
 
 Measured 2026-09-29 tuning campaign (fp8 + MTP4 era, since retired): decode
 **37.4-40.6 tok/s, acceptance ~54 %**; the 256k ctx arm measured 34.1-36.5;

scripts/frank-primary.env

@@ -1,13 +1,20 @@
 # frank-primary.env — tcclaviger RDNA4 vLLM production config (dankserver :7770).
 #
-# 2026-10-02 shape D: fp8 KV, MAXLEN 200704 x SEQS 3, the KV pool cut to ~2
+# 2026-10-02 shape E: fp8 KV, MAXLEN 200704 x SEQS 3, pool 3.15e9/rank, and
-# full-length contexts. The window shrinks from the native 262144 to 200704 and
+# --expert-precise-memory ON. The window shrink from the native 262144 and the
-# the pool from 6.1e9 to 3.15e9; --max-num-seqs stays 3 because SEQS=2 CANNOT
+# --max-num-seqs 3 lock both come from shape D (SEQS=2 CANNOT boot this model —
-# boot this model (measured 2x2 below). Same digest-pinned tcclaviger RDNA4
+# the measured 2x2 is below); shape E adds the precise-overhead plan, which
-# fork. Rollback for the fp8+MTP4 era is git tag frank-primary-fp8-mtp-20261001.
+# trades the default 7.60 GiB compile headroom for the fitted 5.25 GiB and puts
+# the 2.35 GiB difference into the expert arena: 6,496 -> 7,522 slots (+16%).
+# The pool is deliberately NOT cut (option A): the pool is not an arena lever
+# (measured), so 3.15e9 keeps shape D's 1.99x concurrency while the arena grows.
+# Same digest-pinned tcclaviger RDNA4 fork. Rollback for the fp8+MTP4 era is git
+# tag frank-primary-fp8-mtp-20261001.
 #
 # HIP reads bare env vars TEMP, HOST, GPU_MAX_HEAP_SIZE — every key here is
-# prefixed FRANK_PRIMARY_ so none can leak.
+# prefixed FRANK_PRIMARY_ so none can leak. The one bare name we WANT inside the
+# container (GPU_MAX_HW_QUEUES) is held here under the prefix and passed on the
+# run line as `-e GPU_MAX_HW_QUEUES=…`.
 #
 
 # --- runtime ----------------------------------------------------------------
@@ -29,7 +36,7 @@ FRANK_PRIMARY_PUBLISH=7770:7770
 # hoc experiment identity (both hold the GPUs / name registrations).
 FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
 
-# --- qualified engine shape (2026-10-02, shape D) ---------------------------
+# --- qualified engine shape (2026-10-02, shape E) ---------------------------
 # WHAT THE SLOT COUNT DOES AND DOES NOT DO. The endpoint advertises 3 slots
 # (SEQS 3), but the POOL, not the slot count, decides how many full-length
 # conversations are resident. MEASURED LIVE 2026-10-02 at this shape:
@@ -66,7 +73,9 @@ FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
 # d(reserve) 0.57 GiB = d(arena) 0.57 GiB = +248 slots. The 6.1e9 -> 3.15e9
 # pool cut buys ZERO arena; it buys runtime HEADROOM (the shape it replaced
 # died with 0 bytes free on a real 995-token request). To grow the arena,
-# lower MAXLEN — not KV_BYTES.
+# lower MAXLEN — not KV_BYTES. The OTHER arena lever is the overhead term:
+# --expert-precise-memory (shape E) takes it from 7.60 to 5.25 GiB and lands
+# those 2.35 GiB in the arena — 14.87 -> 17.22 GiB / 6,496 -> 7,522 slots.
 #
 # GATHER 32x64 stays: R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64
 # measured +18.7% decode vs default geometry and the win grows with depth.
@@ -75,18 +84,59 @@ FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
 # 1.36 s); 1024-4096 are interchangeable on decode; 12288 costs 2.5% and
 # arena. 1024 keeps the workspace floor so the arena stays max. (Side note
 # measured 2026-10-02: MNBT 2048 compiles the first graph in 26 s vs 172 s at
-# 1024 — a startup-cost lever, not a correctness one.)
+# 1024 — a startup-cost lever, not a correctness one. The compile RANGE is part
+# of the cache key, so FRANK_PRIMARY_CACHE_DIR pays for whichever rung
+# is chosen exactly once.)
 # PREFETCH IS ABSENT BY LAW: expert-cache telemetry measured the prefetcher at
 # 36% precision (36,720 slabs prefetched, 13,369 served) — roughly half the
 # gather traffic was waste. Never re-add --expert-prefetch without a fresh A/B.
 # TunableOp stays OFF: CLAV_TUNABLEOP_SWEEP silently skips under expert offload
 # unless CLAV_TUNABLEOP_WITH_OFFLOAD=1, and forcing it corrupts numerics.
-# --expert-precise-memory stays DROPPED: it crashed the 2 x 262144 bf16 boot
+# --expert-precise-memory is ON (shape E). It selects the fitted overhead
-# (0.13 GiB free of 31.86 GiB, NRestarts=3 at ~6.7 min each). It survives in
+# (5.25 GiB) over the default (7.60 GiB = fitted + compile/graph-capture
-# the serve script as an opt-in; re-enable only with a pool/context reduction
+# headroom); the 2.35 GiB difference lands in the expert arena. MEASURED
-# that pays for the 2.35 GiB it moves into the arena.
+# 2026-10-02 by a boot probe on the shape-D base (production stopped, candidate
+# booted AS the service, plan read off the engine's own log):
+#   runtime overhead  7.60 -> 5.25 GiB   (registry Qwen4Exp…, precise
+#                                         gfx1201 TP2 nbt4096/seqs16/len262144)
+#   expert cache      14.87 -> 17.22 GiB
+#   slots             6,496 -> 7,522     (+1,026 = +16%, 2.344 MiB each)
+#   swap region                 13.7 GiB / 5,986 slots, 133-134 per layer
+#   boot                        healthy in 420 s, no OOM, no knife-edge
+# The flag was called a non-option after it crash-looped the 2 x 262144 bf16
+# boot (0.13 GiB free of 31.86 GiB, NRestarts=3 at ~6.7 min each) — that shape
+# was already at the ceiling, so precise-memory was the last 2.35 GiB over, not
+# the cause. At 200704 x 3 it fits with room; the run line also reports
+# 'expert_precise_memory': True in non-default args, so a silent argparse drop
+# would be visible in the boot log.
+FRANK_PRIMARY_PRECISE_MEMORY=1
+# GPU_MAX_HW_QUEUES caps the HSA hardware queues per GPU (AMD's own tuning
+# guide recommends 2 for this class of workload; the runtime default on Linux
+# is 4). Verified a REAL knob, not a no-op: the string is present in
+# /opt/rocm/core-10.0/lib/libhsa-runtime64.so. NOT A/B MEASURED — Frank's call,
+# shipped as a deliberate CPU-side tune alongside the 32x64 gather geometry.
+FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2
+# Persistent compile cache. MEASURED 2026-10-02: the image ships NO
+# /root/.cache at all (`ls: cannot access '/root/.cache'` in a fresh
+# container), and after our boot it holds /root/.cache/vllm at 1,154,714,333 B
+# in 4,665 files plus /root/.cache/comgr at 229 MB — both inside the
+# container's ephemeral layer, so every `podman rm -f` (every deploy, restart
+# and reboot) throws them away and re-pays the build. The engine's own log
+# names the loss: `torch.compile took 181.20 s in total`, of which 'Compiling a
+# graph for compile range (1, 1024) takes 173.19 s', followed by 'saved AOT
+# compiled function to /root/.cache/vllm/torch_compile_cache/…'. Mounting a
+# host dir at /root/.cache persists the vLLM AOT/torch_compile tree AND the
+# comgr HIP-kernel JIT cache, and hides nothing (the image has no such dir).
+# Safe to grow/prune: entries are keyed by version + config hash, so a stale
+# entry is a miss, never a wrong answer.
+FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary
 FRANK_PRIMARY_MAXLEN=200704
 FRANK_PRIMARY_SEQS=3
+# The POOL is deliberately kept at shape D's 3.15e9 (option A): because the pool
+# is not an arena lever, raising it back from a cut buys concurrency only, and
+# 3.15e9 preserves the measured 1.99x while precise-memory grows the arena. A
+# 2.2e9 pool was probed in the same window (277,686 tokens / 1.38x) and rejected
+# as the shipped value; it remains the conservative arm if free VRAM ever binds.
 FRANK_PRIMARY_KV_BYTES=3150000000
 FRANK_PRIMARY_KV_DTYPE=fp8
 FRANK_PRIMARY_MNBT=1024

scripts/serve-frank-primary.sh

@@ -11,7 +11,7 @@
 # every flag here was qualified by boot probe on 2026-09-29 — see
 # frank-primary.env for the qualification record).
 #
-# Shape (2026-10-02 shape D): MAXLEN 200704 x 3 slots, fp8 KV, KV_BYTES
+# Shape (2026-10-02 shape E): MAXLEN 200704 x 3 slots, fp8 KV, KV_BYTES
 # 3,150,000,000 PER RANK — the POOL, not the slot count, caps resident
 # conversations at ~2 full-length (measured: 398,658 tokens = 1.99x);
 # shorter requests use the spare blocks. --max-num-seqs stays 3 because SEQS=2
@@ -21,8 +21,18 @@
 # decode, --max-num-batched-tokens 1024 (Frank's lock), MoE gather geometry
 # R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (+18.7% decode, grows
 # with depth), prefix caching, expert prefetch OFF (36% precision measured),
-# TunableOp off. The native-262144 and fp8+MTP4 shapes are retired to git tag
+# TunableOp off. Shape E adds --expert-precise-memory (fitted 5.25 GiB overhead
-# frank-primary-fp8-mtp-20261001.
+# instead of the 7.60 GiB default -> expert arena 17.22 GiB / 7,522 slots, +16%)
+# and GPU_MAX_HW_QUEUES=2. The native-262144 and fp8+MTP4 shapes are retired to
+# git tag frank-primary-fp8-mtp-20261001.
+#
+# THE COMPILE CACHE IS MOUNTED, and that is a boot-cost fix, not a tuning knob:
+# the image ships no /root/.cache at all, our first boot writes 1.1 GB of vLLM
+# AOT/torch_compile artifacts plus 229 MB of comgr JIT kernels into the
+# container's ephemeral layer, and `podman rm -f` deletes them — so every
+# deploy/restart/reboot re-pays `torch.compile took 181.20 s in total` (173.19 s
+# of it one graph). Mounting FRANK_PRIMARY_CACHE_DIR at /root/.cache makes all
+# later boots load the compiled artifact instead, and hides nothing.
 #
 # The root unit (scripts/frank-primary.service, User=dank,
 # LimitMEMLOCK=infinity) grants the pin budget for the tiered-expert host
@@ -98,6 +108,15 @@ done
 LOG=/home/dank/logs/frank_primary.log
 mkdir -p "$(dirname "$LOG")"
 
+# Persistent compile cache (see the header). Create it before the mount so even
+# the first, cold boot has a target; unset -> no mount, exactly the old
+# ephemeral behaviour. It carries /root/.cache/vllm (AOT + torch_compile) and
+# /root/.cache/comgr (HIP kernel JIT).
+CACHE_DIR="${FRANK_PRIMARY_CACHE_DIR:-}"
+if [ -n "$CACHE_DIR" ]; then
+    mkdir -p "$CACHE_DIR"
+fi
+
 # --device /dev/dri is the WHOLE DIRECTORY (individual renderD nodes enumerate
 # without the card nodes and HIP refuses contexts — "No CUDA GPUs are
 # available" while device_count() says 2). --ipc=host for the engine's shared
@@ -109,8 +128,10 @@ podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
   --cap-add SYS_PTRACE \
   --device /dev/kfd --device /dev/dri --device /dev/mem \
   -e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \
+  ${FRANK_PRIMARY_GPU_MAX_HW_QUEUES:+-e GPU_MAX_HW_QUEUES="$FRANK_PRIMARY_GPU_MAX_HW_QUEUES"} \
   ${FRANK_PRIMARY_GATHER_CHUNKS:+-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"} \
   ${FRANK_PRIMARY_GATHER_LANES:+-e R4D_LRU_GATHER_LANES="$FRANK_PRIMARY_GATHER_LANES"} \
+  ${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache} \
   -v "$FRANK_PRIMARY_MODEL_DIR":/models:ro \
   "$FRANK_PRIMARY_IMAGE" /models \
   --served-model-name "$FRANK_PRIMARY_SERVED_NAME" --port 7770 \

serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md

@@ -275,7 +275,7 @@ Depth caveat: every number above is at a near-empty KV. Deep-context
 behavior (the gather win growing with depth, bf16's decode cost at depth)
 is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap.
 
-### The deployed shape (2026-10-02: 3 slots x 200704, fp8, pool sized for 2 contexts)
+### The deployed shape (2026-10-02 shape E: 3 slots x 200704, fp8, precise-memory arena)
 
 ```
 FRANK_PRIMARY_MAXLEN=200704   200704-token window
@@ -283,10 +283,66 @@ FRANK_PRIMARY_SEQS=3          --max-num-seqs 3 (SEQS=2 CANNOT boot - below)
 FRANK_PRIMARY_KV_BYTES=3150000000   PER RANK (fp8, 7,892 B/token -> 1.99x)
 FRANK_PRIMARY_KV_DTYPE=fp8    --kv-cache-dtype fp8 (quantized)
 FRANK_PRIMARY_MNBT=1024       --max-num-batched-tokens 1024
+FRANK_PRIMARY_PRECISE_MEMORY=1      --expert-precise-memory (overhead 7.60 -> 5.25 GiB)
+FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2   -e GPU_MAX_HW_QUEUES=2 (HSA queues/GPU)
+FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary   mounted at /root/.cache
 R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64  (container env)
 NO --speculative-config, NO --expert-prefetch
 ```
 
+**`--expert-precise-memory` is the arena lever that does not touch the window.**
+The planner's overhead term is the only other place the arena can come from, and
+the flag swaps the default registry figure for the fitted one. Measured
+2026-10-02 by a boot probe on the shape-D base (production stopped, candidate
+booted AS the service; numbers are the engine's own plan lines):
+
+| plan line | shape D | **shape E** |
+|---|---|---|
+| `runtime overhead (non-torch + buffers + activation)` | 7.60 GiB | **5.25 GiB** (`precise (gfx1201 TP2 nbt4096/seqs16/len262144)`) |
+| `expert cache space` | 14.87 GiB | **17.22 GiB** |
+| slots | 6,496 | **7,522** (+1,026, +16%) |
+| `swap region` | - | 13.7 GiB / 5,986 slots, 133-134 per layer |
+| boot | - | healthy in 420 s, no OOM |
+
+`'expert_precise_memory': True` also appears in `non-default args`, so a silent
+argparse drop is visible in the boot log. The flag lands **in the arena, not the
+pool** - `--kv-cache-memory-bytes` is untouched at 3.15e9/rank (option A), because
+the pool is not an arena lever and cutting it further would only cost
+concurrency: a 2.2e9 pool was probed in the same window and reports 277,686
+tokens / **1.38x**, so it is the conservative fallback if free VRAM ever binds,
+not the shipped value.
+
+**The compile cache is a boot-cost fix, and it was worth 173 s per boot.**
+Measured 2026-10-02: the image ships **no `/root/.cache` at all** (a fresh
+container: `No such file or directory`), and after a boot that directory holds
+`vllm` at **1,154,714,333 B in 4,665 files** plus `comgr` at 229 MB, all inside
+the container's ephemeral layer. Every
+`podman rm -f` - i.e. every deploy, restart and reboot - discards it, and the
+engine re-pays:
+
+```
+Using cache directory: /root/.cache/vllm/torch_compile_cache/b265bd396a/rank_0_0/backbone
+Dynamo bytecode transform time: 4.99 s
+Compiling a graph for compile range (1, 1024) takes 173.19 s
+saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/…
+torch.compile took 181.20 s in total
+init engine (profile, create kv cache, warmup model) took 217.26 s (compilation: 181.20 s)
+```
+
+Mounting a host directory at `/root/.cache` makes every later boot load the AOT
+artifact instead. The mount is deliberately the **parent**, not
+`/root/.cache/vllm`: `VLLM_CACHE_ROOT` defaults to the `vllm` child, and
+`comgr` is the HIP-kernel JIT cache, so one mount persists both — and it hides
+nothing, because the image ships no `/root/.cache`. The cache key
+includes the compile range, so the MNBT choice costs exactly one compilation.
+Entries are version- and config-hashed: a stale entry is a miss, never a wrong
+answer.
+
+`GPU_MAX_HW_QUEUES=2` caps the HSA hardware queues per GPU. Verified a real knob
+rather than a no-op - the literal is read out of
+`/opt/rocm/core-10.0/lib/libhsa-runtime64.so` - and shipped deliberately
+alongside the 32x64 gather geometry. It is **not** A/B measured.
+
 **The pool, not the slot count, sets how many conversations live at once.**
 Measured live at this shape: `GPU KV cache size: 398,658 tokens, Maximum
 concurrency for 200,704 tokens per request: 1.99x` (3,146,129,408 B reserved).
@@ -337,25 +393,27 @@ max_model_len** - not from the `--kv-cache-memory-bytes` value:
 D(reserve) 0.57 GiB = D(arena) 0.57 GiB = +248 slots. The 6.1e9 -> 3.15e9
 pool cut (2.75 GiB) bought **zero** arena slots. It buys runtime HEADROOM,
 which is still worth having - the shape it replaced died with `0 bytes free`
-on a real 995-token request. Grow the arena by lowering `max_model_len`; the
+on a real 995-token request. Grow the arena by lowering `max_model_len`, or by
-pool is the headroom lever.
+taking the overhead term with `--expert-precise-memory` (below); the pool is the
+headroom lever only.
 
-This replaced the 2026-10-01 shape — 2 x 262144 at bf16 KV with
+#### The other arena lever: `--expert-precise-memory` (overhead 7.60 -> 5.25 GiB)
-`--expert-precise-memory` — which crash-looped at graph capture: the arena is
+
-DEVICE-resident, the precise-memory flag drops the plan's overhead 7.60 ->
+The flag is now ON. It was refused on 2026-10-01 for a reason that does not
-5.25 GiB, and that shape had 0.13 GiB to spare (weights 6.28 + pool 6.80 +
+generalize: the 2 x 262144 bf16 shape was already at the ceiling (weights 6.28 +
-graphs 0.55 + arena 16.65 + non-torch ~1.45 = 31.73 of 31.86 GiB,
+pool 6.80 + graphs 0.55 + arena 16.65 + non-torch ~1.45 = 31.73 of 31.86 GiB,
-NRestarts=3 at ~6.7 min each). The plain bf16 2 x 262144 boot also did not
+NRestarts=3 at ~6.7 min each), so the 2.35 GiB the flag moves into the arena was
-stay up: it left 0 bytes free at runtime and a real client request (995-token
+the last stack over the line, not the cause. The plain bf16 2 x 262144 boot also
-prompt, max_tokens=261149) killed the worker with `Tried to allocate 50.00 MiB
+did not stay up: 0 bytes free at runtime, and a real client request (995-token
+prompt, `max_tokens=261149`) killed the worker with `Tried to allocate 50.00 MiB
 ... 0 bytes free`.
 
-The correction is a pool cut plus a window trim, not a downgrade. Three
+At 200704 x 3 the flag fits with room and is worth **+1,026 slots (+16%)**. What
-full-length 200704-token contexts need ~4.75e9 B; pinning the pool at 3.15e9
+made it affordable is the window trim, not a pool cut: the arena is
-instead admits **1.99** of them. That is deliberate: the pool is the
+DEVICE-resident, the pool is not part of the budget's arithmetic, and the 200704
-runtime-HEADROOM lever (the shape this replaced died with 0 bytes free), while
+window's KV reserve (1.91 GiB) plus a smaller attention footprint is what the
-the arena is sized by `max_model_len` alone. A third full-length request queues;
+plan needed. The pool stays at 3.15e9/rank deliberately (option A) - cutting it
-short requests use the spare blocks.
+further would cost concurrency (2.2e9 -> 1.38x) and buy no slots.
 
 Boot-probed 2026-10-02 (transient systemd units at the production uid +
 memlock, full-log capture; the rootless bench probe had masked the production
@@ -366,9 +424,10 @@ memory environment). The night searched two concurrency contracts:
 | 3 x 262144, bf16, KV 11,047,010,304 | **OOM at graph capture** — `Tried to allocate 76.00 MiB ... 0 bytes free`, 30.43 GiB of 31.86 GiB already allocated by PyTorch |
 | 3 x 245760, bf16 | **OOM identically** |
 | 3 x 237568, bf16 | PASS — KV 718,606 tokens |
-| 2 x 262144, bf16 + `--expert-precise-memory` | **DEPLOYED then crashed** — 0.13 GiB free at boot, NRestarts=3 |
+| 2 x 262144, bf16 + `--expert-precise-memory` | **DEPLOYED then crashed** — 0.13 GiB free at boot, NRestarts=3 (the ceiling, not the flag) |
 | 2 x 262144, bf16 (no precise flag) | did not stay up either: 0 bytes free at runtime, worker killed by a real request |
-| **3 x 200704, fp8, pool sized for 2 contexts** | **DEPLOYED (2026-10-02)**: 398,658 tokens = 1.99x concurrency, arena 14.87 GiB / 6,496 slots, the 2.75 GiB/rank pool cut returned as runtime headroom |
+| 3 x 200704, fp8, pool sized for 2 contexts | **DEPLOYED 2026-10-02 (shape D)**: 398,658 tokens = 1.99x concurrency, arena 14.87 GiB / 6,496 slots, the 2.75 GiB/rank pool cut returned as runtime headroom |
+| **3 x 200704, fp8, + `--expert-precise-memory`** | **DEPLOYED 2026-10-02 (shape E)**: same 1.99x pool, overhead 5.25 GiB, arena 17.22 GiB / 7,522 slots, boot healthy in 420 s |
 
 Six full-length contexts cannot fit at any KV dtype: the 6-seq graph/activation
 footprint leaves only 9.18 GiB for the pool, and 6 × 262144 needs 12.47 GB even

serverAppConfigs-docs/LLM/index.md

@@ -4,11 +4,14 @@ Documentation for the LLM inference fleet on the two hosts (2026-10-02):
 **Dankserver runs the tcclaviger RDNA4 vLLM fork on frank-primary** (the
 digest-pinned `tcclaviger/vllm` image driving `podman run` directly, serving
 orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both Radeon AI PRO R9700s
-on :7770, TP=2 with automatic tiered-expert offload, **shape D since
+on :7770, TP=2 with automatic tiered-expert offload, **shape E since
 2026-10-02: 3 slots x 200704 ctx** with the KV pool sized for **~2 full-length
 conversations** — measured 398,658 tokens = 1.99x — fp8 KV, no speculative
 decode, MNBT 1024, MoE gather 32x64, util 0.98, reasoning/tool parsers wired
-(qwen3 / qwen3_xml) with preserve_thinking on. `--max-num-seqs` must stay 3:
+(qwen3 / qwen3_xml) with preserve_thinking on, plus **`--expert-precise-memory`
+(the arena grows to 17.22 GiB / 7,522 slots, +16%), `GPU_MAX_HW_QUEUES=2`, and a
+mounted cache dir** (the image ships no `/root/.cache` at all; unmounted, every boot re-pays
+`torch.compile took 181.20 s`). `--max-num-seqs` must stay 3:
 SEQS 2 cannot compile this model. Rollback frank-primary-fp8-mtp-20261001),
 and Dankclaw
 runs two endpoints — **ROCmFPX llama.cpp (frank-large, port 7770:
@@ -34,7 +37,8 @@ gfx1201, 32 GiB each]
 podman run, frank-primary, port 7770]
         C[Qwen3.8-Flash-Next FP8, no spec decode
 TP=2, expert offload automatic, 3 x 200704 ctx,
-fp8 KV pool ~2 full-length, MNBT 1024, gather 32x64, qwen3 parsers]
+fp8 KV pool ~2 full-length, MNBT 1024, gather 32x64, qwen3 parsers,
+expert-precise-memory arena 7,522 slots, /root/.cache mounted]
     end
 
     subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
@@ -81,13 +85,15 @@ frank-small, port 7771]
   mmproj for vision, ~100 tok/s decode, ~2.7 GiB of the card left free.
   The KV arithmetic, the flash-attn dependency, rollback to Q6_K
 - [[dankserver-qwen38-fp8-expert-offload-tuning]] — **ACTIVE since
-  2026-10-02 (shape D)** (root systemd unit, port 7770): the tcclaviger RDNA4
+  2026-10-02 (shape E)** (root systemd unit, port 7770): the tcclaviger RDNA4
   vLLM fork serving orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both
   R9700s, TP=2 with automatic expert offload — `--max-model-len 200704` x 3
   slots, fp8 KV, pool 3,150,000,000 B/rank (measured 398,658 tokens = 1.99x),
-  MNBT 1024, gather 32x64, no spec decode, prefetch off. Why `--max-num-seqs`
+  MNBT 1024, gather 32x64, no spec decode, prefetch off, plus
-  must stay 3 (SEQS 2 cannot compile this model), why the arena lever is
+  `--expert-precise-memory` (arena 17.22 GiB / 7,522 slots), GPU_MAX_HW_QUEUES=2
-  `max_model_len` rather than the KV pool, and the measured envelope
+  and a mounted compile cache. Why `--max-num-seqs`
+  must stay 3 (SEQS 2 cannot compile this model), why the arena's levers are
+  `max_model_len` and the overhead term rather than the KV pool, and the measured envelope
 - [[frank-primary-glm53-flash-w4a16]] — RETIRED 2026-09-30 → 10-01 (the
   failed GLM experiment; frank-primary runs Qwen FP8 again, see the intro):
   the tcclaviger RDNA4 vLLM fork

serverAppConfigs-docs/log.md

@@ -1106,3 +1106,85 @@ and then answers with nothing, and curl reports "Empty reply from server" (52).
 - **Operating lesson: `deploy.sh <target>` with no `--service` is a fleet-wide
   drain.** It restarts every service whose recorded revision is behind HEAD and
   whose paths changed since. To touch one service, name it.
+
+## 2026-10-02 — frank-primary shape E: `--expert-precise-memory` (+1,026 arena slots), a mounted compile cache, and the P2P answer
+
+Shape D left one question open — the plan's overhead term was reserving 2.35 GiB
+of compile headroom the boot never used — and the answer to Frank's free-VRAM
+question arrived together with a boot-cost bug that had been costing 181 s on
+every restart since the cutover.
+
+- **`--expert-precise-memory` is real and it fits (option A).** Boot-probed on
+  the shape-D base with production stopped, the candidate booted AS the service
+  (`FRANK_PRIMARY_ENV_FILE=/tmp/probe-b.env bash scripts/serve-frank-primary.sh`,
+  trap-guaranteed restore), the numbers read off the engine's own plan lines:
+
+  | plan line | shape D | shape E |
+  |---|---|---|
+  | `runtime overhead (non-torch + buffers + activation)` | 7.60 GiB | **5.25 GiB** (`precise (gfx1201 TP2 nbt4096/seqs16/len262144)`) |
+  | `expert cache space` | 14.87 GiB | **17.22 GiB** |
+  | slots | 6,496 | **7,522** (+1,026, +16 %) |
+  | `swap region` | — | 13.7 GiB / 5,986 slots, 133-134 per layer |
+  | boot | — | **healthy in 420 s**, no OOM |
+
+  The flag also prints `'expert_precise_memory': True` in `non-default args`, so
+  a silent argparse drop cannot hide. It was refused on 2026-10-01 only because
+  the 2 x 262144 bf16 shape was already 0.13 GiB under the ceiling — there the
+  flag was the last stack over the line, not the cause.
+- **The pool was NOT cut (option A) because the pool is not an arena lever.**
+  A 2.2e9 arm was probed in the same window and reports `277,686 tokens,
+  1.38x` — measured, rejected as the shipped value, kept as the conservative
+  fallback. With 3.15e9/rank the endpoint keeps shape D's 1.99x concurrency and
+  gains the arena, which is the shape D follow-up had actually been looking for.
+- **THE COMPILE CACHE WAS BEING THROWN AWAY ON EVERY RESTART — 181 s each.**
+  Measured: a fresh container has **no `/root/.cache` at all** (`No such file or
+  directory`); after our boot it holds `/root/.cache/vllm` at
+  **1,154,714,333 B in 4,665 files** plus `/root/.cache/comgr` at 229 MB, inside
+  the ephemeral layer, and `podman rm -f` (every deploy, restart, reboot)
+  discards them. The engine names the loss itself: `Compiling a graph for compile
+  range (1, 1024) takes 173.19 s` ... `torch.compile took 181.20 s in total` ...
+  `init engine (profile, create kv cache, warmup model) took 217.26 s
+  (compilation: 181.20 s)`. The serve script now mounts
+  `FRANK_PRIMARY_CACHE_DIR` at `/root/.cache` — the **parent**, so the vLLM
+  AOT/torch_compile tree (`VLLM_CACHE_ROOT`, default `/root/.cache/vllm`) and
+  the comgr JIT cache both persist, and nothing is hidden because the image has
+  no such directory. Same class of fix as the recipe's
+  `/cache/{triton,vllm,inductor}` volumes; the compile RANGE is part of the
+  cache key, so an MNBT change costs one recompilation, not a permanent tax.
+- **`GPU_MAX_HW_QUEUES=2`** (HSA hardware queues per GPU) added at Frank's
+  direction. Verified a real knob rather than a no-op — the literal is read from
+  `/opt/rocm/core-10.0/lib/libhsa-runtime64.so` — and shipped deliberately
+  alongside the 32x64 gather geometry. **Not A/B measured.**
+- **P2P: the hardware supports it, and the engine is already using it.** The
+  live log shows `r4d all-reduce INSTALLED (rank=0 max=49152KB finegrained=T…)`
+  on BOTH ranks and `Using ['R4D_AR', 'PYNCCL'] all-reduce backends` — R4D's
+  P2P path, with RCCL as fallback, measured at **45.274 GB/s bidirectional**.
+  Root ports `00:01.1`/`00:01.3` both report ACS `ReqRedir+`, so P2P hairpins at
+  the root complex; there is no XGMI. The old FAIL predates the 2026-09-20 board
+  swap. Kernel support confirmed: `CONFIG_HSA_AMD_P2P=y`, `CONFIG_PCI_P2PDMA=y`,
+  `CONFIG_HSA_AMD=y`, `CONFIG_ZONE_DEVICE=y`, `iommu=pt` on the cmdline.
+- **Recipe diff (Frank's 3 x R9700 compose vs this podman run), ranked:**
+  - *Acted on:* persistent compile caches (above — the one real operational
+    gap), and `GPU_MAX_HW_QUEUES`.
+  - *Retracted:* pinned `--compilation-config cudagraph_capture_sizes` was on my
+    list; measured, it is **not** a gap here — capture is 3 PIECEWISE + 2 FULL
+    graphs, `Graph capturing finished in 3 secs, took 0.31 GiB` (0.11 GiB on the
+    lighter rank). No 15-52-size ladder to cut.
+  - *Deliberate divergence, keep ours:* no MTP (the recipe has it; measured
+    fastest here), `qwen3_xml` vs `qwen3_coder` (ours verified live), no
+    `--override-generation-config` (server-side sampling defaults), published
+    port + Caddy rather than `network_mode: host`.
+  - *Non-gaps:* `shm_size 32gb` (ours is `--ipc=host`), `memlock -1` (same),
+    `SYS_PTRACE` (same), `/dev/kfd` + `/dev/dri` (same), `--enable-chunked-prefill`
+    (V1 default), `plecache` volume (the flag is not even passed), and
+    `ROCR_VISIBLE_DEVICES=1,2,3` vs our `HIP_VISIBLE_DEVICES=0,1` — verified
+    equivalent here: `rocm-smi --showid` puts the two R9700s at GPU[0]/GPU[1] and
+    the 7700X iGPU at GPU[2], so our indices are the right two cards.
+- **Harness note, because it cost two probe attempts:** a probe that reported an
+  instantaneous candidate death was a logging artifact of the inline SSH script,
+  not a crash — the container and its serve script must be polled by recorded id
+  and their logs dumped on the exit path, or a failure looks like a bad flag.
+  The flag was fine both times.
+- Live: `health=200`, `NRestarts=0`, arena 17.22 GiB / 7,522 slots, KV 398,658
+  tokens / 1.99x. Docs: [[dankserver-qwen38-fp8-expert-offload-tuning]].
+

tests/frank_primary_unit.bats

@@ -119,10 +119,25 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
     grep -q '^FRANK_PRIMARY_KV_BYTES=3150000000$' "$ENV_FILE"
     grep -q '^FRANK_PRIMARY_MNBT=1024$' "$ENV_FILE"
     grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
-    # --expert-precise-memory is NOT set: it put the bf16 2 x 262144 boot
+    # --expert-precise-memory IS set (shape E, 2026-10-02). It was refused
-    # 0.13 GiB from the 31.86 GiB ceiling (arena 16.65 device-resident) and
+    # while the 2 x 262144 bf16 boot sat 0.13 GiB under the 31.86 GiB ceiling
-    # crash-looped at graph capture. The serve script keeps the opt-in flag.
+    # (arena 16.65 device-resident, NRestarts=3 at ~6.7 min each) — there it was
-    ! grep -q '^FRANK_PRIMARY_PRECISE_MEMORY=' "$ENV_FILE"
+    # the last 2.35 GiB over the line, not the cause. Probed on the shape-D base
+    # with production stopped: boots healthy in 420 s, plan reports runtime
+    # overhead 5.25 GiB (not 7.60), expert cache 17.22 GiB / 7,522 slots
+    # (+1,026 / +16% over 6,496). The pool is NOT cut with it (option A): the
+    # pool is not an arena lever, so 3.15e9 keeps the measured 1.99x.
+    grep -q '^FRANK_PRIMARY_PRECISE_MEMORY=1$' "$ENV_FILE"
+    # GPU_MAX_HW_QUEUES caps HSA hw queues per GPU; real knob (the literal is in
+    # libhsa-runtime64.so), passed inside the container as a bare name — the env
+    # file keeps only the prefixed form.
+    grep -q '^FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2$' "$ENV_FILE"
+    # The compile cache is a BOOT-COST fix: the image ships no /root/.cache at
+    # all, our boot writes 1.1 GB of vLLM AOT artifacts (4,665 files) plus
+    # 229 MB of comgr JIT kernels into the ephemeral layer, and every
+    # `podman rm -f` throws them away — re-paying `torch.compile took 181.20 s`
+    # (173.19 s of it one graph) on every deploy, restart and reboot.
+    grep -q '^FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary$' "$ENV_FILE"
     grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE"
     grep -q '^FRANK_PRIMARY_GATHER_LANES=64$' "$ENV_FILE"
     ! grep -q '^FRANK_PRIMARY_SPEC=' "$ENV_FILE"
@@ -150,6 +165,13 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
     # moves the plan's `room` term and therefore the expert arena: ~1,025
     # more slots at 2.344 MiB each.
     grep -q -- '${FRANK_PRIMARY_PRECISE_MEMORY:+--expert-precise-memory}' "$SERVE"
+    # Shape E host/CPU wiring: the cache dir is created before it is mounted
+    # (a cold boot must have a target) and the mount lands on /root/.cache,
+    # whose vllm/ child is the path VLLM_CACHE_ROOT defaults to — the engine log
+    # names it as `Using cache directory: /root/.cache/vllm/torch_compile_cache/…`.
+    grep -q -- 'mkdir -p "$CACHE_DIR"' "$SERVE"
+    grep -q -- '${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache}' "$SERVE"
+    grep -q -- '-e GPU_MAX_HW_QUEUES="$FRANK_PRIMARY_GPU_MAX_HW_QUEUES"' "$SERVE"
 }
 
 @test "tools and reasoning are wired — the client contract is served" {