frank-primary: shape E — --expert-precise-memory (arena 6,496->7,522 slots), GPU_MAX_HW_QUEUES=2, and a mounted /root/.cache

Measured by boot probe on the shape-D base (production stopped, candidate
booted AS the service), plan numbers read off the engine's own log:

  runtime overhead   7.60 -> 5.25 GiB (precise gfx1201 TP2 registry)
  expert cache      14.87 -> 17.22 GiB
  slots              6,496 -> 7,522 (+1,026, +16%)
  swap region                13.7 GiB / 5,986 slots
  boot                       healthy in 420 s, no OOM

'--expert-precise-memory': True also appears in non-default args, so a silent
argparse drop is visible. The pool is NOT cut (option A) because the pool is
not an arena lever: 3.15e9/rank keeps shape D's measured 1.99x concurrency,
and a 2.2e9 arm probed in the same window (277,686 tokens, 1.38x) is recorded
as the conservative fallback.

Boot-cost bug found while diffing Frank's 3xR9700 recipe: the image ships no
/root/.cache at all, our boot writes 1.1 GB of vLLM AOT/torch_compile
artifacts (4,665 files) plus 229 MB of comgr JIT kernels into the ephemeral
layer, and podman rm -f discards them — so every deploy/restart/reboot re-pays
'torch.compile took 181.20 s in total' (173.19 s of it one graph). Mounting
FRANK_PRIMARY_CACHE_DIR at /root/.cache persists both and hides nothing.

GPU_MAX_HW_QUEUES=2 added at Frank's direction (real HSA knob); not A/B
measured. Retracted my own recipe-diff claim that cudagraph_capture_sizes
needed pinning: capture is 3 PIECEWISE + 2 FULL graphs, 3 s, 0.31 GiB.

Tests: 367/367.
This commit is contained in:
Frank committed 2026-10-02 14:37:57 -07:00
1 parent 812193aa63
commit 1ac4d48371
7 files changed
+314 -51

No files matched your search

+26 -4
View File
@@ -119,10 +119,25 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
grep -q '^FRANK_PRIMARY_KV_BYTES=3150000000$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_MNBT=1024$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
# --expert-precise-memory is NOT set: it put the bf16 2 x 262144 boot
# 0.13 GiB from the 31.86 GiB ceiling (arena 16.65 device-resident) and
# crash-looped at graph capture. The serve script keeps the opt-in flag.
! grep -q '^FRANK_PRIMARY_PRECISE_MEMORY=' "$ENV_FILE"
# --expert-precise-memory IS set (shape E, 2026-10-02). It was refused
# while the 2 x 262144 bf16 boot sat 0.13 GiB under the 31.86 GiB ceiling
# (arena 16.65 device-resident, NRestarts=3 at ~6.7 min each) — there it was
# the last 2.35 GiB over the line, not the cause. Probed on the shape-D base
# with production stopped: boots healthy in 420 s, plan reports runtime
# overhead 5.25 GiB (not 7.60), expert cache 17.22 GiB / 7,522 slots
# (+1,026 / +16% over 6,496). The pool is NOT cut with it (option A): the
# pool is not an arena lever, so 3.15e9 keeps the measured 1.99x.
grep -q '^FRANK_PRIMARY_PRECISE_MEMORY=1$' "$ENV_FILE"
# GPU_MAX_HW_QUEUES caps HSA hw queues per GPU; real knob (the literal is in
# libhsa-runtime64.so), passed inside the container as a bare name — the env
# file keeps only the prefixed form.
grep -q '^FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2$' "$ENV_FILE"
# The compile cache is a BOOT-COST fix: the image ships no /root/.cache at
# all, our boot writes 1.1 GB of vLLM AOT artifacts (4,665 files) plus
# 229 MB of comgr JIT kernels into the ephemeral layer, and every
# `podman rm -f` throws them away — re-paying `torch.compile took 181.20 s`
# (173.19 s of it one graph) on every deploy, restart and reboot.
grep -q '^FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_GATHER_LANES=64$' "$ENV_FILE"
! grep -q '^FRANK_PRIMARY_SPEC=' "$ENV_FILE"
@@ -150,6 +165,13 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
# moves the plan's `room` term and therefore the expert arena: ~1,025
# more slots at 2.344 MiB each.
grep -q -- '${FRANK_PRIMARY_PRECISE_MEMORY:+--expert-precise-memory}' "$SERVE"
# Shape E host/CPU wiring: the cache dir is created before it is mounted
# (a cold boot must have a target) and the mount lands on /root/.cache,
# whose vllm/ child is the path VLLM_CACHE_ROOT defaults to — the engine log
# names it as `Using cache directory: /root/.cache/vllm/torch_compile_cache/…`.
grep -q -- 'mkdir -p "$CACHE_DIR"' "$SERVE"
grep -q -- '${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache}' "$SERVE"
grep -q -- '-e GPU_MAX_HW_QUEUES="$FRANK_PRIMARY_GPU_MAX_HW_QUEUES"' "$SERVE"
}
@test "tools and reasoning are wired — the client contract is served" {