frank-primary: shape E — --expert-precise-memory (arena 6,496->7,522 slots), GPU_MAX_HW_QUEUES=2, and a mounted /root/.cache
Measured by boot probe on the shape-D base (production stopped, candidate booted AS the service), plan numbers read off the engine's own log: runtime overhead 7.60 -> 5.25 GiB (precise gfx1201 TP2 registry) expert cache 14.87 -> 17.22 GiB slots 6,496 -> 7,522 (+1,026, +16%) swap region 13.7 GiB / 5,986 slots boot healthy in 420 s, no OOM '--expert-precise-memory': True also appears in non-default args, so a silent argparse drop is visible. The pool is NOT cut (option A) because the pool is not an arena lever: 3.15e9/rank keeps shape D's measured 1.99x concurrency, and a 2.2e9 arm probed in the same window (277,686 tokens, 1.38x) is recorded as the conservative fallback. Boot-cost bug found while diffing Frank's 3xR9700 recipe: the image ships no /root/.cache at all, our boot writes 1.1 GB of vLLM AOT/torch_compile artifacts (4,665 files) plus 229 MB of comgr JIT kernels into the ephemeral layer, and podman rm -f discards them — so every deploy/restart/reboot re-pays 'torch.compile took 181.20 s in total' (173.19 s of it one graph). Mounting FRANK_PRIMARY_CACHE_DIR at /root/.cache persists both and hides nothing. GPU_MAX_HW_QUEUES=2 added at Frank's direction (real HSA knob); not A/B measured. Retracted my own recipe-diff claim that cudagraph_capture_sizes needed pinning: capture is 3 PIECEWISE + 2 FULL graphs, 3 s, 0.31 GiB. Tests: 367/367.
This commit is contained in:
Frank
+63
-13
@@ -1,13 +1,20 @@
|
||||
# frank-primary.env — tcclaviger RDNA4 vLLM production config (dankserver :7770).
|
||||
#
|
||||
# 2026-10-02 shape D: fp8 KV, MAXLEN 200704 x SEQS 3, the KV pool cut to ~2
|
||||
# full-length contexts. The window shrinks from the native 262144 to 200704 and
|
||||
# the pool from 6.1e9 to 3.15e9; --max-num-seqs stays 3 because SEQS=2 CANNOT
|
||||
# boot this model (measured 2x2 below). Same digest-pinned tcclaviger RDNA4
|
||||
# fork. Rollback for the fp8+MTP4 era is git tag frank-primary-fp8-mtp-20261001.
|
||||
# 2026-10-02 shape E: fp8 KV, MAXLEN 200704 x SEQS 3, pool 3.15e9/rank, and
|
||||
# --expert-precise-memory ON. The window shrink from the native 262144 and the
|
||||
# --max-num-seqs 3 lock both come from shape D (SEQS=2 CANNOT boot this model —
|
||||
# the measured 2x2 is below); shape E adds the precise-overhead plan, which
|
||||
# trades the default 7.60 GiB compile headroom for the fitted 5.25 GiB and puts
|
||||
# the 2.35 GiB difference into the expert arena: 6,496 -> 7,522 slots (+16%).
|
||||
# The pool is deliberately NOT cut (option A): the pool is not an arena lever
|
||||
# (measured), so 3.15e9 keeps shape D's 1.99x concurrency while the arena grows.
|
||||
# Same digest-pinned tcclaviger RDNA4 fork. Rollback for the fp8+MTP4 era is git
|
||||
# tag frank-primary-fp8-mtp-20261001.
|
||||
#
|
||||
# HIP reads bare env vars TEMP, HOST, GPU_MAX_HEAP_SIZE — every key here is
|
||||
# prefixed FRANK_PRIMARY_ so none can leak.
|
||||
# prefixed FRANK_PRIMARY_ so none can leak. The one bare name we WANT inside the
|
||||
# container (GPU_MAX_HW_QUEUES) is held here under the prefix and passed on the
|
||||
# run line as `-e GPU_MAX_HW_QUEUES=…`.
|
||||
#
|
||||
|
||||
# --- runtime ----------------------------------------------------------------
|
||||
@@ -29,7 +36,7 @@ FRANK_PRIMARY_PUBLISH=7770:7770
|
||||
# hoc experiment identity (both hold the GPUs / name registrations).
|
||||
FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
|
||||
|
||||
# --- qualified engine shape (2026-10-02, shape D) ---------------------------
|
||||
# --- qualified engine shape (2026-10-02, shape E) ---------------------------
|
||||
# WHAT THE SLOT COUNT DOES AND DOES NOT DO. The endpoint advertises 3 slots
|
||||
# (SEQS 3), but the POOL, not the slot count, decides how many full-length
|
||||
# conversations are resident. MEASURED LIVE 2026-10-02 at this shape:
|
||||
@@ -66,7 +73,9 @@ FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
|
||||
# d(reserve) 0.57 GiB = d(arena) 0.57 GiB = +248 slots. The 6.1e9 -> 3.15e9
|
||||
# pool cut buys ZERO arena; it buys runtime HEADROOM (the shape it replaced
|
||||
# died with 0 bytes free on a real 995-token request). To grow the arena,
|
||||
# lower MAXLEN — not KV_BYTES.
|
||||
# lower MAXLEN — not KV_BYTES. The OTHER arena lever is the overhead term:
|
||||
# --expert-precise-memory (shape E) takes it from 7.60 to 5.25 GiB and lands
|
||||
# those 2.35 GiB in the arena — 14.87 -> 17.22 GiB / 6,496 -> 7,522 slots.
|
||||
#
|
||||
# GATHER 32x64 stays: R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64
|
||||
# measured +18.7% decode vs default geometry and the win grows with depth.
|
||||
@@ -75,18 +84,59 @@ FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
|
||||
# 1.36 s); 1024-4096 are interchangeable on decode; 12288 costs 2.5% and
|
||||
# arena. 1024 keeps the workspace floor so the arena stays max. (Side note
|
||||
# measured 2026-10-02: MNBT 2048 compiles the first graph in 26 s vs 172 s at
|
||||
# 1024 — a startup-cost lever, not a correctness one.)
|
||||
# 1024 — a startup-cost lever, not a correctness one. The compile RANGE is part
|
||||
# of the cache key, so FRANK_PRIMARY_CACHE_DIR pays for whichever rung
|
||||
# is chosen exactly once.)
|
||||
# PREFETCH IS ABSENT BY LAW: expert-cache telemetry measured the prefetcher at
|
||||
# 36% precision (36,720 slabs prefetched, 13,369 served) — roughly half the
|
||||
# gather traffic was waste. Never re-add --expert-prefetch without a fresh A/B.
|
||||
# TunableOp stays OFF: CLAV_TUNABLEOP_SWEEP silently skips under expert offload
|
||||
# unless CLAV_TUNABLEOP_WITH_OFFLOAD=1, and forcing it corrupts numerics.
|
||||
# --expert-precise-memory stays DROPPED: it crashed the 2 x 262144 bf16 boot
|
||||
# (0.13 GiB free of 31.86 GiB, NRestarts=3 at ~6.7 min each). It survives in
|
||||
# the serve script as an opt-in; re-enable only with a pool/context reduction
|
||||
# that pays for the 2.35 GiB it moves into the arena.
|
||||
# --expert-precise-memory is ON (shape E). It selects the fitted overhead
|
||||
# (5.25 GiB) over the default (7.60 GiB = fitted + compile/graph-capture
|
||||
# headroom); the 2.35 GiB difference lands in the expert arena. MEASURED
|
||||
# 2026-10-02 by a boot probe on the shape-D base (production stopped, candidate
|
||||
# booted AS the service, plan read off the engine's own log):
|
||||
# runtime overhead 7.60 -> 5.25 GiB (registry Qwen4Exp…, precise
|
||||
# gfx1201 TP2 nbt4096/seqs16/len262144)
|
||||
# expert cache 14.87 -> 17.22 GiB
|
||||
# slots 6,496 -> 7,522 (+1,026 = +16%, 2.344 MiB each)
|
||||
# swap region 13.7 GiB / 5,986 slots, 133-134 per layer
|
||||
# boot healthy in 420 s, no OOM, no knife-edge
|
||||
# The flag was called a non-option after it crash-looped the 2 x 262144 bf16
|
||||
# boot (0.13 GiB free of 31.86 GiB, NRestarts=3 at ~6.7 min each) — that shape
|
||||
# was already at the ceiling, so precise-memory was the last 2.35 GiB over, not
|
||||
# the cause. At 200704 x 3 it fits with room; the run line also reports
|
||||
# 'expert_precise_memory': True in non-default args, so a silent argparse drop
|
||||
# would be visible in the boot log.
|
||||
FRANK_PRIMARY_PRECISE_MEMORY=1
|
||||
# GPU_MAX_HW_QUEUES caps the HSA hardware queues per GPU (AMD's own tuning
|
||||
# guide recommends 2 for this class of workload; the runtime default on Linux
|
||||
# is 4). Verified a REAL knob, not a no-op: the string is present in
|
||||
# /opt/rocm/core-10.0/lib/libhsa-runtime64.so. NOT A/B MEASURED — Frank's call,
|
||||
# shipped as a deliberate CPU-side tune alongside the 32x64 gather geometry.
|
||||
FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2
|
||||
# Persistent compile cache. MEASURED 2026-10-02: the image ships NO
|
||||
# /root/.cache at all (`ls: cannot access '/root/.cache'` in a fresh
|
||||
# container), and after our boot it holds /root/.cache/vllm at 1,154,714,333 B
|
||||
# in 4,665 files plus /root/.cache/comgr at 229 MB — both inside the
|
||||
# container's ephemeral layer, so every `podman rm -f` (every deploy, restart
|
||||
# and reboot) throws them away and re-pays the build. The engine's own log
|
||||
# names the loss: `torch.compile took 181.20 s in total`, of which 'Compiling a
|
||||
# graph for compile range (1, 1024) takes 173.19 s', followed by 'saved AOT
|
||||
# compiled function to /root/.cache/vllm/torch_compile_cache/…'. Mounting a
|
||||
# host dir at /root/.cache persists the vLLM AOT/torch_compile tree AND the
|
||||
# comgr HIP-kernel JIT cache, and hides nothing (the image has no such dir).
|
||||
# Safe to grow/prune: entries are keyed by version + config hash, so a stale
|
||||
# entry is a miss, never a wrong answer.
|
||||
FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary
|
||||
FRANK_PRIMARY_MAXLEN=200704
|
||||
FRANK_PRIMARY_SEQS=3
|
||||
# The POOL is deliberately kept at shape D's 3.15e9 (option A): because the pool
|
||||
# is not an arena lever, raising it back from a cut buys concurrency only, and
|
||||
# 3.15e9 preserves the measured 1.99x while precise-memory grows the arena. A
|
||||
# 2.2e9 pool was probed in the same window (277,686 tokens / 1.38x) and rejected
|
||||
# as the shipped value; it remains the conservative arm if free VRAM ever binds.
|
||||
FRANK_PRIMARY_KV_BYTES=3150000000
|
||||
FRANK_PRIMARY_KV_DTYPE=fp8
|
||||
FRANK_PRIMARY_MNBT=1024
|
||||
|
||||
@@ -11,7 +11,7 @@
|
||||
# every flag here was qualified by boot probe on 2026-09-29 — see
|
||||
# frank-primary.env for the qualification record).
|
||||
#
|
||||
# Shape (2026-10-02 shape D): MAXLEN 200704 x 3 slots, fp8 KV, KV_BYTES
|
||||
# Shape (2026-10-02 shape E): MAXLEN 200704 x 3 slots, fp8 KV, KV_BYTES
|
||||
# 3,150,000,000 PER RANK — the POOL, not the slot count, caps resident
|
||||
# conversations at ~2 full-length (measured: 398,658 tokens = 1.99x);
|
||||
# shorter requests use the spare blocks. --max-num-seqs stays 3 because SEQS=2
|
||||
@@ -21,8 +21,18 @@
|
||||
# decode, --max-num-batched-tokens 1024 (Frank's lock), MoE gather geometry
|
||||
# R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (+18.7% decode, grows
|
||||
# with depth), prefix caching, expert prefetch OFF (36% precision measured),
|
||||
# TunableOp off. The native-262144 and fp8+MTP4 shapes are retired to git tag
|
||||
# frank-primary-fp8-mtp-20261001.
|
||||
# TunableOp off. Shape E adds --expert-precise-memory (fitted 5.25 GiB overhead
|
||||
# instead of the 7.60 GiB default -> expert arena 17.22 GiB / 7,522 slots, +16%)
|
||||
# and GPU_MAX_HW_QUEUES=2. The native-262144 and fp8+MTP4 shapes are retired to
|
||||
# git tag frank-primary-fp8-mtp-20261001.
|
||||
#
|
||||
# THE COMPILE CACHE IS MOUNTED, and that is a boot-cost fix, not a tuning knob:
|
||||
# the image ships no /root/.cache at all, our first boot writes 1.1 GB of vLLM
|
||||
# AOT/torch_compile artifacts plus 229 MB of comgr JIT kernels into the
|
||||
# container's ephemeral layer, and `podman rm -f` deletes them — so every
|
||||
# deploy/restart/reboot re-pays `torch.compile took 181.20 s in total` (173.19 s
|
||||
# of it one graph). Mounting FRANK_PRIMARY_CACHE_DIR at /root/.cache makes all
|
||||
# later boots load the compiled artifact instead, and hides nothing.
|
||||
#
|
||||
# The root unit (scripts/frank-primary.service, User=dank,
|
||||
# LimitMEMLOCK=infinity) grants the pin budget for the tiered-expert host
|
||||
@@ -98,6 +108,15 @@ done
|
||||
LOG=/home/dank/logs/frank_primary.log
|
||||
mkdir -p "$(dirname "$LOG")"
|
||||
|
||||
# Persistent compile cache (see the header). Create it before the mount so even
|
||||
# the first, cold boot has a target; unset -> no mount, exactly the old
|
||||
# ephemeral behaviour. It carries /root/.cache/vllm (AOT + torch_compile) and
|
||||
# /root/.cache/comgr (HIP kernel JIT).
|
||||
CACHE_DIR="${FRANK_PRIMARY_CACHE_DIR:-}"
|
||||
if [ -n "$CACHE_DIR" ]; then
|
||||
mkdir -p "$CACHE_DIR"
|
||||
fi
|
||||
|
||||
# --device /dev/dri is the WHOLE DIRECTORY (individual renderD nodes enumerate
|
||||
# without the card nodes and HIP refuses contexts — "No CUDA GPUs are
|
||||
# available" while device_count() says 2). --ipc=host for the engine's shared
|
||||
@@ -109,8 +128,10 @@ podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
|
||||
--cap-add SYS_PTRACE \
|
||||
--device /dev/kfd --device /dev/dri --device /dev/mem \
|
||||
-e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \
|
||||
${FRANK_PRIMARY_GPU_MAX_HW_QUEUES:+-e GPU_MAX_HW_QUEUES="$FRANK_PRIMARY_GPU_MAX_HW_QUEUES"} \
|
||||
${FRANK_PRIMARY_GATHER_CHUNKS:+-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"} \
|
||||
${FRANK_PRIMARY_GATHER_LANES:+-e R4D_LRU_GATHER_LANES="$FRANK_PRIMARY_GATHER_LANES"} \
|
||||
${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache} \
|
||||
-v "$FRANK_PRIMARY_MODEL_DIR":/models:ro \
|
||||
"$FRANK_PRIMARY_IMAGE" /models \
|
||||
--served-model-name "$FRANK_PRIMARY_SERVED_NAME" --port 7770 \
|
||||
|
||||
Reference in New Issue
Block a user