frank-primary: shape E — --expert-precise-memory (arena 6,496->7,522 slots), GPU_MAX_HW_QUEUES=2, and a mounted /root/.cache
Measured by boot probe on the shape-D base (production stopped, candidate booted AS the service), plan numbers read off the engine's own log: runtime overhead 7.60 -> 5.25 GiB (precise gfx1201 TP2 registry) expert cache 14.87 -> 17.22 GiB slots 6,496 -> 7,522 (+1,026, +16%) swap region 13.7 GiB / 5,986 slots boot healthy in 420 s, no OOM '--expert-precise-memory': True also appears in non-default args, so a silent argparse drop is visible. The pool is NOT cut (option A) because the pool is not an arena lever: 3.15e9/rank keeps shape D's measured 1.99x concurrency, and a 2.2e9 arm probed in the same window (277,686 tokens, 1.38x) is recorded as the conservative fallback. Boot-cost bug found while diffing Frank's 3xR9700 recipe: the image ships no /root/.cache at all, our boot writes 1.1 GB of vLLM AOT/torch_compile artifacts (4,665 files) plus 229 MB of comgr JIT kernels into the ephemeral layer, and podman rm -f discards them — so every deploy/restart/reboot re-pays 'torch.compile took 181.20 s in total' (173.19 s of it one graph). Mounting FRANK_PRIMARY_CACHE_DIR at /root/.cache persists both and hides nothing. GPU_MAX_HW_QUEUES=2 added at Frank's direction (real HSA knob); not A/B measured. Retracted my own recipe-diff claim that cudagraph_capture_sizes needed pinning: capture is 3 PIECEWISE + 2 FULL graphs, 3 s, 0.31 GiB. Tests: 367/367.
This commit is contained in:
Frank
@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
|
||||
| Target | Service | Runtime | Model | Port |
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 1024, MoE gather 32×64, **3 slots × 200704 ctx** at KV_BYTES 3,150,000,000/rank — the **pool, not the slot count, caps resident conversations at ~2 full-length** (measured live: `GPU KV cache size: 398,658 tokens, Maximum concurrency for 200,704 tokens per request: 1.99×`), and shorter requests use the spare blocks. **`--max-num-seqs` must stay 3: SEQS=2 cannot boot this model** — torch.fx `ConstraintViolationError` on `query_start_loc`, measured at BOTH MNBT 1024 and 2048. The pool cut buys runtime **headroom, NOT arena**: the arena is sized by `max_model_len` via the registry KV reserve (262144 → 6,248 slots; 200704 → 6,496) — util 0.98, and the client contract wired: `--reasoning-parser …
|
||||
| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 1024, MoE gather 32×64, **`--expert-precise-memory`** (fitted 5.25 GiB overhead, not the 7.60 GiB default → expert arena **17.22 GiB / 7,522 slots**, +16 %), **`GPU_MAX_HW_QUEUES=2`**, **mounted cache dir** at `/root/.cache` (the image ships no `/root/.cache` at all; unmounted, every boot re-pays `torch.compile took 181.20 s`), **3 slots × 200704 ctx** at KV_BYTES 3,150,000,000/rank — the **pool, not the slot count, caps resident conversations at ~2 full-length** (measured live: `GPU KV cache size: 398,658 tokens, Maximum concurrency for 200,704 tokens per request: 1.99×`), and shorter requests use the spare blocks. **`--max-num-seqs` must stay 3: SEQS=2 cannot boot this model** — torch.fx `ConstraintViolationError` on `query_start_loc`, measured at BOTH MNBT 1024 and 2048. The pool cut buys runtime **headroom, NOT arena**: the arena is sized by `max_model_len` via the registry KV reserve (262144 → 6,248 slots; 200704 → 6,496) plus the overhead term (`--expert-precise-memory` → 7,522) — util 0.98, and the client contract wired: `--reasoning-parser …
|
||||
| Dankclaw | `frank-large` | **ROCmFPX llama.cpp fork** (`kingjones30/ROCmFPX` @ `36e9acd`, gfx1151, built by `scripts/build-rocmfpx.sh`; bare `llama-server`, root systemd unit) | `kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF` **`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf`** (26.28 GiB, ftype 115 ≈ 8.39 bpw: 353 F32 + 187 Q8_0 + 311 `GGML_TYPE_Q8_0_ROCMFPX` tensors — only this fork can read them), **MTP spec decode from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` (n-max 4, pinned to ROCm0)**, **1 session × 262,144 (native window)** with **f16 KV**, the bundled BF16 mmproj for vision, thinking on with the card's recipe (temp 1.0 / top_p 0.95 / top_k 20). Measured 2026-10-02: decode **17.0–17.1 t/s** with MTP vs **7.6 without (+123 %)**, draft acceptance 0.462, prefill 325–351 t/s, prompt-cache replay 0.5 s, boot→healthy 16 s, GTT 64.5 GiB. Cut over 2026-10-02 from gufo; rollback tag `frank-large-gufo-q4kxl-20261001` | 7770 |
|
||||
| Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** (9B dense qwen3_5 hybrid, 6.4 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **3 × 180,224** (pool 540,672 — NOT the native window; three contexts cost 81,920 tokens per conversation) with **asymmetric KV f16 K + q8_0 V** under flash attention, 21,684 used / **2,441 free** of 24,576 MiB, 106.8 tok/s decode, 3,814 t/s prefill. Resized twice 2026-10-02 (Q6_K → Q5_K_M → 3 slots + vision → this pool); rollback `models/MiMo-Q6K/` or `PARALLEL=2 CTX_SIZE=524288` | 7771 |
|
||||
|
||||
@@ -453,16 +453,39 @@ sizes that input buffer `max_num_reqs + 1` and `_dummy_run` sets
|
||||
((1,1024) vs (1,2048)). `--enforce-eager` bypasses the path but drops CUDA
|
||||
graphs. The same failure class killed the 2026-10-01 bf16 2×262144 shape.
|
||||
|
||||
**The expert arena's lever is `max_model_len`, not `--kv-cache-memory-bytes`.**
|
||||
**The expert arena has TWO levers: `max_model_len` and the overhead term.**
|
||||
The fork's planner sizes expert cache as `budget − non-expert − KV reserve −
|
||||
graphs − overhead`, and the KV reserve is registry-priced at 9190 B/token ×
|
||||
`max_model_len`: 262144 → 2.48 GiB reserve → arena 14.30 GiB / **6,248 slots**;
|
||||
200704 → 1.91 GiB → 14.87 GiB / **6,496 slots**. Δreserve 0.57 GiB = Δarena
|
||||
0.57 GiB. The 6.1e9 → 3.15e9 pool cut bought **zero** arena; it buys runtime
|
||||
headroom (the shape it replaced died with `0 bytes free` on a real 995-token
|
||||
request). Grow the arena by lowering `max_model_len`. The arena (2.344 MiB per
|
||||
slot) is the decode-critical region — cutting it for KV headroom collapses the
|
||||
offload hit rate.
|
||||
request). The second lever is `--expert-precise-memory`, **ON since
|
||||
2026-10-02**: it swaps the registry's default overhead (7.60 GiB = fitted +
|
||||
compile/graph-capture headroom) for the fitted figure, and the 2.35 GiB
|
||||
difference lands in the arena — measured plan lines at 200704 × 3: overhead
|
||||
5.25 GiB, expert cache **17.22 GiB / 7,522 slots** (+1,026 / +16 %), swap region
|
||||
13.7 GiB / 5,986 slots, boot healthy in 420 s. The pool stayed at 3.15e9
|
||||
deliberately: because the pool is not an arena lever, cutting it would cost
|
||||
concurrency (a 2.2e9 probe reports 277,686 tokens / **1.38×**) and buy no slots.
|
||||
The arena (2.344 MiB per slot) is the decode-critical region — cutting it for KV
|
||||
headroom collapses the offload hit rate.
|
||||
|
||||
**A container's compile cache does NOT survive `podman rm -f`, and on this
|
||||
endpoint that is 181 s per boot.** Measured 2026-10-02: the image ships **no
|
||||
`/root/.cache` at all** (fresh container: `No such file or directory`); after a
|
||||
boot it holds `/root/.cache/vllm` at **1,154,714,333 B in 4,665 files** plus
|
||||
`/root/.cache/comgr` at 229 MB, all in the ephemeral layer, and the engine
|
||||
prints `torch.compile took 181.20 s in total` (173.19 s of it one graph,
|
||||
`Compiling a graph for compile range (1, 1024)`). Every deploy, restart and
|
||||
reboot re-pays it. The serve script now mounts
|
||||
`FRANK_PRIMARY_CACHE_DIR` at `/root/.cache` — the parent, so the vLLM AOT /
|
||||
torch_compile tree (`VLLM_CACHE_ROOT`, whose default is
|
||||
`/root/.cache/vllm`) and the comgr HIP-kernel JIT cache both persist, and
|
||||
nothing in the image is hidden because the image has no such directory. The
|
||||
compile RANGE is part of the cache key, so an MNBT change costs one
|
||||
recompilation, not a permanent tax. Same class of fix as the recipe's
|
||||
`/cache/{triton,vllm,inductor}` volumes.
|
||||
|
||||
Measured 2026-09-29 tuning campaign (fp8 + MTP4 era, since retired): decode
|
||||
**37.4-40.6 tok/s, acceptance ~54 %**; the 256k ctx arm measured 34.1-36.5;
|
||||
|
||||
+63
-13
@@ -1,13 +1,20 @@
|
||||
# frank-primary.env — tcclaviger RDNA4 vLLM production config (dankserver :7770).
|
||||
#
|
||||
# 2026-10-02 shape D: fp8 KV, MAXLEN 200704 x SEQS 3, the KV pool cut to ~2
|
||||
# full-length contexts. The window shrinks from the native 262144 to 200704 and
|
||||
# the pool from 6.1e9 to 3.15e9; --max-num-seqs stays 3 because SEQS=2 CANNOT
|
||||
# boot this model (measured 2x2 below). Same digest-pinned tcclaviger RDNA4
|
||||
# fork. Rollback for the fp8+MTP4 era is git tag frank-primary-fp8-mtp-20261001.
|
||||
# 2026-10-02 shape E: fp8 KV, MAXLEN 200704 x SEQS 3, pool 3.15e9/rank, and
|
||||
# --expert-precise-memory ON. The window shrink from the native 262144 and the
|
||||
# --max-num-seqs 3 lock both come from shape D (SEQS=2 CANNOT boot this model —
|
||||
# the measured 2x2 is below); shape E adds the precise-overhead plan, which
|
||||
# trades the default 7.60 GiB compile headroom for the fitted 5.25 GiB and puts
|
||||
# the 2.35 GiB difference into the expert arena: 6,496 -> 7,522 slots (+16%).
|
||||
# The pool is deliberately NOT cut (option A): the pool is not an arena lever
|
||||
# (measured), so 3.15e9 keeps shape D's 1.99x concurrency while the arena grows.
|
||||
# Same digest-pinned tcclaviger RDNA4 fork. Rollback for the fp8+MTP4 era is git
|
||||
# tag frank-primary-fp8-mtp-20261001.
|
||||
#
|
||||
# HIP reads bare env vars TEMP, HOST, GPU_MAX_HEAP_SIZE — every key here is
|
||||
# prefixed FRANK_PRIMARY_ so none can leak.
|
||||
# prefixed FRANK_PRIMARY_ so none can leak. The one bare name we WANT inside the
|
||||
# container (GPU_MAX_HW_QUEUES) is held here under the prefix and passed on the
|
||||
# run line as `-e GPU_MAX_HW_QUEUES=…`.
|
||||
#
|
||||
|
||||
# --- runtime ----------------------------------------------------------------
|
||||
@@ -29,7 +36,7 @@ FRANK_PRIMARY_PUBLISH=7770:7770
|
||||
# hoc experiment identity (both hold the GPUs / name registrations).
|
||||
FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
|
||||
|
||||
# --- qualified engine shape (2026-10-02, shape D) ---------------------------
|
||||
# --- qualified engine shape (2026-10-02, shape E) ---------------------------
|
||||
# WHAT THE SLOT COUNT DOES AND DOES NOT DO. The endpoint advertises 3 slots
|
||||
# (SEQS 3), but the POOL, not the slot count, decides how many full-length
|
||||
# conversations are resident. MEASURED LIVE 2026-10-02 at this shape:
|
||||
@@ -66,7 +73,9 @@ FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
|
||||
# d(reserve) 0.57 GiB = d(arena) 0.57 GiB = +248 slots. The 6.1e9 -> 3.15e9
|
||||
# pool cut buys ZERO arena; it buys runtime HEADROOM (the shape it replaced
|
||||
# died with 0 bytes free on a real 995-token request). To grow the arena,
|
||||
# lower MAXLEN — not KV_BYTES.
|
||||
# lower MAXLEN — not KV_BYTES. The OTHER arena lever is the overhead term:
|
||||
# --expert-precise-memory (shape E) takes it from 7.60 to 5.25 GiB and lands
|
||||
# those 2.35 GiB in the arena — 14.87 -> 17.22 GiB / 6,496 -> 7,522 slots.
|
||||
#
|
||||
# GATHER 32x64 stays: R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64
|
||||
# measured +18.7% decode vs default geometry and the win grows with depth.
|
||||
@@ -75,18 +84,59 @@ FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
|
||||
# 1.36 s); 1024-4096 are interchangeable on decode; 12288 costs 2.5% and
|
||||
# arena. 1024 keeps the workspace floor so the arena stays max. (Side note
|
||||
# measured 2026-10-02: MNBT 2048 compiles the first graph in 26 s vs 172 s at
|
||||
# 1024 — a startup-cost lever, not a correctness one.)
|
||||
# 1024 — a startup-cost lever, not a correctness one. The compile RANGE is part
|
||||
# of the cache key, so FRANK_PRIMARY_CACHE_DIR pays for whichever rung
|
||||
# is chosen exactly once.)
|
||||
# PREFETCH IS ABSENT BY LAW: expert-cache telemetry measured the prefetcher at
|
||||
# 36% precision (36,720 slabs prefetched, 13,369 served) — roughly half the
|
||||
# gather traffic was waste. Never re-add --expert-prefetch without a fresh A/B.
|
||||
# TunableOp stays OFF: CLAV_TUNABLEOP_SWEEP silently skips under expert offload
|
||||
# unless CLAV_TUNABLEOP_WITH_OFFLOAD=1, and forcing it corrupts numerics.
|
||||
# --expert-precise-memory stays DROPPED: it crashed the 2 x 262144 bf16 boot
|
||||
# (0.13 GiB free of 31.86 GiB, NRestarts=3 at ~6.7 min each). It survives in
|
||||
# the serve script as an opt-in; re-enable only with a pool/context reduction
|
||||
# that pays for the 2.35 GiB it moves into the arena.
|
||||
# --expert-precise-memory is ON (shape E). It selects the fitted overhead
|
||||
# (5.25 GiB) over the default (7.60 GiB = fitted + compile/graph-capture
|
||||
# headroom); the 2.35 GiB difference lands in the expert arena. MEASURED
|
||||
# 2026-10-02 by a boot probe on the shape-D base (production stopped, candidate
|
||||
# booted AS the service, plan read off the engine's own log):
|
||||
# runtime overhead 7.60 -> 5.25 GiB (registry Qwen4Exp…, precise
|
||||
# gfx1201 TP2 nbt4096/seqs16/len262144)
|
||||
# expert cache 14.87 -> 17.22 GiB
|
||||
# slots 6,496 -> 7,522 (+1,026 = +16%, 2.344 MiB each)
|
||||
# swap region 13.7 GiB / 5,986 slots, 133-134 per layer
|
||||
# boot healthy in 420 s, no OOM, no knife-edge
|
||||
# The flag was called a non-option after it crash-looped the 2 x 262144 bf16
|
||||
# boot (0.13 GiB free of 31.86 GiB, NRestarts=3 at ~6.7 min each) — that shape
|
||||
# was already at the ceiling, so precise-memory was the last 2.35 GiB over, not
|
||||
# the cause. At 200704 x 3 it fits with room; the run line also reports
|
||||
# 'expert_precise_memory': True in non-default args, so a silent argparse drop
|
||||
# would be visible in the boot log.
|
||||
FRANK_PRIMARY_PRECISE_MEMORY=1
|
||||
# GPU_MAX_HW_QUEUES caps the HSA hardware queues per GPU (AMD's own tuning
|
||||
# guide recommends 2 for this class of workload; the runtime default on Linux
|
||||
# is 4). Verified a REAL knob, not a no-op: the string is present in
|
||||
# /opt/rocm/core-10.0/lib/libhsa-runtime64.so. NOT A/B MEASURED — Frank's call,
|
||||
# shipped as a deliberate CPU-side tune alongside the 32x64 gather geometry.
|
||||
FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2
|
||||
# Persistent compile cache. MEASURED 2026-10-02: the image ships NO
|
||||
# /root/.cache at all (`ls: cannot access '/root/.cache'` in a fresh
|
||||
# container), and after our boot it holds /root/.cache/vllm at 1,154,714,333 B
|
||||
# in 4,665 files plus /root/.cache/comgr at 229 MB — both inside the
|
||||
# container's ephemeral layer, so every `podman rm -f` (every deploy, restart
|
||||
# and reboot) throws them away and re-pays the build. The engine's own log
|
||||
# names the loss: `torch.compile took 181.20 s in total`, of which 'Compiling a
|
||||
# graph for compile range (1, 1024) takes 173.19 s', followed by 'saved AOT
|
||||
# compiled function to /root/.cache/vllm/torch_compile_cache/…'. Mounting a
|
||||
# host dir at /root/.cache persists the vLLM AOT/torch_compile tree AND the
|
||||
# comgr HIP-kernel JIT cache, and hides nothing (the image has no such dir).
|
||||
# Safe to grow/prune: entries are keyed by version + config hash, so a stale
|
||||
# entry is a miss, never a wrong answer.
|
||||
FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary
|
||||
FRANK_PRIMARY_MAXLEN=200704
|
||||
FRANK_PRIMARY_SEQS=3
|
||||
# The POOL is deliberately kept at shape D's 3.15e9 (option A): because the pool
|
||||
# is not an arena lever, raising it back from a cut buys concurrency only, and
|
||||
# 3.15e9 preserves the measured 1.99x while precise-memory grows the arena. A
|
||||
# 2.2e9 pool was probed in the same window (277,686 tokens / 1.38x) and rejected
|
||||
# as the shipped value; it remains the conservative arm if free VRAM ever binds.
|
||||
FRANK_PRIMARY_KV_BYTES=3150000000
|
||||
FRANK_PRIMARY_KV_DTYPE=fp8
|
||||
FRANK_PRIMARY_MNBT=1024
|
||||
|
||||
@@ -11,7 +11,7 @@
|
||||
# every flag here was qualified by boot probe on 2026-09-29 — see
|
||||
# frank-primary.env for the qualification record).
|
||||
#
|
||||
# Shape (2026-10-02 shape D): MAXLEN 200704 x 3 slots, fp8 KV, KV_BYTES
|
||||
# Shape (2026-10-02 shape E): MAXLEN 200704 x 3 slots, fp8 KV, KV_BYTES
|
||||
# 3,150,000,000 PER RANK — the POOL, not the slot count, caps resident
|
||||
# conversations at ~2 full-length (measured: 398,658 tokens = 1.99x);
|
||||
# shorter requests use the spare blocks. --max-num-seqs stays 3 because SEQS=2
|
||||
@@ -21,8 +21,18 @@
|
||||
# decode, --max-num-batched-tokens 1024 (Frank's lock), MoE gather geometry
|
||||
# R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (+18.7% decode, grows
|
||||
# with depth), prefix caching, expert prefetch OFF (36% precision measured),
|
||||
# TunableOp off. The native-262144 and fp8+MTP4 shapes are retired to git tag
|
||||
# frank-primary-fp8-mtp-20261001.
|
||||
# TunableOp off. Shape E adds --expert-precise-memory (fitted 5.25 GiB overhead
|
||||
# instead of the 7.60 GiB default -> expert arena 17.22 GiB / 7,522 slots, +16%)
|
||||
# and GPU_MAX_HW_QUEUES=2. The native-262144 and fp8+MTP4 shapes are retired to
|
||||
# git tag frank-primary-fp8-mtp-20261001.
|
||||
#
|
||||
# THE COMPILE CACHE IS MOUNTED, and that is a boot-cost fix, not a tuning knob:
|
||||
# the image ships no /root/.cache at all, our first boot writes 1.1 GB of vLLM
|
||||
# AOT/torch_compile artifacts plus 229 MB of comgr JIT kernels into the
|
||||
# container's ephemeral layer, and `podman rm -f` deletes them — so every
|
||||
# deploy/restart/reboot re-pays `torch.compile took 181.20 s in total` (173.19 s
|
||||
# of it one graph). Mounting FRANK_PRIMARY_CACHE_DIR at /root/.cache makes all
|
||||
# later boots load the compiled artifact instead, and hides nothing.
|
||||
#
|
||||
# The root unit (scripts/frank-primary.service, User=dank,
|
||||
# LimitMEMLOCK=infinity) grants the pin budget for the tiered-expert host
|
||||
@@ -98,6 +108,15 @@ done
|
||||
LOG=/home/dank/logs/frank_primary.log
|
||||
mkdir -p "$(dirname "$LOG")"
|
||||
|
||||
# Persistent compile cache (see the header). Create it before the mount so even
|
||||
# the first, cold boot has a target; unset -> no mount, exactly the old
|
||||
# ephemeral behaviour. It carries /root/.cache/vllm (AOT + torch_compile) and
|
||||
# /root/.cache/comgr (HIP kernel JIT).
|
||||
CACHE_DIR="${FRANK_PRIMARY_CACHE_DIR:-}"
|
||||
if [ -n "$CACHE_DIR" ]; then
|
||||
mkdir -p "$CACHE_DIR"
|
||||
fi
|
||||
|
||||
# --device /dev/dri is the WHOLE DIRECTORY (individual renderD nodes enumerate
|
||||
# without the card nodes and HIP refuses contexts — "No CUDA GPUs are
|
||||
# available" while device_count() says 2). --ipc=host for the engine's shared
|
||||
@@ -109,8 +128,10 @@ podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
|
||||
--cap-add SYS_PTRACE \
|
||||
--device /dev/kfd --device /dev/dri --device /dev/mem \
|
||||
-e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \
|
||||
${FRANK_PRIMARY_GPU_MAX_HW_QUEUES:+-e GPU_MAX_HW_QUEUES="$FRANK_PRIMARY_GPU_MAX_HW_QUEUES"} \
|
||||
${FRANK_PRIMARY_GATHER_CHUNKS:+-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"} \
|
||||
${FRANK_PRIMARY_GATHER_LANES:+-e R4D_LRU_GATHER_LANES="$FRANK_PRIMARY_GATHER_LANES"} \
|
||||
${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache} \
|
||||
-v "$FRANK_PRIMARY_MODEL_DIR":/models:ro \
|
||||
"$FRANK_PRIMARY_IMAGE" /models \
|
||||
--served-model-name "$FRANK_PRIMARY_SERVED_NAME" --port 7770 \
|
||||
|
||||
@@ -275,7 +275,7 @@ Depth caveat: every number above is at a near-empty KV. Deep-context
|
||||
behavior (the gather win growing with depth, bf16's decode cost at depth)
|
||||
is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap.
|
||||
|
||||
### The deployed shape (2026-10-02: 3 slots x 200704, fp8, pool sized for 2 contexts)
|
||||
### The deployed shape (2026-10-02 shape E: 3 slots x 200704, fp8, precise-memory arena)
|
||||
|
||||
```
|
||||
FRANK_PRIMARY_MAXLEN=200704 200704-token window
|
||||
@@ -283,10 +283,66 @@ FRANK_PRIMARY_SEQS=3 --max-num-seqs 3 (SEQS=2 CANNOT boot - below)
|
||||
FRANK_PRIMARY_KV_BYTES=3150000000 PER RANK (fp8, 7,892 B/token -> 1.99x)
|
||||
FRANK_PRIMARY_KV_DTYPE=fp8 --kv-cache-dtype fp8 (quantized)
|
||||
FRANK_PRIMARY_MNBT=1024 --max-num-batched-tokens 1024
|
||||
FRANK_PRIMARY_PRECISE_MEMORY=1 --expert-precise-memory (overhead 7.60 -> 5.25 GiB)
|
||||
FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2 -e GPU_MAX_HW_QUEUES=2 (HSA queues/GPU)
|
||||
FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary mounted at /root/.cache
|
||||
R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (container env)
|
||||
NO --speculative-config, NO --expert-prefetch
|
||||
```
|
||||
|
||||
**`--expert-precise-memory` is the arena lever that does not touch the window.**
|
||||
The planner's overhead term is the only other place the arena can come from, and
|
||||
the flag swaps the default registry figure for the fitted one. Measured
|
||||
2026-10-02 by a boot probe on the shape-D base (production stopped, candidate
|
||||
booted AS the service; numbers are the engine's own plan lines):
|
||||
|
||||
| plan line | shape D | **shape E** |
|
||||
|---|---|---|
|
||||
| `runtime overhead (non-torch + buffers + activation)` | 7.60 GiB | **5.25 GiB** (`precise (gfx1201 TP2 nbt4096/seqs16/len262144)`) |
|
||||
| `expert cache space` | 14.87 GiB | **17.22 GiB** |
|
||||
| slots | 6,496 | **7,522** (+1,026, +16%) |
|
||||
| `swap region` | - | 13.7 GiB / 5,986 slots, 133-134 per layer |
|
||||
| boot | - | healthy in 420 s, no OOM |
|
||||
|
||||
`'expert_precise_memory': True` also appears in `non-default args`, so a silent
|
||||
argparse drop is visible in the boot log. The flag lands **in the arena, not the
|
||||
pool** - `--kv-cache-memory-bytes` is untouched at 3.15e9/rank (option A), because
|
||||
the pool is not an arena lever and cutting it further would only cost
|
||||
concurrency: a 2.2e9 pool was probed in the same window and reports 277,686
|
||||
tokens / **1.38x**, so it is the conservative fallback if free VRAM ever binds,
|
||||
not the shipped value.
|
||||
|
||||
**The compile cache is a boot-cost fix, and it was worth 173 s per boot.**
|
||||
Measured 2026-10-02: the image ships **no `/root/.cache` at all** (a fresh
|
||||
container: `No such file or directory`), and after a boot that directory holds
|
||||
`vllm` at **1,154,714,333 B in 4,665 files** plus `comgr` at 229 MB, all inside
|
||||
the container's ephemeral layer. Every
|
||||
`podman rm -f` - i.e. every deploy, restart and reboot - discards it, and the
|
||||
engine re-pays:
|
||||
|
||||
```
|
||||
Using cache directory: /root/.cache/vllm/torch_compile_cache/b265bd396a/rank_0_0/backbone
|
||||
Dynamo bytecode transform time: 4.99 s
|
||||
Compiling a graph for compile range (1, 1024) takes 173.19 s
|
||||
saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/…
|
||||
torch.compile took 181.20 s in total
|
||||
init engine (profile, create kv cache, warmup model) took 217.26 s (compilation: 181.20 s)
|
||||
```
|
||||
|
||||
Mounting a host directory at `/root/.cache` makes every later boot load the AOT
|
||||
artifact instead. The mount is deliberately the **parent**, not
|
||||
`/root/.cache/vllm`: `VLLM_CACHE_ROOT` defaults to the `vllm` child, and
|
||||
`comgr` is the HIP-kernel JIT cache, so one mount persists both — and it hides
|
||||
nothing, because the image ships no `/root/.cache`. The cache key
|
||||
includes the compile range, so the MNBT choice costs exactly one compilation.
|
||||
Entries are version- and config-hashed: a stale entry is a miss, never a wrong
|
||||
answer.
|
||||
|
||||
`GPU_MAX_HW_QUEUES=2` caps the HSA hardware queues per GPU. Verified a real knob
|
||||
rather than a no-op - the literal is read out of
|
||||
`/opt/rocm/core-10.0/lib/libhsa-runtime64.so` - and shipped deliberately
|
||||
alongside the 32x64 gather geometry. It is **not** A/B measured.
|
||||
|
||||
**The pool, not the slot count, sets how many conversations live at once.**
|
||||
Measured live at this shape: `GPU KV cache size: 398,658 tokens, Maximum
|
||||
concurrency for 200,704 tokens per request: 1.99x` (3,146,129,408 B reserved).
|
||||
@@ -337,25 +393,27 @@ max_model_len** - not from the `--kv-cache-memory-bytes` value:
|
||||
D(reserve) 0.57 GiB = D(arena) 0.57 GiB = +248 slots. The 6.1e9 -> 3.15e9
|
||||
pool cut (2.75 GiB) bought **zero** arena slots. It buys runtime HEADROOM,
|
||||
which is still worth having - the shape it replaced died with `0 bytes free`
|
||||
on a real 995-token request. Grow the arena by lowering `max_model_len`; the
|
||||
pool is the headroom lever.
|
||||
on a real 995-token request. Grow the arena by lowering `max_model_len`, or by
|
||||
taking the overhead term with `--expert-precise-memory` (below); the pool is the
|
||||
headroom lever only.
|
||||
|
||||
This replaced the 2026-10-01 shape — 2 x 262144 at bf16 KV with
|
||||
`--expert-precise-memory` — which crash-looped at graph capture: the arena is
|
||||
DEVICE-resident, the precise-memory flag drops the plan's overhead 7.60 ->
|
||||
5.25 GiB, and that shape had 0.13 GiB to spare (weights 6.28 + pool 6.80 +
|
||||
graphs 0.55 + arena 16.65 + non-torch ~1.45 = 31.73 of 31.86 GiB,
|
||||
NRestarts=3 at ~6.7 min each). The plain bf16 2 x 262144 boot also did not
|
||||
stay up: it left 0 bytes free at runtime and a real client request (995-token
|
||||
prompt, max_tokens=261149) killed the worker with `Tried to allocate 50.00 MiB
|
||||
#### The other arena lever: `--expert-precise-memory` (overhead 7.60 -> 5.25 GiB)
|
||||
|
||||
The flag is now ON. It was refused on 2026-10-01 for a reason that does not
|
||||
generalize: the 2 x 262144 bf16 shape was already at the ceiling (weights 6.28 +
|
||||
pool 6.80 + graphs 0.55 + arena 16.65 + non-torch ~1.45 = 31.73 of 31.86 GiB,
|
||||
NRestarts=3 at ~6.7 min each), so the 2.35 GiB the flag moves into the arena was
|
||||
the last stack over the line, not the cause. The plain bf16 2 x 262144 boot also
|
||||
did not stay up: 0 bytes free at runtime, and a real client request (995-token
|
||||
prompt, `max_tokens=261149`) killed the worker with `Tried to allocate 50.00 MiB
|
||||
... 0 bytes free`.
|
||||
|
||||
The correction is a pool cut plus a window trim, not a downgrade. Three
|
||||
full-length 200704-token contexts need ~4.75e9 B; pinning the pool at 3.15e9
|
||||
instead admits **1.99** of them. That is deliberate: the pool is the
|
||||
runtime-HEADROOM lever (the shape this replaced died with 0 bytes free), while
|
||||
the arena is sized by `max_model_len` alone. A third full-length request queues;
|
||||
short requests use the spare blocks.
|
||||
At 200704 x 3 the flag fits with room and is worth **+1,026 slots (+16%)**. What
|
||||
made it affordable is the window trim, not a pool cut: the arena is
|
||||
DEVICE-resident, the pool is not part of the budget's arithmetic, and the 200704
|
||||
window's KV reserve (1.91 GiB) plus a smaller attention footprint is what the
|
||||
plan needed. The pool stays at 3.15e9/rank deliberately (option A) - cutting it
|
||||
further would cost concurrency (2.2e9 -> 1.38x) and buy no slots.
|
||||
|
||||
Boot-probed 2026-10-02 (transient systemd units at the production uid +
|
||||
memlock, full-log capture; the rootless bench probe had masked the production
|
||||
@@ -366,9 +424,10 @@ memory environment). The night searched two concurrency contracts:
|
||||
| 3 x 262144, bf16, KV 11,047,010,304 | **OOM at graph capture** — `Tried to allocate 76.00 MiB ... 0 bytes free`, 30.43 GiB of 31.86 GiB already allocated by PyTorch |
|
||||
| 3 x 245760, bf16 | **OOM identically** |
|
||||
| 3 x 237568, bf16 | PASS — KV 718,606 tokens |
|
||||
| 2 x 262144, bf16 + `--expert-precise-memory` | **DEPLOYED then crashed** — 0.13 GiB free at boot, NRestarts=3 |
|
||||
| 2 x 262144, bf16 + `--expert-precise-memory` | **DEPLOYED then crashed** — 0.13 GiB free at boot, NRestarts=3 (the ceiling, not the flag) |
|
||||
| 2 x 262144, bf16 (no precise flag) | did not stay up either: 0 bytes free at runtime, worker killed by a real request |
|
||||
| **3 x 200704, fp8, pool sized for 2 contexts** | **DEPLOYED (2026-10-02)**: 398,658 tokens = 1.99x concurrency, arena 14.87 GiB / 6,496 slots, the 2.75 GiB/rank pool cut returned as runtime headroom |
|
||||
| 3 x 200704, fp8, pool sized for 2 contexts | **DEPLOYED 2026-10-02 (shape D)**: 398,658 tokens = 1.99x concurrency, arena 14.87 GiB / 6,496 slots, the 2.75 GiB/rank pool cut returned as runtime headroom |
|
||||
| **3 x 200704, fp8, + `--expert-precise-memory`** | **DEPLOYED 2026-10-02 (shape E)**: same 1.99x pool, overhead 5.25 GiB, arena 17.22 GiB / 7,522 slots, boot healthy in 420 s |
|
||||
|
||||
Six full-length contexts cannot fit at any KV dtype: the 6-seq graph/activation
|
||||
footprint leaves only 9.18 GiB for the pool, and 6 × 262144 needs 12.47 GB even
|
||||
|
||||
@@ -4,11 +4,14 @@ Documentation for the LLM inference fleet on the two hosts (2026-10-02):
|
||||
**Dankserver runs the tcclaviger RDNA4 vLLM fork on frank-primary** (the
|
||||
digest-pinned `tcclaviger/vllm` image driving `podman run` directly, serving
|
||||
orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both Radeon AI PRO R9700s
|
||||
on :7770, TP=2 with automatic tiered-expert offload, **shape D since
|
||||
on :7770, TP=2 with automatic tiered-expert offload, **shape E since
|
||||
2026-10-02: 3 slots x 200704 ctx** with the KV pool sized for **~2 full-length
|
||||
conversations** — measured 398,658 tokens = 1.99x — fp8 KV, no speculative
|
||||
decode, MNBT 1024, MoE gather 32x64, util 0.98, reasoning/tool parsers wired
|
||||
(qwen3 / qwen3_xml) with preserve_thinking on. `--max-num-seqs` must stay 3:
|
||||
(qwen3 / qwen3_xml) with preserve_thinking on, plus **`--expert-precise-memory`
|
||||
(the arena grows to 17.22 GiB / 7,522 slots, +16%), `GPU_MAX_HW_QUEUES=2`, and a
|
||||
mounted cache dir** (the image ships no `/root/.cache` at all; unmounted, every boot re-pays
|
||||
`torch.compile took 181.20 s`). `--max-num-seqs` must stay 3:
|
||||
SEQS 2 cannot compile this model. Rollback frank-primary-fp8-mtp-20261001),
|
||||
and Dankclaw
|
||||
runs two endpoints — **ROCmFPX llama.cpp (frank-large, port 7770:
|
||||
@@ -34,7 +37,8 @@ gfx1201, 32 GiB each]
|
||||
podman run, frank-primary, port 7770]
|
||||
C[Qwen3.8-Flash-Next FP8, no spec decode
|
||||
TP=2, expert offload automatic, 3 x 200704 ctx,
|
||||
fp8 KV pool ~2 full-length, MNBT 1024, gather 32x64, qwen3 parsers]
|
||||
fp8 KV pool ~2 full-length, MNBT 1024, gather 32x64, qwen3 parsers,
|
||||
expert-precise-memory arena 7,522 slots, /root/.cache mounted]
|
||||
end
|
||||
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
@@ -81,13 +85,15 @@ frank-small, port 7771]
|
||||
mmproj for vision, ~100 tok/s decode, ~2.7 GiB of the card left free.
|
||||
The KV arithmetic, the flash-attn dependency, rollback to Q6_K
|
||||
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — **ACTIVE since
|
||||
2026-10-02 (shape D)** (root systemd unit, port 7770): the tcclaviger RDNA4
|
||||
2026-10-02 (shape E)** (root systemd unit, port 7770): the tcclaviger RDNA4
|
||||
vLLM fork serving orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both
|
||||
R9700s, TP=2 with automatic expert offload — `--max-model-len 200704` x 3
|
||||
slots, fp8 KV, pool 3,150,000,000 B/rank (measured 398,658 tokens = 1.99x),
|
||||
MNBT 1024, gather 32x64, no spec decode, prefetch off. Why `--max-num-seqs`
|
||||
must stay 3 (SEQS 2 cannot compile this model), why the arena lever is
|
||||
`max_model_len` rather than the KV pool, and the measured envelope
|
||||
MNBT 1024, gather 32x64, no spec decode, prefetch off, plus
|
||||
`--expert-precise-memory` (arena 17.22 GiB / 7,522 slots), GPU_MAX_HW_QUEUES=2
|
||||
and a mounted compile cache. Why `--max-num-seqs`
|
||||
must stay 3 (SEQS 2 cannot compile this model), why the arena's levers are
|
||||
`max_model_len` and the overhead term rather than the KV pool, and the measured envelope
|
||||
- [[frank-primary-glm53-flash-w4a16]] — RETIRED 2026-09-30 → 10-01 (the
|
||||
failed GLM experiment; frank-primary runs Qwen FP8 again, see the intro):
|
||||
the tcclaviger RDNA4 vLLM fork
|
||||
|
||||
@@ -1106,3 +1106,85 @@ and then answers with nothing, and curl reports "Empty reply from server" (52).
|
||||
- **Operating lesson: `deploy.sh <target>` with no `--service` is a fleet-wide
|
||||
drain.** It restarts every service whose recorded revision is behind HEAD and
|
||||
whose paths changed since. To touch one service, name it.
|
||||
|
||||
## 2026-10-02 — frank-primary shape E: `--expert-precise-memory` (+1,026 arena slots), a mounted compile cache, and the P2P answer
|
||||
|
||||
Shape D left one question open — the plan's overhead term was reserving 2.35 GiB
|
||||
of compile headroom the boot never used — and the answer to Frank's free-VRAM
|
||||
question arrived together with a boot-cost bug that had been costing 181 s on
|
||||
every restart since the cutover.
|
||||
|
||||
- **`--expert-precise-memory` is real and it fits (option A).** Boot-probed on
|
||||
the shape-D base with production stopped, the candidate booted AS the service
|
||||
(`FRANK_PRIMARY_ENV_FILE=/tmp/probe-b.env bash scripts/serve-frank-primary.sh`,
|
||||
trap-guaranteed restore), the numbers read off the engine's own plan lines:
|
||||
|
||||
| plan line | shape D | shape E |
|
||||
|---|---|---|
|
||||
| `runtime overhead (non-torch + buffers + activation)` | 7.60 GiB | **5.25 GiB** (`precise (gfx1201 TP2 nbt4096/seqs16/len262144)`) |
|
||||
| `expert cache space` | 14.87 GiB | **17.22 GiB** |
|
||||
| slots | 6,496 | **7,522** (+1,026, +16 %) |
|
||||
| `swap region` | — | 13.7 GiB / 5,986 slots, 133-134 per layer |
|
||||
| boot | — | **healthy in 420 s**, no OOM |
|
||||
|
||||
The flag also prints `'expert_precise_memory': True` in `non-default args`, so
|
||||
a silent argparse drop cannot hide. It was refused on 2026-10-01 only because
|
||||
the 2 x 262144 bf16 shape was already 0.13 GiB under the ceiling — there the
|
||||
flag was the last stack over the line, not the cause.
|
||||
- **The pool was NOT cut (option A) because the pool is not an arena lever.**
|
||||
A 2.2e9 arm was probed in the same window and reports `277,686 tokens,
|
||||
1.38x` — measured, rejected as the shipped value, kept as the conservative
|
||||
fallback. With 3.15e9/rank the endpoint keeps shape D's 1.99x concurrency and
|
||||
gains the arena, which is the shape D follow-up had actually been looking for.
|
||||
- **THE COMPILE CACHE WAS BEING THROWN AWAY ON EVERY RESTART — 181 s each.**
|
||||
Measured: a fresh container has **no `/root/.cache` at all** (`No such file or
|
||||
directory`); after our boot it holds `/root/.cache/vllm` at
|
||||
**1,154,714,333 B in 4,665 files** plus `/root/.cache/comgr` at 229 MB, inside
|
||||
the ephemeral layer, and `podman rm -f` (every deploy, restart, reboot)
|
||||
discards them. The engine names the loss itself: `Compiling a graph for compile
|
||||
range (1, 1024) takes 173.19 s` ... `torch.compile took 181.20 s in total` ...
|
||||
`init engine (profile, create kv cache, warmup model) took 217.26 s
|
||||
(compilation: 181.20 s)`. The serve script now mounts
|
||||
`FRANK_PRIMARY_CACHE_DIR` at `/root/.cache` — the **parent**, so the vLLM
|
||||
AOT/torch_compile tree (`VLLM_CACHE_ROOT`, default `/root/.cache/vllm`) and
|
||||
the comgr JIT cache both persist, and nothing is hidden because the image has
|
||||
no such directory. Same class of fix as the recipe's
|
||||
`/cache/{triton,vllm,inductor}` volumes; the compile RANGE is part of the
|
||||
cache key, so an MNBT change costs one recompilation, not a permanent tax.
|
||||
- **`GPU_MAX_HW_QUEUES=2`** (HSA hardware queues per GPU) added at Frank's
|
||||
direction. Verified a real knob rather than a no-op — the literal is read from
|
||||
`/opt/rocm/core-10.0/lib/libhsa-runtime64.so` — and shipped deliberately
|
||||
alongside the 32x64 gather geometry. **Not A/B measured.**
|
||||
- **P2P: the hardware supports it, and the engine is already using it.** The
|
||||
live log shows `r4d all-reduce INSTALLED (rank=0 max=49152KB finegrained=T…)`
|
||||
on BOTH ranks and `Using ['R4D_AR', 'PYNCCL'] all-reduce backends` — R4D's
|
||||
P2P path, with RCCL as fallback, measured at **45.274 GB/s bidirectional**.
|
||||
Root ports `00:01.1`/`00:01.3` both report ACS `ReqRedir+`, so P2P hairpins at
|
||||
the root complex; there is no XGMI. The old FAIL predates the 2026-09-20 board
|
||||
swap. Kernel support confirmed: `CONFIG_HSA_AMD_P2P=y`, `CONFIG_PCI_P2PDMA=y`,
|
||||
`CONFIG_HSA_AMD=y`, `CONFIG_ZONE_DEVICE=y`, `iommu=pt` on the cmdline.
|
||||
- **Recipe diff (Frank's 3 x R9700 compose vs this podman run), ranked:**
|
||||
- *Acted on:* persistent compile caches (above — the one real operational
|
||||
gap), and `GPU_MAX_HW_QUEUES`.
|
||||
- *Retracted:* pinned `--compilation-config cudagraph_capture_sizes` was on my
|
||||
list; measured, it is **not** a gap here — capture is 3 PIECEWISE + 2 FULL
|
||||
graphs, `Graph capturing finished in 3 secs, took 0.31 GiB` (0.11 GiB on the
|
||||
lighter rank). No 15-52-size ladder to cut.
|
||||
- *Deliberate divergence, keep ours:* no MTP (the recipe has it; measured
|
||||
fastest here), `qwen3_xml` vs `qwen3_coder` (ours verified live), no
|
||||
`--override-generation-config` (server-side sampling defaults), published
|
||||
port + Caddy rather than `network_mode: host`.
|
||||
- *Non-gaps:* `shm_size 32gb` (ours is `--ipc=host`), `memlock -1` (same),
|
||||
`SYS_PTRACE` (same), `/dev/kfd` + `/dev/dri` (same), `--enable-chunked-prefill`
|
||||
(V1 default), `plecache` volume (the flag is not even passed), and
|
||||
`ROCR_VISIBLE_DEVICES=1,2,3` vs our `HIP_VISIBLE_DEVICES=0,1` — verified
|
||||
equivalent here: `rocm-smi --showid` puts the two R9700s at GPU[0]/GPU[1] and
|
||||
the 7700X iGPU at GPU[2], so our indices are the right two cards.
|
||||
- **Harness note, because it cost two probe attempts:** a probe that reported an
|
||||
instantaneous candidate death was a logging artifact of the inline SSH script,
|
||||
not a crash — the container and its serve script must be polled by recorded id
|
||||
and their logs dumped on the exit path, or a failure looks like a bad flag.
|
||||
The flag was fine both times.
|
||||
- Live: `health=200`, `NRestarts=0`, arena 17.22 GiB / 7,522 slots, KV 398,658
|
||||
tokens / 1.99x. Docs: [[dankserver-qwen38-fp8-expert-offload-tuning]].
|
||||
|
||||
|
||||
@@ -119,10 +119,25 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
||||
grep -q '^FRANK_PRIMARY_KV_BYTES=3150000000$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_MNBT=1024$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
|
||||
# --expert-precise-memory is NOT set: it put the bf16 2 x 262144 boot
|
||||
# 0.13 GiB from the 31.86 GiB ceiling (arena 16.65 device-resident) and
|
||||
# crash-looped at graph capture. The serve script keeps the opt-in flag.
|
||||
! grep -q '^FRANK_PRIMARY_PRECISE_MEMORY=' "$ENV_FILE"
|
||||
# --expert-precise-memory IS set (shape E, 2026-10-02). It was refused
|
||||
# while the 2 x 262144 bf16 boot sat 0.13 GiB under the 31.86 GiB ceiling
|
||||
# (arena 16.65 device-resident, NRestarts=3 at ~6.7 min each) — there it was
|
||||
# the last 2.35 GiB over the line, not the cause. Probed on the shape-D base
|
||||
# with production stopped: boots healthy in 420 s, plan reports runtime
|
||||
# overhead 5.25 GiB (not 7.60), expert cache 17.22 GiB / 7,522 slots
|
||||
# (+1,026 / +16% over 6,496). The pool is NOT cut with it (option A): the
|
||||
# pool is not an arena lever, so 3.15e9 keeps the measured 1.99x.
|
||||
grep -q '^FRANK_PRIMARY_PRECISE_MEMORY=1$' "$ENV_FILE"
|
||||
# GPU_MAX_HW_QUEUES caps HSA hw queues per GPU; real knob (the literal is in
|
||||
# libhsa-runtime64.so), passed inside the container as a bare name — the env
|
||||
# file keeps only the prefixed form.
|
||||
grep -q '^FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2$' "$ENV_FILE"
|
||||
# The compile cache is a BOOT-COST fix: the image ships no /root/.cache at
|
||||
# all, our boot writes 1.1 GB of vLLM AOT artifacts (4,665 files) plus
|
||||
# 229 MB of comgr JIT kernels into the ephemeral layer, and every
|
||||
# `podman rm -f` throws them away — re-paying `torch.compile took 181.20 s`
|
||||
# (173.19 s of it one graph) on every deploy, restart and reboot.
|
||||
grep -q '^FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_GATHER_LANES=64$' "$ENV_FILE"
|
||||
! grep -q '^FRANK_PRIMARY_SPEC=' "$ENV_FILE"
|
||||
@@ -150,6 +165,13 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
||||
# moves the plan's `room` term and therefore the expert arena: ~1,025
|
||||
# more slots at 2.344 MiB each.
|
||||
grep -q -- '${FRANK_PRIMARY_PRECISE_MEMORY:+--expert-precise-memory}' "$SERVE"
|
||||
# Shape E host/CPU wiring: the cache dir is created before it is mounted
|
||||
# (a cold boot must have a target) and the mount lands on /root/.cache,
|
||||
# whose vllm/ child is the path VLLM_CACHE_ROOT defaults to — the engine log
|
||||
# names it as `Using cache directory: /root/.cache/vllm/torch_compile_cache/…`.
|
||||
grep -q -- 'mkdir -p "$CACHE_DIR"' "$SERVE"
|
||||
grep -q -- '${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache}' "$SERVE"
|
||||
grep -q -- '-e GPU_MAX_HW_QUEUES="$FRANK_PRIMARY_GPU_MAX_HW_QUEUES"' "$SERVE"
|
||||
}
|
||||
|
||||
@test "tools and reasoning are wired — the client contract is served" {
|
||||
|
||||
Reference in New Issue
Block a user