From 1ac4d48371fa580bcd7789e7ba5b31ca8932827c Mon Sep 17 00:00:00 2001 From: Frank Date: Fri, 2 Oct 2026 14:37:57 -0700 Subject: [PATCH] =?UTF-8?q?frank-primary:=20shape=20E=20=E2=80=94=20--expe?= =?UTF-8?q?rt-precise-memory=20(arena=206,496->7,522=20slots),=20GPU=5FMAX?= =?UTF-8?q?=5FHW=5FQUEUES=3D2,=20and=20a=20mounted=20/root/.cache?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Measured by boot probe on the shape-D base (production stopped, candidate booted AS the service), plan numbers read off the engine's own log: runtime overhead 7.60 -> 5.25 GiB (precise gfx1201 TP2 registry) expert cache 14.87 -> 17.22 GiB slots 6,496 -> 7,522 (+1,026, +16%) swap region 13.7 GiB / 5,986 slots boot healthy in 420 s, no OOM '--expert-precise-memory': True also appears in non-default args, so a silent argparse drop is visible. The pool is NOT cut (option A) because the pool is not an arena lever: 3.15e9/rank keeps shape D's measured 1.99x concurrency, and a 2.2e9 arm probed in the same window (277,686 tokens, 1.38x) is recorded as the conservative fallback. Boot-cost bug found while diffing Frank's 3xR9700 recipe: the image ships no /root/.cache at all, our boot writes 1.1 GB of vLLM AOT/torch_compile artifacts (4,665 files) plus 229 MB of comgr JIT kernels into the ephemeral layer, and podman rm -f discards them — so every deploy/restart/reboot re-pays 'torch.compile took 181.20 s in total' (173.19 s of it one graph). Mounting FRANK_PRIMARY_CACHE_DIR at /root/.cache persists both and hides nothing. GPU_MAX_HW_QUEUES=2 added at Frank's direction (real HSA knob); not A/B measured. Retracted my own recipe-diff claim that cudagraph_capture_sizes needed pinning: capture is 3 PIECEWISE + 2 FULL graphs, 3 s, 0.31 GiB. Tests: 367/367. --- AGENTS.md | 33 ++++++- scripts/frank-primary.env | 76 ++++++++++++--- scripts/serve-frank-primary.sh | 27 +++++- ...server-qwen38-fp8-expert-offload-tuning.md | 97 +++++++++++++++---- serverAppConfigs-docs/LLM/index.md | 20 ++-- serverAppConfigs-docs/log.md | 82 ++++++++++++++++ tests/frank_primary_unit.bats | 30 +++++- 7 files changed, 314 insertions(+), 51 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 519f114..d8b574d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| -| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 1024, MoE gather 32×64, **3 slots × 200704 ctx** at KV_BYTES 3,150,000,000/rank — the **pool, not the slot count, caps resident conversations at ~2 full-length** (measured live: `GPU KV cache size: 398,658 tokens, Maximum concurrency for 200,704 tokens per request: 1.99×`), and shorter requests use the spare blocks. **`--max-num-seqs` must stay 3: SEQS=2 cannot boot this model** — torch.fx `ConstraintViolationError` on `query_start_loc`, measured at BOTH MNBT 1024 and 2048. The pool cut buys runtime **headroom, NOT arena**: the arena is sized by `max_model_len` via the registry KV reserve (262144 → 6,248 slots; 200704 → 6,496) — util 0.98, and the client contract wired: `--reasoning-parser … +| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 1024, MoE gather 32×64, **`--expert-precise-memory`** (fitted 5.25 GiB overhead, not the 7.60 GiB default → expert arena **17.22 GiB / 7,522 slots**, +16 %), **`GPU_MAX_HW_QUEUES=2`**, **mounted cache dir** at `/root/.cache` (the image ships no `/root/.cache` at all; unmounted, every boot re-pays `torch.compile took 181.20 s`), **3 slots × 200704 ctx** at KV_BYTES 3,150,000,000/rank — the **pool, not the slot count, caps resident conversations at ~2 full-length** (measured live: `GPU KV cache size: 398,658 tokens, Maximum concurrency for 200,704 tokens per request: 1.99×`), and shorter requests use the spare blocks. **`--max-num-seqs` must stay 3: SEQS=2 cannot boot this model** — torch.fx `ConstraintViolationError` on `query_start_loc`, measured at BOTH MNBT 1024 and 2048. The pool cut buys runtime **headroom, NOT arena**: the arena is sized by `max_model_len` via the registry KV reserve (262144 → 6,248 slots; 200704 → 6,496) plus the overhead term (`--expert-precise-memory` → 7,522) — util 0.98, and the client contract wired: `--reasoning-parser … | Dankclaw | `frank-large` | **ROCmFPX llama.cpp fork** (`kingjones30/ROCmFPX` @ `36e9acd`, gfx1151, built by `scripts/build-rocmfpx.sh`; bare `llama-server`, root systemd unit) | `kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF` **`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf`** (26.28 GiB, ftype 115 ≈ 8.39 bpw: 353 F32 + 187 Q8_0 + 311 `GGML_TYPE_Q8_0_ROCMFPX` tensors — only this fork can read them), **MTP spec decode from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` (n-max 4, pinned to ROCm0)**, **1 session × 262,144 (native window)** with **f16 KV**, the bundled BF16 mmproj for vision, thinking on with the card's recipe (temp 1.0 / top_p 0.95 / top_k 20). Measured 2026-10-02: decode **17.0–17.1 t/s** with MTP vs **7.6 without (+123 %)**, draft acceptance 0.462, prefill 325–351 t/s, prompt-cache replay 0.5 s, boot→healthy 16 s, GTT 64.5 GiB. Cut over 2026-10-02 from gufo; rollback tag `frank-large-gufo-q4kxl-20261001` | 7770 | | Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** (9B dense qwen3_5 hybrid, 6.4 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **3 × 180,224** (pool 540,672 — NOT the native window; three contexts cost 81,920 tokens per conversation) with **asymmetric KV f16 K + q8_0 V** under flash attention, 21,684 used / **2,441 free** of 24,576 MiB, 106.8 tok/s decode, 3,814 t/s prefill. Resized twice 2026-10-02 (Q6_K → Q5_K_M → 3 slots + vision → this pool); rollback `models/MiMo-Q6K/` or `PARALLEL=2 CTX_SIZE=524288` | 7771 | @@ -453,16 +453,39 @@ sizes that input buffer `max_num_reqs + 1` and `_dummy_run` sets ((1,1024) vs (1,2048)). `--enforce-eager` bypasses the path but drops CUDA graphs. The same failure class killed the 2026-10-01 bf16 2×262144 shape. -**The expert arena's lever is `max_model_len`, not `--kv-cache-memory-bytes`.** +**The expert arena has TWO levers: `max_model_len` and the overhead term.** The fork's planner sizes expert cache as `budget − non-expert − KV reserve − graphs − overhead`, and the KV reserve is registry-priced at 9190 B/token × `max_model_len`: 262144 → 2.48 GiB reserve → arena 14.30 GiB / **6,248 slots**; 200704 → 1.91 GiB → 14.87 GiB / **6,496 slots**. Δreserve 0.57 GiB = Δarena 0.57 GiB. The 6.1e9 → 3.15e9 pool cut bought **zero** arena; it buys runtime headroom (the shape it replaced died with `0 bytes free` on a real 995-token -request). Grow the arena by lowering `max_model_len`. The arena (2.344 MiB per -slot) is the decode-critical region — cutting it for KV headroom collapses the -offload hit rate. +request). The second lever is `--expert-precise-memory`, **ON since +2026-10-02**: it swaps the registry's default overhead (7.60 GiB = fitted + +compile/graph-capture headroom) for the fitted figure, and the 2.35 GiB +difference lands in the arena — measured plan lines at 200704 × 3: overhead +5.25 GiB, expert cache **17.22 GiB / 7,522 slots** (+1,026 / +16 %), swap region +13.7 GiB / 5,986 slots, boot healthy in 420 s. The pool stayed at 3.15e9 +deliberately: because the pool is not an arena lever, cutting it would cost +concurrency (a 2.2e9 probe reports 277,686 tokens / **1.38×**) and buy no slots. +The arena (2.344 MiB per slot) is the decode-critical region — cutting it for KV +headroom collapses the offload hit rate. + +**A container's compile cache does NOT survive `podman rm -f`, and on this +endpoint that is 181 s per boot.** Measured 2026-10-02: the image ships **no +`/root/.cache` at all** (fresh container: `No such file or directory`); after a +boot it holds `/root/.cache/vllm` at **1,154,714,333 B in 4,665 files** plus +`/root/.cache/comgr` at 229 MB, all in the ephemeral layer, and the engine +prints `torch.compile took 181.20 s in total` (173.19 s of it one graph, +`Compiling a graph for compile range (1, 1024)`). Every deploy, restart and +reboot re-pays it. The serve script now mounts +`FRANK_PRIMARY_CACHE_DIR` at `/root/.cache` — the parent, so the vLLM AOT / +torch_compile tree (`VLLM_CACHE_ROOT`, whose default is +`/root/.cache/vllm`) and the comgr HIP-kernel JIT cache both persist, and +nothing in the image is hidden because the image has no such directory. The +compile RANGE is part of the cache key, so an MNBT change costs one +recompilation, not a permanent tax. Same class of fix as the recipe's +`/cache/{triton,vllm,inductor}` volumes. Measured 2026-09-29 tuning campaign (fp8 + MTP4 era, since retired): decode **37.4-40.6 tok/s, acceptance ~54 %**; the 256k ctx arm measured 34.1-36.5; diff --git a/scripts/frank-primary.env b/scripts/frank-primary.env index 018c5d8..6dd151f 100644 --- a/scripts/frank-primary.env +++ b/scripts/frank-primary.env @@ -1,13 +1,20 @@ # frank-primary.env — tcclaviger RDNA4 vLLM production config (dankserver :7770). # -# 2026-10-02 shape D: fp8 KV, MAXLEN 200704 x SEQS 3, the KV pool cut to ~2 -# full-length contexts. The window shrinks from the native 262144 to 200704 and -# the pool from 6.1e9 to 3.15e9; --max-num-seqs stays 3 because SEQS=2 CANNOT -# boot this model (measured 2x2 below). Same digest-pinned tcclaviger RDNA4 -# fork. Rollback for the fp8+MTP4 era is git tag frank-primary-fp8-mtp-20261001. +# 2026-10-02 shape E: fp8 KV, MAXLEN 200704 x SEQS 3, pool 3.15e9/rank, and +# --expert-precise-memory ON. The window shrink from the native 262144 and the +# --max-num-seqs 3 lock both come from shape D (SEQS=2 CANNOT boot this model — +# the measured 2x2 is below); shape E adds the precise-overhead plan, which +# trades the default 7.60 GiB compile headroom for the fitted 5.25 GiB and puts +# the 2.35 GiB difference into the expert arena: 6,496 -> 7,522 slots (+16%). +# The pool is deliberately NOT cut (option A): the pool is not an arena lever +# (measured), so 3.15e9 keeps shape D's 1.99x concurrency while the arena grows. +# Same digest-pinned tcclaviger RDNA4 fork. Rollback for the fp8+MTP4 era is git +# tag frank-primary-fp8-mtp-20261001. # # HIP reads bare env vars TEMP, HOST, GPU_MAX_HEAP_SIZE — every key here is -# prefixed FRANK_PRIMARY_ so none can leak. +# prefixed FRANK_PRIMARY_ so none can leak. The one bare name we WANT inside the +# container (GPU_MAX_HW_QUEUES) is held here under the prefix and passed on the +# run line as `-e GPU_MAX_HW_QUEUES=…`. # # --- runtime ---------------------------------------------------------------- @@ -29,7 +36,7 @@ FRANK_PRIMARY_PUBLISH=7770:7770 # hoc experiment identity (both hold the GPUs / name registrations). FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8" -# --- qualified engine shape (2026-10-02, shape D) --------------------------- +# --- qualified engine shape (2026-10-02, shape E) --------------------------- # WHAT THE SLOT COUNT DOES AND DOES NOT DO. The endpoint advertises 3 slots # (SEQS 3), but the POOL, not the slot count, decides how many full-length # conversations are resident. MEASURED LIVE 2026-10-02 at this shape: @@ -66,7 +73,9 @@ FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8" # d(reserve) 0.57 GiB = d(arena) 0.57 GiB = +248 slots. The 6.1e9 -> 3.15e9 # pool cut buys ZERO arena; it buys runtime HEADROOM (the shape it replaced # died with 0 bytes free on a real 995-token request). To grow the arena, -# lower MAXLEN — not KV_BYTES. +# lower MAXLEN — not KV_BYTES. The OTHER arena lever is the overhead term: +# --expert-precise-memory (shape E) takes it from 7.60 to 5.25 GiB and lands +# those 2.35 GiB in the arena — 14.87 -> 17.22 GiB / 6,496 -> 7,522 slots. # # GATHER 32x64 stays: R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 # measured +18.7% decode vs default geometry and the win grows with depth. @@ -75,18 +84,59 @@ FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8" # 1.36 s); 1024-4096 are interchangeable on decode; 12288 costs 2.5% and # arena. 1024 keeps the workspace floor so the arena stays max. (Side note # measured 2026-10-02: MNBT 2048 compiles the first graph in 26 s vs 172 s at -# 1024 — a startup-cost lever, not a correctness one.) +# 1024 — a startup-cost lever, not a correctness one. The compile RANGE is part +# of the cache key, so FRANK_PRIMARY_CACHE_DIR pays for whichever rung +# is chosen exactly once.) # PREFETCH IS ABSENT BY LAW: expert-cache telemetry measured the prefetcher at # 36% precision (36,720 slabs prefetched, 13,369 served) — roughly half the # gather traffic was waste. Never re-add --expert-prefetch without a fresh A/B. # TunableOp stays OFF: CLAV_TUNABLEOP_SWEEP silently skips under expert offload # unless CLAV_TUNABLEOP_WITH_OFFLOAD=1, and forcing it corrupts numerics. -# --expert-precise-memory stays DROPPED: it crashed the 2 x 262144 bf16 boot -# (0.13 GiB free of 31.86 GiB, NRestarts=3 at ~6.7 min each). It survives in -# the serve script as an opt-in; re-enable only with a pool/context reduction -# that pays for the 2.35 GiB it moves into the arena. +# --expert-precise-memory is ON (shape E). It selects the fitted overhead +# (5.25 GiB) over the default (7.60 GiB = fitted + compile/graph-capture +# headroom); the 2.35 GiB difference lands in the expert arena. MEASURED +# 2026-10-02 by a boot probe on the shape-D base (production stopped, candidate +# booted AS the service, plan read off the engine's own log): +# runtime overhead 7.60 -> 5.25 GiB (registry Qwen4Exp…, precise +# gfx1201 TP2 nbt4096/seqs16/len262144) +# expert cache 14.87 -> 17.22 GiB +# slots 6,496 -> 7,522 (+1,026 = +16%, 2.344 MiB each) +# swap region 13.7 GiB / 5,986 slots, 133-134 per layer +# boot healthy in 420 s, no OOM, no knife-edge +# The flag was called a non-option after it crash-looped the 2 x 262144 bf16 +# boot (0.13 GiB free of 31.86 GiB, NRestarts=3 at ~6.7 min each) — that shape +# was already at the ceiling, so precise-memory was the last 2.35 GiB over, not +# the cause. At 200704 x 3 it fits with room; the run line also reports +# 'expert_precise_memory': True in non-default args, so a silent argparse drop +# would be visible in the boot log. +FRANK_PRIMARY_PRECISE_MEMORY=1 +# GPU_MAX_HW_QUEUES caps the HSA hardware queues per GPU (AMD's own tuning +# guide recommends 2 for this class of workload; the runtime default on Linux +# is 4). Verified a REAL knob, not a no-op: the string is present in +# /opt/rocm/core-10.0/lib/libhsa-runtime64.so. NOT A/B MEASURED — Frank's call, +# shipped as a deliberate CPU-side tune alongside the 32x64 gather geometry. +FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2 +# Persistent compile cache. MEASURED 2026-10-02: the image ships NO +# /root/.cache at all (`ls: cannot access '/root/.cache'` in a fresh +# container), and after our boot it holds /root/.cache/vllm at 1,154,714,333 B +# in 4,665 files plus /root/.cache/comgr at 229 MB — both inside the +# container's ephemeral layer, so every `podman rm -f` (every deploy, restart +# and reboot) throws them away and re-pays the build. The engine's own log +# names the loss: `torch.compile took 181.20 s in total`, of which 'Compiling a +# graph for compile range (1, 1024) takes 173.19 s', followed by 'saved AOT +# compiled function to /root/.cache/vllm/torch_compile_cache/…'. Mounting a +# host dir at /root/.cache persists the vLLM AOT/torch_compile tree AND the +# comgr HIP-kernel JIT cache, and hides nothing (the image has no such dir). +# Safe to grow/prune: entries are keyed by version + config hash, so a stale +# entry is a miss, never a wrong answer. +FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary FRANK_PRIMARY_MAXLEN=200704 FRANK_PRIMARY_SEQS=3 +# The POOL is deliberately kept at shape D's 3.15e9 (option A): because the pool +# is not an arena lever, raising it back from a cut buys concurrency only, and +# 3.15e9 preserves the measured 1.99x while precise-memory grows the arena. A +# 2.2e9 pool was probed in the same window (277,686 tokens / 1.38x) and rejected +# as the shipped value; it remains the conservative arm if free VRAM ever binds. FRANK_PRIMARY_KV_BYTES=3150000000 FRANK_PRIMARY_KV_DTYPE=fp8 FRANK_PRIMARY_MNBT=1024 diff --git a/scripts/serve-frank-primary.sh b/scripts/serve-frank-primary.sh index e2c7572..70e0703 100755 --- a/scripts/serve-frank-primary.sh +++ b/scripts/serve-frank-primary.sh @@ -11,7 +11,7 @@ # every flag here was qualified by boot probe on 2026-09-29 — see # frank-primary.env for the qualification record). # -# Shape (2026-10-02 shape D): MAXLEN 200704 x 3 slots, fp8 KV, KV_BYTES +# Shape (2026-10-02 shape E): MAXLEN 200704 x 3 slots, fp8 KV, KV_BYTES # 3,150,000,000 PER RANK — the POOL, not the slot count, caps resident # conversations at ~2 full-length (measured: 398,658 tokens = 1.99x); # shorter requests use the spare blocks. --max-num-seqs stays 3 because SEQS=2 @@ -21,8 +21,18 @@ # decode, --max-num-batched-tokens 1024 (Frank's lock), MoE gather geometry # R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (+18.7% decode, grows # with depth), prefix caching, expert prefetch OFF (36% precision measured), -# TunableOp off. The native-262144 and fp8+MTP4 shapes are retired to git tag -# frank-primary-fp8-mtp-20261001. +# TunableOp off. Shape E adds --expert-precise-memory (fitted 5.25 GiB overhead +# instead of the 7.60 GiB default -> expert arena 17.22 GiB / 7,522 slots, +16%) +# and GPU_MAX_HW_QUEUES=2. The native-262144 and fp8+MTP4 shapes are retired to +# git tag frank-primary-fp8-mtp-20261001. +# +# THE COMPILE CACHE IS MOUNTED, and that is a boot-cost fix, not a tuning knob: +# the image ships no /root/.cache at all, our first boot writes 1.1 GB of vLLM +# AOT/torch_compile artifacts plus 229 MB of comgr JIT kernels into the +# container's ephemeral layer, and `podman rm -f` deletes them — so every +# deploy/restart/reboot re-pays `torch.compile took 181.20 s in total` (173.19 s +# of it one graph). Mounting FRANK_PRIMARY_CACHE_DIR at /root/.cache makes all +# later boots load the compiled artifact instead, and hides nothing. # # The root unit (scripts/frank-primary.service, User=dank, # LimitMEMLOCK=infinity) grants the pin budget for the tiered-expert host @@ -98,6 +108,15 @@ done LOG=/home/dank/logs/frank_primary.log mkdir -p "$(dirname "$LOG")" +# Persistent compile cache (see the header). Create it before the mount so even +# the first, cold boot has a target; unset -> no mount, exactly the old +# ephemeral behaviour. It carries /root/.cache/vllm (AOT + torch_compile) and +# /root/.cache/comgr (HIP kernel JIT). +CACHE_DIR="${FRANK_PRIMARY_CACHE_DIR:-}" +if [ -n "$CACHE_DIR" ]; then + mkdir -p "$CACHE_DIR" +fi + # --device /dev/dri is the WHOLE DIRECTORY (individual renderD nodes enumerate # without the card nodes and HIP refuses contexts — "No CUDA GPUs are # available" while device_count() says 2). --ipc=host for the engine's shared @@ -109,8 +128,10 @@ podman run -d --name "$FRANK_PRIMARY_CONTAINER" \ --cap-add SYS_PTRACE \ --device /dev/kfd --device /dev/dri --device /dev/mem \ -e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \ + ${FRANK_PRIMARY_GPU_MAX_HW_QUEUES:+-e GPU_MAX_HW_QUEUES="$FRANK_PRIMARY_GPU_MAX_HW_QUEUES"} \ ${FRANK_PRIMARY_GATHER_CHUNKS:+-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"} \ ${FRANK_PRIMARY_GATHER_LANES:+-e R4D_LRU_GATHER_LANES="$FRANK_PRIMARY_GATHER_LANES"} \ + ${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache} \ -v "$FRANK_PRIMARY_MODEL_DIR":/models:ro \ "$FRANK_PRIMARY_IMAGE" /models \ --served-model-name "$FRANK_PRIMARY_SERVED_NAME" --port 7770 \ diff --git a/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md b/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md index ecffe15..f1ab43d 100644 --- a/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md +++ b/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md @@ -275,7 +275,7 @@ Depth caveat: every number above is at a near-empty KV. Deep-context behavior (the gather win growing with depth, bf16's decode cost at depth) is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap. -### The deployed shape (2026-10-02: 3 slots x 200704, fp8, pool sized for 2 contexts) +### The deployed shape (2026-10-02 shape E: 3 slots x 200704, fp8, precise-memory arena) ``` FRANK_PRIMARY_MAXLEN=200704 200704-token window @@ -283,10 +283,66 @@ FRANK_PRIMARY_SEQS=3 --max-num-seqs 3 (SEQS=2 CANNOT boot - below) FRANK_PRIMARY_KV_BYTES=3150000000 PER RANK (fp8, 7,892 B/token -> 1.99x) FRANK_PRIMARY_KV_DTYPE=fp8 --kv-cache-dtype fp8 (quantized) FRANK_PRIMARY_MNBT=1024 --max-num-batched-tokens 1024 +FRANK_PRIMARY_PRECISE_MEMORY=1 --expert-precise-memory (overhead 7.60 -> 5.25 GiB) +FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2 -e GPU_MAX_HW_QUEUES=2 (HSA queues/GPU) +FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary mounted at /root/.cache R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (container env) NO --speculative-config, NO --expert-prefetch ``` +**`--expert-precise-memory` is the arena lever that does not touch the window.** +The planner's overhead term is the only other place the arena can come from, and +the flag swaps the default registry figure for the fitted one. Measured +2026-10-02 by a boot probe on the shape-D base (production stopped, candidate +booted AS the service; numbers are the engine's own plan lines): + +| plan line | shape D | **shape E** | +|---|---|---| +| `runtime overhead (non-torch + buffers + activation)` | 7.60 GiB | **5.25 GiB** (`precise (gfx1201 TP2 nbt4096/seqs16/len262144)`) | +| `expert cache space` | 14.87 GiB | **17.22 GiB** | +| slots | 6,496 | **7,522** (+1,026, +16%) | +| `swap region` | - | 13.7 GiB / 5,986 slots, 133-134 per layer | +| boot | - | healthy in 420 s, no OOM | + +`'expert_precise_memory': True` also appears in `non-default args`, so a silent +argparse drop is visible in the boot log. The flag lands **in the arena, not the +pool** - `--kv-cache-memory-bytes` is untouched at 3.15e9/rank (option A), because +the pool is not an arena lever and cutting it further would only cost +concurrency: a 2.2e9 pool was probed in the same window and reports 277,686 +tokens / **1.38x**, so it is the conservative fallback if free VRAM ever binds, +not the shipped value. + +**The compile cache is a boot-cost fix, and it was worth 173 s per boot.** +Measured 2026-10-02: the image ships **no `/root/.cache` at all** (a fresh +container: `No such file or directory`), and after a boot that directory holds +`vllm` at **1,154,714,333 B in 4,665 files** plus `comgr` at 229 MB, all inside +the container's ephemeral layer. Every +`podman rm -f` - i.e. every deploy, restart and reboot - discards it, and the +engine re-pays: + +``` +Using cache directory: /root/.cache/vllm/torch_compile_cache/b265bd396a/rank_0_0/backbone +Dynamo bytecode transform time: 4.99 s +Compiling a graph for compile range (1, 1024) takes 173.19 s +saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/… +torch.compile took 181.20 s in total +init engine (profile, create kv cache, warmup model) took 217.26 s (compilation: 181.20 s) +``` + +Mounting a host directory at `/root/.cache` makes every later boot load the AOT +artifact instead. The mount is deliberately the **parent**, not +`/root/.cache/vllm`: `VLLM_CACHE_ROOT` defaults to the `vllm` child, and +`comgr` is the HIP-kernel JIT cache, so one mount persists both — and it hides +nothing, because the image ships no `/root/.cache`. The cache key +includes the compile range, so the MNBT choice costs exactly one compilation. +Entries are version- and config-hashed: a stale entry is a miss, never a wrong +answer. + +`GPU_MAX_HW_QUEUES=2` caps the HSA hardware queues per GPU. Verified a real knob +rather than a no-op - the literal is read out of +`/opt/rocm/core-10.0/lib/libhsa-runtime64.so` - and shipped deliberately +alongside the 32x64 gather geometry. It is **not** A/B measured. + **The pool, not the slot count, sets how many conversations live at once.** Measured live at this shape: `GPU KV cache size: 398,658 tokens, Maximum concurrency for 200,704 tokens per request: 1.99x` (3,146,129,408 B reserved). @@ -337,25 +393,27 @@ max_model_len** - not from the `--kv-cache-memory-bytes` value: D(reserve) 0.57 GiB = D(arena) 0.57 GiB = +248 slots. The 6.1e9 -> 3.15e9 pool cut (2.75 GiB) bought **zero** arena slots. It buys runtime HEADROOM, which is still worth having - the shape it replaced died with `0 bytes free` -on a real 995-token request. Grow the arena by lowering `max_model_len`; the -pool is the headroom lever. +on a real 995-token request. Grow the arena by lowering `max_model_len`, or by +taking the overhead term with `--expert-precise-memory` (below); the pool is the +headroom lever only. -This replaced the 2026-10-01 shape — 2 x 262144 at bf16 KV with -`--expert-precise-memory` — which crash-looped at graph capture: the arena is -DEVICE-resident, the precise-memory flag drops the plan's overhead 7.60 -> -5.25 GiB, and that shape had 0.13 GiB to spare (weights 6.28 + pool 6.80 + -graphs 0.55 + arena 16.65 + non-torch ~1.45 = 31.73 of 31.86 GiB, -NRestarts=3 at ~6.7 min each). The plain bf16 2 x 262144 boot also did not -stay up: it left 0 bytes free at runtime and a real client request (995-token -prompt, max_tokens=261149) killed the worker with `Tried to allocate 50.00 MiB +#### The other arena lever: `--expert-precise-memory` (overhead 7.60 -> 5.25 GiB) + +The flag is now ON. It was refused on 2026-10-01 for a reason that does not +generalize: the 2 x 262144 bf16 shape was already at the ceiling (weights 6.28 + +pool 6.80 + graphs 0.55 + arena 16.65 + non-torch ~1.45 = 31.73 of 31.86 GiB, +NRestarts=3 at ~6.7 min each), so the 2.35 GiB the flag moves into the arena was +the last stack over the line, not the cause. The plain bf16 2 x 262144 boot also +did not stay up: 0 bytes free at runtime, and a real client request (995-token +prompt, `max_tokens=261149`) killed the worker with `Tried to allocate 50.00 MiB ... 0 bytes free`. -The correction is a pool cut plus a window trim, not a downgrade. Three -full-length 200704-token contexts need ~4.75e9 B; pinning the pool at 3.15e9 -instead admits **1.99** of them. That is deliberate: the pool is the -runtime-HEADROOM lever (the shape this replaced died with 0 bytes free), while -the arena is sized by `max_model_len` alone. A third full-length request queues; -short requests use the spare blocks. +At 200704 x 3 the flag fits with room and is worth **+1,026 slots (+16%)**. What +made it affordable is the window trim, not a pool cut: the arena is +DEVICE-resident, the pool is not part of the budget's arithmetic, and the 200704 +window's KV reserve (1.91 GiB) plus a smaller attention footprint is what the +plan needed. The pool stays at 3.15e9/rank deliberately (option A) - cutting it +further would cost concurrency (2.2e9 -> 1.38x) and buy no slots. Boot-probed 2026-10-02 (transient systemd units at the production uid + memlock, full-log capture; the rootless bench probe had masked the production @@ -366,9 +424,10 @@ memory environment). The night searched two concurrency contracts: | 3 x 262144, bf16, KV 11,047,010,304 | **OOM at graph capture** — `Tried to allocate 76.00 MiB ... 0 bytes free`, 30.43 GiB of 31.86 GiB already allocated by PyTorch | | 3 x 245760, bf16 | **OOM identically** | | 3 x 237568, bf16 | PASS — KV 718,606 tokens | -| 2 x 262144, bf16 + `--expert-precise-memory` | **DEPLOYED then crashed** — 0.13 GiB free at boot, NRestarts=3 | +| 2 x 262144, bf16 + `--expert-precise-memory` | **DEPLOYED then crashed** — 0.13 GiB free at boot, NRestarts=3 (the ceiling, not the flag) | | 2 x 262144, bf16 (no precise flag) | did not stay up either: 0 bytes free at runtime, worker killed by a real request | -| **3 x 200704, fp8, pool sized for 2 contexts** | **DEPLOYED (2026-10-02)**: 398,658 tokens = 1.99x concurrency, arena 14.87 GiB / 6,496 slots, the 2.75 GiB/rank pool cut returned as runtime headroom | +| 3 x 200704, fp8, pool sized for 2 contexts | **DEPLOYED 2026-10-02 (shape D)**: 398,658 tokens = 1.99x concurrency, arena 14.87 GiB / 6,496 slots, the 2.75 GiB/rank pool cut returned as runtime headroom | +| **3 x 200704, fp8, + `--expert-precise-memory`** | **DEPLOYED 2026-10-02 (shape E)**: same 1.99x pool, overhead 5.25 GiB, arena 17.22 GiB / 7,522 slots, boot healthy in 420 s | Six full-length contexts cannot fit at any KV dtype: the 6-seq graph/activation footprint leaves only 9.18 GiB for the pool, and 6 × 262144 needs 12.47 GB even diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index 74620ce..eda91a3 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -4,11 +4,14 @@ Documentation for the LLM inference fleet on the two hosts (2026-10-02): **Dankserver runs the tcclaviger RDNA4 vLLM fork on frank-primary** (the digest-pinned `tcclaviger/vllm` image driving `podman run` directly, serving orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both Radeon AI PRO R9700s -on :7770, TP=2 with automatic tiered-expert offload, **shape D since +on :7770, TP=2 with automatic tiered-expert offload, **shape E since 2026-10-02: 3 slots x 200704 ctx** with the KV pool sized for **~2 full-length conversations** — measured 398,658 tokens = 1.99x — fp8 KV, no speculative decode, MNBT 1024, MoE gather 32x64, util 0.98, reasoning/tool parsers wired -(qwen3 / qwen3_xml) with preserve_thinking on. `--max-num-seqs` must stay 3: +(qwen3 / qwen3_xml) with preserve_thinking on, plus **`--expert-precise-memory` +(the arena grows to 17.22 GiB / 7,522 slots, +16%), `GPU_MAX_HW_QUEUES=2`, and a +mounted cache dir** (the image ships no `/root/.cache` at all; unmounted, every boot re-pays +`torch.compile took 181.20 s`). `--max-num-seqs` must stay 3: SEQS 2 cannot compile this model. Rollback frank-primary-fp8-mtp-20261001), and Dankclaw runs two endpoints — **ROCmFPX llama.cpp (frank-large, port 7770: @@ -34,7 +37,8 @@ gfx1201, 32 GiB each] podman run, frank-primary, port 7770] C[Qwen3.8-Flash-Next FP8, no spec decode TP=2, expert offload automatic, 3 x 200704 ctx, -fp8 KV pool ~2 full-length, MNBT 1024, gather 32x64, qwen3 parsers] +fp8 KV pool ~2 full-length, MNBT 1024, gather 32x64, qwen3 parsers, +expert-precise-memory arena 7,522 slots, /root/.cache mounted] end subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04] @@ -81,13 +85,15 @@ frank-small, port 7771] mmproj for vision, ~100 tok/s decode, ~2.7 GiB of the card left free. The KV arithmetic, the flash-attn dependency, rollback to Q6_K - [[dankserver-qwen38-fp8-expert-offload-tuning]] — **ACTIVE since - 2026-10-02 (shape D)** (root systemd unit, port 7770): the tcclaviger RDNA4 + 2026-10-02 (shape E)** (root systemd unit, port 7770): the tcclaviger RDNA4 vLLM fork serving orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both R9700s, TP=2 with automatic expert offload — `--max-model-len 200704` x 3 slots, fp8 KV, pool 3,150,000,000 B/rank (measured 398,658 tokens = 1.99x), - MNBT 1024, gather 32x64, no spec decode, prefetch off. Why `--max-num-seqs` - must stay 3 (SEQS 2 cannot compile this model), why the arena lever is - `max_model_len` rather than the KV pool, and the measured envelope + MNBT 1024, gather 32x64, no spec decode, prefetch off, plus + `--expert-precise-memory` (arena 17.22 GiB / 7,522 slots), GPU_MAX_HW_QUEUES=2 + and a mounted compile cache. Why `--max-num-seqs` + must stay 3 (SEQS 2 cannot compile this model), why the arena's levers are + `max_model_len` and the overhead term rather than the KV pool, and the measured envelope - [[frank-primary-glm53-flash-w4a16]] — RETIRED 2026-09-30 → 10-01 (the failed GLM experiment; frank-primary runs Qwen FP8 again, see the intro): the tcclaviger RDNA4 vLLM fork diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 0452f0a..9da0f25 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -1106,3 +1106,85 @@ and then answers with nothing, and curl reports "Empty reply from server" (52). - **Operating lesson: `deploy.sh ` with no `--service` is a fleet-wide drain.** It restarts every service whose recorded revision is behind HEAD and whose paths changed since. To touch one service, name it. + +## 2026-10-02 — frank-primary shape E: `--expert-precise-memory` (+1,026 arena slots), a mounted compile cache, and the P2P answer + +Shape D left one question open — the plan's overhead term was reserving 2.35 GiB +of compile headroom the boot never used — and the answer to Frank's free-VRAM +question arrived together with a boot-cost bug that had been costing 181 s on +every restart since the cutover. + +- **`--expert-precise-memory` is real and it fits (option A).** Boot-probed on + the shape-D base with production stopped, the candidate booted AS the service + (`FRANK_PRIMARY_ENV_FILE=/tmp/probe-b.env bash scripts/serve-frank-primary.sh`, + trap-guaranteed restore), the numbers read off the engine's own plan lines: + + | plan line | shape D | shape E | + |---|---|---| + | `runtime overhead (non-torch + buffers + activation)` | 7.60 GiB | **5.25 GiB** (`precise (gfx1201 TP2 nbt4096/seqs16/len262144)`) | + | `expert cache space` | 14.87 GiB | **17.22 GiB** | + | slots | 6,496 | **7,522** (+1,026, +16 %) | + | `swap region` | — | 13.7 GiB / 5,986 slots, 133-134 per layer | + | boot | — | **healthy in 420 s**, no OOM | + + The flag also prints `'expert_precise_memory': True` in `non-default args`, so + a silent argparse drop cannot hide. It was refused on 2026-10-01 only because + the 2 x 262144 bf16 shape was already 0.13 GiB under the ceiling — there the + flag was the last stack over the line, not the cause. +- **The pool was NOT cut (option A) because the pool is not an arena lever.** + A 2.2e9 arm was probed in the same window and reports `277,686 tokens, + 1.38x` — measured, rejected as the shipped value, kept as the conservative + fallback. With 3.15e9/rank the endpoint keeps shape D's 1.99x concurrency and + gains the arena, which is the shape D follow-up had actually been looking for. +- **THE COMPILE CACHE WAS BEING THROWN AWAY ON EVERY RESTART — 181 s each.** + Measured: a fresh container has **no `/root/.cache` at all** (`No such file or + directory`); after our boot it holds `/root/.cache/vllm` at + **1,154,714,333 B in 4,665 files** plus `/root/.cache/comgr` at 229 MB, inside + the ephemeral layer, and `podman rm -f` (every deploy, restart, reboot) + discards them. The engine names the loss itself: `Compiling a graph for compile + range (1, 1024) takes 173.19 s` ... `torch.compile took 181.20 s in total` ... + `init engine (profile, create kv cache, warmup model) took 217.26 s + (compilation: 181.20 s)`. The serve script now mounts + `FRANK_PRIMARY_CACHE_DIR` at `/root/.cache` — the **parent**, so the vLLM + AOT/torch_compile tree (`VLLM_CACHE_ROOT`, default `/root/.cache/vllm`) and + the comgr JIT cache both persist, and nothing is hidden because the image has + no such directory. Same class of fix as the recipe's + `/cache/{triton,vllm,inductor}` volumes; the compile RANGE is part of the + cache key, so an MNBT change costs one recompilation, not a permanent tax. +- **`GPU_MAX_HW_QUEUES=2`** (HSA hardware queues per GPU) added at Frank's + direction. Verified a real knob rather than a no-op — the literal is read from + `/opt/rocm/core-10.0/lib/libhsa-runtime64.so` — and shipped deliberately + alongside the 32x64 gather geometry. **Not A/B measured.** +- **P2P: the hardware supports it, and the engine is already using it.** The + live log shows `r4d all-reduce INSTALLED (rank=0 max=49152KB finegrained=T…)` + on BOTH ranks and `Using ['R4D_AR', 'PYNCCL'] all-reduce backends` — R4D's + P2P path, with RCCL as fallback, measured at **45.274 GB/s bidirectional**. + Root ports `00:01.1`/`00:01.3` both report ACS `ReqRedir+`, so P2P hairpins at + the root complex; there is no XGMI. The old FAIL predates the 2026-09-20 board + swap. Kernel support confirmed: `CONFIG_HSA_AMD_P2P=y`, `CONFIG_PCI_P2PDMA=y`, + `CONFIG_HSA_AMD=y`, `CONFIG_ZONE_DEVICE=y`, `iommu=pt` on the cmdline. +- **Recipe diff (Frank's 3 x R9700 compose vs this podman run), ranked:** + - *Acted on:* persistent compile caches (above — the one real operational + gap), and `GPU_MAX_HW_QUEUES`. + - *Retracted:* pinned `--compilation-config cudagraph_capture_sizes` was on my + list; measured, it is **not** a gap here — capture is 3 PIECEWISE + 2 FULL + graphs, `Graph capturing finished in 3 secs, took 0.31 GiB` (0.11 GiB on the + lighter rank). No 15-52-size ladder to cut. + - *Deliberate divergence, keep ours:* no MTP (the recipe has it; measured + fastest here), `qwen3_xml` vs `qwen3_coder` (ours verified live), no + `--override-generation-config` (server-side sampling defaults), published + port + Caddy rather than `network_mode: host`. + - *Non-gaps:* `shm_size 32gb` (ours is `--ipc=host`), `memlock -1` (same), + `SYS_PTRACE` (same), `/dev/kfd` + `/dev/dri` (same), `--enable-chunked-prefill` + (V1 default), `plecache` volume (the flag is not even passed), and + `ROCR_VISIBLE_DEVICES=1,2,3` vs our `HIP_VISIBLE_DEVICES=0,1` — verified + equivalent here: `rocm-smi --showid` puts the two R9700s at GPU[0]/GPU[1] and + the 7700X iGPU at GPU[2], so our indices are the right two cards. +- **Harness note, because it cost two probe attempts:** a probe that reported an + instantaneous candidate death was a logging artifact of the inline SSH script, + not a crash — the container and its serve script must be polled by recorded id + and their logs dumped on the exit path, or a failure looks like a bad flag. + The flag was fine both times. +- Live: `health=200`, `NRestarts=0`, arena 17.22 GiB / 7,522 slots, KV 398,658 + tokens / 1.99x. Docs: [[dankserver-qwen38-fp8-expert-offload-tuning]]. + diff --git a/tests/frank_primary_unit.bats b/tests/frank_primary_unit.bats index 2be46f7..6b7f3db 100755 --- a/tests/frank_primary_unit.bats +++ b/tests/frank_primary_unit.bats @@ -119,10 +119,25 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh" grep -q '^FRANK_PRIMARY_KV_BYTES=3150000000$' "$ENV_FILE" grep -q '^FRANK_PRIMARY_MNBT=1024$' "$ENV_FILE" grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE" - # --expert-precise-memory is NOT set: it put the bf16 2 x 262144 boot - # 0.13 GiB from the 31.86 GiB ceiling (arena 16.65 device-resident) and - # crash-looped at graph capture. The serve script keeps the opt-in flag. - ! grep -q '^FRANK_PRIMARY_PRECISE_MEMORY=' "$ENV_FILE" + # --expert-precise-memory IS set (shape E, 2026-10-02). It was refused + # while the 2 x 262144 bf16 boot sat 0.13 GiB under the 31.86 GiB ceiling + # (arena 16.65 device-resident, NRestarts=3 at ~6.7 min each) — there it was + # the last 2.35 GiB over the line, not the cause. Probed on the shape-D base + # with production stopped: boots healthy in 420 s, plan reports runtime + # overhead 5.25 GiB (not 7.60), expert cache 17.22 GiB / 7,522 slots + # (+1,026 / +16% over 6,496). The pool is NOT cut with it (option A): the + # pool is not an arena lever, so 3.15e9 keeps the measured 1.99x. + grep -q '^FRANK_PRIMARY_PRECISE_MEMORY=1$' "$ENV_FILE" + # GPU_MAX_HW_QUEUES caps HSA hw queues per GPU; real knob (the literal is in + # libhsa-runtime64.so), passed inside the container as a bare name — the env + # file keeps only the prefixed form. + grep -q '^FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2$' "$ENV_FILE" + # The compile cache is a BOOT-COST fix: the image ships no /root/.cache at + # all, our boot writes 1.1 GB of vLLM AOT artifacts (4,665 files) plus + # 229 MB of comgr JIT kernels into the ephemeral layer, and every + # `podman rm -f` throws them away — re-paying `torch.compile took 181.20 s` + # (173.19 s of it one graph) on every deploy, restart and reboot. + grep -q '^FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary$' "$ENV_FILE" grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE" grep -q '^FRANK_PRIMARY_GATHER_LANES=64$' "$ENV_FILE" ! grep -q '^FRANK_PRIMARY_SPEC=' "$ENV_FILE" @@ -150,6 +165,13 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh" # moves the plan's `room` term and therefore the expert arena: ~1,025 # more slots at 2.344 MiB each. grep -q -- '${FRANK_PRIMARY_PRECISE_MEMORY:+--expert-precise-memory}' "$SERVE" + # Shape E host/CPU wiring: the cache dir is created before it is mounted + # (a cold boot must have a target) and the mount lands on /root/.cache, + # whose vllm/ child is the path VLLM_CACHE_ROOT defaults to — the engine log + # names it as `Using cache directory: /root/.cache/vllm/torch_compile_cache/…`. + grep -q -- 'mkdir -p "$CACHE_DIR"' "$SERVE" + grep -q -- '${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache}' "$SERVE" + grep -q -- '-e GPU_MAX_HW_QUEUES="$FRANK_PRIMARY_GPU_MAX_HW_QUEUES"' "$SERVE" } @test "tools and reasoning are wired — the client contract is served" {