Frank
57f33e5289
Live after the 1ac4d48 deploy: 'expert_precise_memory': True, runtime overhead
5.25 GiB, expert cache space 17.22 GiB / 7522 slots, swap region 13.7 GiB /
5986 slots, GPU KV cache 398,658 tokens / 1.99x, pool 3,150,000,000, MNBT 1024
-- the probe's projection is now production.
Compile cache proven on consecutive boots of that shape: cold 'Compiling a
graph for compile range (1, 1024) takes 179.24 s' / 'torch.compile took
188.26 s'; warm 'Directly load the compiled graph(s) ... from the cache, took
2.143 s' / 'torch.compile took 3.97 s in total' / 'init engine ... took 41.13 s
(compilation: 3.98 s)'. Host dir 1.4 GB / 4,707 files. Time to /health stays
payload-bound (490 s warm), so the claim is ~184 s of inductor/AOT build per
restart, not a proportional boot cut.
Tests: 367/367.
171 lines
11 KiB
Bash
171 lines
11 KiB
Bash
# frank-primary.env — tcclaviger RDNA4 vLLM production config (dankserver :7770).
|
|
#
|
|
# 2026-10-02 shape E: fp8 KV, MAXLEN 200704 x SEQS 3, pool 3.15e9/rank, and
|
|
# --expert-precise-memory ON. The window shrink from the native 262144 and the
|
|
# --max-num-seqs 3 lock both come from shape D (SEQS=2 CANNOT boot this model —
|
|
# the measured 2x2 is below); shape E adds the precise-overhead plan, which
|
|
# trades the default 7.60 GiB compile headroom for the fitted 5.25 GiB and puts
|
|
# the 2.35 GiB difference into the expert arena: 6,496 -> 7,522 slots (+16%).
|
|
# The pool is deliberately NOT cut (option A): the pool is not an arena lever
|
|
# (measured), so 3.15e9 keeps shape D's 1.99x concurrency while the arena grows.
|
|
# Same digest-pinned tcclaviger RDNA4 fork. Rollback for the fp8+MTP4 era is git
|
|
# tag frank-primary-fp8-mtp-20261001.
|
|
#
|
|
# HIP reads bare env vars TEMP, HOST, GPU_MAX_HEAP_SIZE — every key here is
|
|
# prefixed FRANK_PRIMARY_ so none can leak. The one bare name we WANT inside the
|
|
# container (GPU_MAX_HW_QUEUES) is held here under the prefix and passed on the
|
|
# run line as `-e GPU_MAX_HW_QUEUES=…`.
|
|
#
|
|
|
|
# --- runtime ----------------------------------------------------------------
|
|
# Digest-pinned: the floating :latest tag moved once already. Local alias
|
|
# qwen-fp8-runtime:tuned-20260929 (image ID 3322dbbbc96d) points at the same
|
|
# bytes in rootless storage.
|
|
FRANK_PRIMARY_IMAGE=docker.io/tcclaviger/vllm@sha256:da4029666668b90a0386c1ed23ea370c79916ad4e1a6ad1dd9db098b9fa069ea
|
|
|
|
# --- payload ----------------------------------------------------------------
|
|
FRANK_PRIMARY_MODEL_DIR=/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8
|
|
FRANK_PRIMARY_MODEL_SHARDS=131
|
|
|
|
# --- identity ---------------------------------------------------------------
|
|
# The harness models.yml id stays frank-primary (provider 192.168.0.235:7770).
|
|
FRANK_PRIMARY_CONTAINER=frank-primary-qwen
|
|
FRANK_PRIMARY_SERVED_NAME=frank-primary
|
|
FRANK_PRIMARY_PUBLISH=7770:7770
|
|
# Containers removed before every start: the retired R9V identity and the ad
|
|
# hoc experiment identity (both hold the GPUs / name registrations).
|
|
FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
|
|
|
|
# --- qualified engine shape (2026-10-02, shape E) ---------------------------
|
|
# WHAT THE SLOT COUNT DOES AND DOES NOT DO. The endpoint advertises 3 slots
|
|
# (SEQS 3), but the POOL, not the slot count, decides how many full-length
|
|
# conversations are resident. MEASURED LIVE 2026-10-02 at this shape:
|
|
# GPU KV cache size: 398,658 tokens,
|
|
# Maximum concurrency for 200,704 tokens per request: 1.99x
|
|
# (3,146,129,408 B reserved — 7,892 B/token). The per-token price is
|
|
# SHAPE-DEPENDENT, not a constant: the 262144 shape priced at 7,701 B/token
|
|
# (6,100,000,000 B -> 792,069 tokens), ~2.5% cheaper, so never extrapolate a
|
|
# concurrency figure across geometries. Shorter requests use the spare blocks,
|
|
# and a third LONG request queues rather than being refused. Verified: three
|
|
# concurrent 67-token requests answered in parallel, ~4.1 s each.
|
|
#
|
|
# WHY NOT --max-num-seqs 2 (measured 2026-10-02, single-variable boot probes
|
|
# against the then-deployed known-good shape, production stopped for each):
|
|
# SEQS 2 + MNBT 1024 -> Engine core initialization failed
|
|
# SEQS 2 + MNBT 2048 -> Engine core initialization failed
|
|
# SEQS 3 + MNBT 1024 -> healthy (792,069 tokens, 3.02x, arena 6,248)
|
|
# SEQS 3 + MNBT 2048 -> healthy (the deployed incumbent)
|
|
# The failure is torch.fx ConstraintViolationError: query_start_loc.size()[0]
|
|
# "marked dynamic but your code specialized it to be a constant (3)", raised
|
|
# from AOT build_guards(strict_error=True) during profile_run -> _dummy_run.
|
|
# model_runner.py sizes that input buffer max_num_reqs + 1 and _dummy_run sets
|
|
# num_reqs = min(max_num_batched_tokens, max_num_reqs), so the width is
|
|
# SEQS + 1 for ANY MNBT >= SEQS: 3 at SEQS 2, 4 at SEQS 3. MNBT only moves the
|
|
# compile RANGE ((1,1024) vs (1,2048)) and cannot dodge it — the 2x2 above is
|
|
# why MNBT is not a lever here. --enforce-eager bypasses the path but loses
|
|
# CUDA graphs. Same failure class killed the 2026-10-01 bf16 2 x 262144 shape.
|
|
#
|
|
# THE ARENA LEVER IS max_model_len, NOT KV_BYTES (measured 2026-10-02). The
|
|
# planner sizes expert cache as budget - non-expert - KV reserve - graphs -
|
|
# overhead, and the KV reserve is registry-priced at 9190 B/token x max_model_len:
|
|
# MAXLEN 262144 -> reserve 2.48 GiB -> arena 14.30 GiB / 6,248 slots
|
|
# MAXLEN 200704 -> reserve 1.91 GiB -> arena 14.87 GiB / 6,496 slots
|
|
# d(reserve) 0.57 GiB = d(arena) 0.57 GiB = +248 slots. The 6.1e9 -> 3.15e9
|
|
# pool cut buys ZERO arena; it buys runtime HEADROOM (the shape it replaced
|
|
# died with 0 bytes free on a real 995-token request). To grow the arena,
|
|
# lower MAXLEN — not KV_BYTES. The OTHER arena lever is the overhead term:
|
|
# --expert-precise-memory (shape E) takes it from 7.60 to 5.25 GiB and lands
|
|
# those 2.35 GiB in the arena — 14.87 -> 17.22 GiB / 6,496 -> 7,522 slots.
|
|
#
|
|
# GATHER 32x64 stays: R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64
|
|
# measured +18.7% decode vs default geometry and the win grows with depth.
|
|
# Do not drop it.
|
|
# MNBT 1024 is Frank's lock: MNBT 512 doubles TTFT at 4064 ctx (2.46 s vs
|
|
# 1.36 s); 1024-4096 are interchangeable on decode; 12288 costs 2.5% and
|
|
# arena. 1024 keeps the workspace floor so the arena stays max. (Side note
|
|
# measured 2026-10-02: MNBT 2048 compiles the first graph in 26 s vs 172 s at
|
|
# 1024 — a startup-cost lever, not a correctness one. The compile RANGE is part
|
|
# of the cache key, so FRANK_PRIMARY_CACHE_DIR pays for whichever rung
|
|
# is chosen exactly once.)
|
|
# PREFETCH IS ABSENT BY LAW: expert-cache telemetry measured the prefetcher at
|
|
# 36% precision (36,720 slabs prefetched, 13,369 served) — roughly half the
|
|
# gather traffic was waste. Never re-add --expert-prefetch without a fresh A/B.
|
|
# TunableOp stays OFF: CLAV_TUNABLEOP_SWEEP silently skips under expert offload
|
|
# unless CLAV_TUNABLEOP_WITH_OFFLOAD=1, and forcing it corrupts numerics.
|
|
# --expert-precise-memory is ON (shape E). It selects the fitted overhead
|
|
# (5.25 GiB) over the default (7.60 GiB = fitted + compile/graph-capture
|
|
# headroom); the 2.35 GiB difference lands in the expert arena. MEASURED
|
|
# 2026-10-02 by a boot probe on the shape-D base (production stopped, candidate
|
|
# booted AS the service, plan read off the engine's own log):
|
|
# runtime overhead 7.60 -> 5.25 GiB (registry Qwen4Exp…, precise
|
|
# gfx1201 TP2 nbt4096/seqs16/len262144)
|
|
# expert cache 14.87 -> 17.22 GiB
|
|
# slots 6,496 -> 7,522 (+1,026 = +16%, 2.344 MiB each)
|
|
# swap region 13.7 GiB / 5,986 slots, 133-134 per layer
|
|
# boot healthy in 420 s, no OOM, no knife-edge
|
|
# The flag was called a non-option after it crash-looped the 2 x 262144 bf16
|
|
# boot (0.13 GiB free of 31.86 GiB, NRestarts=3 at ~6.7 min each) — that shape
|
|
# was already at the ceiling, so precise-memory was the last 2.35 GiB over, not
|
|
# the cause. At 200704 x 3 it fits with room; the run line also reports
|
|
# 'expert_precise_memory': True in non-default args, so a silent argparse drop
|
|
# would be visible in the boot log.
|
|
FRANK_PRIMARY_PRECISE_MEMORY=1
|
|
# GPU_MAX_HW_QUEUES caps the HSA hardware queues per GPU (AMD's own tuning
|
|
# guide recommends 2 for this class of workload; the runtime default on Linux
|
|
# is 4). Verified a REAL knob, not a no-op: the string is present in
|
|
# /opt/rocm/core-10.0/lib/libhsa-runtime64.so. NOT A/B MEASURED — Frank's call,
|
|
# shipped as a deliberate CPU-side tune alongside the 32x64 gather geometry.
|
|
FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2
|
|
# Persistent compile cache. MEASURED 2026-10-02: the image ships NO
|
|
# /root/.cache at all (`ls: cannot access '/root/.cache'` in a fresh
|
|
# container), and after our boot it holds /root/.cache/vllm at 1,154,714,333 B
|
|
# in 4,665 files plus /root/.cache/comgr at 229 MB — both inside the
|
|
# container's ephemeral layer, so every `podman rm -f` (every deploy, restart
|
|
# and reboot) throws them away and re-pays the build. The engine's own log
|
|
# names the loss: `torch.compile took 181.20 s in total`, of which 'Compiling a
|
|
# graph for compile range (1, 1024) takes 173.19 s', followed by 'saved AOT
|
|
# compiled function to /root/.cache/vllm/torch_compile_cache/…'. Mounting a
|
|
# host dir at /root/.cache persists the vLLM AOT/torch_compile tree AND the
|
|
# comgr HIP-kernel JIT cache, and hides nothing (the image has no such dir).
|
|
# PROVEN LIVE 2026-10-02, two boots of this exact shape:
|
|
# cold `Compiling a graph for compile range (1, 1024) takes 179.24 s`
|
|
# `torch.compile took 188.26 s in total`
|
|
# warm `Directly load the compiled graph(s) for compile range (1, 1024)
|
|
# from the cache, took 2.143 s`
|
|
# `torch.compile took 3.97 s in total`
|
|
# `init engine (profile, create kv cache, warmup model) took 41.13 s
|
|
# (compilation: 3.98 s)`
|
|
# (Total time to /health is dominated by the 172.8 GiB payload load, not the
|
|
# compile — the saving is the ~184 s of inductor/AOT build.)
|
|
# Safe to grow/prune: entries are keyed by version + config hash, so a stale
|
|
# entry is a miss, never a wrong answer.
|
|
FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary
|
|
FRANK_PRIMARY_MAXLEN=200704
|
|
FRANK_PRIMARY_SEQS=3
|
|
# The POOL is deliberately kept at shape D's 3.15e9 (option A): because the pool
|
|
# is not an arena lever, raising it back from a cut buys concurrency only, and
|
|
# 3.15e9 preserves the measured 1.99x while precise-memory grows the arena. A
|
|
# 2.2e9 pool was probed in the same window (277,686 tokens / 1.38x) and rejected
|
|
# as the shipped value; it remains the conservative arm if free VRAM ever binds.
|
|
FRANK_PRIMARY_KV_BYTES=3150000000
|
|
FRANK_PRIMARY_KV_DTYPE=fp8
|
|
FRANK_PRIMARY_MNBT=1024
|
|
FRANK_PRIMARY_GATHER_CHUNKS=32
|
|
FRANK_PRIMARY_GATHER_LANES=64
|
|
FRANK_PRIMARY_UTIL=0.98
|
|
|
|
# --- client contract: reasoning split, tools, preserve_thinking ----------------
|
|
# Without --reasoning-parser the engine leaves the thinking text INLINE in
|
|
# `content` with a dangling `</think>` and a null `reasoning` field (measured
|
|
# 2026-10-01: content = 'We need answer...Brief.\n</think>\n\n391') — the
|
|
# malformed thinking block clients choke on. The qwen3 parser splits it
|
|
# properly. The GLM rollback (d91379d) had dropped all three of these; the
|
|
# working values are the ones ecaad36 shipped for this model.
|
|
FRANK_PRIMARY_REASONING_PARSER=qwen3
|
|
# Tools: --enable-auto-tool-choice + the qwen3_xml parser is what makes
|
|
# tool_calls work here (without it, tool_choice auto returns 400).
|
|
FRANK_PRIMARY_TOOL_PARSER=qwen3_xml
|
|
# preserve_thinking keeps prior thinking blocks in the rendered prompt across
|
|
# turns; enable_thinking makes thinking the server-side default. Verified the
|
|
# payload's tokenizer_config template consumes both keys.
|
|
FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS='{"enable_thinking": true, "preserve_thinking": true}' |