Frank
d57174c8d3
Three engine knobs that were impossible to set without editing the run line:
FRANK_PRIMARY_SPEC (--speculative-config), FRANK_PRIMARY_COMPCONFIG
(--compilation-config) and FRANK_PRIMARY_GEN_CONFIG
(--override-generation-config). Each holds the whole JSON value; empty or
unset means the flag is absent, so the shipped shape E argv is unchanged.
The seam re-pins the law that justified the omission: "speculative decode is
ABSENT BY MEASUREMENT" compared the wrong pair — c1248d4 retired MTP by
weighing fp8+MTP4 (37.4-40.6 t/s, 54% acceptance) against the bf16 no-spec
boot (36.4-39.4), bf16 then proved unbootable, and fp8 came back WITHOUT MTP.
fp8+spec was never measured against fp8+no-spec on one shape, the no-spec
incumbent runs ~30 t/s under the fleet's 35 t/s target, and every vendor
flash-next recipe ships MTP3.
Also in the seam: a dead assertion is deleted (a failing `! grep` mid-test is
exempt from bats' errexit, so the pre-precise-memory `! grep
--expert-precise-memory` never ran), and the knob test asserts by RENDERING the
flag from the sourced env instead of grepping for an assignment.
207 lines
13 KiB
Bash
207 lines
13 KiB
Bash
# frank-primary.env — tcclaviger RDNA4 vLLM production config (dankserver :7770).
|
|
#
|
|
# 2026-10-02 shape E: fp8 KV, MAXLEN 200704 x SEQS 3, pool 3.15e9/rank, and
|
|
# --expert-precise-memory ON. The window shrink from the native 262144 and the
|
|
# --max-num-seqs 3 lock both come from shape D (SEQS=2 CANNOT boot this model —
|
|
# the measured 2x2 is below); shape E adds the precise-overhead plan, which
|
|
# trades the default 7.60 GiB compile headroom for the fitted 5.25 GiB and puts
|
|
# the 2.35 GiB difference into the expert arena: 6,496 -> 7,522 slots (+16%).
|
|
# The pool is deliberately NOT cut (option A): the pool is not an arena lever
|
|
# (measured), so 3.15e9 keeps shape D's 1.99x concurrency while the arena grows.
|
|
# Same digest-pinned tcclaviger RDNA4 fork. Rollback for the fp8+MTP4 era is git
|
|
# tag frank-primary-fp8-mtp-20261001.
|
|
#
|
|
# HIP reads bare env vars TEMP, HOST, GPU_MAX_HEAP_SIZE — every key here is
|
|
# prefixed FRANK_PRIMARY_ so none can leak. The one bare name we WANT inside the
|
|
# container (GPU_MAX_HW_QUEUES) is held here under the prefix and passed on the
|
|
# run line as `-e GPU_MAX_HW_QUEUES=…`.
|
|
#
|
|
|
|
# --- runtime ----------------------------------------------------------------
|
|
# Digest-pinned: the floating :latest tag moved once already. Local alias
|
|
# qwen-fp8-runtime:tuned-20260929 (image ID 3322dbbbc96d) points at the same
|
|
# bytes in rootless storage.
|
|
FRANK_PRIMARY_IMAGE=docker.io/tcclaviger/vllm@sha256:da4029666668b90a0386c1ed23ea370c79916ad4e1a6ad1dd9db098b9fa069ea
|
|
|
|
# --- payload ----------------------------------------------------------------
|
|
FRANK_PRIMARY_MODEL_DIR=/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8
|
|
FRANK_PRIMARY_MODEL_SHARDS=131
|
|
|
|
# --- identity ---------------------------------------------------------------
|
|
# The harness models.yml id stays frank-primary (provider 192.168.0.235:7770).
|
|
FRANK_PRIMARY_CONTAINER=frank-primary-qwen
|
|
FRANK_PRIMARY_SERVED_NAME=frank-primary
|
|
FRANK_PRIMARY_PUBLISH=7770:7770
|
|
# Containers removed before every start: the retired R9V identity and the ad
|
|
# hoc experiment identity (both hold the GPUs / name registrations).
|
|
FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
|
|
|
|
# --- qualified engine shape (2026-10-02, shape E) ---------------------------
|
|
# WHAT THE SLOT COUNT DOES AND DOES NOT DO. The endpoint advertises 3 slots
|
|
# (SEQS 3), but the POOL, not the slot count, decides how many full-length
|
|
# conversations are resident. MEASURED LIVE 2026-10-02 at this shape:
|
|
# GPU KV cache size: 398,658 tokens,
|
|
# Maximum concurrency for 200,704 tokens per request: 1.99x
|
|
# (3,146,129,408 B reserved — 7,892 B/token). The per-token price is
|
|
# SHAPE-DEPENDENT, not a constant: the 262144 shape priced at 7,701 B/token
|
|
# (6,100,000,000 B -> 792,069 tokens), ~2.5% cheaper, so never extrapolate a
|
|
# concurrency figure across geometries. Shorter requests use the spare blocks,
|
|
# and a third LONG request queues rather than being refused. Verified: three
|
|
# concurrent 67-token requests answered in parallel, ~4.1 s each.
|
|
#
|
|
# WHY NOT --max-num-seqs 2 (measured 2026-10-02, single-variable boot probes
|
|
# against the then-deployed known-good shape, production stopped for each):
|
|
# SEQS 2 + MNBT 1024 -> Engine core initialization failed
|
|
# SEQS 2 + MNBT 2048 -> Engine core initialization failed
|
|
# SEQS 3 + MNBT 1024 -> healthy (792,069 tokens, 3.02x, arena 6,248)
|
|
# SEQS 3 + MNBT 2048 -> healthy (the deployed incumbent)
|
|
# The failure is torch.fx ConstraintViolationError: query_start_loc.size()[0]
|
|
# "marked dynamic but your code specialized it to be a constant (3)", raised
|
|
# from AOT build_guards(strict_error=True) during profile_run -> _dummy_run.
|
|
# model_runner.py sizes that input buffer max_num_reqs + 1 and _dummy_run sets
|
|
# num_reqs = min(max_num_batched_tokens, max_num_reqs), so the width is
|
|
# SEQS + 1 for ANY MNBT >= SEQS: 3 at SEQS 2, 4 at SEQS 3. MNBT only moves the
|
|
# compile RANGE ((1,1024) vs (1,2048)) and cannot dodge it — the 2x2 above is
|
|
# why MNBT is not a lever here. --enforce-eager bypasses the path but loses
|
|
# CUDA graphs. Same failure class killed the 2026-10-01 bf16 2 x 262144 shape.
|
|
#
|
|
# THE ARENA LEVER IS max_model_len, NOT KV_BYTES (measured 2026-10-02). The
|
|
# planner sizes expert cache as budget - non-expert - KV reserve - graphs -
|
|
# overhead, and the KV reserve is registry-priced at 9190 B/token x max_model_len:
|
|
# MAXLEN 262144 -> reserve 2.48 GiB -> arena 14.30 GiB / 6,248 slots
|
|
# MAXLEN 200704 -> reserve 1.91 GiB -> arena 14.87 GiB / 6,496 slots
|
|
# d(reserve) 0.57 GiB = d(arena) 0.57 GiB = +248 slots. The 6.1e9 -> 3.15e9
|
|
# pool cut buys ZERO arena; it buys runtime HEADROOM (the shape it replaced
|
|
# died with 0 bytes free on a real 995-token request). To grow the arena,
|
|
# lower MAXLEN — not KV_BYTES. The OTHER arena lever is the overhead term:
|
|
# --expert-precise-memory (shape E) takes it from 7.60 to 5.25 GiB and lands
|
|
# those 2.35 GiB in the arena — 14.87 -> 17.22 GiB / 6,496 -> 7,522 slots.
|
|
#
|
|
# GATHER 32x64 stays: R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64
|
|
# measured +18.7% decode vs default geometry and the win grows with depth.
|
|
# Do not drop it.
|
|
# MNBT 1024 is Frank's lock: MNBT 512 doubles TTFT at 4064 ctx (2.46 s vs
|
|
# 1.36 s); 1024-4096 are interchangeable on decode; 12288 costs 2.5% and
|
|
# arena. 1024 keeps the workspace floor so the arena stays max. (Side note
|
|
# measured 2026-10-02: MNBT 2048 compiles the first graph in 26 s vs 172 s at
|
|
# 1024 — a startup-cost lever, not a correctness one. The compile RANGE is part
|
|
# of the cache key, so FRANK_PRIMARY_CACHE_DIR pays for whichever rung
|
|
# is chosen exactly once.)
|
|
# PREFETCH IS ABSENT BY LAW: expert-cache telemetry measured the prefetcher at
|
|
# 36% precision (36,720 slabs prefetched, 13,369 served) — roughly half the
|
|
# gather traffic was waste. Never re-add --expert-prefetch without a fresh A/B.
|
|
# TunableOp stays OFF: CLAV_TUNABLEOP_SWEEP silently skips under expert offload
|
|
# unless CLAV_TUNABLEOP_WITH_OFFLOAD=1, and forcing it corrupts numerics.
|
|
# --expert-precise-memory is ON (shape E). It selects the fitted overhead
|
|
# (5.25 GiB) over the default (7.60 GiB = fitted + compile/graph-capture
|
|
# headroom); the 2.35 GiB difference lands in the expert arena. MEASURED
|
|
# 2026-10-02 by a boot probe on the shape-D base (production stopped, candidate
|
|
# booted AS the service, plan read off the engine's own log):
|
|
# runtime overhead 7.60 -> 5.25 GiB (registry Qwen4Exp…, precise
|
|
# gfx1201 TP2 nbt4096/seqs16/len262144)
|
|
# expert cache 14.87 -> 17.22 GiB
|
|
# slots 6,496 -> 7,522 (+1,026 = +16%, 2.344 MiB each)
|
|
# swap region 13.7 GiB / 5,986 slots, 133-134 per layer
|
|
# boot healthy in 420 s, no OOM, no knife-edge
|
|
# The flag was called a non-option after it crash-looped the 2 x 262144 bf16
|
|
# boot (0.13 GiB free of 31.86 GiB, NRestarts=3 at ~6.7 min each) — that shape
|
|
# was already at the ceiling, so precise-memory was the last 2.35 GiB over, not
|
|
# the cause. At 200704 x 3 it fits with room; the run line also reports
|
|
# 'expert_precise_memory': True in non-default args, so a silent argparse drop
|
|
# would be visible in the boot log.
|
|
FRANK_PRIMARY_PRECISE_MEMORY=1
|
|
# GPU_MAX_HW_QUEUES caps the HSA hardware queues per GPU (AMD's own tuning
|
|
# guide recommends 2 for this class of workload; the runtime default on Linux
|
|
# is 4). Verified a REAL knob, not a no-op: the string is present in
|
|
# /opt/rocm/core-10.0/lib/libhsa-runtime64.so. NOT A/B MEASURED — Frank's call,
|
|
# shipped as a deliberate CPU-side tune alongside the 32x64 gather geometry.
|
|
FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2
|
|
# Persistent compile cache. MEASURED 2026-10-02: the image ships NO
|
|
# /root/.cache at all (`ls: cannot access '/root/.cache'` in a fresh
|
|
# container), and after our boot it holds /root/.cache/vllm at 1,154,714,333 B
|
|
# in 4,665 files plus /root/.cache/comgr at 229 MB — both inside the
|
|
# container's ephemeral layer, so every `podman rm -f` (every deploy, restart
|
|
# and reboot) throws them away and re-pays the build. The engine's own log
|
|
# names the loss: `torch.compile took 181.20 s in total`, of which 'Compiling a
|
|
# graph for compile range (1, 1024) takes 173.19 s', followed by 'saved AOT
|
|
# compiled function to /root/.cache/vllm/torch_compile_cache/…'. Mounting a
|
|
# host dir at /root/.cache persists the vLLM AOT/torch_compile tree AND the
|
|
# comgr HIP-kernel JIT cache, and hides nothing (the image has no such dir).
|
|
# PROVEN LIVE 2026-10-02, two boots of this exact shape:
|
|
# cold `Compiling a graph for compile range (1, 1024) takes 179.24 s`
|
|
# `torch.compile took 188.26 s in total`
|
|
# warm `Directly load the compiled graph(s) for compile range (1, 1024)
|
|
# from the cache, took 2.143 s`
|
|
# `torch.compile took 3.97 s in total`
|
|
# `init engine (profile, create kv cache, warmup model) took 41.13 s
|
|
# (compilation: 3.98 s)`
|
|
# (Total time to /health is dominated by the 172.8 GiB payload load, not the
|
|
# compile — the saving is the ~184 s of inductor/AOT build.)
|
|
# Safe to grow/prune: entries are keyed by version + config hash, so a stale
|
|
# entry is a miss, never a wrong answer.
|
|
FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary
|
|
FRANK_PRIMARY_MAXLEN=200704
|
|
FRANK_PRIMARY_SEQS=3
|
|
# The POOL is deliberately kept at shape D's 3.15e9 (option A): because the pool
|
|
# is not an arena lever, raising it back from a cut buys concurrency only, and
|
|
# 3.15e9 preserves the measured 1.99x while precise-memory grows the arena. A
|
|
# 2.2e9 pool was probed in the same window (277,686 tokens / 1.38x) and rejected
|
|
# as the shipped value; it remains the conservative arm if free VRAM ever binds.
|
|
FRANK_PRIMARY_KV_BYTES=3150000000
|
|
FRANK_PRIMARY_KV_DTYPE=fp8
|
|
FRANK_PRIMARY_MNBT=1024
|
|
FRANK_PRIMARY_GATHER_CHUNKS=32
|
|
FRANK_PRIMARY_GATHER_LANES=64
|
|
FRANK_PRIMARY_UTIL=0.98
|
|
|
|
# --- optional engine knobs (unset == the flag is absent) ---------------------
|
|
# FRANK_PRIMARY_SPEC holds the WHOLE --speculative-config JSON; setting it adds
|
|
# the flag. THE RETIREMENT VERDICT WAS INVALID: c1248d4 (2026-10-02) retired MTP
|
|
# by comparing fp8+MTP4 (37.4-40.6 tok/s, 54% acceptance, 09-29) against the
|
|
# bf16 no-spec boot (36.4-39.4) — and bf16 then proved unbootable, so fp8 came
|
|
# back without MTP and fp8+spec was NEVER compared against fp8+no-spec on the
|
|
# same shape. The no-spec incumbent measures ~30 tok/s single-stream (below the
|
|
# fleet's 35 tok/s target); every vendor flash-next recipe ships MTP3; and our
|
|
# MTP4 run used the fork's default GREEDY draft sampling, while the recipes pass
|
|
# draft_sample_method=probabilistic (greedy drafts are rejected more often under
|
|
# temp>0 — the plausible cause of the 54%). MTP also re-adds drafter workspace:
|
|
# the old note says the 7.2 GiB/rank KV ceiling WAS the MTP4 workspace, and
|
|
# --expert-precise-memory spends that same budget, so an MTP arm must be tested
|
|
# with precise memory BOTH on and off.
|
|
# MTP3: '{"method":"mtp","num_speculative_tokens":3,"draft_sample_method":"probabilistic"}'
|
|
FRANK_PRIMARY_SPEC=
|
|
# FRANK_PRIMARY_COMPCONFIG holds the whole --compilation-config JSON. The
|
|
# capture-ladder law: the piecewise cudagraph ladder must contain
|
|
# num_speculative_tokens + 1 (MTP3 -> 4 -> ladder 4,8,12; MTP4 -> 5,10,15;
|
|
# MTP5 -> 6,12,18,24), because every draft step proposes n+1 tokens. The default
|
|
# ladder is multiples of 1+4 up to 512 then powers of two, so an MTP3 shape can
|
|
# miss its own step size. Pinning the ladder is ALSO the candidate fix for the
|
|
# SEQS>3 wall: query_start_loc is specialised to SEQS+1 and that constant must
|
|
# land on a captured graph size (3 is not in the default ladder, 4 is —
|
|
# measured 2026-10-02, and the vendor runs SEQS 16 on this model/TP2).
|
|
# combo_kernels false/true are BOTH in the shipped recipes, so it is not a law.
|
|
# MTP3 at SEQS 3: '{"cudagraph_mode":"PIECEWISE","cudagraph_capture_sizes":[1,4,8,12],"max_cudagraph_capture_size":12}'
|
|
FRANK_PRIMARY_COMPCONFIG=
|
|
# FRANK_PRIMARY_GEN_CONFIG holds the whole --override-generation-config JSON.
|
|
# The payload's HF generation_config.json sets temperature 1.0 / top_p 0.95 /
|
|
# top_k 20 and the engine WARNS about it on every boot, so a client that sends
|
|
# no sampling fields gets creativity it never asked for. This is the one knob
|
|
# that changes ANSWERS, not speed.
|
|
# '{"max_tokens":65536,"temperature":0.7,"top_p":0.95,"top_k":20}'
|
|
FRANK_PRIMARY_GEN_CONFIG=
|
|
|
|
# --- client contract: reasoning split, tools, preserve_thinking ----------------
|
|
# Without --reasoning-parser the engine leaves the thinking text INLINE in
|
|
# `content` with a dangling `</think>` and a null `reasoning` field (measured
|
|
# 2026-10-01: content = 'We need answer...Brief.\n</think>\n\n391') — the
|
|
# malformed thinking block clients choke on. The qwen3 parser splits it
|
|
# properly. The GLM rollback (d91379d) had dropped all three of these; the
|
|
# working values are the ones ecaad36 shipped for this model.
|
|
FRANK_PRIMARY_REASONING_PARSER=qwen3
|
|
# Tools: --enable-auto-tool-choice + the qwen3_xml parser is what makes
|
|
# tool_calls work here (without it, tool_choice auto returns 400).
|
|
FRANK_PRIMARY_TOOL_PARSER=qwen3_xml
|
|
# preserve_thinking keeps prior thinking blocks in the rendered prompt across
|
|
# turns; enable_thinking makes thinking the server-side default. Verified the
|
|
# payload's tokenizer_config template consumes both keys.
|
|
FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS='{"enable_thinking": true, "preserve_thinking": true}' |