Frank
c1248d4a5d
Overnight redo campaign verdict + Frank's call: the fp8+MTP4 shape is retired to git tag frank-primary-fp8-mtp-20261001. The new incumbent: KV cache unquantized (auto), no speculative decode, KV_BYTES 11,047,010,304/rank (engine-measured 798,280 tokens, 3.05x concurrency at 262,144-token requests), MNBT 1024, MoE gather 32x64 (+18.7% decode, grows with depth). Measured at the deployed shape: 39.38/38.79/36.42 tok/s (166/1087/4064 ctx). Seam re-pinned: the 7.2 GiB/rank KV ceiling was the MTP4 workspace, not VRAM; spec decode absent by measurement; bf16 trade documented (clean cache + 262k windows for ~14% decode at 4064 ctx).
71 lines
3.8 KiB
Bash
71 lines
3.8 KiB
Bash
# frank-primary.env — tcclaviger RDNA4 vLLM production config (dankserver :7770).
|
|
#
|
|
# 2026-10-01 cutover to the BF16-KV no-spec incumbent (the overnight redo
|
|
# campaign's verdict, Frank's call): same digest-pinned engine and payload as
|
|
# the 2026-09-29 cutover, but the KV cache is UNQUANTIZED (auto = bf16) and
|
|
# speculative decode is GONE. Git tag frank-primary-fp8-mtp-20261001 is the
|
|
# last pre-cutover commit; the old shape (fp8 KV + MTP4) is in history and
|
|
# restorable from the tag.
|
|
#
|
|
# Every number here was QUALIFIED by boot probe on 2026-10-01 (not inherited):
|
|
# * MAXLEN 262144 x SEQS 3 (786,432 tokens) at bf16 KV boots at
|
|
# KV_BYTES 11,047,010,304 PER RANK: the engine measured "KV cache size:
|
|
# 798,280 tokens; Maximum concurrency for 262,144 tokens per request:
|
|
# 3.05x". The pool holds all three full windows with an 11,848-token
|
|
# margin. Measured decode at the winner shape: 39.38/38.79/36.42 tok/s
|
|
# (166/1087/4064 ctx).
|
|
# * KV_BYTES is PER RANK. The old "7.2 GiB ceiling, 9/16 GiB OOM" law was
|
|
# the MTP4 WORKSPACE: the spec-era pool measured 729,545 tokens because
|
|
# the drafter reserved ~2.6 GiB/rank beside the pool. No-spec bf16 takes
|
|
# 11.05 GiB/rank and boots clean — the ceiling was spec's, not VRAM's.
|
|
# * bf16 KV doubles the attention-KV bytes/token (measured 14,047 vs
|
|
# 7,926) and costs ~14% decode vs fp8 at 4064 ctx (49.25 -> 36.42 with
|
|
# the gather win) — traded deliberately for clean cache + 262k windows.
|
|
# * GATHER 32x64 is the night's biggest lever: R4D_LRU_GATHER_CHUNKS=32 /
|
|
# R4D_LRU_GATHER_LANES=64 measured +18.7% decode vs default geometry,
|
|
# and the win grows with depth (39.5@166 -> 54.1@1087 -> 49.3@4064 fp8
|
|
# era). Do not drop it.
|
|
# * MNBT 1024 is Frank's lock: MNBT 512 doubles TTFT at 4064 ctx (2.46 s
|
|
# vs 1.36 s); 1024-4096 are interchangeable on decode; 12288 costs 2.5%
|
|
# and arena. 1024 keeps the workspace floor so the arena stays max.
|
|
# * PREFETCH IS ABSENT BY LAW: expert-cache telemetry measured the
|
|
# prefetcher at 36% precision (36,720 slabs prefetched, 13,369 served) —
|
|
# roughly half the gather traffic was waste. Never re-add
|
|
# --expert-prefetch without a fresh A/B.
|
|
# * TunableOp stays OFF: CLAV_TUNABLEOP_SWEEP silently skips under expert
|
|
# offload unless CLAV_TUNABLEOP_WITH_OFFLOAD=1, and forcing it corrupts
|
|
# numerics.
|
|
#
|
|
# HIP reads bare env vars TEMP, HOST, GPU_MAX_HEAP_SIZE — every key here is
|
|
# prefixed FRANK_PRIMARY_ so none can leak.
|
|
#
|
|
|
|
# --- runtime ----------------------------------------------------------------
|
|
# Digest-pinned: the floating :latest tag moved once already. Local alias
|
|
# qwen-fp8-runtime:tuned-20260929 (image ID 3322dbbbc96d) points at the same
|
|
# bytes in rootless storage.
|
|
FRANK_PRIMARY_IMAGE=docker.io/tcclaviger/vllm@sha256:da4029666668b90a0386c1ed23ea370c79916ad4e1a6ad1dd9db098b9fa069ea
|
|
|
|
# --- payload ----------------------------------------------------------------
|
|
FRANK_PRIMARY_MODEL_DIR=/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8
|
|
FRANK_PRIMARY_MODEL_SHARDS=131
|
|
|
|
# --- identity ---------------------------------------------------------------
|
|
# The harness models.yml id stays frank-primary (provider 192.168.0.235:7770).
|
|
FRANK_PRIMARY_CONTAINER=frank-primary-qwen
|
|
FRANK_PRIMARY_SERVED_NAME=frank-primary
|
|
FRANK_PRIMARY_PUBLISH=7770:7770
|
|
# Containers removed before every start: the retired R9V identity and the ad
|
|
# hoc experiment identity (both hold the GPUs / name registrations).
|
|
FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
|
|
|
|
# --- qualified engine shape ---------------------------------------------------
|
|
FRANK_PRIMARY_MAXLEN=262144
|
|
FRANK_PRIMARY_SEQS=3
|
|
FRANK_PRIMARY_KV_BYTES=11047010304
|
|
FRANK_PRIMARY_KV_DTYPE=auto
|
|
FRANK_PRIMARY_MNBT=1024
|
|
FRANK_PRIMARY_GATHER_CHUNKS=32
|
|
FRANK_PRIMARY_GATHER_LANES=64
|
|
FRANK_PRIMARY_UTIL=0.98
|