Frank
d91379cff2
DFlash2 experiment declared failed 2026-10-01: the drafter serves and drafts (12.8 tok/s proposed) but acceptance is pinned at 1.0-1.38 on ALL text classes including verbatim repetition — the aux-hidden-state capture semantics are wrong (single-stream entering-state vs post-layer/comb is the untested alternative). At ~1.2 acceptance the 8-token verify tax makes decode 5x SLOWER than the 9.9-10.0 baseline (1.6-1.8 tok/s measured). Restores the verified 37-40 tok/s shape: qwen3.8-flash-next FP8, MTP4, prefetch off, 200k x 3 seqs, fp8 KV.
62 lines
3.4 KiB
Bash
62 lines
3.4 KiB
Bash
# frank-primary.env — tcclaviger RDNA4 vLLM production config (dankserver :7770).
|
|
#
|
|
# 2026-09-29 cutover from the R9V radiance-vLLM fork (its launcher/shim/state
|
|
# machinery is retired; /home/dank/r9v-state/{q8xl,uncensored} stay on disk as
|
|
# the rollback target, and git tag frank-primary-r9v-q8-20260929 is the last
|
|
# pre-cutover commit). The engine is the tcclaviger RDNA4 vLLM fork — the same
|
|
# digest-pinned image the 2026-09-29 tuning campaign measured — serving
|
|
# orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (~172.8 GiB, 131 shards) with
|
|
# the MTP4 drafter on the two R9700s.
|
|
#
|
|
# Every number here was QUALIFIED by boot probe on 2026-09-29 (not inherited):
|
|
# * MAXLEN 200000 x SEQS 3 (600k total) at KV q8/q8 fits with headroom:
|
|
# the engine measured "GPU KV cache size: 729,545 tokens; Maximum
|
|
# concurrency for 200,000 tokens per request: 3.65x" at KV_BYTES 7.2 GiB
|
|
# per rank. KV_BYTES is PER RANK — sizing it to the global requirement
|
|
# OOMs the worker (measured twice: 16 GiB and 9 GiB both die with ~7.5 GiB
|
|
# free; 7.2 GiB boots).
|
|
# * --kv-cache-dtype fp8 is the fleet's q8/q8 KV ask (e4m3 for BOTH K and V
|
|
# — the stock flag quantizes them together, which is exactly what was
|
|
# requested here). Accepted by the attention path; qualified with a
|
|
# completion probe.
|
|
# * PREFETCH IS ABSENT BY LAW: expert-cache telemetry measured the
|
|
# prefetcher at 36% precision (36,720 slabs prefetched, 13,369 served) —
|
|
# roughly half the gather traffic was waste; prefetch off is the tuned
|
|
# incumbent and the +40-51% decode came from deleting the flag. Never
|
|
# re-add --expert-prefetch without a fresh A/B.
|
|
# * TunableOp stays OFF: CLAV_TUNABLEOP_SWEEP silently skips under expert
|
|
# offload unless CLAV_TUNABLEOP_WITH_OFFLOAD=1, and forcing it corrupts
|
|
# numerics (repetition loops, early halts, invalid measurements).
|
|
# Decode at the tuned shape: 40.57/37.44/37.57 tok/s (166/1087/4064 ctx) at
|
|
# the 32k window; 42.94/34.55/36.09 at this 200k shape (arena 6,032 slots).
|
|
#
|
|
# HIP reads bare env vars TEMP, HOST, GPU_MAX_HEAP_SIZE — every key here is
|
|
# prefixed FRANK_PRIMARY_ so none can leak.
|
|
|
|
# --- runtime ----------------------------------------------------------------
|
|
# Digest-pinned: the floating :latest tag moved once already. Local alias
|
|
# qwen-fp8-runtime:tuned-20260929 (image ID 3322dbbbc96d) points at the same
|
|
# bytes in rootless storage.
|
|
FRANK_PRIMARY_IMAGE=docker.io/tcclaviger/vllm@sha256:da4029666668b90a0386c1ed23ea370c79916ad4e1a6ad1dd9db098b9fa069ea
|
|
|
|
# --- payload ----------------------------------------------------------------
|
|
FRANK_PRIMARY_MODEL_DIR=/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8
|
|
FRANK_PRIMARY_MODEL_SHARDS=131
|
|
|
|
# --- identity ---------------------------------------------------------------
|
|
# The harness models.yml id stays frank-primary (provider 192.168.0.235:7770).
|
|
FRANK_PRIMARY_CONTAINER=frank-primary-qwen
|
|
FRANK_PRIMARY_SERVED_NAME=frank-primary
|
|
FRANK_PRIMARY_PUBLISH=7770:7770
|
|
# Containers removed before every start: the retired R9V identity and the ad
|
|
# hoc experiment identity (both hold the GPUs / name registrations).
|
|
FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
|
|
|
|
# --- qualified engine shape ---------------------------------------------------
|
|
FRANK_PRIMARY_MAXLEN=200000
|
|
FRANK_PRIMARY_SEQS=3
|
|
FRANK_PRIMARY_KV_BYTES=7730934784
|
|
FRANK_PRIMARY_KV_DTYPE=fp8
|
|
FRANK_PRIMARY_SPEC=4
|
|
FRANK_PRIMARY_UTIL=0.98
|