Frank
ecaad36654
The llama.cpp b11030 trio becomes the vLLM 0.30 trio ahead of the FP8 cutover on :7770 — same port, same alias, no client moves. Serve shape: locally built ROCm gfx1201 image (localhost/frank-primary:vllm030) execs via podman as dank from the root unit (LimitMEMLOCK=infinity legalizes the pin). UVA offload: --offload-backend uva pins the PLE family (47.75 GiB, 138 tensors: 131 n-gram table shards + 7 per-layer conv/proj) + routed experts (114.86 GiB, 150,528 tensors) in host RAM — 162.60 GiB combined, ~81.3 GiB per rank under TP=2, hence --cpu-offload-gb 82 (80/card strands 2.6 GiB of experts on the cards). Parsers per the host's own Ornith-era precedent: qwen3_xml tools + qwen3 reasoning (vLLM 0.30 emits the channel as `reasoning` — reasoning_content is renamed server-side) + enable_thinking/preserve_thinking template kwargs. Spec decode stays env-wired and OFF at bring-up. Deployer deploy_llm_primary rewritten: image gate, index-driven shard-count gate (131), stale-container cleanup, port wait, and the false-green hardening — health is only green when the unit's own MainPID is live behind the answer (2026-09-26 lesson). Seam tests rewritten to the vLLM contract (15 gates): offload arithmetic, parser wiring, no-llama.cpp-ism, deployer gates. Suite 349/349 green.
97 lines
4.8 KiB
Bash
97 lines
4.8 KiB
Bash
# frank-primary.env — tunables for the vLLM 0.30 frank-primary endpoint
|
|
# (Qwen3.8-Flash-Next-FP8, 2x R9700 gfx1201, dankserver :7770).
|
|
#
|
|
# Paths and numbers only — NEVER credentials. Do not name a tunable TEMP, HOST
|
|
# or GPU_MAX_HEAP_SIZE: the HIP runtime reads those bare names (2026-08-13).
|
|
#
|
|
# Container paths: the model tree is mounted at /models (host
|
|
# /home/dank/models); the compile caches live under /cache (host
|
|
# /home/dank/.cache/frank-primary-vllm, keyed on model + stack + graph knobs —
|
|
# never share it across configurations).
|
|
|
|
# --- identity ---------------------------------------------------------------
|
|
SERVED_NAME=frank-primary
|
|
BIND_HOST=0.0.0.0
|
|
PORT=7770
|
|
|
|
# --- model ------------------------------------------------------------------
|
|
# Host-side dir: /home/dank/models/Qwen3.8-Flash-Next-FP8 (131 safetensors
|
|
# shards, 172.78 GiB, verified against model.safetensors.index.json).
|
|
MODEL_DIR=/models/Qwen3.8-Flash-Next-FP8
|
|
|
|
# --- placement --------------------------------------------------------------
|
|
# Fixed 2-card tensor parallel (the R9700 pair; the Raphael iGPU is index 2
|
|
# and excluded by the visibility pins).
|
|
ROCR_VISIBLE_DEVICES=0,1
|
|
HIP_VISIBLE_DEVICES=0,1
|
|
TP=2
|
|
# 0.97 of 31.86 GiB/card (34,208,743,424 B) — leaves ~1 GiB for the ROCm
|
|
# baseline. Raise only with a measured reason.
|
|
GPU_UTIL=0.97
|
|
# The model's native window.
|
|
CTX_SIZE=262144
|
|
# Sequences the scheduler may hold. 8 is the proven batch shape on this host
|
|
# (radiance era); revisit with a bench after bring-up.
|
|
MAX_SEQS=8
|
|
|
|
# --- weight offloading (the UVA shape) ---------------------------------------
|
|
# --offload-backend uva: matched weights live in PINNED host RAM and are read
|
|
# in place over PCIe by the kernels (zero-copy), instead of being resident
|
|
# on the cards. is_uva_available() = is_pin_memory_available() on ROCm; the
|
|
# pin needs the unit's LimitMEMLOCK=infinity (a root unit grants it via
|
|
# PID1 — the dankserver user session defaults to 8 MiB and cannot raise it).
|
|
# If pinning is unavailable the offloader falls back to a copy path — the
|
|
# load log MUST be checked for the UVA path, not just /health.
|
|
# --cpu-offload-gb is PER GPU (GiB). 2x R9700 total pinned budget ≈ 160 GiB of
|
|
# the 246 GiB host RAM; the rest covers the OS, llama.cpp-era page cache
|
|
# drain and the pinned staging buffers. Do not raise without checking
|
|
# MemAvailable at load time.
|
|
# --cpu-offload-params: name SEGMENTS (dot-delimited components) of tensors
|
|
# to offload. Measured from the payload headers (2026-09-26): `ple` matches
|
|
# 138 tensors = 131 n-gram table shards + 7 per-layer conv/proj weights,
|
|
# 47.75 GiB; `experts` matches 150,528 tensors, 114.86 GiB. Combined
|
|
# 162.60 GiB — with TP=2 each rank needs ~81.3 GiB of pinned host RAM,
|
|
# hence OFFLOAD_GB=82 per GPU (164 GiB total; 80/card left 2.6 GiB of
|
|
# experts resident, inflating the on-card weight share). Resident after
|
|
# the full offload: 10.16 GiB of dense/attention/MTP/ViT/lm_head weights.
|
|
# Matching is greedy-up-to-budget in traversal order; unmatched params are
|
|
# never offloaded.
|
|
OFFLOAD_BACKEND=uva
|
|
OFFLOAD_GB=82
|
|
OFFLOAD_PARAMS=ple,experts
|
|
|
|
# --- engine -----------------------------------------------------------------
|
|
# No aiter wheel in this image (gfx9-only upstream); the AMD qwen4_exp path
|
|
# runs its in-tree triton kernels. Deterministic rather than relying on the
|
|
# availability probe.
|
|
VLLM_ROCM_USE_AITER=0
|
|
# Compile/inductor/triton caches (container /cache, TRITON_CACHE_DIR etc.)
|
|
CACHE_ROOT=/cache
|
|
|
|
# --- parsers / template (the harness wire contract) ---------------------------
|
|
# vLLM 0.30 emits the thinking channel as `reasoning` (the deprecated
|
|
# `reasoning_content` is renamed server-side, chat_completion/protocol.py:554).
|
|
# The qwen3 reasoning parser extracts the template's think-block tags into
|
|
# that channel; the qwen3_xml tool parser structures tool-call XML output into
|
|
# tool_calls (auto-tool-choice enables the structured path for API clients).
|
|
# The family precedent is the host's own Ornith-era serve scripts
|
|
# (~/vllm/failedAttempts/OrnithFP8.sh); the FP8 tokenizer's template
|
|
# consumes enable_thinking + preserve_thinking (verified in tokenizer_config).
|
|
TOOL_PARSER=qwen3_xml
|
|
REASONING_PARSER=qwen3
|
|
CHAT_TEMPLATE_KWARGS='{"enable_thinking": true, "preserve_thinking": true}'
|
|
|
|
# --- speculative decode ------------------------------------------------------
|
|
# The hybrid stack (GatedDeltaNet + QSA short-conv) uses the v1 engine's
|
|
# mamba-capable attention backends, negotiated by the model, not by env.
|
|
# MTP drafting comes from the model's own mtp.* weights via
|
|
# --speculative-config, only when SPEC_CONFIG is non-empty. Bring-up runs
|
|
# with it empty on purpose: measure the base shape clean, then A/B; never
|
|
# change two knobs at once.
|
|
SPEC_CONFIG=
|
|
|
|
|
|
# --- container ---------------------------------------------------------------
|
|
IMAGE=localhost/frank-primary:vllm030
|
|
CONTAINER_NAME=frank-primary-vllm
|