frank-primary: R9V radiance cutover wiring (launcher-driven serve + state gates)
Replace the vLLM 0.30 UVA-offload wiring (never deployed; llama.cpp b11030 endpoint replaced by Frank's order, downtime window open) with the R9V radiance-vLLM shape on the same root unit: - serve-frank-primary.sh: layer the on-host qualified setup state (104 tuned R9V_* keys, /home/dank/r9v-state/uncensored/setup.json) under repo production overrides, then exec the fork's own launcher through the docker->podman shim and block on docker wait so a crashed engine is a nonzero ExecStart (launch.sh --detach alone would fake green). Gates: pinned image by OCI manifest digest in RepoDigests, PLE at pinned byte count (stat -c %s), all GGUF shards + mmproj. - frank-primary.env: thin production overrides only (LAN bind 0.0.0.0, :7770, alias frank-primary, container frank-primary-r9v, GPU BDFs); tuned engine shape stays in the setup state by design. - frank-primary.service: same root-unit shape (User=dank, LimitMEMLOCK=infinity for the tiered-expert pin), plus ExecStop podman stop — docker wait cannot stop the container child. - deploy-remote.sh deploy_llm_primary: state-driven gates (state ready, image digest present in rootless storage, PLE bytes, shard files) replace the fp030 shard-index gate; false-green MainPID check kept. - tests/frank_primary_unit.bats: seam rewritten for the R9V shape.
This commit is contained in:
Frank committed
1 parent
baba84fc90
commit
095128b669
5 files changed
+282
-349
No files matched your search
+19
-95
@@ -1,96 +1,20 @@
|
||||
# frank-primary.env — tunables for the vLLM 0.30 frank-primary endpoint
|
||||
# (Qwen3.8-Flash-Next-FP8, 2x R9700 gfx1201, dankserver :7770).
|
||||
# frank-primary.env — R9V radiance-vLLM production overrides (dankserver :7770).
|
||||
#
|
||||
# Paths and numbers only — NEVER credentials. Do not name a tunable TEMP, HOST
|
||||
# or GPU_MAX_HEAP_SIZE: the HIP runtime reads those bare names (2026-08-13).
|
||||
#
|
||||
# Container paths: the model tree is mounted at /models (host
|
||||
# /home/dank/models); the compile caches live under /cache (host
|
||||
# /home/dank/.cache/frank-primary-vllm, keyed on model + stack + graph knobs —
|
||||
# never share it across configurations).
|
||||
|
||||
# --- identity ---------------------------------------------------------------
|
||||
SERVED_NAME=frank-primary
|
||||
BIND_HOST=0.0.0.0
|
||||
PORT=7770
|
||||
|
||||
# --- model ------------------------------------------------------------------
|
||||
# Host-side dir: /home/dank/models/Qwen3.8-Flash-Next-FP8 (131 safetensors
|
||||
# shards, 172.78 GiB, verified against model.safetensors.index.json).
|
||||
MODEL_DIR=/models/Qwen3.8-Flash-Next-FP8
|
||||
|
||||
# --- placement --------------------------------------------------------------
|
||||
# Fixed 2-card tensor parallel (the R9700 pair; the Raphael iGPU is index 2
|
||||
# and excluded by the visibility pins).
|
||||
ROCR_VISIBLE_DEVICES=0,1
|
||||
HIP_VISIBLE_DEVICES=0,1
|
||||
TP=2
|
||||
# 0.97 of 31.86 GiB/card (34,208,743,424 B) — leaves ~1 GiB for the ROCm
|
||||
# baseline. Raise only with a measured reason.
|
||||
GPU_UTIL=0.97
|
||||
# The model's native window.
|
||||
CTX_SIZE=262144
|
||||
# Sequences the scheduler may hold. 8 is the proven batch shape on this host
|
||||
# (radiance era); revisit with a bench after bring-up.
|
||||
MAX_SEQS=8
|
||||
|
||||
# --- weight offloading (the UVA shape) ---------------------------------------
|
||||
# --offload-backend uva: matched weights live in PINNED host RAM and are read
|
||||
# in place over PCIe by the kernels (zero-copy), instead of being resident
|
||||
# on the cards. is_uva_available() = is_pin_memory_available() on ROCm; the
|
||||
# pin needs the unit's LimitMEMLOCK=infinity (a root unit grants it via
|
||||
# PID1 — the dankserver user session defaults to 8 MiB and cannot raise it).
|
||||
# If pinning is unavailable the offloader falls back to a copy path — the
|
||||
# load log MUST be checked for the UVA path, not just /health.
|
||||
# --cpu-offload-gb is PER GPU (GiB). 2x R9700 total pinned budget ≈ 160 GiB of
|
||||
# the 246 GiB host RAM; the rest covers the OS, llama.cpp-era page cache
|
||||
# drain and the pinned staging buffers. Do not raise without checking
|
||||
# MemAvailable at load time.
|
||||
# --cpu-offload-params: name SEGMENTS (dot-delimited components) of tensors
|
||||
# to offload. Measured from the payload headers (2026-09-26): `ple` matches
|
||||
# 138 tensors = 131 n-gram table shards + 7 per-layer conv/proj weights,
|
||||
# 47.75 GiB; `experts` matches 150,528 tensors, 114.86 GiB. Combined
|
||||
# 162.60 GiB — with TP=2 each rank needs ~81.3 GiB of pinned host RAM,
|
||||
# hence OFFLOAD_GB=82 per GPU (164 GiB total; 80/card left 2.6 GiB of
|
||||
# experts resident, inflating the on-card weight share). Resident after
|
||||
# the full offload: 10.16 GiB of dense/attention/MTP/ViT/lm_head weights.
|
||||
# Matching is greedy-up-to-budget in traversal order; unmatched params are
|
||||
# never offloaded.
|
||||
OFFLOAD_BACKEND=uva
|
||||
OFFLOAD_GB=82
|
||||
OFFLOAD_PARAMS=ple,experts
|
||||
|
||||
# --- engine -----------------------------------------------------------------
|
||||
# No aiter wheel in this image (gfx9-only upstream); the AMD qwen4_exp path
|
||||
# runs its in-tree triton kernels. Deterministic rather than relying on the
|
||||
# availability probe.
|
||||
VLLM_ROCM_USE_AITER=0
|
||||
# Compile/inductor/triton caches (container /cache, TRITON_CACHE_DIR etc.)
|
||||
CACHE_ROOT=/cache
|
||||
|
||||
# --- parsers / template (the harness wire contract) ---------------------------
|
||||
# vLLM 0.30 emits the thinking channel as `reasoning` (the deprecated
|
||||
# `reasoning_content` is renamed server-side, chat_completion/protocol.py:554).
|
||||
# The qwen3 reasoning parser extracts the template's think-block tags into
|
||||
# that channel; the qwen3_xml tool parser structures tool-call XML output into
|
||||
# tool_calls (auto-tool-choice enables the structured path for API clients).
|
||||
# The family precedent is the host's own Ornith-era serve scripts
|
||||
# (~/vllm/failedAttempts/OrnithFP8.sh); the FP8 tokenizer's template
|
||||
# consumes enable_thinking + preserve_thinking (verified in tokenizer_config).
|
||||
TOOL_PARSER=qwen3_xml
|
||||
REASONING_PARSER=qwen3
|
||||
CHAT_TEMPLATE_KWARGS='{"enable_thinking": true, "preserve_thinking": true}'
|
||||
|
||||
# --- speculative decode ------------------------------------------------------
|
||||
# The hybrid stack (GatedDeltaNet + QSA short-conv) uses the v1 engine's
|
||||
# mamba-capable attention backends, negotiated by the model, not by env.
|
||||
# MTP drafting comes from the model's own mtp.* weights via
|
||||
# --speculative-config, only when SPEC_CONFIG is non-empty. Bring-up runs
|
||||
# with it empty on purpose: measure the base shape clean, then A/B; never
|
||||
# change two knobs at once.
|
||||
SPEC_CONFIG=
|
||||
|
||||
|
||||
# --- container ---------------------------------------------------------------
|
||||
IMAGE=localhost/frank-primary:vllm030
|
||||
CONTAINER_NAME=frank-primary-vllm
|
||||
# The full tuned engine shape (104 R9V_* keys: tiered-expert placement,
|
||||
# CPU offload budgets 146.25/78.75 GiB per rank, CED on, MTP4, TP=2,
|
||||
# qwen3_coder tool parser, 196608 model len) lives on-host in the R9V setup
|
||||
# state — /home/dank/r9v-state/uncensored/setup.json — written by the
|
||||
# profile's setup run (PASS=28/FAIL=0, all 20 artifacts sha256-verified).
|
||||
# serve-frank-primary.sh layers that state first and THIS FILE second, so the
|
||||
# keys below win. Never copy tuned values from the state into this file —
|
||||
# the state is the single source of truth for engine shape.
|
||||
R9V_HOME=/home/dank/GitPackages/radiance-vllm-qwen4exp
|
||||
R9V_STATE_FILE=/home/dank/r9v-state/uncensored/setup.json
|
||||
# Fleet identity: LAN bind, the single frank-primary port, the harness's
|
||||
# model id, our own container name (the launcher default r9v-qwen38-flash-next
|
||||
# is the scratch identity).
|
||||
R9V_HOST_BIND=0.0.0.0
|
||||
R9V_HOST_PORT=7770
|
||||
R9V_SERVED_MODEL_NAME=frank-primary
|
||||
R9V_CONTAINER_NAME=frank-primary-r9v
|
||||
R9V_EXPECTED_GPU_BDFS=0000:03:00.0,0000:07:00.0
|
||||
Reference in new issue
Block a user