frank-primary: cutover state to Qwen3.8-Flash-Next Q8_0 (6 shards, PLE pinned, MTP4, sharp template)
- env: STATE_FILE -> r9v-state/q8xl; add dev-overlay adapter mount (quantized inject shard dequant, fork 363d606), FULL_MUTABLE_CACHE=0 (80-slot tiered vs the overlay's uncensored 160-slot injection), and the qwen-sharp-v22.4.0.jinja chat template via the launcher's new R9V_CHAT_TEMPLATE_FILE/KWARGS passthrough (fork 30d4a32) - serve/deploy: shard gates extended to SHARD4/5/6 (unset-safe, the uncensored rollback state carries none) - seams: STATE_FILE/LLM_PRIMARY_STATE -> q8xl
This commit is contained in:
1 parent
928fd81dbd
commit
b06895d725
4 files changed
+61
-30
No files matched your search
@@ -1,15 +1,16 @@
|
||||
# frank-primary.env — R9V radiance-vLLM production overrides (dankserver :7770).
|
||||
#
|
||||
# The full tuned engine shape (104 R9V_* keys: tiered-expert placement,
|
||||
# CPU offload budgets 146.25/78.75 GiB per rank, CED on, MTP4, TP=2,
|
||||
# qwen3_coder tool parser, 196608 model len) lives on-host in the R9V setup
|
||||
# state — /home/dank/r9v-state/uncensored/setup.json — written by the
|
||||
# profile's setup run (PASS=28/FAIL=0, all 20 artifacts sha256-verified).
|
||||
# serve-frank-primary.sh layers that state first and THIS FILE second, so the
|
||||
# keys below win. Never copy tuned values from the state into this file —
|
||||
# the state is the single source of truth for engine shape.
|
||||
# The full tuned engine shape (tiered-expert placement, CPU offload budgets
|
||||
# per rank, MTP4 drafter, TP=2, qwen3_coder tool parser, 131072 model len,
|
||||
# 80-slot tiered expert cache, PLE pinned residency 54.4 GB) lives on-host
|
||||
# in the R9V setup state — /home/dank/r9v-state/q8xl/setup.json — written by
|
||||
# the q8 profile's setup run (ready: true; doctor PASS, artifacts
|
||||
# sha256-verified at fetch time). serve-frank-primary.sh layers that state
|
||||
# first and THIS FILE second, so the keys below win. Never copy tuned values
|
||||
# from the state into this file — the state is the single source of truth
|
||||
# for engine shape.
|
||||
R9V_HOME=/home/dank/GitPackages/radiance-vllm-qwen4exp
|
||||
R9V_STATE_FILE=/home/dank/r9v-state/uncensored/setup.json
|
||||
R9V_STATE_FILE=/home/dank/r9v-state/q8xl/setup.json
|
||||
# Fleet identity: LAN bind, the single frank-primary port, the harness's
|
||||
# model id, our own container name (the launcher default r9v-qwen38-flash-next
|
||||
# is the scratch identity).
|
||||
@@ -18,3 +19,23 @@ R9V_HOST_PORT=7770
|
||||
R9V_SERVED_MODEL_NAME=frank-primary
|
||||
R9V_CONTAINER_NAME=frank-primary-r9v
|
||||
R9V_EXPECTED_GPU_BDFS=0000:03:00.0,0000:07:00.0
|
||||
# The Q8_0 payload quantizes BOTH fused hyper-connection shards (the IQ4_XS
|
||||
# package ships inject as F32), so the shard-uniformity check rejects the
|
||||
# fused projection at load. This dev-overlay adapter dequantizes a quantized
|
||||
# inject shard in-stream (fork commit 363d606): opt-in mount, uncensored
|
||||
# payloads pass through unchanged.
|
||||
R9V_DEV_QWEN4_EXP_ADAPTER_PY=/home/dank/GitPackages/radiance-vllm-qwen4exp/runtimes/dev-overrides/qwen4_exp.py
|
||||
# The runtime overlay's env.always injects R9V_FULL_MUTABLE_CACHE=1 — the
|
||||
# qualified 160-slot uncensored IQ4_XS cache geometry — which the q8
|
||||
# profile's 80-slot tiered cache fails with a baseline mismatch. This flips
|
||||
# that injection off for this payload (launch.sh passes it after the overlay
|
||||
# envs; docker last-wins).
|
||||
R9V_FULL_MUTABLE_CACHE=0
|
||||
# The Qwen-Sharp template (the frank-large fleet law): folds reasoning
|
||||
# efforts (high/xhigh/max -> xhigh, minimal/low -> low) and defaults
|
||||
# preserve_thinking true. --chat-template wins over the tokenizer dir's
|
||||
# metadata/chat_template.jinja; request-level kwargs still win per request.
|
||||
# Braces in values get the same single-quote law as parentheses (source
|
||||
# parses assignment values).
|
||||
R9V_CHAT_TEMPLATE_FILE=/home/dank/serverAppConfigs/scripts/qwen-sharp-v22.4.0.jinja
|
||||
R9V_CHAT_TEMPLATE_KWARGS='{"preserve_thinking":true}'
|
||||
Reference in new issue
Block a user