Frank
b06895d725
- env: STATE_FILE -> r9v-state/q8xl; add dev-overlay adapter mount (quantized inject shard dequant, fork 363d606), FULL_MUTABLE_CACHE=0 (80-slot tiered vs the overlay's uncensored 160-slot injection), and the qwen-sharp-v22.4.0.jinja chat template via the launcher's new R9V_CHAT_TEMPLATE_FILE/KWARGS passthrough (fork 30d4a32) - serve/deploy: shard gates extended to SHARD4/5/6 (unset-safe, the uncensored rollback state carries none) - seams: STATE_FILE/LLM_PRIMARY_STATE -> q8xl
41 lines
2.4 KiB
Bash
41 lines
2.4 KiB
Bash
# frank-primary.env — R9V radiance-vLLM production overrides (dankserver :7770).
|
|
#
|
|
# The full tuned engine shape (tiered-expert placement, CPU offload budgets
|
|
# per rank, MTP4 drafter, TP=2, qwen3_coder tool parser, 131072 model len,
|
|
# 80-slot tiered expert cache, PLE pinned residency 54.4 GB) lives on-host
|
|
# in the R9V setup state — /home/dank/r9v-state/q8xl/setup.json — written by
|
|
# the q8 profile's setup run (ready: true; doctor PASS, artifacts
|
|
# sha256-verified at fetch time). serve-frank-primary.sh layers that state
|
|
# first and THIS FILE second, so the keys below win. Never copy tuned values
|
|
# from the state into this file — the state is the single source of truth
|
|
# for engine shape.
|
|
R9V_HOME=/home/dank/GitPackages/radiance-vllm-qwen4exp
|
|
R9V_STATE_FILE=/home/dank/r9v-state/q8xl/setup.json
|
|
# Fleet identity: LAN bind, the single frank-primary port, the harness's
|
|
# model id, our own container name (the launcher default r9v-qwen38-flash-next
|
|
# is the scratch identity).
|
|
R9V_HOST_BIND=0.0.0.0
|
|
R9V_HOST_PORT=7770
|
|
R9V_SERVED_MODEL_NAME=frank-primary
|
|
R9V_CONTAINER_NAME=frank-primary-r9v
|
|
R9V_EXPECTED_GPU_BDFS=0000:03:00.0,0000:07:00.0
|
|
# The Q8_0 payload quantizes BOTH fused hyper-connection shards (the IQ4_XS
|
|
# package ships inject as F32), so the shard-uniformity check rejects the
|
|
# fused projection at load. This dev-overlay adapter dequantizes a quantized
|
|
# inject shard in-stream (fork commit 363d606): opt-in mount, uncensored
|
|
# payloads pass through unchanged.
|
|
R9V_DEV_QWEN4_EXP_ADAPTER_PY=/home/dank/GitPackages/radiance-vllm-qwen4exp/runtimes/dev-overrides/qwen4_exp.py
|
|
# The runtime overlay's env.always injects R9V_FULL_MUTABLE_CACHE=1 — the
|
|
# qualified 160-slot uncensored IQ4_XS cache geometry — which the q8
|
|
# profile's 80-slot tiered cache fails with a baseline mismatch. This flips
|
|
# that injection off for this payload (launch.sh passes it after the overlay
|
|
# envs; docker last-wins).
|
|
R9V_FULL_MUTABLE_CACHE=0
|
|
# The Qwen-Sharp template (the frank-large fleet law): folds reasoning
|
|
# efforts (high/xhigh/max -> xhigh, minimal/low -> low) and defaults
|
|
# preserve_thinking true. --chat-template wins over the tokenizer dir's
|
|
# metadata/chat_template.jinja; request-level kwargs still win per request.
|
|
# Braces in values get the same single-quote law as parentheses (source
|
|
# parses assignment values).
|
|
R9V_CHAT_TEMPLATE_FILE=/home/dank/serverAppConfigs/scripts/qwen-sharp-v22.4.0.jinja
|
|
R9V_CHAT_TEMPLATE_KWARGS='{"preserve_thinking":true}' |