Frank
f18716563a
- frank-primary.env: R9V_PLE_READY_TIMEOUT=1800 (PLE registration measured 740s under page-cache churn; engine default 600s killed a prod boot 2026-09-28; launch.sh 2de0357 forwards it) - runbook: prefill/decode bench matrix section (chunk ceiling 4096, 80-slot policy max, MTP6 3.7x worse, prefix-caching evidence + recommendation), fork commit list, residency wording - index: q8 runbook ACTIVE, uncensored RETIRED (rollback); log entry incl. the 9h downtime incident
47 lines
2.8 KiB
Bash
47 lines
2.8 KiB
Bash
# frank-primary.env — R9V radiance-vLLM production overrides (dankserver :7770).
|
|
#
|
|
# The full tuned engine shape (tiered-expert placement, CPU offload budgets
|
|
# per rank, MTP4 drafter, TP=2, qwen3_coder tool parser, 131072 model len,
|
|
# 80-slot tiered expert cache, PLE pinned residency 54.4 GB) lives on-host
|
|
# in the R9V setup state — /home/dank/r9v-state/q8xl/setup.json — written by
|
|
# the q8 profile's setup run (ready: true; doctor PASS, artifacts
|
|
# sha256-verified at fetch time). serve-frank-primary.sh layers that state
|
|
# first and THIS FILE second, so the keys below win. Never copy tuned values
|
|
# from the state into this file — the state is the single source of truth
|
|
# for engine shape.
|
|
R9V_HOME=/home/dank/GitPackages/radiance-vllm-qwen4exp
|
|
R9V_STATE_FILE=/home/dank/r9v-state/q8xl/setup.json
|
|
# Fleet identity: LAN bind, the single frank-primary port, the harness's
|
|
# model id, our own container name (the launcher default r9v-qwen38-flash-next
|
|
# is the scratch identity).
|
|
R9V_HOST_BIND=0.0.0.0
|
|
R9V_HOST_PORT=7770
|
|
R9V_SERVED_MODEL_NAME=frank-primary
|
|
R9V_CONTAINER_NAME=frank-primary-r9v
|
|
R9V_EXPECTED_GPU_BDFS=0000:03:00.0,0000:07:00.0
|
|
# PLE registration (54.4 GB mmap -> HIP) can exceed the engine's 600 s
|
|
# VLLM_PLE_OFFLOAD_READY_TIMEOUT default under page-cache churn (measured
|
|
# 740 s on 2026-09-28, right after a bench teardown; a production boot
|
|
# failed on it the same morning). launch.sh 2de0357 forwards this to the
|
|
# engine; 1800 s covers the slowest observed registration with margin.
|
|
R9V_PLE_READY_TIMEOUT=1800
|
|
# The Q8_0 payload quantizes BOTH fused hyper-connection shards (the IQ4_XS
|
|
# package ships inject as F32), so the shard-uniformity check rejects the
|
|
# fused projection at load. This dev-overlay adapter dequantizes a quantized
|
|
# inject shard in-stream (fork commit 363d606): opt-in mount, uncensored
|
|
# payloads pass through unchanged.
|
|
R9V_DEV_QWEN4_EXP_ADAPTER_PY=/home/dank/GitPackages/radiance-vllm-qwen4exp/runtimes/dev-overrides/qwen4_exp.py
|
|
# The runtime overlay's env.always injects R9V_FULL_MUTABLE_CACHE=1 — the
|
|
# qualified 160-slot uncensored IQ4_XS cache geometry — which the q8
|
|
# profile's 80-slot tiered cache fails with a baseline mismatch. This flips
|
|
# that injection off for this payload (launch.sh passes it after the overlay
|
|
# envs; docker last-wins).
|
|
R9V_FULL_MUTABLE_CACHE=0
|
|
# The Qwen-Sharp template (the frank-large fleet law): folds reasoning
|
|
# efforts (high/xhigh/max -> xhigh, minimal/low -> low) and defaults
|
|
# preserve_thinking true. --chat-template wins over the tokenizer dir's
|
|
# metadata/chat_template.jinja; request-level kwargs still win per request.
|
|
# Braces in values get the same single-quote law as parentheses (source
|
|
# parses assignment values).
|
|
R9V_CHAT_TEMPLATE_FILE=/home/dank/serverAppConfigs/scripts/qwen-sharp-v22.4.0.jinja
|
|
R9V_CHAT_TEMPLATE_KWARGS='{"preserve_thinking":true}' |