165 lines
9.7 KiB
Bash
165 lines
9.7 KiB
Bash
# frank-primary.env — radiance (StillDeadcode, Apache-2.0) serving the
|
|
# OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 checkpoint at NATIVE precision
|
|
# (modelopt block-FP8 throughout, no requantisation) on BOTH R9700s (TP=2),
|
|
# dankserver :7770.
|
|
#
|
|
# 2026-10-05 cutover from FreeToken (PR #587 ROCm, one card, experts in
|
|
# pinned host RAM; git tag frank-primary-freetoken-pr587-20261005 is the
|
|
# pre-cutover state — its unit/env/serve trio and Dockerfile are all in the
|
|
# tag and rebuildable). Before that: tcclaviger vLLM fork (tag
|
|
# frank-primary-vllm-shapeG-20261003).
|
|
#
|
|
# WHY RADIANCE: the engine is C++/HIP with libr4d kernels built for
|
|
# gfx1201, drives `podman run` directly from the digest-pinned
|
|
# stilldeadcode/radiance image (287 MiB; host needs only the amdgpu driver),
|
|
# and its qwen4exp plugin reads the modelopt FP8 encoding NATIVELY
|
|
# (`X.weight_scale_inv` is the scale plane — checkpoint.h). The PLE n-gram
|
|
# table (51B params, 47.7 GiB) is served from pinned host RAM via io_uring
|
|
# reads — Frank's call: "Native FP8. Experts from ram. Pin the PLE to ram."
|
|
# The 4-bit-codebook publisher shape (qwen3.8-next-flash-fp8-iq4r-moe.rad,
|
|
# --host-pool-mib 12288) remains the documented fallback if the native
|
|
# container proves unbootable.
|
|
#
|
|
# PAYLOAD: /home/dank/models/rad/qwen4exp-fp8-native.rad — a rad-convert
|
|
# NO-RECIPE pure copy of the official 131-shard checkpoint (every weight
|
|
# keeps its block-FP8 encoding; ~161 GiB). The convert runs against a
|
|
# staging dir whose config.json drops `quantization_config`: the descriptor
|
|
# string is only the plugin SELECTION key (spec §2.4 — exact match, no
|
|
# fallback), and no shipped plugin claims (qwen4exp, "fp8"); the shipped
|
|
# qwen4exp plugin claims (qwen4exp, "") which is what recipe-built
|
|
# containers carry. Per-weight encodings are data, not the descriptor —
|
|
# removing the key changes nothing about what the weights hold. Built by
|
|
# scripts/build-frank-primary-rad.sh (staging + convert + rad-info).
|
|
#
|
|
# SERVING FACTS (docs/GUIDE.md, docs/DOCKER.md, compose/flashnext.yaml):
|
|
# - The HTTP listener binds AFTER the full load (main.cpp: engine.start()
|
|
# then engine.serve()) — during load the port refuses connections, so a
|
|
# bare curl /health gate is honest. /health == /ping: 200 JSON while
|
|
# serving, 503 only while stopping (core/server/admin.cpp:385).
|
|
# - Reasoning: --reasoning-format defaults to auto, emitting thinking in
|
|
# `reasoning_content` (config.cpp). Tool calls: the qwen3.8 dialect, from
|
|
# the chat format. Both need no parser flags.
|
|
# - Sampling: the OpenAI endpoint accepts temperature, top_p, top_k, min_p,
|
|
# typical_p, presence_penalty, frequency_penalty, repetition_penalty,
|
|
# repeat_last_n, chat_template_kwargs, reasoning_effort,
|
|
# enable_thinking, preserve_thinking (core/server/oai.cpp:194-205) —
|
|
# models.yml's extraBody set is fully inside it. Unknown params are
|
|
# refused (400), which is why nothing unverified rides the wire.
|
|
# - MTP: the official checkpoint carries the mtp.* head (3,101 tensors,
|
|
# mtp_num_hidden_layers=1) and the plugin probes draft depth 3 — the
|
|
# publisher's throughput peak for this model. Stated explicitly with
|
|
# --num-speculative-tokens 3.
|
|
# - The fused TP all-reduce is declared over max-num-seqs x
|
|
# (1 + --num-speculative-tokens) rows; 3 x 3 = 9 rows fits either wire.
|
|
# Engine pin: Codeberg tag v1.0.13 (commit 53282fa), Docker Hub tag 1.0.13,
|
|
# index digest below. REJECTED UPGRADE: 1.2.0 (sha256:665775ed1d90de6dbb
|
|
# 25558e07fc60d20328f1346f96f3c6e00845eb763f01fa, tag 1.2.0, commit
|
|
# e95d397) was benched 2026-10-06 on this exact shape (fp8-native rad,
|
|
# vram pair, spec 2, seqs 3, 43.6k greedy probe): decode 29.21 t/s and
|
|
# draft acceptance 0.7123 vs the 1.0.13 control's 30.08 and 0.7549.
|
|
# Acceptance on the greedy harness is deterministic (the control
|
|
# reproduced the recorded 0.755 exactly), so the drop is a real numerics
|
|
# change — the 1.2.0 MoE weight-rotation change (92b8aaf) is the named
|
|
# suspect; decode loss tracks the acceptance ratio (0.943 vs 0.945).
|
|
# The vram-pair translation itself is exact (1.2.0 boot ledger took
|
|
# vram_weights 25.46 GiB / vram_kv 3.46 GiB per card byte-identical to
|
|
# the 1.0.13 ledger at EVC 0.82). Re-test 1.2.0+ when the rotation change
|
|
# is untangled; 1.2.0 also REMOVED --expert-vs-cache-ratio in favour of
|
|
# the --vram-weights-mib / --vram-kv-mib pair (core/config.cpp) — this
|
|
# file already uses the pair, which BOTH pins accept (rollback symmetry).
|
|
FRANK_PRIMARY_IMAGE=stilldeadcode/radiance@sha256:4c3f484564bb2872784e46b0a59cbd71becc9c1e746e301a3596ffae253c322e
|
|
# The .rad container FILE (mounted under /models) and the state dir for the
|
|
# prefix-cache disk tier (mounted rw at /data). The model dir holds ONLY
|
|
# .rad files; the raw HF download and its staging dir live beside it and
|
|
# are never mounted.
|
|
FRANK_PRIMARY_MODEL_DIR=/home/dank/models/rad
|
|
FRANK_PRIMARY_MODEL_FILE=qwen4exp-fp8-native.rad
|
|
FRANK_PRIMARY_STATE_DIR=/home/dank/radiance-state
|
|
# The state dir's tune/ subdirectory holds rad-tune's kernel-selection
|
|
# cache (the engine's selector reads $RADIANCE_HOME/tune/<machine>.tune
|
|
# at boot; $RADIANCE_HOME defaults to /opt/radiance, in the image, so the
|
|
# cache must ride a host mount — serve script mounts it, see the runbook's
|
|
# "Tuning levers").
|
|
# Container identity is the SERVICE name (Frank's call, 2026-10-05): one
|
|
# name across systemctl and podman ps. The runtime changes; the name does
|
|
# not. The legacy list retires every predecessor identity that could hold
|
|
# the GPUs or the port.
|
|
FRANK_PRIMARY_CONTAINER=frank-primary
|
|
FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-qwen frank-primary-r9v qwen-fp8 glm-probe"
|
|
FRANK_PRIMARY_SERVED_NAME=frank-primary
|
|
FRANK_PRIMARY_PORT=7770
|
|
# Frank's chat-template call: froggeric's fixed Qwen3.8 template (the
|
|
# repo's root chat_template.jinja, vendored here; the checkpoint's own
|
|
# template is baked into the .rad by rad-convert and this overrides it).
|
|
# Radiance derives the reasoning/tool-call reply format from it.
|
|
FRANK_PRIMARY_TEMPLATE=/home/dank/serverAppConfigs/scripts/qwen38-froggeric-v22.5.jinja
|
|
|
|
# --- engine shape (first-boot conservative; the boot ledger arbitrates) --
|
|
# Tensor parallel: BOTH R9700s. The board swap (2026-09-20, ProArt B850
|
|
# x8/x8 on root ports) made collectives legal on this host; radiance's TP
|
|
# wire is libr4d's own, not RCCL.
|
|
FRANK_PRIMARY_TP=2
|
|
# Lossy WHT 6-bit wire for large all-reduces (publisher's production choice:
|
|
# +5.6-6.3% prefill, ~-1.3% 8-stream decode at TP2). min-kb stays the
|
|
# engine default.
|
|
FRANK_PRIMARY_TP_WIRE=wht6
|
|
# Decode slots. 2 is Frank's first-boot value ("use parallelism = 2 to
|
|
# start"); raise only after a real-traffic look at slot occupancy and the
|
|
# KV pool's headroom line. The recurrent-state cost scales with it (54
|
|
# MiB/copy/card, 2 copies with speculation).
|
|
FRANK_PRIMARY_MAX_SEQS=3
|
|
# Served window. Native is 262,144; 200,000 is the publisher's production
|
|
# value (compose) and what models.yml's contextWindow must quote — a longer
|
|
# prompt is refused by the engine, never truncated.
|
|
FRANK_PRIMARY_MAX_MODEL_LEN=200000
|
|
FRANK_PRIMARY_PLACEMENT=expert_tiered
|
|
# Pinned host pool for the expert rows that do not fit VRAM. Native FP8
|
|
# experts are 112.8 GiB total (56.4 GiB per rank slice) vs the 4-bit
|
|
# publisher shape's ~29 GiB — this pool is the cost of native precision.
|
|
# The boot ledger prints the pool's coverage ("the pool backs N% of the
|
|
# worst case ... M full-length sessions"); resize from that line, not from
|
|
# arithmetic.
|
|
FRANK_PRIMARY_HOST_POOL_MIB=40960
|
|
# VRAM left unclaimed per card (publisher default 96 MiB).
|
|
FRANK_PRIMARY_GPU_HEADROOM_MIB=96
|
|
# VRAM split of claimed memory per card, MiB (0 = derive from the card).
|
|
# Engine 1.2.0 REMOVED --expert-vs-cache-ratio; these two knobs replace
|
|
# it. The values transcribe the live 1.0.13 boot ledger at ratio 0.82:
|
|
# vram_weights 25.46 GiB, vram_kv 3.46 GiB per card -> 26071 / 3543.
|
|
# Bracket arms scale the KV term linearly in (1-r) from the 0.82 anchor
|
|
# with weights fixed: ratio 0.90 -> KV 1968, ratio 0.95 -> KV 984. The
|
|
# boot ledger arbitrates; re-derive after any payload change.
|
|
FRANK_PRIMARY_VRAM_WEIGHTS_MIB=26071
|
|
FRANK_PRIMARY_VRAM_KV_MIB=3543
|
|
# KV cache in FP8 (7.32 KiB/token at this model's 12 full-attention layers,
|
|
# 2 KV heads, 256 head_dim, K+V).
|
|
FRANK_PRIMARY_KV_DTYPE=fp8
|
|
# Spec decode: the checkpoint's own MTP head at the publisher's depth-3
|
|
# operating point (probe() returns 3 when mtp_num_hidden_layers > 0; the
|
|
# flag states it so the ledger records the intent).
|
|
FRANK_PRIMARY_SPEC_TOKENS=2
|
|
FRANK_PRIMARY_MAX_BATCHED_TOKENS=2048
|
|
# Prefix cache tiers (publisher pattern; disk tier sized for 2 slots).
|
|
FRANK_PRIMARY_PREFIX_HOST_MIB=4096
|
|
FRANK_PRIMARY_PREFIX_DISK_MIB=65536
|
|
# Frank's pin law: the PLE n-gram table (47.7 GiB) serves from pinned host
|
|
# RAM the cards read. `ram` REFUSES the start if the memlock limit is too
|
|
# small — the unit's LimitMEMLOCK=infinity is the other half of this law.
|
|
FRANK_PRIMARY_NGRAM_PLACEMENT=ram
|
|
# TTM preflight: the pinned ledger (2 x 40 GiB pools + 47.7 GiB PLE + 4 GiB
|
|
# prefix host) is ~132 GiB. ttm's pages_limit/pages_max cap the GPU-mappable
|
|
# pinned pages; the current cap (32,252,917 pages = ~123 GiB) is BELOW the
|
|
# plan, so the serve script refuses to start until both are raised. This is
|
|
# the floor it enforces (in MiB), with ~8 GiB of margin.
|
|
FRANK_PRIMARY_TTM_MIN_MIB=140000
|
|
# Escape hatch for boot-ladder experiments. Unset -> no extra argv. Keep
|
|
# multi-flag values as one string; the serve script expands it unquoted on
|
|
# purpose.
|
|
FRANK_PRIMARY_EXTRA_ARGS=
|
|
# The .rad's minimum size in MiB (deployer gate; a truncated convert is a
|
|
# guaranteed long-timeout boot instead of a fast failure). The converted
|
|
# container measured 177,109 MiB (1,857,127,997,440 B = 173.0 GiB; the
|
|
# plan-only estimate 172.76 GiB covers the data planes but not the metadata
|
|
# tail) - floor sits just under the payload.
|
|
FRANK_PRIMARY_RAD_MIN_MIB=177000
|