Frank
1919ad7f31
Replace the FreeToken PR587 endpoint (single-GPU: its TP=2 dies in the pynccl.cu ROCm JIT) with the radiance engine on both R9700s, TP=2 via libr4d's fused one-shot all-reduce. Official Qwen/Qwen3.8-Flash-Next-FP8 payload at native block-FP8 precision: - scripts/build-frank-primary-rad.sh: staging (quantization_config drop = plugin SELECTION law) -> checkpoint reshape -> rad-convert with ONE --quant rule (PLE rtn fixed-scale); plan-only first. - scripts/stack-qwen38-official-fp8.py: the two value-preserving reshapes (experts byte-stacked into the plugin's stacked layout; PLE's 128 scaleless fp8 shards decoded to one bf16 shard with the checkpoint's own scale; the rtn rule re-encodes at that same scale - codes round-trip exactly and land the one-element scale plane the plugin declares). - scripts/hfget.sh + verified download (131/131 shards sha256-match). - froggeric v22.5 chat template vendored (--override-chat-template). - serve script (pinned ledger: 2x40 GiB pools + 47.7 GiB PLE + prefix host), TTM preflight fixed for kernels with pages_limit only (no pages_max - an absent knob is skipped, not read as zero); pages_limit raised to 52,428,800 pages = 200 GiB on the host pre-cutover. - root unit, relinked; deployer gates image digest + .rad floor 158,000 MiB + health; prometheus re-adds the /metrics job. - rollback: tag frank-primary-freetoken-pr587-20261005 (FreeToken trio + Dockerfile); payload + image stay on disk until the window closes. deploy-remote.sh also carries the parallel session's frank-small payload gates (swift15) - committed here per Frank's direction to commit the tree.
136 lines
7.8 KiB
Bash
136 lines
7.8 KiB
Bash
# frank-primary.env — radiance (StillDeadcode, Apache-2.0) serving the
|
|
# OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 checkpoint at NATIVE precision
|
|
# (modelopt block-FP8 throughout, no requantisation) on BOTH R9700s (TP=2),
|
|
# dankserver :7770.
|
|
#
|
|
# 2026-10-05 cutover from FreeToken (PR #587 ROCm, one card, experts in
|
|
# pinned host RAM; git tag frank-primary-freetoken-pr587-20261005 is the
|
|
# pre-cutover state — its unit/env/serve trio and Dockerfile are all in the
|
|
# tag and rebuildable). Before that: tcclaviger vLLM fork (tag
|
|
# frank-primary-vllm-shapeG-20261003).
|
|
#
|
|
# WHY RADIANCE: the engine is C++/HIP with libr4d kernels built for
|
|
# gfx1201, drives `podman run` directly from the digest-pinned
|
|
# stilldeadcode/radiance image (287 MiB; host needs only the amdgpu driver),
|
|
# and its qwen4exp plugin reads the modelopt FP8 encoding NATIVELY
|
|
# (`X.weight_scale_inv` is the scale plane — checkpoint.h). The PLE n-gram
|
|
# table (51B params, 47.7 GiB) is served from pinned host RAM via io_uring
|
|
# reads — Frank's call: "Native FP8. Experts from ram. Pin the PLE to ram."
|
|
# The 4-bit-codebook publisher shape (qwen3.8-next-flash-fp8-iq4r-moe.rad,
|
|
# --host-pool-mib 12288) remains the documented fallback if the native
|
|
# container proves unbootable.
|
|
#
|
|
# PAYLOAD: /home/dank/models/rad/qwen4exp-fp8-native.rad — a rad-convert
|
|
# NO-RECIPE pure copy of the official 131-shard checkpoint (every weight
|
|
# keeps its block-FP8 encoding; ~161 GiB). The convert runs against a
|
|
# staging dir whose config.json drops `quantization_config`: the descriptor
|
|
# string is only the plugin SELECTION key (spec §2.4 — exact match, no
|
|
# fallback), and no shipped plugin claims (qwen4exp, "fp8"); the shipped
|
|
# qwen4exp plugin claims (qwen4exp, "") which is what recipe-built
|
|
# containers carry. Per-weight encodings are data, not the descriptor —
|
|
# removing the key changes nothing about what the weights hold. Built by
|
|
# scripts/build-frank-primary-rad.sh (staging + convert + rad-info).
|
|
#
|
|
# SERVING FACTS (docs/GUIDE.md, docs/DOCKER.md, compose/flashnext.yaml):
|
|
# - The HTTP listener binds AFTER the full load (main.cpp: engine.start()
|
|
# then engine.serve()) — during load the port refuses connections, so a
|
|
# bare curl /health gate is honest. /health == /ping: 200 JSON while
|
|
# serving, 503 only while stopping (core/server/admin.cpp:385).
|
|
# - Reasoning: --reasoning-format defaults to auto, emitting thinking in
|
|
# `reasoning_content` (config.cpp). Tool calls: the qwen3.8 dialect, from
|
|
# the chat format. Both need no parser flags.
|
|
# - Sampling: the OpenAI endpoint accepts temperature, top_p, top_k, min_p,
|
|
# typical_p, presence_penalty, frequency_penalty, repetition_penalty,
|
|
# repeat_last_n, chat_template_kwargs, reasoning_effort,
|
|
# enable_thinking, preserve_thinking (core/server/oai.cpp:194-205) —
|
|
# models.yml's extraBody set is fully inside it. Unknown params are
|
|
# refused (400), which is why nothing unverified rides the wire.
|
|
# - MTP: the official checkpoint carries the mtp.* head (3,101 tensors,
|
|
# mtp_num_hidden_layers=1) and the plugin probes draft depth 3 — the
|
|
# publisher's throughput peak for this model. Stated explicitly with
|
|
# --num-speculative-tokens 3.
|
|
# - The fused TP all-reduce is declared over max-num-seqs x
|
|
# (1 + --num-speculative-tokens) rows; 2 x 4 = 8 rows fits either wire.
|
|
FRANK_PRIMARY_IMAGE=stilldeadcode/radiance@sha256:4c3f484564bb2872784e46b0a59cbd71becc9c1e746e301a3596ffae253c322e
|
|
# The .rad container FILE (mounted under /models) and the state dir for the
|
|
# prefix-cache disk tier (mounted rw at /data). The model dir holds ONLY
|
|
# .rad files; the raw HF download and its staging dir live beside it and
|
|
# are never mounted.
|
|
FRANK_PRIMARY_MODEL_DIR=/home/dank/models/rad
|
|
FRANK_PRIMARY_MODEL_FILE=qwen4exp-fp8-native.rad
|
|
FRANK_PRIMARY_STATE_DIR=/home/dank/radiance-state
|
|
# Container identity is the SERVICE name (Frank's call, 2026-10-05): one
|
|
# name across systemctl and podman ps. The runtime changes; the name does
|
|
# not. The legacy list retires every predecessor identity that could hold
|
|
# the GPUs or the port.
|
|
FRANK_PRIMARY_CONTAINER=frank-primary
|
|
FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-qwen frank-primary-r9v qwen-fp8 glm-probe"
|
|
FRANK_PRIMARY_SERVED_NAME=frank-primary
|
|
FRANK_PRIMARY_PORT=7770
|
|
# Frank's chat-template call: froggeric's fixed Qwen3.8 template (the
|
|
# repo's root chat_template.jinja, vendored here; the checkpoint's own
|
|
# template is baked into the .rad by rad-convert and this overrides it).
|
|
# Radiance derives the reasoning/tool-call reply format from it.
|
|
FRANK_PRIMARY_TEMPLATE=/home/dank/serverAppConfigs/scripts/qwen38-froggeric-v22.5.jinja
|
|
|
|
# --- engine shape (first-boot conservative; the boot ledger arbitrates) --
|
|
# Tensor parallel: BOTH R9700s. The board swap (2026-09-20, ProArt B850
|
|
# x8/x8 on root ports) made collectives legal on this host; radiance's TP
|
|
# wire is libr4d's own, not RCCL.
|
|
FRANK_PRIMARY_TP=2
|
|
# Lossy WHT 6-bit wire for large all-reduces (publisher's production choice:
|
|
# +5.6-6.3% prefill, ~-1.3% 8-stream decode at TP2). min-kb stays the
|
|
# engine default.
|
|
FRANK_PRIMARY_TP_WIRE=wht6
|
|
# Decode slots. 2 is Frank's first-boot value ("use parallelism = 2 to
|
|
# start"); raise only after a real-traffic look at slot occupancy and the
|
|
# KV pool's headroom line. The recurrent-state cost scales with it (54
|
|
# MiB/copy/card, 2 copies with speculation).
|
|
FRANK_PRIMARY_MAX_SEQS=2
|
|
# Served window. Native is 262,144; 200,000 is the publisher's production
|
|
# value (compose) and what models.yml's contextWindow must quote — a longer
|
|
# prompt is refused by the engine, never truncated.
|
|
FRANK_PRIMARY_MAX_MODEL_LEN=200000
|
|
FRANK_PRIMARY_PLACEMENT=expert_tiered
|
|
# Pinned host pool for the expert rows that do not fit VRAM. Native FP8
|
|
# experts are 112.8 GiB total (56.4 GiB per rank slice) vs the 4-bit
|
|
# publisher shape's ~29 GiB — this pool is the cost of native precision.
|
|
# The boot ledger prints the pool's coverage ("the pool backs N% of the
|
|
# worst case ... M full-length sessions"); resize from that line, not from
|
|
# arithmetic.
|
|
FRANK_PRIMARY_HOST_POOL_MIB=40960
|
|
# VRAM left unclaimed per card (publisher default 96 MiB).
|
|
FRANK_PRIMARY_GPU_HEADROOM_MIB=96
|
|
# Weights share of claimed VRAM (publisher production value; higher = more
|
|
# prefix-cache hit rate, less KV headroom).
|
|
FRANK_PRIMARY_EXPERT_VS_CACHE=0.82
|
|
# KV cache in FP8 (7.32 KiB/token at this model's 12 full-attention layers,
|
|
# 2 KV heads, 256 head_dim, K+V).
|
|
FRANK_PRIMARY_KV_DTYPE=fp8
|
|
# Spec decode: the checkpoint's own MTP head at the publisher's depth-3
|
|
# operating point (probe() returns 3 when mtp_num_hidden_layers > 0; the
|
|
# flag states it so the ledger records the intent).
|
|
FRANK_PRIMARY_SPEC_TOKENS=3
|
|
FRANK_PRIMARY_MAX_BATCHED_TOKENS=2048
|
|
# Prefix cache tiers (publisher pattern; disk tier sized for 2 slots).
|
|
FRANK_PRIMARY_PREFIX_HOST_MIB=4096
|
|
FRANK_PRIMARY_PREFIX_DISK_MIB=65536
|
|
# Frank's pin law: the PLE n-gram table (47.7 GiB) serves from pinned host
|
|
# RAM the cards read. `ram` REFUSES the start if the memlock limit is too
|
|
# small — the unit's LimitMEMLOCK=infinity is the other half of this law.
|
|
FRANK_PRIMARY_NGRAM_PLACEMENT=ram
|
|
# TTM preflight: the pinned ledger (2 x 40 GiB pools + 47.7 GiB PLE + 4 GiB
|
|
# prefix host) is ~132 GiB. ttm's pages_limit/pages_max cap the GPU-mappable
|
|
# pinned pages; the current cap (32,252,917 pages = ~123 GiB) is BELOW the
|
|
# plan, so the serve script refuses to start until both are raised. This is
|
|
# the floor it enforces (in MiB), with ~8 GiB of margin.
|
|
FRANK_PRIMARY_TTM_MIN_MIB=140000
|
|
# Escape hatch for boot-ladder experiments. Unset -> no extra argv. Keep
|
|
# multi-flag values as one string; the serve script expands it unquoted on
|
|
# purpose.
|
|
FRANK_PRIMARY_EXTRA_ARGS=
|
|
# The .rad's minimum size in MiB (deployer gate; a truncated convert is a
|
|
# guaranteed long-timeout boot instead of a fast failure). The pure copy of
|
|
# the 172.8 GB checkpoint should land ~161 GiB.
|
|
FRANK_PRIMARY_RAD_MIN_MIB=158000
|